Find strings in a csv and create a new list with it?

I have a csv file that looks something like this:

mike,123,184,2.6,543
mark,124,284,2.6,541
jane,178,384,2.6,542
zack,165,484,2.6,544
mike,178,584,2.6,545
zack,112,684,2.6,546
zack,145,784,2.6,547
mike,110,984,2.6,548

I want to be able to search that csv file for mike and create a list for that parameter only and be able to plot let say the 2nd column vs the last column or something of that nature. What would be the best way to do this if I have a very large csv file?

Posted Oct 24, 2016, 5:38 PM UTC · edited

666
novel_yet_trivial•

What have you tried? We will help you if you are stuck but we will not write code for you. In this case, the csv module would be very helpful for you.

1
IsitwhenipeeOP•

For starters I tried to just read one row to see if I can do that. I used this:

import csv
csv_file = csv.reader(open('data2.csv', "r"), delimiter=",")
for row in csv_file:
    print (row[300])

However I got an error saying that it is out of range. Even though the data file has about 2 million rows.

EDIT: So I also tried this, since I realized row means columns

import csv
desiredData = 'mike'
with open('data2.csv', 'rb') as csvfile:
    content = csv.reader(csvfile, delimiter=',')
    for row in content:
        if desiredData == row[0]:
            print (row)

and it worked! However, my question is when I do store this in a list, how do I access each column for plotting?

1
novel_yet_trivial•

print (row[300]) is trying to print the 300th column in the first row.

You have a good start. You just need to use the 1st column as a check to add to a list:

for row in csv_file:
    if row[0] == 'mike': #if the first column value is 'mike'
        data.append(row[1]) #add the value from the second column to the 'data' list
3
IsitwhenipeeOP•

Yes, if you see my edit I actually figured out but i am only reading it. I tried to use your code to append it, however, it is taking for ever to store the data into data.

import csv
desiredData = 'IMU'
data = []
with open('data2.csv', 'r') as csvfile:
    content = csv.reader(csvfile, delimiter=',')
    for row in content:
        if desiredData == row[0]:
            data.append(row[1])
print (data)
1
chrisutpg•

I believe you need to do..

if row[0] == desiredData:

the way you are doing it, I believe you are looping the entire file for each condition, where as the above way will evaluate the condition as it loops the file. I could be completely wrong here, take with a grain of salt.

2
novel_yet_trivial•

Forever to store it or forever to print it? It should read it in less than 10 seconds (assuming the file is ~50MB and you have a fairly modern computer). However printing that amount of data to the screen can take a very long time. Maybe try:

print(len(data), 'rows extracted')
2
IsitwhenipeeOP•

Even though it is taking forever to store the data I tried to plot it but I am getting this error: ValueError: x and y must have same first dimension

import csv
import matplotlib.pyplot as plt

desiredData = 'IMU'
data = []
with open('data2.csv', 'r') as csvfile:
    content = csv.reader(csvfile, delimiter=',')
    for row in content:
        if desiredData == row[0]:
            data.append(row[1])
fig1 = plt.figure(figsize= (10,10))
ax = fig1.add_subplot(111)
mSec = data[:3]
Alt = data[:4]
ax.plot(mSec, Alt, label='Alt(m)', color = 'r')
plt.legend(loc='best',prop={'size':10})
plt.grid(True)
ax.set_xlabel('Time (s)')
ax.set_ylabel('Alt (m)')
1
novel_yet_trivial•

'data' is only the second column. What are you trying to plot?

2