Python，从csvs处理数组并插入数据库

Question

I want to read in multiple csv files as arrays and duplicate the rows of those arrays based on the numeric value of the first entry in each row, (if the value is 1, it isn't duplicated, but if the value is 3 that row is represented 3 times). After manipulating the arrays I want to insert them into a db table.

sample csv files:

mult, n1, n2, n3, n4
1, 23.2, 55, 0, 1.1
3, 6.6, 0.2, 5, 9
1, 2.2, 5, 8, 9
2, 3.3, 10, 2, 2

mult, n1, n2, n3, n4
2, 23.2, 55, 0, 1.1
3, 6.6, 0.2, 5, 9
1, 2.2, 5, 8, 9
1, 3.3, 10, 2, 2

desired outcome

[[1, 23.2, 55, 0, 1.1],
[3, 6.6, 0.2, 5, 9],
[3, 6.6, 0.2, 5, 9],
[3, 6.6, 0.2, 5, 9],
[1, 2.2, 5, 8, 9],
[2, 3.3, 10, 2, 2],
[2, 3.3, 10, 2, 2]]

[[2, 23.2 55, 0, 1.1],
[2, 23.2 55, 0, 1,1],
[3, 6.6, 0.2, 5, 9],
[3, 6.6, 0.2, 5, 9],
[3, 6.6, 0.2, 5, 9],
[1, 2.2, 5, 8, 9],
[1, 3.3, 10, 2, 2]]

Initially I read the csvs in as a list and then had a for loop to insert each row into the db based on the number in row[0]

basic code snippet of what currently works:

import csv, os, glob
import psycopg2

path = "/home/user/Desktop/files/*.csv"

for fname in glob.glob(path):
    self.readFile(fname)

with open(filename, 'r') as f:
    arr= list(csv.reader(f)) 
    iter_arr = iter(arr)
    next(iter_arr)

    for row in iter_arr:                   
        mult = int(float(row[0]))
        for i in range (mult):
            try:
                self.cur.execute("INSERT INTO csv_table VALUES (%s, %s, %s, %s, %s)", row)

             except Exception, exc:     
                 locked = True
                 print ("%s", exc)     

The above code works in that it will load the correct amount of rows in the database table, but I thought it would be more useful to duplicate the rows before loading them to the database so that I can manipulate the data in the arrays further if I need to, like changing or adding values.

I asked a question earlier about using numpy, which allowed me to manipulate some randomly generated arrays correctly, but instead of duplicating rows as separate elements it is duplicating them within each other. I can't figure out how to resize it to get it to work, plus it seems like. There Resizing does not seem to work and I get a ('%s', TypeError('not all arguments converted during string formatting',))

a = ([list(map(float, row)) for row in csv.reader(f)])
aa = np.asarray(a)
result = ([np.tile(aa[i], aa[i, 1].astype(int)) for i in range(aa.shape[0])])result = np.asarray(result)

Outcome

[[1, 23.2, 55, 0, 1.1],
[3, 6.6, 0.2, 5, 9,
3, 6.6, 0.2, 5, 9,
3, 6.6, 0.2, 5, 9],
[1, 2.2, 5, 8, 9],
[2, 3.3, 10, 2, 2,
2, 3.3, 10, 2, 2]]

[[2, 23.2 55, 0, 1.1,
2, 23.2 55, 0, 1,1],
[3, 6.6, 0.2, 5, 9,
3, 6.6, 0.2, 5, 9,
3, 6.6, 0.2, 5, 9],
[1, 2.2, 5, 8, 9],
[1, 3.3, 10, 2, 2]]

Answer 1

Will this work for you? I converted your string from above into a list, then looped through each line and appended it to the final array

a = """mult, n1, n2, n3, n4
1, 23.2, 55, 0, 1.1
3, 6.6, 0.2, 5, 9
1, 2.2, 5, 8, 9
2, 3.3, 10, 2, 2
2, 23.2, 55, 0, 1.1
3, 6.6, 0.2, 5, 9
1, 2.2, 5, 8, 9
1, 3.3, 10, 2, 2"""

a = a.split('\n')

final = a[0]
for line in a[1:]:
    for i in range(int(line[0])):
        final.append(line)

Answer 2

You can create a generator function that accepts an iterable of rows (such as a csv.reader ), inspects each row to determine how many times it should be repeated, and yields each row the required number of times.

import csv


def generate_rows(rows):
    for row in rows:
        num_repeats = int(row[0])
        for _ in range(num_repeats):
            yield row


if __name__ == '__main__':
    with open('test.csv', newline='') as f:
        reader = csv.reader(f)
        next(reader)    # skip headers
        for row in generate_rows(reader):
            print(row)

Given your first example csv, the program produces this output:

['1', '23.2', '55', '0', '1.1']
['3', '6.6', '0.2', '5', '9']
['3', '6.6', '0.2', '5', '9']
['3', '6.6', '0.2', '5', '9']
['1', '2.2', '5', '8', '9']
['2', '3.3', '10', '2', '2']
['2', '3.3', '10', '2', '2']

If you want to collect the output in a list, just call list on the generator function:

rows = list(generate_rows(iterable))