Python：将 CSV 列解析为 np.array 的行和列索引？

Question

My CSV file is formatted as such:

Id,Prediction
r1_c1,1
r3_c1,3
...

When I read the csv file as such:

df = pd.read_csv('data/data_train.csv', delimiter=',')

I can get a matrix of size (N,1), where N is the number of values present in my input CSV file. Do notice that there are some missing values inside my input CSV file, therefore I cannot do a simple np.reshape

Is there a fancy function or procedure, within pandas or np that fills a matrix A such that A[i][j] = v_ij , where v_ij is the value with associated 'Id' equal to ri_cj ?

One could do it evidently with a for loop, but consider when the size of the input CSV file is rather large... one would be interested in leveraging the parallelism/vectorization implemented in numpy , for example. I couldn't describe my problem with keywords, so apologies if I couldn't find the associated documentation.

Answer 1

First, while reading csv file with pd.read_csv read Prediction column as pre-defined dtype (in the case of this example it's np.int32 ) just to make it more efficient.

Then run the fallowing code:

# Parse Id column string values as integer pairs of output array indices.
indices = np.empty((len(df), 2), np.int32)
for i, id_string in enumerate(df.Id):
    id_parts = id_string.split('_')
    indices[i, :] = int(id_parts[0][1:]), int(id_parts[1][1:])

# Infer shape of output array (OPTIONAL: used if output shape is not given).
n, m = np.max(indices, axis=0) + 1

# Create predictions output array with the same dtype.
out = np.zeros((n, m), df.dtypes.Prediction)

# Assign Prediction column values.
out[indices[:, 0], indices[:, 1]] = df.Prediction

Small test data:

    Id  Prediction
0   r0_c2   5
1   r1_c1   8
2   r2_c0   7

Output (without specifying output shape (n, m) ):

[[0 0 5]
 [0 8 0]
 [7 0 0]]

I've also timed-it using %%timeit function while running 900000 rows of data with output shape of (1000, 1000) and here are the results:

1 loop, best of 3: 1.61 s per loop

I hope this answers your question, good luck!