将大量索引（从Pandas数据帧）加载到稀疏矩阵的快速方法？

Question

I've got a large Pandas dataframe with 1.500.000 rows, and one column contains lists with numbers. You can imagine it like this

df = pd.DataFrame({'lists' : [[0, 1, 2], [6, 7, 8], [3, 4, 5]]})

but way bigger. in the end I want a matrix, that looks like this

[1, 1, 1, 0, 0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 1, 1, 1]
[0, 0, 0, 1, 1, 1, 0, 0, 0]

so the row index of the df is the row index of the matrix and the numbers in the list are the column indexes that need to be set to True.

The matrix will be of shape 1.500.000 x 30.000, but this takes too much RAM, so I save the matrix with lil_matrix(), and then later I can form the matrix batch for batch.

The way I do this right now is the following:

sparse_matrix = sparse.lil_matrix((1.500.000, 30.000), dtype=bool)
list_with_lists = df["lists"].tolist()
for i, list in enumerate(list_with_lists):
    for number in list:
        sparse_matrix[i, number] = True

It works, but it takes a couple of minutes and I really hope there is a faster way as this takes too much times. Does anyone know a faster way?

Answer 1

Not sure how this will work with the scipy.sparse.lil_matrix , but try using advanced indexing:

rows = np.arange(m.shape[0])[:, np.newaxis]
cols = df['lists'].tolist()
m[rows, cols] = 1

Basically, we're saying set every [row, column] pair found here to True . row looks like [[1], [2], [3], ..., N] for a N * M matrix and cols is your series.

With a test case

import pandas as pd
import numpy as np

df = pd.DataFrame({'lists' : [[0, 1, 2], [6, 7, 8], [3, 4, 5]]})

m = np.zeros((3, 9), dtype=bool)

rows = np.arange(m.shape[0])[:, np.newaxis]
cols = df['lists'].tolist()
m[rows, cols] = True

print(m.view(np.int8))

I get

[[1 1 1 0 0 0 0 0 0]
 [0 0 0 0 0 0 1 1 1]
 [0 0 0 1 1 1 0 0 0]]

Answer 2

You can try dok_matrix with its update function. You will need to prepare a list of the form ((row_idx, col_idx), val) and pass it in the update function. Here, I tried to use the map and reduce to create the list.

from itertools import chain
from scipy import sparse

df = pd.DataFrame({'lists' : [[0, 1, 2], [6, 7, 8], [3, 4, 5]]})
sparse_matrix = sparse.dok_matrix((1500000, 30000), dtype=bool)
list_with_lists = df["lists"].tolist()


update_list = chain.from_iterable(map(lambda l, r: [((r, i), 1) for i in l], 
                                      list_with_lists, 
                                      range(len(list_with_lists))))    

# update_list 
[((0, 0), 1),
 ((0, 1), 1),
 ((0, 2), 1),
 ((1, 6), 1),
 ((1, 7), 1),
 ((1, 8), 1),
 ((2, 3), 1),
 ((2, 4), 1),
 ((2, 5), 1)]

sparse_matrix.update(update_list)

Timing

Setup

from numpy.random import randint
df = pd.DataFrame({'lists' : [randint(0, 30000, 10) for i in range(10000)]})
list_with_lists = df["lists"].tolist()

Using lil_matrix with double-loop update

sparse_matrix = sparse.lil_matrix((1500000, 30000), dtype=bool)
%%timeit  # OP's original way of adding using lil_matrix
for i, list in enumerate(list_with_lists):
    for number in list:
        sparse_matrix[i, number] = True
635 ms ± 26.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Using dok_matrix with update .

sparse_matrix = sparse.dok_matrix((1500000, 30000), dtype=bool)
%%timeit # updating using `update` function in dok_matrix
update_list = chain.from_iterable(map(lambda l, r: [((r, i), 1) for i in l], 
                              list_with_lists, 
                              range(len(list_with_lists))))    
sparse_matrix.update(update_list)
48.7 ms ± 6.84 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

---

dok_matrix , however, may be slow in other matrix operations.