[英]Substitute for numpy broadcasting using scipy.sparse.csc_matrix
I have in my code the following expression: 我的代码中有以下表达式:
a = (b / x[:, np.newaxis]).sum(axis=1)
where b
is an ndarray of shape (M, N)
, and x
is an ndarray of shape (M,)
. 其中b
是形状的ndarray (M, N)
, x
是形状的ndarray (M,)
。 Now, b
is actually sparse, so for memory efficiency I would like to substitute in a scipy.sparse.csc_matrix
or csr_matrix
. 现在, b
实际上是稀疏的,所以为了提高内存效率,我想在scipy.sparse.csc_matrix
或csr_matrix
。 However, broadcasting in this way is not implemented (even though division or multiplication is guaranteed to maintain sparsity) (the entries of x
are non-zero), and raises a NotImplementedError
. 但是,没有实现这种方式的广播(即使保证分割或乘法保持稀疏性)( x
的条目非零),并引发NotImplementedError
。 Is there a sparse
function I'm not aware of that would do what I want? 是否有sparse
功能我不知道会做我想要的? ( dot()
would sum along the wrong axis.) ( dot()
将沿错误的轴相加。)
If b
is in CSC format, then b.data
has the non-zero entries of b
, and b.indices
has the row index of each of the non-zero entries, so you can do your division as: 如果b
是CSC格式,则b.data
具有b
的非零条目,并且b.indices
具有每个非零条目的行索引,因此您可以将您的除法视为:
b.data /= np.take(x, b.indices)
It's hackier than Warren's elegant solution, but it will probably also be faster in most settings: 它比Warren的优雅解决方案更为讨厌,但在大多数情况下它可能也会更快:
b = sps.rand(1000, 1000, density=0.01, format='csc')
x = np.random.rand(1000)
def row_divide_col_reduce(b, x):
data = b.data.copy() / np.take(x, b.indices)
ret = sps.csc_matrix((data, b.indices.copy(), b.indptr.copy()),
shape=b.shape)
return ret.sum(axis=1)
def row_divide_col_reduce_bis(b, x):
d = sps.spdiags(1.0/x, 0, len(x), len(x))
return (d * b).sum(axis=1)
In [2]: %timeit row_divide_col_reduce(b, x)
1000 loops, best of 3: 210 us per loop
In [3]: %timeit row_divide_col_reduce_bis(b, x)
1000 loops, best of 3: 697 us per loop
In [4]: np.allclose(row_divide_col_reduce(b, x),
...: row_divide_col_reduce_bis(b, x))
Out[4]: True
You can cut the time almost in half in the above example if you do the division in-place, ie: 如果你就地进行划分,你可以在上面的例子中将时间减少一半,即:
def row_divide_col_reduce(b, x):
b.data /= np.take(x, b.indices)
return b.sum(axis=1)
In [2]: %timeit row_divide_col_reduce(b, x)
10000 loops, best of 3: 131 us per loop
To implement a = (b / x[:, np.newaxis]).sum(axis=1)
, you can use a = b.sum(axis=1).A1 / x
. 要实现a = (b / x[:, np.newaxis]).sum(axis=1)
,可以使用a = b.sum(axis=1).A1 / x
。 The A1
attribute returns the 1D ndarray, so the result is a 1D ndarray, not a matrix
. A1
属性返回1D ndarray,因此结果是1D ndarray,而不是matrix
。 This concise expression works because you are both scaling by x
and summing along axis 1. For example: 这个简洁的表达式有效,因为您既可以按x
缩放, 也可以沿轴1求和。例如:
In [190]: b
Out[190]:
<3x3 sparse matrix of type '<type 'numpy.float64'>'
with 5 stored elements in Compressed Sparse Row format>
In [191]: b.A
Out[191]:
array([[ 1., 0., 2.],
[ 0., 3., 0.],
[ 4., 0., 5.]])
In [192]: x
Out[192]: array([ 2., 3., 4.])
In [193]: b.sum(axis=1).A1 / x
Out[193]: array([ 1.5 , 1. , 2.25])
More generally, if you want to scale the rows of a sparse matrix with a vector x
, you could multiply b
on the left with a sparse matrix containing 1.0/x
on the diagonal. 更一般地说,如果要使用向量x
缩放稀疏矩阵的行,可以将左侧的b
乘以对角线上包含1.0/x
的稀疏矩阵。 The function scipy.sparse.spdiags
can be used to create such a matrix. 函数scipy.sparse.spdiags
可用于创建这样的矩阵。 For example: 例如:
In [71]: from scipy.sparse import csc_matrix, spdiags
In [72]: b = csc_matrix([[1,0,2],[0,3,0],[4,0,5]], dtype=np.float64)
In [73]: b.A
Out[73]:
array([[ 1., 0., 2.],
[ 0., 3., 0.],
[ 4., 0., 5.]])
In [74]: x = array([2., 3., 4.])
In [75]: d = spdiags(1.0/x, 0, len(x), len(x))
In [76]: d.A
Out[76]:
array([[ 0.5 , 0. , 0. ],
[ 0. , 0.33333333, 0. ],
[ 0. , 0. , 0.25 ]])
In [77]: p = d * b
In [78]: p.A
Out[78]:
array([[ 0.5 , 0. , 1. ],
[ 0. , 1. , 0. ],
[ 1. , 0. , 1.25]])
In [79]: a = p.sum(axis=1)
In [80]: a
Out[80]:
matrix([[ 1.5 ],
[ 1. ],
[ 2.25]])
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.