稀疏与密集向量PySpark

Question

How can I know whether or not I should use a sparse or dense representation in PySpark? I understand the differences between them (sparse saves memory by only storing the non-zero indices and values), but performance-wise, are there any general heuristics that describe when to use sparse vectors over dense ones?

Is there a general "cutoff" dimension and percent of 0 values beyond which it is generally better to use sparse vectors? If not, how should I go about making the decision? Thanks.

Answer 1

The thing to remember is that pyspark.ml.linalg.Vector and pyspark.mllib.linalg.Vector are just compatibility layer between Python and Java API. There are not full featured or optimized linear algebra utilities and you shouldn't use them as such. The available operations are either not designed for performance or just convert to standard NumPy array under the covers.

When used with other ml / mllib tools there will be serialized and converted to Java equivalents so Python representation performance is mostly inconsequential.

This means that the biggest real concern is storage and a simple rule of thumb is:

• If on average half of the entries is zero it is better to use SparseVector .
• Otherwise it is better to use DenseVector .