使用pyspark SparseVectors解析所有零稀疏向量

Question

In pyspark, if I generate a sparse vector that represents an all zero vector and then stringify it it works as expected:

>>> res = Vectors.stringify(SparseVector(4, [], []))
'(4,[],[])'

But then the parse method fails to load this back:

>>> SparseVector.parse(res)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../spark-1.5.2-bin-hadoop2.4/python/pyspark/mllib/linalg/__init__.py", line 545, in parse
    raise ValueError("Unable to parse indices from %s." % new_s)
ValueError: Unable to parse indices from .

Anyone knows of the way to solve this?

Answer 1

This is a bug described by SPARK-14739 . The simplest workaround for now is to use ast module instead:

import ast
from pyspark.mllib.linalg import SparseVector

def parse_sparse(s):
    return SparseVector(*ast.literal_eval(s.strip()))

parse_sparse("(1, [], [])")
## SparseVector(1, {})

parse_sparse("(5, [1, 3], [0.4, -0.1])")
## SparseVector(5, {1: 0.4, 3: -0.1})