MXNET-無效的類型&#39; <type 'numpy.ndarray'> &#39;用於數據，應為NDArray，numpy.ndarray，

Question

我在使用mxnet進行基本IO時遇到麻煩。 我正在嘗試使用mxnet.io.NDArrayIter讀取內存數據集以在mxnet中進行訓練。 我有下面的代碼（為簡潔起見，簡明扼要），該代碼對代碼進行預處理並嘗試對其進行迭代（很大程度上基於本教程 ）：

import csv
import mxnet as mx
import numpy as np

from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.pipeline import Pipeline


with open('data.csv', 'r') as data_file:
    data = list(csv.reader(data_file))

labels = np.array(map(lambda x: x[1], data)) # one-hot encoded classes
data = map(lambda x: x[0], data) # raw text in need of pre-processing

transformer = Pipeline(steps=(('count_vectorizer', CountVectorizer()),
                              ('tfidf_transformer', TfidfTransformer())))

preprocessed_data = np.array([np.array(row) for row in transformer.fit_transform(data)])

training_data = mx.io.NDArrayIter(data=preprocessed_data, label=labels, batch_size=50)

for i, batch in enumerate(training_data):
    print(batch)

執行此代碼時，出現以下錯誤：

    Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/mxnet/io.py", line 510, in _init_data
    data[k] = array(v)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/utils.py", line 146, in array
    return _array(source_array, ctx=ctx, dtype=dtype)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/ndarray.py", line 2245, in array
    arr[:] = source_array
  File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/ndarray.py", line 437, in __setitem__
    self._set_nd_basic_indexing(key, value)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/ndarray.py", line 698, in _set_nd_basic_indexing
    self._sync_copyfrom(value)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/ndarray.py", line 856, in _sync_copyfrom
    source_array = np.ascontiguousarray(source_array, dtype=self.dtype)
  File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 581, in ascontiguousarray
    return array(a, dtype, copy=False, order='C', ndmin=1)
TypeError: float() argument must be a string or a number, not 'csr_matrix'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "mxnet_test.py", line 20, in <module>
    training_data = mx.io.NDArrayIter(data=preprocessed_data, label=labels, batch_size=50)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/io.py", line 643, in __init__
    self.data = _init_data(data, allow_empty=False, default_name=data_name)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/io.py", line 513, in _init_data
    "should be NDArray, numpy.ndarray or h5py.Dataset")
TypeError: Invalid type '<class 'numpy.ndarray'>' for data, should be NDArray, numpy.ndarray or h5py.Dataset

我不明白，我的數據被轉換為numpy.ndarray創建之前NDArrayIter實例。 有人願意提供有關如何在mxnet讀取數據的mxnet嗎？

上面的代碼當前使用以下版本：

• mxnet-1.1.0
• numpy的-1.14.2

Answer 1

在user2357112的幫助下，此問題通過使用Python 3中的異常鏈接來查找異常（正在更新中）而得以解決：

transformer管道返回的是scipy.sparse.csr_matrix矩陣的numpy.array ，而不是numpy.array 。 通過添加更改以下行以改為使用toarray方法進行轉換，腳本將運行。

preprocessed_data = np.array([row.toarray() for row in transformer.fit_transform(data)])

最佳解決方案 ：在toarray上使用scipy.sparse.csr_matrix時，在內存消耗方面效率低下。 在mxnet 1.10版本中，可以使用mxnet.nd.sparse.array來更有效地存儲數據：

...
preprocessed_data = mx.nd.sparse.array(transformer.fit_transform(data))

training_data = mx.io.NDArrayIter(data=preprocessed_data, label=preprocessed_labels, batch_size=5, last_batch_handle='discard')

for i, batch in enumerate(training_data):
    print(batch)

與唯一需要注意的是，人們必須使用last_batch_handle='discard'關鍵字參數在NDArrayIter （功能last_batch_handle 這里 ）