[英]Convert pandas dataframe to NumPy array
How do I convert a pandas dataframe into a NumPy array?如何将 pandas dataframe 转换为 Z3B7F949B2343F9E5390E29F6EF5E1778Z 数组?
DataFrame: DataFrame:
import numpy as np
import pandas as pd
index = [1, 2, 3, 4, 5, 6, 7]
a = [np.nan, np.nan, np.nan, 0.1, 0.1, 0.1, 0.1]
b = [0.2, np.nan, 0.2, 0.2, 0.2, np.nan, np.nan]
c = [np.nan, 0.5, 0.5, np.nan, 0.5, 0.5, np.nan]
df = pd.DataFrame({'A': a, 'B': b, 'C': c}, index=index)
df = df.rename_axis('ID')
gives给
label A B C
ID
1 NaN 0.2 NaN
2 NaN NaN 0.5
3 NaN 0.2 0.5
4 0.1 0.2 NaN
5 0.1 0.2 0.5
6 0.1 NaN 0.5
7 0.1 NaN NaN
I would like to convert this to a NumPy array, like so:我想将其转换为 Z3B7F949B2343F9E5390E29F6EF5E1778Z 数组,如下所示:
array([[ nan, 0.2, nan],
[ nan, nan, 0.5],
[ nan, 0.2, 0.5],
[ 0.1, 0.2, nan],
[ 0.1, 0.2, 0.5],
[ 0.1, nan, 0.5],
[ 0.1, nan, nan]])
Also, is it possible to preserve the dtypes, like this?另外,是否可以像这样保留数据类型?
array([[ 1, nan, 0.2, nan],
[ 2, nan, nan, 0.5],
[ 3, nan, 0.2, 0.5],
[ 4, 0.1, 0.2, nan],
[ 5, 0.1, 0.2, 0.5],
[ 6, 0.1, nan, 0.5],
[ 7, 0.1, nan, nan]],
dtype=[('ID', '<i4'), ('A', '<f8'), ('B', '<f8'), ('B', '<f8')])
df.to_numpy()
is better than df.values
, here's why. df.to_numpy()
比df.values
好,这就是原因。 * * It's time to deprecate your usage of values
and as_matrix()
.是时候弃用values
和as_matrix()
了。
pandas v0.24.0
introduced two new methods for obtaining NumPy arrays from pandas objects: pandas v0.24.0
引入了两种从 pandas 对象获取 NumPy 数组的新方法:
to_numpy()
, which is defined on Index
, Series
, and DataFrame
objects, and to_numpy()
,在Index
、 Series
和DataFrame
对象上定义,以及array
, which is defined on Index
and Series
objects only. array
,仅在Index
和Series
对象上定义。 If you visit the v0.24 docs for .values
, you will see a big red warning that says:如果您访问 .values 的.values
文档,您会看到一个大红色警告,上面写着:
Warning: We recommend using
DataFrame.to_numpy()
instead.警告:我们建议改用DataFrame.to_numpy()
。
See this section of the v0.24.0 release notes , and this answer for more information.请参阅v0.24.0 发行说明的这一部分,以及此答案以获取更多信息。
* - to_numpy()
is my recommended method for any production code that needs to run reliably for many versions into the future. * - to_numpy()
是我推荐的任何生产代码的方法,这些生产代码需要在未来的许多版本中可靠运行。 However if you're just making a scratchpad in jupyter or the terminal, using .values
to save a few milliseconds of typing is a permissable exception.但是,如果您只是在 jupyter 或终端中制作暂存器,则使用.values
来节省几毫秒的输入时间是一个允许的例外。 You can always add the fit n finish later.您可以随时添加适合 n 完成以后。
to_numpy()
实现更好的一致性: to_numpy()
In the spirit of better consistency throughout the API, a new method to_numpy
has been introduced to extract the underlying NumPy array from DataFrames.本着在整个 API 中保持更好一致性的精神,引入了一种新方法to_numpy
来从 DataFrames 中提取底层 NumPy 数组。
# Setup
df = pd.DataFrame(data={'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]},
index=['a', 'b', 'c'])
# Convert the entire DataFrame
df.to_numpy()
# array([[1, 4, 7],
# [2, 5, 8],
# [3, 6, 9]])
# Convert specific columns
df[['A', 'C']].to_numpy()
# array([[1, 7],
# [2, 8],
# [3, 9]])
As mentioned above, this method is also defined on Index
and Series
objects (see here ).如上所述,此方法也在Index
和Series
对象上定义(参见此处)。
df.index.to_numpy()
# array(['a', 'b', 'c'], dtype=object)
df['A'].to_numpy()
# array([1, 2, 3])
By default, a view is returned, so any modifications made will affect the original.默认情况下,会返回一个视图,因此所做的任何修改都会影响原始视图。
v = df.to_numpy()
v[0, 0] = -1
df
A B C
a -1 4 7
b 2 5 8
c 3 6 9
If you need a copy instead, use to_numpy(copy=True)
.如果您需要副本,请使用to_numpy(copy=True)
。
If you're using pandas 1.x, chances are you'll be dealing with extension types a lot more.如果您使用的是 pandas 1.x,那么您可能会更多地处理扩展类型。 You'll have to be a little more careful that these extension types are correctly converted.您必须更加小心这些扩展类型是否正确转换。
a = pd.array([1, 2, None], dtype="Int64")
a
<IntegerArray>
[1, 2, <NA>]
Length: 3, dtype: Int64
# Wrong
a.to_numpy()
# array([1, 2, <NA>], dtype=object) # yuck, objects
# Correct
a.to_numpy(dtype='float', na_value=np.nan)
# array([ 1., 2., nan])
# Also correct
a.to_numpy(dtype='int', na_value=-1)
# array([ 1, 2, -1])
This is called out in the docs .这在 docs 中被提及。
dtypes
in the result...如果您需要结果中的dtypes
... As shown in another answer, DataFrame.to_records
is a good way to do this.如另一个答案所示, DataFrame.to_records
是一种很好的方法。
df.to_records()
# rec.array([('a', 1, 4, 7), ('b', 2, 5, 8), ('c', 3, 6, 9)],
# dtype=[('index', 'O'), ('A', '<i8'), ('B', '<i8'), ('C', '<i8')])
This cannot be done with to_numpy
, unfortunately.不幸的是,这不能用to_numpy
完成。 However, as an alternative, you can use np.rec.fromrecords
:但是,作为替代方案,您可以使用np.rec.fromrecords
:
v = df.reset_index()
np.rec.fromrecords(v, names=v.columns.tolist())
# rec.array([('a', 1, 4, 7), ('b', 2, 5, 8), ('c', 3, 6, 9)],
# dtype=[('index', '<U1'), ('A', '<i8'), ('B', '<i8'), ('C', '<i8')])
Performance wise, it's nearly the same (actually, using rec.fromrecords
is a bit faster).性能方面,几乎相同(实际上,使用rec.fromrecords
会快一点)。
df2 = pd.concat([df] * 10000)
%timeit df2.to_records()
%%timeit
v = df2.reset_index()
np.rec.fromrecords(v, names=v.columns.tolist())
12.9 ms ± 511 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
9.56 ms ± 291 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
to_numpy()
(in addition to array
) was added as a result of discussions under two GitHub issues GH19954 and GH23623 .由于在两个 GitHub 问题GH19954和GH23623下的讨论,添加了to_numpy()
(除了array
)。
Specifically, the docs mention the rationale:具体来说,文档提到了理由:
[...] with
.values
it was unclear whether the returned value would be the actual array, some transformation of it, or one of pandas custom arrays (likeCategorical
). [...].values
不清楚返回的值是实际数组、它的一些转换,还是熊猫自定义数组之一(如Categorical
)。 For example, withPeriodIndex
,.values
generates a newndarray
of period objects each time.例如,使用PeriodIndex
,.values
生成一个新的周期对象ndarray
。 [...] [...]
to_numpy
aims to improve the consistency of the API, which is a major step in the right direction. to_numpy
旨在提高 API 的一致性,这是朝着正确方向迈出的重要一步。 .values
will not be deprecated in the current version, but I expect this may happen at some point in the future, so I would urge users to migrate towards the newer API, as soon as you can. .values
在当前版本中不会被弃用,但我预计这可能会在未来的某个时候发生,所以我会敦促用户尽快迁移到更新的 API。
DataFrame.values
has inconsistent behaviour, as already noted.如前所述, DataFrame.values
的行为不一致。
DataFrame.get_values()
is simply a wrapper around DataFrame.values
, so everything said above applies. DataFrame.get_values()
只是DataFrame.values
的一个包装器,所以上面所说的一切都适用。
DataFrame.as_matrix()
is deprecated now, do NOT use! DataFrame.as_matrix()
现在已弃用,请勿使用!
To convert a pandas dataframe (df) to a numpy ndarray, use this code:要将 pandas 数据帧 (df) 转换为 numpy ndarray,请使用以下代码:
df.values
array([[nan, 0.2, nan],
[nan, nan, 0.5],
[nan, 0.2, 0.5],
[0.1, 0.2, nan],
[0.1, 0.2, 0.5],
[0.1, nan, 0.5],
[0.1, nan, nan]])
Note : The .as_matrix()
method used in this answer is deprecated.注意:此答案中使用的.as_matrix()
方法已弃用。 Pandas 0.23.4 warns:熊猫 0.23.4 警告:
Method
.as_matrix
will be removed in a future version.方法.as_matrix
将在未来版本中删除。 Use .values instead.请改用 .values。
Pandas has something built in... Pandas 内置了一些东西...
numpy_matrix = df.as_matrix()
gives给
array([[nan, 0.2, nan],
[nan, nan, 0.5],
[nan, 0.2, 0.5],
[0.1, 0.2, nan],
[0.1, 0.2, 0.5],
[0.1, nan, 0.5],
[0.1, nan, nan]])
I would just chain the DataFrame.reset_index() and DataFrame.values functions to get the Numpy representation of the dataframe, including the index:我将链接DataFrame.reset_index()和DataFrame.values函数来获取数据帧的 Numpy 表示,包括索引:
In [8]: df
Out[8]:
A B C
0 -0.982726 0.150726 0.691625
1 0.617297 -0.471879 0.505547
2 0.417123 -1.356803 -1.013499
3 -0.166363 -0.957758 1.178659
4 -0.164103 0.074516 -0.674325
5 -0.340169 -0.293698 1.231791
6 -1.062825 0.556273 1.508058
7 0.959610 0.247539 0.091333
[8 rows x 3 columns]
In [9]: df.reset_index().values
Out[9]:
array([[ 0. , -0.98272574, 0.150726 , 0.69162512],
[ 1. , 0.61729734, -0.47187926, 0.50554728],
[ 2. , 0.4171228 , -1.35680324, -1.01349922],
[ 3. , -0.16636303, -0.95775849, 1.17865945],
[ 4. , -0.16410334, 0.0745164 , -0.67432474],
[ 5. , -0.34016865, -0.29369841, 1.23179064],
[ 6. , -1.06282542, 0.55627285, 1.50805754],
[ 7. , 0.95961001, 0.24753911, 0.09133339]])
To get the dtypes we'd need to transform this ndarray into a structured array using view :要获得 dtypes,我们需要使用view将此 ndarray 转换为结构化数组:
In [10]: df.reset_index().values.ravel().view(dtype=[('index', int), ('A', float), ('B', float), ('C', float)])
Out[10]:
array([( 0, -0.98272574, 0.150726 , 0.69162512),
( 1, 0.61729734, -0.47187926, 0.50554728),
( 2, 0.4171228 , -1.35680324, -1.01349922),
( 3, -0.16636303, -0.95775849, 1.17865945),
( 4, -0.16410334, 0.0745164 , -0.67432474),
( 5, -0.34016865, -0.29369841, 1.23179064),
( 6, -1.06282542, 0.55627285, 1.50805754),
( 7, 0.95961001, 0.24753911, 0.09133339),
dtype=[('index', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])
You can use the to_records
method, but have to play around a bit with the dtypes if they are not what you want from the get go.您可以使用to_records
方法,但如果它们不是您从一开始就想要的,则必须使用 dtypes。 In my case, having copied your DF from a string, the index type is string (represented by an object
dtype in pandas):在我的例子中,从字符串中复制了你的 DF,索引类型是字符串(由 pandas 中的object
dtype 表示):
In [102]: df
Out[102]:
label A B C
ID
1 NaN 0.2 NaN
2 NaN NaN 0.5
3 NaN 0.2 0.5
4 0.1 0.2 NaN
5 0.1 0.2 0.5
6 0.1 NaN 0.5
7 0.1 NaN NaN
In [103]: df.index.dtype
Out[103]: dtype('object')
In [104]: df.to_records()
Out[104]:
rec.array([(1, nan, 0.2, nan), (2, nan, nan, 0.5), (3, nan, 0.2, 0.5),
(4, 0.1, 0.2, nan), (5, 0.1, 0.2, 0.5), (6, 0.1, nan, 0.5),
(7, 0.1, nan, nan)],
dtype=[('index', '|O8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])
In [106]: df.to_records().dtype
Out[106]: dtype([('index', '|O8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])
Converting the recarray dtype does not work for me, but one can do this in Pandas already:转换 recarray dtype 对我不起作用,但已经可以在 Pandas 中执行此操作:
In [109]: df.index = df.index.astype('i8')
In [111]: df.to_records().view([('ID', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])
Out[111]:
rec.array([(1, nan, 0.2, nan), (2, nan, nan, 0.5), (3, nan, 0.2, 0.5),
(4, 0.1, 0.2, nan), (5, 0.1, 0.2, 0.5), (6, 0.1, nan, 0.5),
(7, 0.1, nan, nan)],
dtype=[('ID', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])
Note that Pandas does not set the name of the index properly (to ID
) in the exported record array (a bug?), so we profit from the type conversion to also correct for that.请注意,Pandas 没有在导出的记录数组中正确设置索引的名称(到ID
)(一个错误?),所以我们从类型转换中受益,也纠正了这一点。
At the moment Pandas has only 8-byte integers, i8
, and floats, f8
(see this issue ).目前,Pandas 只有 8 字节整数i8
和浮点数f8
(请参阅本期)。
It seems like df.to_records()
will work for you.似乎df.to_records()
会为您工作。 The exact feature you're looking for was requested and to_records
pointed to as an alternative.您正在寻找的确切功能已被请求,并且to_records
指出作为替代。
I tried this out locally using your example, and that call yields something very similar to the output you were looking for:我使用您的示例在本地进行了尝试,该调用产生的输出与您正在寻找的输出非常相似:
rec.array([(1, nan, 0.2, nan), (2, nan, nan, 0.5), (3, nan, 0.2, 0.5),
(4, 0.1, 0.2, nan), (5, 0.1, 0.2, 0.5), (6, 0.1, nan, 0.5),
(7, 0.1, nan, nan)],
dtype=[(u'ID', '<i8'), (u'A', '<f8'), (u'B', '<f8'), (u'C', '<f8')])
Note that this is a recarray
rather than an array
.请注意,这是一个recarray
而不是array
。 You could move the result in to regular numpy array by calling its constructor as np.array(df.to_records())
.您可以通过将其构造函数调用为np.array(df.to_records())
将结果移动到常规 numpy 数组中。
尝试这个:
a = numpy.asarray(df)
Here is my approach to making a structure array from a pandas DataFrame.这是我从 pandas DataFrame 制作结构数组的方法。
Create the data frame创建数据框
import pandas as pd
import numpy as np
import six
NaN = float('nan')
ID = [1, 2, 3, 4, 5, 6, 7]
A = [NaN, NaN, NaN, 0.1, 0.1, 0.1, 0.1]
B = [0.2, NaN, 0.2, 0.2, 0.2, NaN, NaN]
C = [NaN, 0.5, 0.5, NaN, 0.5, 0.5, NaN]
columns = {'A':A, 'B':B, 'C':C}
df = pd.DataFrame(columns, index=ID)
df.index.name = 'ID'
print(df)
A B C
ID
1 NaN 0.2 NaN
2 NaN NaN 0.5
3 NaN 0.2 0.5
4 0.1 0.2 NaN
5 0.1 0.2 0.5
6 0.1 NaN 0.5
7 0.1 NaN NaN
Define function to make a numpy structure array (not a record array) from a pandas DataFrame.定义函数以从 pandas DataFrame 创建一个 numpy 结构数组(不是记录数组)。
def df_to_sarray(df):
"""
Convert a pandas DataFrame object to a numpy structured array.
This is functionally equivalent to but more efficient than
np.array(df.to_array())
:param df: the data frame to convert
:return: a numpy structured array representation of df
"""
v = df.values
cols = df.columns
if six.PY2: # python 2 needs .encode() but 3 does not
types = [(cols[i].encode(), df[k].dtype.type) for (i, k) in enumerate(cols)]
else:
types = [(cols[i], df[k].dtype.type) for (i, k) in enumerate(cols)]
dtype = np.dtype(types)
z = np.zeros(v.shape[0], dtype)
for (i, k) in enumerate(z.dtype.names):
z[k] = v[:, i]
return z
Use reset_index
to make a new data frame that includes the index as part of its data.使用reset_index
创建一个新的数据框,其中包含索引作为其数据的一部分。 Convert that data frame to a structure array.将该数据框转换为结构体数组。
sa = df_to_sarray(df.reset_index())
sa
array([(1L, nan, 0.2, nan), (2L, nan, nan, 0.5), (3L, nan, 0.2, 0.5),
(4L, 0.1, 0.2, nan), (5L, 0.1, 0.2, 0.5), (6L, 0.1, nan, 0.5),
(7L, 0.1, nan, nan)],
dtype=[('ID', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])
EDIT: Updated df_to_sarray to avoid error calling .encode() with python 3. Thanks to Joseph Garvin and halcyon for their comment and solution.编辑:更新 df_to_sarray 以避免使用 python 3 调用 .encode() 时出错。感谢Joseph Garvin和halcyon的评论和解决方案。
A Simpler Way for Example DataFrame:示例 DataFrame 的更简单方法:
df
gbm nnet reg
0 12.097439 12.047437 12.100953
1 12.109811 12.070209 12.095288
2 11.720734 11.622139 11.740523
3 11.824557 11.926414 11.926527
4 11.800868 11.727730 11.729737
5 12.490984 12.502440 12.530894
USE:利用:
np.array(df.to_records().view(type=np.matrix))
GET:得到:
array([[(0, 12.097439 , 12.047437, 12.10095324),
(1, 12.10981081, 12.070209, 12.09528824),
(2, 11.72073428, 11.622139, 11.74052253),
(3, 11.82455653, 11.926414, 11.92652727),
(4, 11.80086775, 11.72773 , 11.72973699),
(5, 12.49098389, 12.50244 , 12.53089367)]],
dtype=(numpy.record, [('index', '<i8'), ('gbm', '<f8'), ('nnet', '<f4'),
('reg', '<f8')]))
Two ways to convert the data-frame to its Numpy-array representation.将数据帧转换为其 Numpy 数组表示的两种方法。
mah_np_array = df.as_matrix(columns=None)
mah_np_array = df.values
Doc: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.as_matrix.html文档: https ://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.as_matrix.html
I went through the answers above.我浏览了上面的答案。 The " as_matrix() " method works but its obsolete now. “ as_matrix() ” 方法有效,但现在已过时。 For me, What worked was " .to_numpy() ".对我来说,有效的是“ .to_numpy() ”。
This returns a multidimensional array.这将返回一个多维数组。 I'll prefer using this method if you're reading data from excel sheet and you need to access data from any index.如果您从 excel 表中读取数据并且需要从任何索引访问数据,我会更喜欢使用此方法。 Hope this helps :)希望这可以帮助 :)
Just had a similar problem when exporting from dataframe to arcgis table and stumbled on a solution from usgs ( https://my.usgs.gov/confluence/display/cdi/pandas.DataFrame+to+ArcGIS+Table ).从数据框导出到 arcgis 表时遇到了类似的问题,偶然发现了 usgs 的解决方案( https://my.usgs.gov/confluence/display/cdi/pandas.DataFrame+to+ArcGIS+Table )。 In short your problem has a similar solution:简而言之,您的问题有类似的解决方案:
df
A B C
ID
1 NaN 0.2 NaN
2 NaN NaN 0.5
3 NaN 0.2 0.5
4 0.1 0.2 NaN
5 0.1 0.2 0.5
6 0.1 NaN 0.5
7 0.1 NaN NaN
np_data = np.array(np.rec.fromrecords(df.values))
np_names = df.dtypes.index.tolist()
np_data.dtype.names = tuple([name.encode('UTF8') for name in np_names])
np_data
array([( nan, 0.2, nan), ( nan, nan, 0.5), ( nan, 0.2, 0.5),
( 0.1, 0.2, nan), ( 0.1, 0.2, 0.5), ( 0.1, nan, 0.5),
( 0.1, nan, nan)],
dtype=(numpy.record, [('A', '<f8'), ('B', '<f8'), ('C', '<f8')]))
A simple way to convert dataframe to numpy array:将数据帧转换为 numpy 数组的简单方法:
import pandas as pd
df = pd.DataFrame({"A": [1, 2], "B": [3, 4]})
df_to_array = df.to_numpy()
array([[1, 3],
[2, 4]])
Use of to_numpy is encouraged to preserve consistency.鼓励使用 to_numpy 以保持一致性。
Reference: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_numpy.html参考: https ://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_numpy.html
Try this:尝试这个:
np.array(df)
array([['ID', nan, nan, nan],
['1', nan, 0.2, nan],
['2', nan, nan, 0.5],
['3', nan, 0.2, 0.5],
['4', 0.1, 0.2, nan],
['5', 0.1, 0.2, 0.5],
['6', 0.1, nan, 0.5],
['7', 0.1, nan, nan]], dtype=object)
Some more information at: [ https://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html] Valid for numpy 1.16.5 and pandas 0.25.2.更多信息位于:[ https://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html]适用于 numpy 1.16.5 和 pandas 0.25.2。
Further to meteore's answer, I found the code除了流星的回答,我找到了代码
df.index = df.index.astype('i8')
doesn't work for me.对我不起作用。 So I put my code here for the convenience of others stuck with this issue.所以我把我的代码放在这里是为了方便其他人遇到这个问题。
city_cluster_df = pd.read_csv(text_filepath, encoding='utf-8')
# the field 'city_en' is a string, when converted to Numpy array, it will be an object
city_cluster_arr = city_cluster_df[['city_en','lat','lon','cluster','cluster_filtered']].to_records()
descr=city_cluster_arr.dtype.descr
# change the field 'city_en' to string type (the index for 'city_en' here is 1 because before the field is the row index of dataframe)
descr[1]=(descr[1][0], "S20")
newArr=city_cluster_arr.astype(np.dtype(descr))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.