[英]pandas read_csv and filter columns with usecols
I have a csv file which isn't coming in correctly with pandas.read_csv
when I filter the columns with usecols
and use multiple indexes.当我使用
pandas.read_csv
过滤列并使用多个索引时,我有一个 csv 文件无法正确输入usecols
。
import pandas as pd
csv = r"""dummy,date,loc,x
bar,20090101,a,1
bar,20090102,a,3
bar,20090103,a,5
bar,20090101,b,1
bar,20090102,b,3
bar,20090103,b,5"""
f = open('foo.csv', 'w')
f.write(csv)
f.close()
df1 = pd.read_csv('foo.csv',
header=0,
names=["dummy", "date", "loc", "x"],
index_col=["date", "loc"],
usecols=["dummy", "date", "loc", "x"],
parse_dates=["date"])
print df1
# Ignore the dummy columns
df2 = pd.read_csv('foo.csv',
index_col=["date", "loc"],
usecols=["date", "loc", "x"], # <----------- Changed
parse_dates=["date"],
header=0,
names=["dummy", "date", "loc", "x"])
print df2
I expect that df1 and df2 should be the same except for the missing dummy column, but the columns come in mislabeled.我希望 df1 和 df2 除了缺少虚拟列之外应该是相同的,但是这些列的标签错误。 Also the date is getting parsed as a date.
日期也被解析为日期。
In [118]: %run test.py
dummy x
date loc
2009-01-01 a bar 1
2009-01-02 a bar 3
2009-01-03 a bar 5
2009-01-01 b bar 1
2009-01-02 b bar 3
2009-01-03 b bar 5
date
date loc
a 1 20090101
3 20090102
5 20090103
b 1 20090101
3 20090102
5 20090103
Using column numbers instead of names give me the same problem.使用列号而不是名称会给我同样的问题。 I can workaround the issue by dropping the dummy column after the read_csv step, but I'm trying to understand what is going wrong.
我可以通过在 read_csv 步骤之后删除虚拟列来解决此问题,但我试图了解出了什么问题。 I'm using pandas 0.10.1.
我正在使用熊猫 0.10.1。
edit: fixed bad header usage.编辑:修复了错误的标头使用。
The solution lies in understanding these two keyword arguments:解决方案在于理解这两个关键字参数:
usecols
) using column names rather than integer indices.usecols
)时,才需要使用名称。 So because you have a header row, passing header=0
is sufficient and additionally passing names
appears to be confusing pd.read_csv
.因此,因为您有一个标题行,所以传递
header=0
就足够了,另外传递names
似乎令人困惑pd.read_csv
。
Removing names
from the second call gives the desired output:从第二个调用中删除
names
会给出所需的输出:
import pandas as pd
from StringIO import StringIO
csv = r"""dummy,date,loc,x
bar,20090101,a,1
bar,20090102,a,3
bar,20090103,a,5
bar,20090101,b,1
bar,20090102,b,3
bar,20090103,b,5"""
df = pd.read_csv(StringIO(csv),
header=0,
index_col=["date", "loc"],
usecols=["date", "loc", "x"],
parse_dates=["date"])
Which gives us:这给了我们:
x
date loc
2009-01-01 a 1
2009-01-02 a 3
2009-01-03 a 5
2009-01-01 b 1
2009-01-02 b 3
2009-01-03 b 5
This code achieves what you want --- also its weird and certainly buggy:这段代码实现了你想要的——它也很奇怪,而且肯定有问题:
I observed that it works when:我观察到它在以下情况下起作用:
a) you specify the index_col
rel. a)您指定
index_col
rel。 to the number of columns you really use -- so its three columns in this example, not four (you drop dummy
and start counting from then onwards)到你真正使用的列数——所以在这个例子中它是三列,而不是四列(你放下
dummy
并从那时起开始计数)
b) same for parse_dates
b)
parse_dates
相同
c) not so for usecols
;) for obvious reasons c)
usecols
不是这样;) 出于显而易见的原因
d) here I adapted the names
to mirror this behaviour d)在这里我修改了
names
以反映这种行为
import pandas as pd
from StringIO import StringIO
csv = """dummy,date,loc,x
bar,20090101,a,1
bar,20090102,a,3
bar,20090103,a,5
bar,20090101,b,1
bar,20090102,b,3
bar,20090103,b,5
"""
df = pd.read_csv(StringIO(csv),
index_col=[0,1],
usecols=[1,2,3],
parse_dates=[0],
header=0,
names=["date", "loc", "", "x"])
print df
which prints哪个打印
x
date loc
2009-01-01 a 1
2009-01-02 a 3
2009-01-03 a 5
2009-01-01 b 1
2009-01-02 b 3
2009-01-03 b 5
If your csv file contains extra data, columns can be deleted from the DataFrame after import.如果您的 csv 文件包含额外数据,则可以在导入后从 DataFrame 中删除列。
import pandas as pd
from StringIO import StringIO
csv = r"""dummy,date,loc,x
bar,20090101,a,1
bar,20090102,a,3
bar,20090103,a,5
bar,20090101,b,1
bar,20090102,b,3
bar,20090103,b,5"""
df = pd.read_csv(StringIO(csv),
index_col=["date", "loc"],
usecols=["dummy", "date", "loc", "x"],
parse_dates=["date"],
header=0,
names=["dummy", "date", "loc", "x"])
del df['dummy']
Which gives us:这给了我们:
x
date loc
2009-01-01 a 1
2009-01-02 a 3
2009-01-03 a 5
2009-01-01 b 1
2009-01-02 b 3
2009-01-03 b 5
You have to just add the index_col=False
parameter您只需添加
index_col=False
参数
df1 = pd.read_csv('foo.csv',
header=0,
index_col=False,
names=["dummy", "date", "loc", "x"],
usecols=["dummy", "date", "loc", "x"],
parse_dates=["date"])
print df1
首先导入 csv 并使用 csv.DictReader 其易于处理...
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.