[英]Encoding error when reading csv file containing pandas dataframe
I used df.to_csv()
to convert a dataframe to csv file. 我使用df.to_csv()
将数据帧转换为csv文件。 Under python 3 the pandas doc states that it defaults to utf-8 encoding. 在python 3下, pandas doc声明其默认为utf-8编码。
However when I run pd.read_csv()
on the same file, I get the error: 但是,当我在同一文件上运行pd.read_csv()
时,出现错误:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xae in position 8: invalid start byte
But using pd.read_csv()
with encoding="ISO-8859-1"
works. 但是将pd.read_csv()
与encoding="ISO-8859-1"
一起使用是pd.read_csv()
。
What is the issue here and how do I resolve it so I can write and load files with consistent encoding? 这是什么问题,我该如何解决,这样才能以一致的编码方式写入和加载文件?
The original .csv
you are trying to read is encoded
in eg ISO-8859-1
. 原来.csv
你试图读取encoded
在如ISO-8859-1
。 That's why it's a UnicodeDecodeError
- python / pandas is trying to decode
the source using utf-8
codec assuming per default the source is unicode
. 这就是为什么它是一个UnicodeDecodeError
-蟒蛇/大熊猫正试图decode
使用源utf-8
编码解码器每默认假设源是unicode
。
Once you indicate the non-default source encoding, pandas will use the proper codec to match the source and decode into the format used internally. 指明非默认源编码后,熊猫将使用适当的编解码器来匹配源并解码为内部使用的格式。
See python docs and more here . 在这里查看python docs和更多内容。 Also very good. 也很好。
请尝试使用encoding ='unicode_escape'读取数据。
Here is a concrete example of pandas using some unknown(?) encoding when not explicitly using the encoding
parameter with pandas.to_csv
. 这是未使用pandas.to_csv
明确使用encoding
参数时,使用某些unknown(?)编码的熊猫的具体示例。
0x92 is ' (looks like an apostrophe) 0x92是'(看起来像撇号)
import pandas
ERRORFILE = r'written_without_encoding_parameter.csv'
NO_ERRORFILE = r'written_WITH_encoding_parameter.csv'
df_dummy = pandas.DataFrame([u"Yo what's up", u"I like your sister’s friend"])
df_dummy.to_csv(ERRORFILE)
df_dummy.to_csv(NO_ERRORFILE, encoding="utf-8")
df_no_error_with_latin = pandas.read_csv(ERRORFILE, encoding="Latin-1")
df_no_error = pandas.read_csv(NO_ERRORFILE)
df_error = pandas.read_csv(ERRORFILE)
>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 18: invalid start byte
So it looks like you have to explicitly use encoding="utf-8"
with to_csv
even though pandas docs say it is using this by default. 因此,即使熊猫文档说默认情况下使用它,您似乎也必须对to_csv
显式使用encoding="utf-8"
。 Or use encoding="Latin-1"
with read_csv
. 或者,将encoding="Latin-1"
与read_csv
一起read_csv
。
Even more frustrating... 更令人沮丧的是...
df_error_even_with_utf8 = pandas.read_csv(ERRORFILE, encoding="utf-8")
>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 18: invalid start byte
I am using Windows 7, Python 3.5, pandas 0.19.2. 我正在使用Windows 7,Python 3.5,熊猫0.19.2。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.