如何读取 Pandas 中的 txt 文件：错误标记数据

Question

问题：我用pandas.read_csv读取txt文件，但是有一些错误。 流程如下图：

1. import pandas as pd

2. txt 文件的路径： './Data/fold2_l25431/test.txt'

3. 例子test.txt的内容：（txt的前三行，读入的时候，想分成三列，'1','2','3'在一列，'persona'在一列, 以及一栏中冒号后的句子）

First line: 1 persona: i am adorkable.
Second line: 2 persona: i am book dumb.
Third line: 3 persona: i am token evil teammate.

4. 代码： pd.read_csv('./Data/fold2_l25431/test.txt')或pd.read_csv('./Data/fold2_l25431/test.txt', sep=" ")

5. ParserError: Error tokenizing data. C error: Expected 8 fields in line 6, saw 9

Answer 1

尝试这个：

 import pandas as pd
 pd.read_csv( 'test.txt',header=None ,on_bad_lines='skip')

Answer 2

我无法重现您的错误。

一般建议：

• 只对 csv 个文件使用 read_csv
• 如果必须使用 txt，请指定分隔符sep=":"和换行符l.neterminator="\n"
• 如果您认为某些数据可能无效，请使用on_bad_lines="skip"并检查您的 output

Answer 3

错误的原因是 SPACE (sep = " ") 。 使用其他东西（如,或|来分隔字段。用逗号更新的表看起来像这样

1, persona:, i am adorkable.
2, persona:, i am book dumb.
3, persona:, i am token evil teammate.
4, persona:, i am never my fault.
5, persona:, i am honor before reason.
6, persona:, i am jerk with a heart of gold.
7, persona:, i am no social skills.
8, persona:, i am bad liar

.. 应该使用此命令pd.read_csv('test1.txt', sep = ",", header = None)

output 将是

0   1   2
1   persona:    i am adorkable.
2   persona:    i am book dumb.
3   persona:    i am token evil teammate.
4   persona:    i am never my fault.
5   persona:    i am honor before reason.
6   persona:    i am jerk with a heart of gold.
7   persona:    i am no social skills.
8   persona:    i am bad liar

Answer 4

您的文件不是 csv，因此您可能必须自己编写 function 才能读取它并拆分为列

我使用io只是为了模拟 memory 中的文件 - 所以每个人都可以复制和测试它 - 但你应该使用open()

text  = '''1 persona: i am adorkable.
2 persona: i am book dumb.
3 persona: i am token evil teammate.
4 persona: i am never my fault.
5 persona: i am honor before reason.
6 persona: i am jerk with a heart of gold.
7 persona: i am no social skills.
8 persona: i am bad liar'''

import io

#f = open('./Data/fold2_l25431/test.txt')
f = io.StringIO(text)

rows = []

for line in f:
    line = line.strip()               # remove '\n'

    first, rest = line.split(' ', 1)  # split only on first space
    second, third = rest.split(': ')  # split on ": "

    rows.append( [first, second, third] )
    
print(rows)    

结果：

[
  ['1', 'persona', 'i am adorkable.'], 
  ['2', 'persona', 'i am book dumb.'], 
  ['3', 'persona', 'i am token evil teammate.'], 
  ['4', 'persona', 'i am never my fault.'], 
  ['5', 'persona', 'i am honor before reason.'], 
  ['6', 'persona', 'i am jerk with a heart of gold.'], 
  ['7', 'persona', 'i am no social skills.'], 
  ['8', 'persona', 'i am bad liar']
]

稍后您可以将此列表转换为DataFrame

import pandas as pd

df = pd.DataFrame(rows, columns=['1', '2', '3'])

print(df)

结果：

   1        2                                3
0  1  persona                  i am adorkable.
1  2  persona                  i am book dumb.
2  3  persona        i am token evil teammate.
3  4  persona             i am never my fault.
4  5  persona        i am honor before reason.
5  6  persona  i am jerk with a heart of gold.
6  7  persona           i am no social skills.
7  8  persona                    i am bad liar