当我尝试在 Jupyter Notebook 中加载 txt 时出现问题

Question

我试图从一个文件夹中加载所有 txt 文件。 当我想将 txt 文件加载到 pandas dataframe 并将它们连接起来时，下面的这段代码大部分时间都有效，但在这种情况下不起作用，我不知道为什么。

这是代码：

path = 'C:/Users/user/Documents/UNIAO'


csv_files = glob.glob(os.path.join(path, "*.txt"))

list_of_dataframes = []
# loop over the list of csv files
for f in csv_files:
    text_file = open(f, "r", encoding='unicode_escape')

    data = text_file.read()
    separator= data[4]

    df = pd.read_csv(f, sep=separator, encoding ='unicode_escape')
    list_of_dataframes.append(df)

这是错误消息：

ParserError                               Traceback (most recent call last)
Cell In [5], line 19
     16 separator = data[4]
---> 19 df = pd.read_csv(f, sep=separator, encoding ='unicode_escape')
     20 print(f)
     23 list_of_dataframes.append(df)

File c:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\util\_decorators.py:311, in deprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper(*args, **kwargs)
    305 if len(args) > num_allow_args:
    306     warnings.warn(
    307         msg.format(arguments=arguments),
    308         FutureWarning,
    309         stacklevel=stacklevel,
    310     )
--> 311 return func(*args, **kwargs)

File c:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py:680, in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, error_bad_lines, warn_bad_lines, on_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options)
    665 kwds_defaults = _refine_defaults_read(
    666     dialect,
    667     delimiter,
   (...)
    676     defaults={"delimiter": ","},
    677 )
...
--> 739     raise ParserError(msg)
    740 elif self.on_bad_lines == self.BadLineHandleMethod.WARN:
    741     base = f"Skipping line {row_num}: "

ParserError: Expected 197 fields in line 11955, saw 198

Answer 1

这个问题可能是由于第 11955 行的一些损坏/丢失的数据，您可以尝试，

对于 Pandas >= 1.3.0

df = pd.read_csv（f，sep=分隔符，编码='unicode_escape'，on_bad_lines='skip'）

对于 Pandas < 1.3.0

df = pd.read_csv(f, sep=separator, encoding ='unicode_escape', error_bad_lines=False)

请注意，这将导致有问题的行被跳过。

有关更多信息，请参阅Pandas 文档