Pandas read_fwf 似乎不尊重编码参数

Question

I wanted to try and use the memory_map parameter to see whether it improved the load time of a file. (I don't really know what the parameter does, but thought I would give it a shot.)

When I try to load the file I get the error UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 295: invalid start byte . I tried setting the encoding parameter (see below) but it doesn't appear to work.

Here is the code:

import pandas as pd
fwf_widths  = [6,2,6,2,14,1,40,1,10,1,10,1,1,3,3,1,1,1,2,1,5,1,10,
               1,10,1,30,2,30,1,18,1,2,1,5,1,2,1,2,3,3,1,1,1,2,1,
               5,1,10,1,10,1,30,2,30,1,18,1,2,1,5,1,2,1,2,3,1,1,1,
               1,1,1,1,2,1,1,1,1,1,1,1,]
pd.read_fwf("MOVEOUTA.ALL.OUT1.txt",
            usecols=range(0,80, 2), 
            widths=fwf_widths,
            encoding='windows-1252',
            memory_map=True)

Am I doing something wrong, or should I raise an issue with pandas (I have version 1.01)?

Edit:

I tried this as well, but continue to receive the same error:

with open("MOVEOUTA.ALL.OUT1.txt", mode='r',encoding='windows-1252', ) as f:
    df = pd.read_fwf(f,
                     usecols=range(0,80, 2), 
                     widths=fwf_widths,
                     memory_map=True)

Answer 1

I don't know whether pandas.read_fwf accepts parameter encoding :

 pandas.read_fwf(filepath_or_buffer, colspecs='infer', widths=None, infer_nrows=100, **kwds)

Read a table of fixed-width formatted lines into DataFrame.

Also supports optionally iterating or breaking of the file into chunks.

Additional help can be found in theonline docs for IO Tools .

The following code snippet should do the job (pass in an instance of StringIO to the filepath_or_buffer parameter):

import pandas as pd
from io import StringIO

with open("MOVEOUTA.ALL.OUT1.txt", mode='r', encoding='windows-1252') as f:
    content = f.read()
 
fwf_widths  = [6,2,6,2,14,1,40,1,10,1,10,1,1,3,3,1,1,1,2,1,5,1,10,
               1,10,1,30,2,30,1,18,1,2,1,5,1,2,1,2,3,3,1,1,1,2,1,
               5,1,10,1,10,1,30,2,30,1,18,1,2,1,5,1,2,1,2,3,1,1,1,
               1,1,1,1,2,1,1,1,1,1,1,1,]
df = pd.read_fwf( StringIO( content),
            usecols=range(0,80, 2),       # ??? this param not tested
            widths=fwf_widths,
            memory_map=True)