UnicodeDecodeError无法解码通过写入和读取（使用熊猫）到文件中解决的字节

Question

I have an excel-like data structure composed of bytes that I was not able to decode.

It is a list that looks like:

my_object = [b'\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1..., ........, b'\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff']

(Note that the last line of my_object is an actual one and is fully written here.)

If I try decoding lines independently I get:

my_object[-1].decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 3: invalid start byte

(Note that I tried several different codecs, including: 'utf8', 'ascii', 'ISO-8859-2', 'gbk', 'latin_1', ...)

However, if I try to save my_object to a file first, using:

f = open('test.xls','wb')
[f.write(my_object[i]) for i in range(len(my_object))]
f.close()

and then open it using pandas like:

import pandas as pd
pd.read_excel('test.xls')

I get the expected result:

     Time (s)  Acceleration x (m/s^2)  Acceleration y (m/s^2)  \
0    0.000000                0.863679                0.196953   
1    0.002500                0.892268                0.206483   
2    0.005001                0.844621                0.196953  
......

This is a nice workaround, however, I really would like to avoid writing and reading from and to the disk to perform such an operation.

Can anyone help?

Thank you in advance.

Answer 1

If you just want pandas to read in an excel file when you already have the raw bytes in memory, you can use the io package to turn a string or bytes into a readable file in memory:

import io
file_bytes = b''.join(my_object)
pd.read_excel(io.BytesIO(file_bytes))