How to upload csv with FastAPI and process it in memory with Pandas?

Question

I am trying to upload a csv file with FastAPI and then load it into pandas.

import pandas as pd
import os
import io, base64

from fastapi import FastAPI, File, UploadFile, Form

app = FastAPI()

@app.post('/uploadfile/')
async def create_data_file(
        experiment: str = Form(...),
        file_type: str = Form(...),
        file_id: str = Form(...),
        data_file: UploadFile = File(...),
        ):
    
    #decoded = base64.b64decode(data_file.file)
    #decoded = io.StringIO(decoded.decode('utf-8'))
    
    print(pd.read_csv(data_file.file, sep='\t'))

    return {'filename': data_file.filename, 
            'experiment':experiment, 
            'file_type': file_type, 
            'file_id': file_id}

I tried using the file.file content directly or converting it with base64 or StringIO . I also tried codec . The error I get with the example code is

AttributeError: 'SpooledTemporaryFile' object has no attribute 'readable'

Answer 1

Changing the encoding to what suits you best, I found this workaround:

from io import StringIO
 
pd.read_csv(StringIO(str(data_file.file.read(), 'utf-16')), encoding='utf-16')

Answer 2

This is a workaround using libraries csv and codecs to create the records which then can be turned into a pandas dataframe:

def to_df(file):
    data = file.file
    data = csv.reader(codecs.iterdecode(data,'utf-8'), delimiter='\t')
    header = data.__next__()
    df = pd.DataFrame(data, columns=header)
    return df