Dask dataframe saving to_csv for incremental data - 高效写入csv

Question

I have an existing code for reading streaming data and storing it using pandas DataFrame (new data comes in every 5 mins), I then capture this Data Category wise (~350 categories).

Next, I write all the new data (as this is to be incrementally stored) using to_csv in a loop.

The Pseudocode is given below:

    for row in parentdf.itertuples(): #insert into <tbl> .
        mycat = row.category # this is the ONLY parameter which is passed to the Key function below.
        try:
            df = FnforExtractingNParsingData(mycat ,NumericParam1,NumericParam1) 
    
            df.insert(0,'NewCol',sym)
            df = df.assign(calculatedCol = functions1(params))
            df = df.assign(calculatedCol1 = functions2(params),20))
            df = df.assign(calculatedCol3 = functions3(More params),20))
            df[20:].to_csv(outfile, mode='a', header=False, index=False)

The category-wise reading and storing in csv takes 2 Mins-Per cycle *. This is close to .34 Seconds for each writing of the 350 Categories incrementally . I am wondering whether I can make the above process faster & efficient by using dask dataframes.

I looked up dask.org and didn't get any clear answers, looked at the use cases as well.

Additional details: I am using Python 3.7 and Pandas 0.25 , Further the above code above doesn't return any errors, even though we have completed good amount of Exception handling already on the above. My key function ie FnforExtractingNParsingData is fairly resilient and is working as desired for a long time.

Answer 1

Sounds like you're reading data into a Pandas DataFrame every 5 minutes and then writing it to disk. The question doesn't mention some key facts:

• how much data is ingested every 5 minutes (10MB or 10TB)?
• where is the code being executed (AWS Lambda or a big cluster of machines)?
• what data operations does FnforExtractingNParsingData perform?

Dask DataFrames can be written to disk as multiple CSV files in parallel, which can be a lot faster than writing a single file with Pandas, but it depends. Dask is overkill for a tiny dataset. Dask can leverage all the CPUs of a single machine, so it can scale up on a single machine better than most people realize. For large datasets, Dask will help a lot. Feel free to provide more details in your question and I can give more specific suggestions.