Formatting multiple text files and writing them to CSV file in Python

Question

I am trying to take the first row from multiple text files and add them to a new row in a CSV file, separated by columns. Currently, I am struggling with:

1. Not overwriting previous data on the CSV file
2. Skipping the header
3. Excluding commas in strings like Went From: 123 Address City, State

Currently, I have multiple text files each with information like date, time, etc. I wrote a bash script that takes each text files and pastes them into a CSV file:

paste -d "," col_Date col_Current_Time col_Weather col_From col_To col_Time_Elapsed > Database.csv

and I get:

2022-02-24,07:47:24,Went from: 2678, 20th Street, Ingleside Village, Tuscaloosa, Alabama 35401 United States, to: 2678 20th Street, Tuscaloosa, Alabama, 35401, United States, 1 s

and I want to get:

Date, Time, From, To, Time Elapsed
2022-02-24,07:47:24,Went from: 2678 20th Street Ingleside Village Tuscaloosa Alabama 35401 United States, to: 2678 20th Street Tuscaloosa Alabama 35401 United States, 1 s

Either I need to remove the commas in the string before, or somehow tell the CSV file to omit these specific commas in the From and To string. I write to each of the text files using this Python code:

with open(col_Date, 'w') as f3:
    print(today, file=f3)
f3.close()

This is my second week of coding, so there is no such thing as over explaining to me. Let me know if you need anything else! Thank you!

Answer 1

To work on csv files, I would recommend to use pandas dataframe.

To install pandas

pip install pandas

In the below Code snippet, we convert our text data into a dictionary containing the field as key & your data as the label.

After that, the already made csv file to which this data is to appended is read by pandas and converted to a pandas dataframe, and the new data is appended to it and flushed to the same/ another csv file.

import pandas
import os.path


# This is the new row to add to the already made csv file
# Convert your text data in a dictionary
# that will depend how your columns are separated in the txt file
df = pd.DataFrame.from_dict(data)

# Read csv to append to
csv_file_path = "data.csv"
if os.path.exists(csv_file_path):
    df = pd.concat([pd.read_csv(csv_file_path), df])
    # Save 
    df.to_csv(csv_file_path, header=True, index=False, columns=labels)

It will be of better if you could include a sample text file, to understand the problem statement better.