Using python to batch insert a column into CSVs using file name information

Question

I have 169 CSV files with an identical structure (90 columns with the same headers) and an identical naming system.

Screenshot of file names

The files are named as such:

• 2019-v-1
• 2019-v-2
• 2019-v-3
• etc.

For each CSV, I would like to add a column, with the header 'Visits', and for the value in that column to be taken from the file name (the number at the end, after the second dash).

So, for example, the first CSV will have a new column called 'Visits', where every row is given the value '1' in that column.

If there is a Python solution, that would be amazing. I don't come from a coding background, and that's the only language I've got somewhat familiar with, but I can't seem to figure this one out myself.

Any help would be massively appreciated - thank you!

Answer 1

import pandas as pd
import os

def csv_folder_input(path,folder):
    path = os.path.join(path,folder)
    os.chdir(path)
    counter=1
    for filename in os.listdir(path):
        if filename.endswith(".csv"):
             with open(filename, 'r') as csvfile:
             counter=counter+1
             df = pd.read_csv(csvfile)
             df['Visits']=int(filename.split('_')[2].split('.')[0])
             df.to_csv(filename)
       
csv_folder_input(your path name,your folder name)

Put in your path name followed by your folder name. I can see that your folder name is 2019-v. enter the appropriate path name prior to the folder and ensure that the correct path format for a MacOS is entered. It should work fine I believe.

Answer 2

First, you need a list of the files:

from os import listdir
from os.path import isfile, join
import csv       # You'll need this for the next step

mypath = 'path/to/csv/directory'
allfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

Then you want to open each file, add the column, and save it again.

from os import listdir
from os.path import isfile, join
import csv

mypath = 'path/to/csv/directory'
allfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

for file in allfiles:

  # This will only work if your filenames are consistent.
  # We split the name into a list, ['2019', 'v', '1.csv']
  #   then take the third item (index 2), and remove the
  #   last 4 characters.

  number_at_end = file.split("-")[2][:-4]

  # We open the file...

  with open(file, newline='') as csvfile:
    reader = csv.reader(csvfile)

  # Add the column name to the first row, and
  # add the value to each row...

  for i, row in enumerate(reader):
    if i == 0:
      row.append('Visits')
    else:
      row.append(number_at_end)

  # and then write the file back.

  with open(file, newline='') as csvfile:
    writer = csv.writer(csvfile)
    writer.writerows(reader)