Pandas：使用正则表达式读取文件

Question

I am trying to read multiple excel files while using wildcards and putting it in saparate dataframes using pandas.

i have read base path and will be using below to access subdirectories:

>>>inputs_path
'C:/Users/ABC/Downloads/Input'

>>>path1 = os.chdir(inputs_path + "/path1")
>>>fls=glob.glob("*.*")
>>>fls

['Zambia_W4.xlsm',
 'Australia_W4.xlsx',
 'France_W4.xlsx',
 'Japan_W3.xlsm',
 'India_W3.xlsx',
 'Italy_W3.xlsx',
 'MEA_W5.xlsx',
 'NE_W5.xlsm',
 'Russia_W5.xlsx',
 'Spain_W2.xlsx']

>>>path2 = os.chdir(inputs_path + "/path2")
>>>fls=glob.glob("*.*")
>>>fls

['Today.xlsm',
 'Yesterday.xlsx',
 'Tomorrow.xlsx']

Right now i am reading them as follows:

>>>df_italy = pd.read_excel("Italy_W3.xlsx",sheet_name='Sheet1')
>>>df_russia = pd.read_excel("Russia_W5.xlsx",sheet_name='Sheet3')
>>>df_france_1 = pd.read_excel("France_W4.xlsx",sheet_name='Sheet1', usecols = 'M, Q', skiprows=4)
>>>df_spain = pd.read_excel("Spain_W2.xlsx",sheet_name='Sheet2',usecols = 'T:U', skiprows=30 )
>>>df_ne = pd.read_excel("NE_W5.xlsm",sheet_name='Sheet2',usecols = 'N,P', skiprows=4 )
>>>df_ne_c = pd.read_excel("NE_W5.xlsm",sheet_name='Sheet1',usecols = 'H:J', skiprows=141 )

Since i have filenames in the list fls, is there a way i could use that list and read files without actually having to use the actual filename since the filename will change as per week number. Also its mandatory to keep the dataframe names as mentioned above while reading the excel files.

i am looking to read the file as

>>>df_italy = pd.read_excel("Italy*.xlsx",sheet_name='Sheet1')

Is there any way to do this?

Answer 1

If your files always have a _ to split on you could create a dictionary with the split value as the key, and the file path as the location.

Lets use Pathlib which was added in Python 3.4+ as it's easier to use with file systems.

Regex Matching FileName.

Assuming your dictionary is created as above with filenames and paths as the values we could do this. You'll need to extend the function to deal with multiple file matches.

import re
from pathlib import path

file_dict = {file.stem : file for file in location.glob('*.xlsx')}

# assume the numbers are paths.
files = {'Zambia_W4.xlsm': 2,
 'Australia_W4.xlsx': 5,
 'France_W4.xlsx': 0,
 'Japan_W3.xlsm': 7,
 'India_W3.xlsx': 2,
 'Italy_W3.xlsx': 6,
 'MEA_W5.xlsx': 7,
 'NE_W5.xlsm': 4,
 'Russia_W5.xlsx': 3,
 'Spain_W2.xlsx': 5}

def file_name_match(file_dict,pattern):

    for name, source in file_dict.items():
        if re.search(pattern,name,flags=re.IGNORECASE):
            return file_dict.get(name)

file_name_match(file_dict,'italy')
output: 6

---

df = pd.read_excel(file_name_match(file_dict,'italy'),sheetname=...)

Answer 2

It might be feasible to simply populate a dictionary of dataframes like this:

my_dfs = {}
for f in fls:
    my_dfs[f.split(“.”)[0]] = pandas.dataframe(f.split(“,”)[0], ...)

You can use a for loop also to just run the job you need to do for each file, which shouldn't require knowledge of the file name. Also, it's possible to also just read all the spreadsheets into one df, and ensure there is an additional column that has the corresponding file name for each row.

Answer 3

The code below assumes you have several files for each country, and need to sort them to find the latest week.

import glob
import os
import re

def find_country_file(country_name):
  all_country_files = glob.glob(os.path.join(inputs_path, '{0}_W*.*'))
  week_numbers = [re.search('W([0-9]+)', x) for x in all_country_files]
  week_numbers = [int(x.group(1)) for x in week_numbers if x is not None]
  latest_week_number = sorted(week_numbers, reversed=True)[0]
  latest_country_file = [x for x in all_country_files if 'W{0}.'.format(latest_week_number) in x]
  return os.path.basename(latest_country_file)


df_italy = pd.read_excel(find_country_file('Italy') , sheet_name='Sheet1')
df_russia = pd.read_excel(find_country_file('Russia'), sheet_name='Sheet3')
df_france_1 = pd.read_excel(find_country_file('France'),sheet_name='Sheet1', usecols = 'M, Q', skiprows=4)
df_spain = pd.read_excel(find_country_file('Spain'),sheet_name='Sheet2',usecols = 'T:U', skiprows=30 )
df_ne = pd.read_excel(find_country_file('NE'),sheet_name='Sheet2',usecols = 'N,P', skiprows=4 )
df_ne_c = pd.read_excel(find_country_file('NE'),sheet_name='Sheet1',usecols = 'H:J', skiprows=141)

the method find_country searches for all files with the country name in the path, uses regex to pull out the week number, sorts them to find the highest number, and then returns the file path from the glob of all country files that matches the latest week found.