python read all files from a folder and write the file name and other info into a txt file

Question

I have 30911 html files. I need to do webscraping and then save the info into a txt file named index.txt. It should look like

filename1, title, t1, date, p1
filename2, title, t1, date, p1
filename3, title, t1, date, p2
and so on...

I only want filename, but output gave me path+filename.

Answer 1

You can use:

path = 'C:/Users/.../.../output/'
#read html files
for filename in glob.glob(os.path.join(path, '*.html')):
    soup = bs4.BeautifulSoup(open(filename).read(), "lxml")
    title = soup.find('h1')
    ticker = soup.find('p')
    d_date = soup.find_all('div', {"id": "a-body"})[0].find_all("p")[2]

    try:
        def find_participant(tag):
            return tag.name == 'p' and tag.find("strong", text=re.compile(r"Executives|Corporate Participants"))

        participants = soup.find(find_participant)
        parti_names = ""
        for parti in participants.find_next_siblings("p"):
            if parti.find("strong", text=re.compile(r"(Operator)")):
                break
            parti_names += parti.get_text(strip=True) + ","
    except:
        indexFile = open('C:/Users/.../output1/' + 'index.txt', 'a+')
        indexFile.write(filename + ', ' + title.get_text(strip=True) + ', '+ ticker.get_text(strip=True) + ', ' + d_date.get_text(strip=True) + ', ' + 'No participants' + '\n')
    else:
        participants = soup.find(find_participant)
        parti_names = ""
        for parti in participants.find_next_siblings("p"):
            if parti.find("strong", text=re.compile(r"(Operator)")):
                break
            parti_names += parti.get_text(strip=True) + ","
        indexFile = open('C:/Users/.../output1/' + 'index.txt', 'a+')
        indexFile.write(os.path.basename(filename) + ', ' + title.get_text(strip=True) + ', '+ ticker.get_text(strip=True) + ', ' + d_date.get_text(strip=True) + ', ' + parti_names + '\n')
        indexFile.close()

Answer 2

Your problem is that filename is filepath in reality, in order to get the filename you could use os module

os.path.basename('filepath')

so in order to write to the file:

indexFile.write(os.path.basename(filename)+ ', ' + title.get_text(strip=True) + ', '+ ticker.get_text(strip=True) + ', ' + d_date.get_text(strip=True) + ', ' + parti_names + '\n')

Answer 3

ntpath is another module used to get base name from path.

>>> import ntpath
>>> ntpath.basename('C:/Users/.../output1/' + 'index.txt')
'index.txt'