Import data into Excel file from a working file in Python

Question

I have a long code for web scraping data into an interim working file and then generating an Excel file where this data will be imported in. I want to split the code into two parts, so that I don't have to run the webscraping every single time if my working file is already created.

How do I reference the working file ("data") in the second part of the code that generates the Excel file?

My current line of code resulted in vanishing all the data from the existing working file:

data=eval(open('data','r',encoding='utf8').read())

Should it be simply this?

data=open('data','r',encoding='utf8')

Answer 1

You may try reading the excel file as the very first step as a DataFrame into your script. If the file exists and it's empty, it will return an empty dataframe and if it doesn't exist you write a simple code to create an empty one. In any case you create a starting dataframe for yourself. After extracting your required data, you simply append your result to your initial dataframe and overwrite the Excel file with your final dataframe. In order to speed the two processes up and to save time, you may use a multi threading programme to the first part of the task where you create the initial dataframe while the second part of your programme scrapes the data.

If this doesn't answer your question, do let me know what is it that you Were looking for exactly.