Loop to download CSV, takes only last value (last url of the list)

Question

I am quite new here but I think you could help me with a basic thing (a loop).

Basically, my scripts runs well but when I open the outputs (.csv files) I see that it took only the last url of the list.

for url in urls:
   for jobboard_name in jobboard_names:
       jobboard_name = requests.get(url)
       for file_name in files_names:
           file = open(file_name, "wb")    
           file.write(jobboard_name.content)
           os.replace(file_name,today.strftime('%d-%m-%Y') + "/" + file_name)

Do you have an idea what I need to implement here? I would like to take all the elements of the list "urls"?

Thank you for your help !

Answer 1

Opening in write binary mode "wb" writes over the file every time you open it. You want to open it in append binary mode "ab" instead:

for file_name in files_names:
    file = open(file_name, "ab")

Alternatively, you could create new files and write binary to them like so (which may be easier than renaming your original files):

for url in urls:
   for jobboard_name in jobboard_names:
       jobboard_name = requests.get(url)
       for file_name in files_names:
           with open(today.strftime('%d-%m-%Y') + "/" + file_name, "wb") as file:    
               file.write(jobboard_name.content)

As @joao pointed out, for jobboard_name in jobboard_names may be the problem - since it doesn't appear to exist (I assumed you had it higher up), all you'll do is write the same file over and over. To avoid that we can modify your original code like so:

for url in urls:
       jobboard_name = requests.get(url)
       for file_name in files_names:
           with open(file_name, "ab") as file:   
               file.write(jobboard_name.content)
           os.replace(file_name,today.strftime('%d-%m-%Y') + "/" + file_name)