Python for loop storing only the last value

Question

My code are scraping data from public site and storing in a df and saving as csv, but my code are not working well.

    for ano in lista_ano:
     for distribuidora in lista_distribuidores:
      for mes in lista_mes:
        scraping = pd.read_html('https://www2.aneel.gov.br/aplicacoes/indicadores_de_qualidade/decFecSegMensal.cfm?mes={}&ano={}&regiao=SE&distribuidora={}&tipo=d'.format(mes,ano,distribuidora))
        dfs= pd.DataFrame(scraping[0])
        dfs.drop(dfs.tail(3).index,inplace=True)
        dfs.drop(dfs.head(2).index,inplace=True)
        dfs = dfs.assign(MES = '{}'.format(mes))
        dfs = dfs.assign(ANO = '{}'.format(ano))
        dfs = dfs.assign(DISTRIBUIDORA = '{}'.format(distribuidora))
        all_dfs = pd.DataFrame(dfs)
        all_dfs.to_csv('final_data.csv', encoding= 'utf-8')

My problem here is my all_dfs.to_csv are creating a new csv for each looping and not storing data in the same local.

Answer 1

You are overwriting the existing csv on each iteration.

To fix it, simply indicate that you want to append instead of write by

all_dfs.to_csv('final_data.csv', encoding='utf-8', mode='a')