How to export to csv the output of every iteration when scraping with a loop in python

Question

I am very new to Python. I am trying to scrape a website and I have created a small code for this:

select = Select(character.find_element_by_id('character-template-choice'))
options = select.options
for index in range(0, len(options) - 0):
    select.select_by_index(index)
    option1 = character.find_elements_by_class_name('pc-stat-value')
    power = []
    for c in option1:
        power.append("Power:" + c.text)
    option2 = character.find_elements_by_class_name(
        'unit-stat-group-stat-label')
    skills = []
    for a in option2:
        skills.append(a.text)
    option3 = character.find_elements_by_class_name(
        'unit-stat-group-stat-value')
    values = []
    for b in option3:
        values.append(b.text)
 test = [power, skills, values]
 print(test)
 df = pd.DataFrame(test).T
 df.to_csv('test2.csv', index=False, header=False)

The issue I have is when I try to export "test" list of lists to csv I only get the last iteration. I want to get the data for every iteration but I don't know how to do so. Can someone help me?

Thank you

Answer 1

You will need to append these three list every iteration to get then all in the end:

select = Select(character.find_element_by_id('character-template-choice'))
options = select.options
test = []  # Create empty list
for index in range(0, len(options) - 0):
    select.select_by_index(index)
    option1 = character.find_elements_by_class_name('pc-stat-value')
    power = []
    for c in option1:
        power.append("Power:" + c.text)
    option2 = character.find_elements_by_class_name(
        'unit-stat-group-stat-label')
    skills = []
    for a in option2:
        skills.append(a.text)
    option3 = character.find_elements_by_class_name(
        'unit-stat-group-stat-value')
    values = []
    for b in option3:
        values.append(b.text)
    test.append([power, skills, values])  # Add a list of lists to your test

print(test)
df = pd.DataFrame(test).T
df.to_csv('test2.csv', index=False, header=False)

This will give you a list test with power, skills, values inside one list for iteration.