从for循环内追加到数据框的下一行

Question

I've created a web scraper that scrapes the Yahoo Finance Summary and Statistics page of a stock for Python programming educational purposes only. It reads from the '1stocklist.csv' in the programs directory which looks like this:

Symbols
SNAP
KO

From there, it adds the new information to new columns in the dataframe as it should. There are a lot of 'for' loops in there and I'm still tweaking it as it's not grabbing some data correctly, but it's fine for now.

My problem is trying to save the dataframe to a new .csv file. The way it outputs right now as you'll see is something like this:

The SNAP row should begin with the 14.02 and everything right, and the next row should be KO beginning with the 51.39 and over.

Any ideas? Just create a 1stocklist.csv file that looks like the above and try it. Thanks!

# Import dependencies
from bs4 import BeautifulSoup
import re, random, time, requests, datetime, csv
import pandas as pd
import numpy as np


# Use Pandas to read the "1stocklist.csv" file. We'll use Pandas so that we can append a 'dataframe' with new
# information we get from the Zacks site to work with in the program and output to the 'data(date).csv' file later
maindf = pd.read_csv('1stocklist.csv', skiprows=1, names=[
# The .csv header names
    "Symbols"
    ]) #, delimiter = ',')

# Setting a time delay will help keep scraping suspicion down and server load down when scraping the Zacks site
timeDelay = random.randrange(2, 8)


# Start scraping Yahoo
print('Beginning to scrape Yahoo Finance site for information ...')
tickerlist = len(maindf['Symbols']) # for progress bar


# Create a progress counter to display how far along in the zacks rank scraping it is
zackscounter = 1

# For every ticker in the stocklist dataframe
for ticker in maindf['Symbols']:

# Print the progress
    print(zackscounter, ' of ', tickerlist, ' - ', ticker) # for seeing which stock it's currently on

# The list of URL's for the stock's different pages to scrape the information from
    summaryurl = 'https://ca.finance.yahoo.com/quote/' + ticker
    statsurl = 'https://ca.finance.yahoo.com/quote/' + ticker + '/key-statistics'

# Define the headers to use in Beautiful Soup 4
    headers = requests.utils.default_headers()
    headers['User-Agent'] = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36'

# Employ random time delay now before starting with the (next) ticker
    time.sleep(timeDelay)





# Use Beautiful Soup 4 to get the info from the first Summary URL page
    page = requests.get(summaryurl, headers=headers)
    soup = BeautifulSoup(page.text, 'html.parser')

    counter = 0 # used to tell which 'td' it's currently looking at
    table = soup.find('div', {'id' :'quote-summary'})
    for i in table.find_all('span'):
        counter += 1
        if counter % 2 == 0: # All Even td's are the metrics/numbers we want
            data_point = i.text
            #print(data_point)
            maindf[column_name] = data_point # Add the data point to the right column
        else:                # All odd td's are the header names
            column_name = i.text
            #print(column_name)





# Use Beautiful Soup 4 to get the info from the second stats URL page
    page = requests.get(statsurl, headers=headers)
    soup = BeautifulSoup(page.text, 'html.parser')
    time.sleep(timeDelay)
# Get all the data in the tables
    counter = 0 # used to tell which 'td' it's currently looking at
    table = soup.find('section', {'data-test' :'qsp-statistics'})
    for i in table.find_all('td'):
        counter += 1
        if counter % 2 == 0: # All Even td's are the metrics/numbers we want
            data_point = i.text
            #print(data_point)
            maindf[column_name] = data_point # Add the data point to the right column
        else:                # All odd td's are the header names
            column_name = i.text
            #print(column_name)





    file_name = 'data_raw.csv'
    if zackscounter == 1:
        maindf.to_csv(file_name, index=False)
    else:
        maindf.to_csv(file_name, index=False, header=False, mode='a')

    zackscounter += 1
    continue

UPDATE:

I know it's something to do with how I'm trying to append the dataframe to the .csv file at the end. My beginning dataframe is just one column with all the ticker symbols in it, then it's trying to add each new column to the dataframe as the program goes along, and fills down to the bottom of the ticker list. What I'm wanting to happen is to just add the column_name header as it should, and then append the appropriate data specific to the one ticker and do that for each ticker in the “Symbols” column of my dataframe. Hope that provides some clarity to the issue?

I've tried using .loc in various ways but to no success. Thanks!

Answer 1

UPDATE WITH ANSWER

I was able to figure it out!

Basically, I changed the first dataframe that reads from the 1stocklist.csv to be its own dataframe, then created a new blank one to work with from within the first for loop. Here is the updated head that I created:

# Use Pandas to read the "1stocklist.csv" file. We'll use Pandas so that we can append a 'dataframe' with new
# information we get from the Zacks site to work with in the program and output to the 'data(date).csv' file later
opening_dataframe = pd.read_csv('1stocklist.csv', skiprows=1, names=[
# The .csv header names
    "Symbols"
    ]) #, delimiter = ',')

# Setting a time delay will help keep scraping suspicion down and server load down when scraping the Zacks site
timeDelay = random.randrange(2, 8)


# Start scraping Yahoo
print('Beginning to scrape Yahoo Finance site for information ...')
tickerlist = len(opening_dataframe['Symbols']) # for progress bar


# Create a progress counter to display how far along in the zacks rank scraping it is
zackscounter = 1

# For every ticker in the stocklist dataframe
for ticker in opening_dataframe['Symbols']:

    maindf = pd.DataFrame(columns=['Symbols'])

    maindf.loc[len(maindf)] = ticker

# Print the progress
    print(zackscounter, ' of ', tickerlist, ' - ', ticker) # for seeing which stock it's currently on

# The list of URL's for the stock's different pages to scrape the information from
    summaryurl = 'https://ca.finance.yahoo.com/quote/' + ticker
    statsurl = 'https://ca.finance.yahoo.com/quote/' + ticker + '/key-statistics'
......
......
......

Notice the "opening_dataframe = ..." name change, and the

maindf = pd.DataFrame(columns=['Symbols']) maindf.loc[len(maindf)] = ticker

part. I also utilize the .loc to add to the next available row in the dataframe. Hopefully this helps someone!