如何通过for循环填充熊猫数据框？

Question

Question is related to this function in particular

def jsontodataframe(): #collect OHLC data from scstrade

    companies = {'Habib Bank Limited':'HBL','Engro Chemical':'ENGRO'}
    url = 'http://www.scstrade.com/stockscreening/SS_CompanySnapShotHP.aspx/chart'

    payload = {"date1":"01/01/2019","date2":"06/01/2019","rows":20,"page":1,"sidx":"trading_Date",
    "sord":"desc"}

    for company in companies:
        payload["par"] = companies[company]
        #print(payload)
        json_data = requests.post(url, json=payload).json() #download the json POST request from scstrade
        json_normalize(json_data)
        df = pd.DataFrame(json_data) #convert the json to pandas dataframe
        df = pd.io.json.json_normalize(json_data['d'], errors='ignore')
        df.columns = ['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Change'] #rename the columns to better names

        df['Date'] = df['Date'].str.strip('/Date()')
        df['Date'] = pd.to_datetime(df['Date'], origin='unix', unit='ms') #convert unix timestamp to pandas datetime and set the index
        df['ID'] = companies[company]
        df.set_index(['ID'], inplace=True)
        print(df.head())

    df.to_csv("OHLC_values.csv") #save .csv file for later usage

Currently the df variable keeps getting overwritten each time and my output is like this:

I thought about using append but that would be a massive performance hit and I want the code to be as efficient as possible (so that I can easily scale it later on). Right now the df.columns line is redundant so should I just define my dataframe outside of the for loop? But that json_normalize function will bring in column names of its own then so thats kinda necessary.

Ideally I just want one big dataframe and then later convert that to one .csv file

Answer 1

Ideally I just want one big dataframe and then later convert that to one .csv file

This could be achieved using pandas.concat

import calendar, requests
import pandas as pd
from pandas.io.json import json_normalize

def jsontodataframe():

    companies = {'Habib Bank Limited':'HBL','Engro Chemical':'ENGRO'}
    url = 'http://www.scstrade.com/stockscreening/SS_CompanySnapShotHP.aspx/chart'

    payload = {"date1":"01/01/2019","date2":"06/01/2019","rows":20,"page":1,"sidx":"trading_Date",
    "sord":"desc"}
    data = []
    for company in companies:
        payload["par"] = companies[company]

        json_data = requests.post(url, json=payload).json() 
        json_normalize(json_data)
        df = pd.DataFrame(json_data) 
        df = pd.io.json.json_normalize(json_data['d'], errors='ignore')
        df.columns = ['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Change'] 

        df['Date'] = df['Date'].str.strip('/Date()')
        df['Date'] = pd.to_datetime(df['Date'], origin='unix', unit='ms') 

        df['Date'] = df['Date'].dt.floor('d') # return only dates

        ### update
        df['Month_as_number'] = df['Date'].dt.month # created a column with a month as number - 5, 11 etc.
        df['Month_as_name'] = df['Month_as_number'].apply(lambda x: calendar.month_abbr[x]) # 5 as May etc
        ###

        df['ID'] = companies[company]
        df.set_index(['ID'], inplace=True)
        data.append(df)
    # save to csv instead of returning dataframe    
    pd.concat(data).to_csv('OHLC_values.csv', index=False)

---

So I edited my initial answer. Now the function saves dataframe to .csv file. Also, I stripped date column to only dates.