简体   繁体   中英

Appending to a scraped data to a JSON file

I am trying to make a json file from the scraped data. However based on my function converToJson() it keeps overwriting the previous entry and not appending. Is it because I'm not iterating over it? For example: The below Json file will just overwrite the first entry every time with new data instead of appending to it.

[{"Volume": "Volume:\\n6,061,086", "Price": "$ 41.88", "Name": "Suncor Energy Inc."}]

def getStockDetails(url, browser):
    
        print(url)
        browser.get(url)
    
        quote_wrapper = browser.find_element_by_css_selector('div.quote-wrapper')
        quote_name = quote_wrapper.find_element_by_class_name(
            "quote-name").find_element_by_tag_name('h2').text
        quote_price = quote_wrapper.find_element_by_class_name("quote-price").text
        quote_volume = quote_wrapper.find_element_by_class_name(
            "quote-volume").text
    
        print("\n")
        print("Quote Name: " + quote_name)
        print("Quote Price: " + quote_price)
        print("Quote Volume: " + quote_volume)
        print("\n")
    
        convertToJson(quote_name,quote_price,quote_volume)
 
  
 def convertToJson(quote_name,quote_price,quote_volume):
        quotesArr = []
        quoteObject = {
            "Name": quote_name,
            "Price": quote_price,
            "Volume": quote_volume
        }
        quotesArr.append(quoteObject)

        with open('trendingQuoteData.json', 'w') as outfile:
            json.dump(quotesArr, outfile)

you need to make variables quotesArr global, put it outside function and write your json when it finished.

quotesArr = []
def convertToJson(quote_name,quote_price,quote_volume):
    quoteObject = {
        "Name": quote_name,
        "Price": quote_price,
        "Volume": quote_volume
    }
    quotesArr.append(quoteObject)

def trendingBot(url, browser):
    browser.get(url)
    trending = getTrendingQuotes(browser)
    for trend in trending:
        getStockDetails(trend, browser)
    # requests finished, write json to file
    with open('trendingQuoteData.json', 'w') as outfile:
        json.dump(quotesArr, outfile)
import json

a = json.loads(jsonStringA)
b = json.loads(jsonStringB)
c = dict(a.items() + b.items())
# or c =  dict(a, **b)

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM