抓取时在单个字典中分组相同的数据

Question

I am trying scrape country name, GDP and population from this website . I am using Scrapy with Python 3.7 . The problem is I am getting all the country data in a dictionary, all the GDP data in a dictionary and all the population data in a dictionary. But I want corresponding country data, GDP and population in a dictionary.

Here is my code:

import scrapy

class DebtByCountriesSpider(scrapy.Spider):
    name = 'debt_by_countries'
    allowed_domains = ['worldpopulationreview.com/countries/countries-by-national-debt']
    start_urls = ['https://worldpopulationreview.com/countries/countries-by-national-debt/']

    def parse(self, response):

        # countries = response.xpath("//td/a/text()").getall()

        countries = response.xpath("//tbody/tr/td/a/text()").getall()
        GDP = response.xpath("//tbody/tr/td[2]/text()").getall()
        population = response.xpath("//tbody/tr/td[3]/text()").getall()


        yield{
            "country_name": countries,
            "GDP": GDP,
            "population": population

        }

Here is the output of my code:

But this is what I want (including the population):

Answer 1

Using zip , we can create a dictionary for each country and yield from there.

for country, gdp, pop in zip(countries, GDP, population):
    yield {"country_name": country, "GDP": gdp, "population": pop}

The reason why your code doesn't work is that the generator is just going to yield a single huge dictionary, where each value is the entire list countries , GDP , and population , respectively. To remedy this, you will want to create a dictionary for each country and yield each element per next call as shown above.

To test the generator, try

gen = parse(response) # or self.parse(response) depending on context
print(next(gen))
print(next(gen))

Each time next is called, the generator will yield a different dictionary corresponding to a new country.