for loop output into single list

Question

I am trying to extract few links, that contain other links and struggling to get the output from the for loop into a single list like stated below.

My code:

pages = ['https://pagetoscrape.com/?page=1',
         'https://pagetoscrape.com/?page=2',
         'https://pagetoscrape.com/?page=3'
        ]

for u in pages:
    response = requests.get(u)
    data = response.content
    soup = BeautifulSoup(data, 'lxml')
    for links in soup.find_all('div', class_='item-to-scrape'):
        link = links.a['href']

        print(link)

Output:

pagetoscrape.com/url1
pagetoscrape.com/url2
pagetoscrape.com/url3
pagetoscrape.com/url4
pagetoscrape.com/url5
pagetoscrape.com/url6
pagetoscrape.com/url7
...

How can I get a list like this to use it later for other operations (another loop that picks the links one by one) similar to the "pages" array variable?:

['pagetoscrape.com/url1', 'pagetoscrape.com/url2', 'pagetoscrape.com/url3', 'pagetoscrape.com/url4', ...]

Answer 1

Build your list with a list comprehension:

link_list = [links.a['href']
                for links in soup.find_all('div', class_='item-to-scrape') ] 

Answer 2

If I understand correctly you want something like this.

pages = ['https://pagetoscrape.com/?page=1',
         'https://pagetoscrape.com/?page=2',
         'https://pagetoscrape.com/?page=3'
        ]

urls = []

for u in pages:
    response = requests.get(u)
    data = response.content
    soup = BeautifulSoup(data, 'lxml')
    for links in soup.find_all('div', class_='item-to-scrape'):
        link = links.a['href']
        urls.append(link)
        print(link)