抓取网站并将仅可见文本导出到文本文档Python 3（Beautiful Soup）

Question

Problem: I am trying to scrape multiple websites using beautifulsoup for only the visible text and then export all of the data to a single text file.

This file will be used as a corpus for finding collocations using NLTK. I'm working with something like this so far but any help would be much appreciated!

import requests
from bs4 import BeautifulSoup
from collections import Counter
urls = ["http://en.wikipedia.org/wiki/Wolfgang_Amadeus_Mozart","http://en.wikipedia.org/wiki/Golf"]
    for url in urls:
    website = requests.get(url)
    soup = BeautifulSoup(website.content)
    text = [''.join(s.findAll(text=True))for s in soup.findAll('p')]
with open('thisisanew.txt','w') as file:
    for item in text:
        print(file, item)

Unfortunately, there are two issues with this: when I try to export the file to a .txt file it is completely blank.

Any ideas?

Answer 1

print(file, item) should be print(item, file=file) .

But don't name your files file as this shadows the file builtin, something like this is better:

with open('thisisanew.txt','w') as outfile:
    for item in text:
        print(item, file=outfile)

To solve the next problem, overwriting the data from the first URL, you can move the file writing code into your loop, and open the file once before entering the loop:

import requests
from bs4 import BeautifulSoup
from collections import Counter
urls = ["http://en.wikipedia.org/wiki/Wolfgang_Amadeus_Mozart","http://en.wikipedia.org/wiki/Golf"]

with open('thisisanew.txt', 'w', encoding='utf-8') as outfile:
    for url in urls:
        website = requests.get(url)
        soup = BeautifulSoup(website.content)
        text = [''.join(s.findAll(text=True))for s in soup.findAll('p')]
        for item in text:
            print(item, file=outfile)

Answer 2

There is another problem: you are collecting the text only from the last url: reassigning the text variable over and over.

Define the text as an empty list before the loop and add new data to it inside:

text = []
for url in urls:
    website = requests.get(url)
    soup = BeautifulSoup(website.content)
    text += [''.join(s.findAll(text=True))for s in soup.findAll('p')]