刮擦并将输出写入文本文件

Question

I coded this scraper using Python 2.7 to fetch links from the first 3 pages of TrueLocal.com.au and write them to a text file.

When I run the program, only the first link is written in the text file. What can I do so that all the URLs returned are written on the file?

import requests
from bs4 import BeautifulSoup

def tru_crawler(max_pages):
    page = 1
    while page <= max_pages:
        url = 'http://www.truelocal.com.au/find/car-rental/' + str(page)
        code = requests.get(url)
        text = code.text
        soup = BeautifulSoup(text)
        for link in soup.findAll('a', {'class':'name'}):
            href = 'http://www.truelocal.com.au' + link.get('href')
            fob = open('c:/test/true.txt', 'w')
            fob.write(href + '\n')
            fob.close()
            print (href)
        page += 1

#Run the function
tru_crawler(3)

Answer 1

Your problem is that for each link, you open the output file, write it, then close the file again. Not only is this inefficient, but unless you open the file in "append" mode each time, it will just get overwritten. What's happening is actually that the last link gets left in the file and everything prior is lost.

The quick fix would be to change the open mode from 'w' to 'a' , but it'd be even better to slightly restructure your program. Right now the tru_crawler function is responsible for both crawling your site and writing output; instead it's better practice to have each function responsible for one thing only.

You can turn your crawl function into a generator that yields links one at a time, and then write the generated output to a file separately. Replace the three fob lines with:

    yield href + '\n'

Then you can do the following:

lines = tru_crawler(3)
filename = 'c:/test/true.txt'
with open(filename, 'w') as handle:
    handle.writelines(lines)

Also note the usage of the with statement ; opening the file using with automatically closes it once that block ends, saving you from having to call close() yourself.

---

Taking the idea of generators and task-separation one step further, you may notice that the tru_crawler function is also responsible for generating the list of URLs to crawl. That too can be separated out, if your crawler accepts an iterable of URLs instead of creating them itself. Something like:

def make_urls(base_url, pages):
    for page in range(1, pages+1):
        yield base_url + str(page)

def crawler(urls):
    for url in urls:
        #fetch, parse, and yield hrefs

Then, instead of calling tru_crawler(3) , it becomes:

urls = make_urls('http://www.truelocal.com.au/find/car_rental/', 3)
lines = crawler(urls)

and then proceed as above.

Now if you want to crawl other sites, you can just change your make_urls call, or create different generators for other URL-patterns, and the rest of your code doesn't need to change!

Answer 2

By default 'w' is truncating mode and you may need append mode. See: https://docs.python.org/2/library/functions.html#open .

Maybe appending your hrefs to a list in while loop and then write to file later would look readable. Or as suggested use yield for efficiency.

Something like

with open('c:/test/true.txt', 'w') as fob:
    fob.writelines(yourlistofhref)

https://docs.python.org/2/library/stdtypes.html#file.writelines