从txt文件解析网址

Question

I am trying to parse a txt file which looks like this:

Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html

I need to read the file and extract the part with the url after 'Disallow' but also ignoring the comments. Thanks in advance.

Answer 1

If you are trying to parse a robots.txt file then you should use the robotparser module:

>>> import robotparser

>>> r = robotparser.RobotFileParser()
>>> r.set_url("http://www.your_url.com/robots.txt")
>>> r.read()

Then just check:

>>> r.can_fetch("*", "/foo.html")
False

Answer 2

Assuming that there's no # in the URLs:

with open('path/to/file') as infile:
    URLs = [line.strip().lstrip("Disallow:").split("#", 1)[0] for line in infile]

Allowing for the existence of # , but assuming that comments beginning with # and the urls are separated by a space:

with open('path/to/file') as infile:
    URLs = [line.strip().lstrip("Disallow:").split(" #", 1)[0] for line in infile]