在python中逐行从txt输入输入到变量

Question

I have a variable DOMAIN which takes url as input. I want to feed it list of URLs one by one from txt file.

My txt file looks like this:

www.yahoo.com
www.google.com
www.bing.com 

I am doing this:

with open('list.txt') as f:
    content = f.readlines()
content = [x.strip() for x in content] 
DOMAIN = content

But the variable DOMAIN takes all URLs once, not separately. It must process one URL as whole and second on another operation.

On a side note, this DOMAIN variable is feed to scrapy for crawling. part of codebase:

from scrapy.selector import HtmlXPathSelector
from scrapy.spider import BaseSpider
from scrapy.http import Request
with open('list.txt') as f:
    content = f.readlines()
# you may also want to remove whitespace characters like `\n` at the end of each line
content = [x.strip() for x in content] 
DOMAIN = content
URL = 'http://%s' % DOMAIN

class MySpider(BaseSpider):
    name = DOMAIN
    allowed_domains = [DOMAIN]
    start_urls = [
        URL
    ]

errors:

scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://['www.google.com', 'www.yahoo.com', 'www.bing.com']>
executing as scrapy runspider spider.py

full working script for single url---

from scrapy.selector import HtmlXPathSelector
from scrapy.spider import BaseSpider
from scrapy.http import Request

DOMAIN = 'google.com'
URL = 'http://%s' % DOMAIN

class MySpider(BaseSpider):
    name = DOMAIN
    allowed_domains = [DOMAIN]
    start_urls = [
        URL
    ]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        for url in hxs.select('//a/@href').extract():
            if not ( url.startswith('http://') or url.startswith('https://') ):
                url= URL + url 
            print url
            yield Request(url, callback=self.parse)

Answer 1

Ok, so you are assigning the list of domains you are creating to DOMAIN.

content = DOMAIN

You then need to concatenate 'http://' to each of these:

with open('list.txt') as f:
    content = f.readlines()
content = [x.strip() for x in content]
domain_list = content
web = 'http://'
start_url = [web + s for s in domain_list]

Then you have a list of all your URL's, which you could use to connect. I'm not sure what you are doing after this, but I think it should involve iterating over the list of start urls?

for url in start_url:
    scrapy.Request(url)  

Hope this helps,

Answer 2

With these lines:

DOMAIN = content
URL = 'http://%s' % DOMAIN

You can made DOMAIN point to the list you have just created from your file, and then concatenated http:// and a string representation of the list, so you get this:

`http://['www.google.com','www.yahoo.com', 'www.bing.com']'

Hence your error. You need a to concatenate the 'http://' to each entry of the list - you can simply do it while you read the file by iterating over the file directly in a list comprehension rather than using readlines() :

with open('list.txt','r') as f:
    url_list = ['http://'+line.strip() for line in f]

Which will yield a list you can then iterate over with scrapy:

['http://www.google.com','http://www.yahoo.com', 'http://www.bing.com']

Note that reading the while file in at once can be considered inefficient if it's a really big file. In that case you could remove the need to read the whole file into a list, and just the requests as you process the file line-by-line:

with open('list.txt','r') as f:
    for line in f:
        url = 'http://'+line
        request = scrapy.http.Request(url)
        # Do something with request here

---

Also, note that you should not use all UPPERCASE for variable names, these are generally used only for constants. Have a look at PEP8 - The Python Style Guidelines for more guidance on naming conventions. Of course these are guidelines, not rules, but it will make is easier for others to follow your code later on if you follow them.