使用urllib2.urlopen（）的未知URL类型

Question

I am trying to do the following:

• open a CSV file containing a list with URLs (GET-Requests)
• read the CSV file and write the entries to a list
• open every single URL and read the answer
• write the answers back to a new CSV file

I get the following error:

Traceback (most recent call last):
  File "C:\Users\l.buinui\Desktop\request2.py", line 16, in <module>
    req = urllib2.urlopen(url)
  File "C:\Python27\lib\urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "C:\Python27\lib\urllib2.py", line 404, in open
    response = self._open(req, data)
  File "C:\Python27\lib\urllib2.py", line 427, in _open
    'unknown_open', req)
  File "C:\Python27\lib\urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "C:\Python27\lib\urllib2.py", line 1247, in unknown_open
    raise URLError('unknown url type: %s' % type)
URLError: <urlopen error unknown url type: ['http>

Here is the code I am using:

import urllib2
import urllib
import csv

# Open and read the source file and write entries to a list called link_list
source_file=open("source_new.csv", "rb")
source = csv.reader(source_file, delimiter=";")
link_list = [row for row in source]
source_file.close()

# Create an output file which contains the answer of the GET-Request
out=open("output.csv", "wb")
output = csv.writer(out, delimiter=";")
for row in link_list:
    url = str(row)
    req = urllib2.urlopen(url)
    output.writerow(req.read())

out.close()

What is going wrong there?

Thanks in advance for any hints.

Cheers

Answer 1

Using the row variable will pass a list element (it contains only one element, the url) to urlopen, but passing row[0] will pass the string containing the url.

The csv.reader returns a list for each row it reads, no matter how many items are in the row.

Answer 2

It's working now. If I directly reference row[0] in the loop there are no problems.

import urllib2
import urllib
import csv

# Open and read the source file and write entries to a list called link_list
source_file=open("source.csv", "rb")
source = csv.reader(source_file, delimiter=";")
link_list = [row for row in source]
source_file.close()

# Create an output file which contains the answer of the GET-Request
out=open("output.csv", "wb")
output = csv.writer(out)
for row in link_list:
    req = urllib2.urlopen(row[0])
    answer = req.read()
    output.writerow([answer])


out.close()