Python通过存储在csv中的链接下载文件

Question

As a newbie in Python (2.7) I`m looking for next suggestion:

I have a csv file with stored http links in one column comma delimited.

http://example.com/file.pdf,
http://example.com/file.xls,
http://example.com/file.xlsx,
http://example.com/file.doc,

The main aim is to loop through all these links and download files by them in original extention and name.

So my search results and help here gave me next script:

import urllib2
import pandas as pd 

links = pd.read_csv('links.csv', sep=',', header =(0))

url = links                   # I know this part wrong by don`n know how to do right

user_agent = 'Mozilla 5.0 (Windows 7; Win64; x64)'

file_name = "tessst"          # here the files name by how to get their original names

u = urllib2.Request(url, headers = {'User-Agent' : user_agent})
req = urllib2.urlopen(u)
f = open(file_name, 'wb')
f.write(req.read())

f.close()

please any help

PS not sure about pandas - maybe csv better?

Answer 1

If I can assume your CSV file to be one column only, containing links then this would work .

import csv, sys
import requests
import urllib2
import os

filename = 'test.csv'
with open(filename, 'rb') as f:
    reader = csv.reader(f)
    try:
        for row in reader:
            if 'http' in row[0]:
                #print row
                rev  = row[0][::-1]
                i  = rev.index('/')
                tmp = rev[0:i]
                #print tmp[::-1]
                rq = urllib2.Request(row[0])
                res = urllib2.urlopen(rq)
                if not os.path.exists("./"+tmp[::-1]):                
                    pdf = open("./" + tmp[::-1], 'wb')
                    pdf.write(res.read())
                    pdf.close()
                else:
                    print "file: ", tmp[::-1], "already exist"
    except csv.Error as e:
        sys.exit('file %s, line %d: %s' % (filename, reader.line_num, e))