WGet下载顺序的逻辑
[英]WGet's Logic in Order of Downloading
问题描述
This is a more general question, but which has wider implications for a data mining project I'm running. 
这是一个更普遍的问题，但它对我正在运行的数据挖掘项目有更广泛的影响。 I have been using wget to mirror archival webpages for analysis. 我一直在使用wget镜像归档网页进行分析。 This is a large amount of data and my current mirroring process has been going on for almost a week. 这是一个大量的数据，我目前的镜像过程已经持续了将近一个星期。 Which has given me a lot of time to watch the readout. 这给了我很多时间来观看读数。
How does wget determine the order in which it downloads pages? wget如何确定下载页面的顺序？ I can't seem to discern a consistant logic to its decision making process (it's not proceeding alphabetically, by date of original site creation, or by file type). 我似乎无法辨别其决策制定过程的一致性逻辑（它不按字母顺序，按原始网站创建日期或文件类型进行）。 As I begin to work with the data, this would be very helpful to grasp. 当我开始处理数据时，这将非常有助于掌握。
FWIW, here is the command that I'm using (it required cookies, and while the site's TOS do allow access 'by any means' I don't want to take any chances) - where SITE = URL: FWIW，这是我正在使用的命令（它需要cookie，而网站的TOS允许以任何方式“访问”我不想冒任何机会） - 其中SITE = URL：
wget -m --cookies=on --keep-session-cookies --load-cookies=cookie3.txt --save-cookies=cookie4.txt --referer=SITE --random-wait --wait=1 --limit-rate=30K --user-agent="Mozilla 4.0" SITE
Edited to Add: In comments to Chown's helpful answer, I refined my question a bit so here it is. 编辑添加：在对Chown的有用答案的评论中，我稍微改进了我的问题，所以在这里。 With larger sites - say epe.lac-bac.gc.ca/100/205/301/ic/cdc/E/Alphabet.asp - I find that it goes through initially creating a directory structure and some of the index.html/default.html pages, but then goes back through the disparate websites a few more times (grabbing a few more images and sub-pages on each pass, for example) 有更大的网站 - 比如epe.lac-bac.gc.ca/100/205/301/ic/cdc/E/Alphabet.asp - 我发现它最初创建了一个目录结构和一些index.html / default.html页面，然后再回到不同的网站几次（例如，在每次传递时抓取更多的图像和子页面）
1 个解决方案
解决方案1
4 已采纳 2011-10-15 23:31:45
From gnu.org wget Recursive Download : 来自gnu.org wget递归下载 ：
- Recursive Download
GNU Wget is capable of traversing parts of the Web (or a single http or ftp server), following links and directory structure.
With http urls, Wget retrieves and parses the html or css from the given url, retrieving the files the document refers to, through markup like href or src, or css uri values specified using the 'url()' functional notation.
Recursive retrieval of http and html/css content is breadth-first .
The maximum depth to which the retrieval may descend is specified with the '-l' option.
When retrieving an ftp url recursively, Wget will retrieve all the data from the given directory tree (including the subdirectories up to the specified depth) on the remote server, creating its mirror image locally.
By default, Wget will create a local directory tree, corresponding to the one found on the remote server.
.... snip ....
Recursive retrieval should be used with care.
From my own very basic testing, it goes in order of appearance from top to bottom of the page when the structure depth is 1: 从我自己的基本测试开始，当结构深度为1时，它按照从页面顶部到底部的外观顺序排列：
[ 16:28 root@host /var/www/html ]# cat index.html
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en-US">
<head>
<link rel="stylesheet" type="text/css" href="style.css">
</head>
<body>
<div style="text-align:center;">
<h2>Mobile Test Page</h2>
</div>
<a href="/c.htm">c</a>
<a href="/a.htm">a</a>
<a href="/b.htm">b</a>
</body>
</html>
[ 16:28 jon@host ~ ]$ wget -m http://98.164.214.224:8000
--2011-10-15 16:28:51-- http://98.164.214.224:8000/
Connecting to 98.164.214.224:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 556 [text/html]
Saving to: "98.164.214.224:8000/index.html"
100%[====================================================================================================================================================================================================>] 556 --.-K/s in 0s
2011-10-15 16:28:51 (19.7 MB/s) - "98.164.214.224:8000/index.html" saved [556/556]
--2011-10-15 16:28:51-- http://98.164.214.224:8000/style.css
Connecting to 98.164.214.224:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 221 [text/css]
Saving to: "98.164.214.224:8000/style.css"
100%[====================================================================================================================================================================================================>] 221 --.-K/s in 0s
2011-10-15 16:28:51 (777 KB/s) - "98.164.214.224:8000/style.css" saved [221/221]
--2011-10-15 16:28:51-- http://98.164.214.224:8000/c.htm
Connecting to 98.164.214.224:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/html]
Saving to: "98.164.214.224:8000/c.htm"
[ <=> ] 0 --.-K/s in 0s
2011-10-15 16:28:51 (0.00 B/s) - "98.164.214.224:8000/c.htm" saved [0/0]
--2011-10-15 16:28:51-- http://98.164.214.224:8000/a.htm
Connecting to 98.164.214.224:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2 [text/html]
Saving to: "98.164.214.224:8000/a.htm"
100%[====================================================================================================================================================================================================>] 2 --.-K/s in 0s
2011-10-15 16:28:51 (102 KB/s) - "98.164.214.224:8000/a.htm" saved [2/2]
--2011-10-15 16:28:51-- http://98.164.214.224:8000/b.htm
Connecting to 98.164.214.224:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2 [text/html]
Saving to: "98.164.214.224:8000/b.htm"
100%[====================================================================================================================================================================================================>] 2 --.-K/s in 0s
2011-10-15 16:28:51 (85.8 KB/s) - "98.164.214.224:8000/b.htm" saved [2/2]
FINISHED --2011-10-15 16:28:51--
Downloaded: 5 files, 781 in 0s (2.15 MB/s)
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.