How to configure heritrix to log all encountered URLs including those which are filtered / not to crawl?
Question
I'm using heritrix 3.1.1-snapshot to crawl / archive some website contents, I need to log all urls encountered in every page it processes, including those urls which are (configured) not to be crawled.
I've been searching for long time and havent gotten positive results :( hope can get some helps here. thanks.
1 answers
solution1
0
http://crawler.archive.org/articles/user_manual/config.html 6.3.1.4节似乎回答了您的问题。
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Directed crawl using Nutch or Heritrix
In Heritrix crawler tool how to extract the contents from crawled urls
Cannot crawl those tweets which contains hashag
How to get a complete list of elements (including those which become visible on scrolling) in java appium?
is it possible to log how the jvm was called (including all -D -Xmx etc)
how to properly configure gradle build to avoid including log4j and slf4j from the resulting jar?
Crawl urls with a certain prefix
Exception encountered with fiji charts even after including all the required jars
How to configure log4j fileappender for all classes?
How do I configure embedded jetty server to log all requests?
Related Tags