Crawl Web Data using Web Crawler

Question

I would like to use a web crawler and crawl a particular website. The website is a learning management system where many student upload their assignments,project presentations and so on. My question is that can i use a web crawler and download the files that have been uploaded in the learning management system. After i download them i would like to create an index on them so as to query the set of documents. User can use my application as a search engine. Can a crawler does this? I know about webeater ( Crawler written in Java )

Answer 1

1. Download the files in Java SingleThread.
2. Parse the files (you can get idea from parse plugins of nutch).
3. Create index with lucene

Answer 2

If you want to use a real webcrawler, user http://www.httrack.com/

It offers you so many options for copying websites or content on webpages including flash. It works on windows and mac.

Then you can do steps 2 and 3 as suggested above.