Creating a knowledge base on top of provided webpages as a feed

Question

I have some issues with my parts of final year projects. We are implementing a plagiarism detection framework. I'm working on internet sources detection part. Currently my internet search algorithm is completed. But I need to enhance it so that internet search delay is reduced.

My idea is like this:

First user is prompt to insert some web links as the initial knowledge feed for the system. Then it crawl through internet and expand it's knowledge

Once the knowledge is fetch System don't need to query internet again. Can someone provide me some guidance to implement it? We are using Java. But any abstract detail will surely help me.

Answer 1

if the server side programming is you hand then you can manage a tabel having a boolean in database which shows whether the details were read before. every time your client connects to server, it will check the boolean first and if boolean was set false then it will mean that there is a need to send updates to client other wise no updates will be sent,

the boolean will become true every time when client downloads any data from server and will become false when ever the database is updated

Answer 2

I'm not quite sure that I understand what you're asking. Anyway:

• if you're looking for a Java Web crawler, then you I recommend that you read this question

• if you're looking for Java libraries to build a knowledge base (KB), then it really depends on (1) what kind of properties your KB should have, and (2) what kind of reasoning capabilities you expect from your KB. One option is to use the Jena framework , but this requires that you're comfortable with Semantic Web formalisms.

Good luck!