Saving Current Progress or State

Question

I'm not sure how to solve a problem I have but if my application is scraping Google keywords for say 1,000 webpages, if I cancel the task, or an exception is thrown, what's the best approach for saving how far the task has got, so when it continues it resumes from it's current position.

Do I serialize an object that contains properties with the current status? I've never had to deal with this and unsure of what to actually search for.

Answer 1

I convert my comment to answer as requested to do.

I'll suggest you to save the state of the task to a file. By State I mean the current status of the work your program is doing.

For example, If your application's task is to parse n number of files. It encountered an exception in middle of it, let's say file index 45 . Save in a temporary file that you were in 45 so when application runs again you query the temporary file to see whether any task pending, If yes then continue from that point.

Even me too don't know any programming pattern says this approach, I just suggested my idea.

Answer 2

Like database do, use a hard disk:

• Plan your future task in a file
• Each time that you finish a task, append some text (task id o whatever)

You have to deal with some concurrency, but I think that you can deal with it

Use XML is perhaps more advanced to save your plan of work, but after that is more simple to record a file with id's of finished tasks.