How to read a file from the disk if less than X days old, if older, refetch the html file

Question

I wish to read an html file off of the internet and cache it. Then when I go back, because I'm debugging, I don't want to hammer the servers with the numerous requests I'll need. I don't want to get my IP banned for slamming the server over and over again just because I'm debugging. So my code needs to look something like:

if ((file > days_old) || !(file exists))
   fetch html file from internet
   save file to disk
else
   read it from the disk

Because there will be multiple files, I'll need to include a variable name in the file name so the file is unique and I can easily look it up again.

I just learned Perl this semester and we only learned the basics & a bit of regex, once I get this I should be mostly fine.

Thanks!

Answer 1

Use an existing module:

• Cache::Cache

• HTTP::Cache::Transparent

If you really want to implement your own, you'll want to look at the If-Modified-Since and ETag HTTP headers to determine when to re-fetch a file, rather than an arbitrary days_old number you suck out of your thumb. You will also have to generate a unique filename, preferably with a hash function, while retaining the original URL to cater for hash collisions.