How to log the browsing history by packet sniffing?

Question

I would like to log my browsing history in a separate file. I would like to do this by logging the network traffic and in specific the HTTP get requests originating from my PC. I would like to do this in Python but I have no idea where to start with.

Answer 1

As I have mentioned, you can use the urlsnarf tool from dsniff as a pretty straight-forward solution. If you aren't looking for a strictly-Python solution, you can easily wrap it from Python.

To get a real-time output, you can run it directly using subprocess module:

import subprocess

p = subprocess.Popen('urlsnarf', stdout = subprocess.PIPE)
try:
    while True:
        l = p.stdout.readline()
        # ...
finally:
    p.terminate()

But that would require your user to have necessary permissions for packet sniffing. If you will want to run it as root, it would be probably better to run urlsnarf separately and just pipe the output through a named pipe.

First, with root permissions (in shell):

mkfifo /home/youruser/tmp/urlsnarf-pipe
chown youruser /home/youruser/tmp/urlsnarf-pipe
urlsnarf > /home/youruser/tmp/urlsnarf-pipe

Then simply read the pipe from within a Python script (run as your user):

f = open('/home/youruser/tmp/urlsnarf-pipe', 'r')
while True:
    l = f.readline()
    # ...