Python recursion error with no recursion?

Question

I'm new to python, and I'm building a web crawler for funsies/educational purposes. I don't use any recursive functions, but I still get the 'RuntimeError: maximum recursion depth exceeded' error. I'm really confused, and kinda feel like I'm missing something obvious or just misunderstanding something. Am I somehow recursing, or could it be related to my large loops? The idea is to crawl the web until you've crawled 10k pages.

UPDATES:

Latest Code is here: http://pastebin.com/4v5GT7ft

Stack Trace is Here: http://pastebin.com/9GzAxZM9

Looks like my issue is trying to call str() on a URL that is not encoded properly. I've tried decoding the URLs and then ecoding them to unicode, but I was never able to do it successfully. Any advice would be greatly appreciated!

Answer 1

The code you gave us doesn't actually run (it's missing all the import statements, and it has indentation errors, and so on), and it requires a JET database we don't have and a third-party module to read it, and it's hardcoded to use pre-existing directories in your home directory.

I've attempted to fix all of that at http://pastebin.com/rCJriEu5 (which requires lxml and bs4 —if you were using a different parsing library or BS3, I can try it that way).

And when I run it, it seems to work. It's 31.73% complete, with no errors yet. Even if I do a sys.setrecusionlimit(50) at the start of the file, it still seems to work (3.67% complete so far).

So, whatever is wrong in your code is apparently in code you haven't showed us.