Python urljoin not removing superflous dots

Question

I'm using urljoin to get the absolute URL of links of a page. For the most part it does a good job at things like resolving relative links, but I notice that for some reason it does not remove superflous dots in some cases. For example:

>>> urljoin("http://x.com","http://x.com/../../X",False)
'http://x.com/../../X'
>>> urljoin("http://x.com","http://x.com/./../X",False)
'http://x.com/./../X'

If I give such an URL to a web browser, it corrects it fine, but if I try to use Python's urlopen() it throws an exception (urllib2.HTTPError: HTTP Error 400: Bad Request).

Is this expected behavior? Is there some other Python function that correctly removes these dots that I should be using instead, or is this a bug?

Answer 1

I think you should use an absolute base and a relative url .
If you call it like this, it removes the dots:

urljoin("http://x.com/a/b/page.html","../../index.html",False)
# result: 'http://x.com/index.html'

urljoin("http://x.com/a/b/page.html","./index.html",False)
# result: 'http://x.com/a/b/index.html'

I found a way to normalize a url in this answer . Example:

urljoin('http://www.example.com/foo/bar/../../baz/bux/', '.')
# result: 'http://www.example.com/baz/bux/'

I think the invalid url handling (too much .. ) can only be handled "manually", like this:

def remove_extra_dots(url):
    parsed = list(urlparse(url))
    dirs = []
    for name in parsed[2].split("/"):
        if name == "..":
            if len(dirs) > 1:
                dirs.pop()
        else:
            dirs.append(name)
    parsed[2] = "/".join(dirs)
    return urlunparse(parsed)

This will eliminate all .. s from the url, even the invalid ones. Examples:

"http://x.com/a/b/c/../../X"  #->  http://x.com/a/X
"http://x.com/a/b/../../X"    #->  http://x.com/X
"http://x.com/../../X"        #->  http://x.com/X

Answer 2

It's a bit of a bug! According to RFC 3986 , those superfluous .. segments should be removed. No Python standard library function I've been able to find really does the job correctly.

---

The existing answer is slightly incorrect in a couple of ways, sadly. URL resolution is a bit more complicated than one might expect.

In addition to the shortcomings of urljoin mentioned there (it doesn't resolve URLs when joining an absolute path, and it doesn't handle excessive .. s), joining an URL with . will remove the last segment. For example, urljoin('http://example.com/dir/./wrong/../file.txt', '.') will result in 'http://example.com/dir/' , removing the file and thus making you have to add it back in. Also, urljoin('http://example.com/dir/..', '.') results in 'http://example.com/dir/' , which is just simply wrong for your case.

Not only that, the remove_extra_dots function provided actually has a bug in it. If the URL ends in a trailing dot segment (which the last bug in the previous paragraph renders impossible, but if it was somehow fixed), a trailing slash won't be added. Consider remove_extra_dots('http://example.com/..') . This should result in 'http://example.com/' , but actually results in 'http://example.com' (note the missing slash). It's a small difference, but a lot of sites do redirects on receiving missing slashes, so you might get unexpected results.

---

The following function completely resolves URLs - that is, both . s and .. s - following RFC 3986. Without having to rely on urljoin , too!

try:
    # Python 3
    from urllib.parse import urlsplit, urlunsplit
except ImportError:
    # Python 2
    from urlparse import urlsplit, urlunsplit

def resolve_url(url):
    parts = list(urlsplit(url))
    segments = parts[2].split('/')
    segments = [segment + '/' for segment in segments[:-1]] + [segments[-1]]
    resolved = []
    for segment in segments:
        if segment in ('../', '..'):
            if resolved[1:]:
                resolved.pop()
        elif segment not in ('./', '.'):
            resolved.append(segment)
    parts[2] = ''.join(resolved)
    return urlunsplit(parts)

You can then call it when you have your complete URL (in your case, after joining) as following.

>>> resolve_url("http://example.com/dir/../../thing/.")
'http://example.com/thing/'

For more information on what works/doesn't work, see a similar answer I wrote on the subject .