Pycurl fails to upload after 301 redirection on Python3

Question

I'm troubleshooting an existing python based Nagios plugin that uses PycURL to test that different actions can be taken on a remote WebDav service (GET,PUT,DELETE). We are having an issue when the service responds with a 301 redirection with the error "411 Length Required".

After checking the headers of the PUT requests for both the original service and to the redirected one, the latter is missing the "Content-Length" header, which is why this is failing. I haven't been able to find if there is an option that needs to be setup that is maybe needed for this to occur.

I am able to fix this in Python2 by adding the filesize using the option "INFILESIZE":

c.setopt(c.INFILESIZE, os.path.getsize(filepath))

The code looks like this:

#!/bin/python2

import pycurl
import os

filepath = '/tmp/testfile'

c = pycurl.Curl()
c.setopt(c.URL, 'http://remote_host.com/filename')

c.setopt(c.UPLOAD, 1)
file = open(filepath)
c.setopt(c.READDATA, file)
c.setopt(c.FOLLOWLOCATION, 1)
c.setopt(c.INFILESIZE, os.path.getsize(filepath))

c.perform()
c.close()

file.close()

However on Python3 (I've tried on 3.4, 3.6 and 3.7) the same code exits with error:

Traceback (most recent call last):
  File "/usr/lib64/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 2: invalid continuation byte
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
pycurl.error: (42, 'operation aborted by callback')

And I'm do not understand if this is an issue of with the reply from the server. But if I remove the INFILESIZE option, then it just fails with the 411 error mentioned above.

If anyone has any clue what I'm doing wrong, it would be greatly appreciated.

Answer 1

PycURL has different handling of files in Python 2 and Python 3. It sounds like you are running into this difference. See this manual page for the description of behavior: http://pycurl.io/docs/latest/files.html

It sounds like on Python 2, libcurl is able to perform a stat(2) call on the open file description to figure out its size. On Python 3 there is no file descriptor being passed but a function, hence stat(2) approach doesn't work and no file length is calculated for you.

To troubleshoot the Unicode decode error, on Python 2 change the code to use READFUNCTION rather than READDATA and see what error you get. If the server returns an error message in the response which is not valid utf-8, Python 3 may fail in the way you describe.