Using python script to send GET request to a server with netcat

Question

I'm running Ubuntu 16.04 and I'm trying to write a python script that makes a GET request to a specified image file given the url. As an example, in the code below:

host is www.google.com

port is 80

u.path is /images/srpr/logo3w.png

proc = Popen(["netcat {} {}".format(host, port)], shell= True)
proc = Popen(["GET {} HTTP/1.1".format(u.path)], shell= True)
proc = Popen(["Host: {}".format(host)], shell= True)
proc = Popen(["Connection: close"], shell= True)
proc = Popen(["\n"], shell= True)

My problem is that I can execute these normally in the terminal, but when I try to run the script it seems like sends the GET request to www.google.com before it takes the specification of u.path . I know it is doing this for two reasons. First, just before the server response comes in I get the following:

/bin/sh: 1: Host:: not found /bin/sh: 1: Connection:: not found

Second, I know that the server response of the image data is a bunch of ugly stuff interpreted as weird Unicode symbols on the terminal, but I'm clearly getting the www.google.com HTML text on the server response.

I was thinking I may need to make it wait to do the HTTP request until the netcat STDIN is open, but I don't know how. Or maybe it's just completing the request because it's sending a \\n somehow? I really don't know.

EDIT: It seems like it actually isn't sending the request to www.google.com . I saved the server response as a .html file and it looks like a cloudfront website

EDIT2: After more research, it seems as if the problem is that since netcat is interactive and so it 'deadlocks' or something like that. I tried to use proc.communicate() but since I need to send multiple lines it doesn't allow it seeing as communicate only allows the initial input to be written to STDIN and then it sends EOF or something along those lines. This led me to trying to use proc.stdin.write but this is apparently also known to cause deadlock with something related to making the Popen commands use subprocess.PIPE for STDIN , STDOUT , and STDERR . It also requires the input to be encoded as a bytes-like object, which I have done but when I send \\r\\n\\r\\n at the end to try to close the connection it doesn't do anything and the STDOUT just contains b'' which I understand to be an empty string in the form of bytes

Answer 1

For anyone that has a similar problem, here is the solution that I found:

#begin the interactive shell of netcat
proc = Popen(['netcat -q -1 {} {}'.format(host, port)], shell=True, stdout=PIPE, stdin=PIPE, stderr=PIPE)

#set file status flags on stdout to non-blocking reads
fcntl.fcntl(proc.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)

#each time we write a diffrent line to the interactive shell
#we need to flush the buffer just to be safe
#credit to http://nigelarmstrong.me/2015/04/python-subprocess/
proc.stdin.write(str.encode('GET %s HTTP/1.1\n' %(path+filename)))
proc.stdin.flush()
proc.stdin.write(str.encode('Host: {}\n'.format(host)))
proc.stdin.flush()
proc.stdin.write(str.encode('Connection: close\n'))
proc.stdin.flush()
proc.stdin.write(str.encode('\r\n\r\n'))
proc.stdin.flush()

#give the server time to respond
proc.wait()

#store the server response (which is bytes-like)
#attempting to decode it results in error since we're recieving data as a mix of text/image
serv_response = proc.stdout.read()