如何读取大文件（socket编程和python）？

Question

I'm a beginner in socket programming and python. I would like to learn how to send a large text file (eg, > 5MB) from the server to client. I keep getting an error that says

Traceback (most recent call last):
  File "fserver.py", line 50, in <module>
    reply = f.read()
ValueError: Mixing iteration and read methods would lose data

Below is a partial of my code. Can someone take a look and give me some hints on how to resolve this issue? Thank you for your time.

myserver.py

#validate filename
        if os.path.exists(filename):
            with open(filename) as f:
                for line in f:
                    reply = f.read()
                    client.send(reply)
            #f = open(filename, 'r')
            #reply = f.read()
            #client.send(piece)
        else:
            reply = 'File not found'
            client.send(reply)

myclient.py

while True:
    print 'Enter a command: list or get <filename>'
    command = raw_input()
    if command.strip() == 'quit':
        break
    client_socket.send(command)

    data = client_socket.recv(socksize)
    print data

Answer 1

The problem here has nothing to do with sockets, or with how big the file is. When you do this:

for line in f:
    reply = f.read()

The for line in f is trying to read one line of the file at a time, and then for each line you're trying to read the entire file. That won't work.

If you didn't get this error (which you won't in many cases), the first time through the loop you would read and ignore the first line, and then read and send everything but the first line (or, possibly, everything but the first, say, 4KB) as one giant reply, and then the loop would be done.

What you want is either one or the other:

for line in f:
    reply = line

… or …

# no for loop
reply = f.read()

---

Meanwhile, on your client side, you're only doing one recv . That's going to get the first 4K (or whatever socksize is) or less, and then you never receive anything else.

What you need is a loop. Like this:

while True:
    data = client_socket.recv(socksize)
    print data

But now you have a new problem. Once the file is done, the client will sit there waiting forever for the next chunk of data, which will never come. So the client needs to know when it's done. And the only way it can know that is if the server puts that information into the data stream.

One way to do this is to send the length before the file. One standardized way to do this is to use the netstring protocol. You can find libraries that do this for you, but it's simple enough to do by hand. Or maybe do something more like HTTP, where the headers are just separated by newlines, and separated from the body by a blank line; then you can use socket.makefile as your protocol implementation. Or even a binary protocol, where you just send the length as four bytes.

There's another problem we might as well fix while we're here: send(reply) doesn't necessarily send the whole reply; it sends anywhere from 1 byte to the whole thing, and returns a number telling you what got sent. The simple fix to that is to use sendall(reply) , which guarantees to send all of it.

And finally: Your server is expecting that each recv will get exactly one command, as sent by send . But sockets don't work that way. Sockets are byte streams, not message streams ; there's nothing preventing recv from getting, say, just half a command, and then your server will break. So, you need some kind of protocol in that direction as well. Again, you could use netstring, or newline-separated messages, or a binary length prefix, but you have to do something .

(The blog post linked above has very simple example code for using binary length prefixes as a protocol.)

Answer 2

你可以for line in file.readlines()做