如何从流附加到文件而不是在Python中覆盖

Question

I am new to python so this maybe a simple question but I have a kafka consumer from which I read in messages. Every time a new message comes in it rewrites the previous message into the order.json file however I want to append it instead. Additionally, I want to make sure the messages do not get read in no faster than every 1 second using possible some sort of pause. Any tips on how to do this would be much appreciated. Here is my current code

for message in consumer:
     with open('order.json', 'w') as file:
          file.write(message.value.decode('UTF-8'))

Answer 1

You want to open your file in append mode . Also, you may not want to open the file on each message, since it can be an expensive operation (eg you will need to change file metadata like modification time every time you close the file):

# open file in append mode, once
with open('order.json', 'a') as file:
    for message in consumer:
        file.write(message.value.decode('UTF-8'))

As for rate limiting, you could start with something simple, like the following:

import time

def ratelimit(it, sleep=1.0):
    for v in it:
        yield v
        time.sleep(sleep)

if __name__ == '__main__':
    for i in ratelimit(range(10)):
        print(i)

This would ensure, there is at least one second delay between successive values from an iterator. Here's a asciicast showing the rate-limiter in action.

Answer 2

Open the file in append mode as ' w' (write mode) truncates the file each time it exists

for message in consumer:
    with open('order.json', 'a') as file:
        file.write(message.value.decode('UTF-8'))

Answer 3

Open outside the loop:

with open('order.json', 'w') as file:
    for message in consumer:         
          file.write(message.value.decode('UTF-8'))

Or open outside and use a if you are going to be appending for each run:

with open('order.json', 'a') as file:
    for message in consumer:         
          file.write(message.value.decode('UTF-8'))