Pickle (Python 3.6) Writes Empty File

Question

I'm trying to teach myself Python and so created a dumb script to check a blogging website to check for new updates and then save metadata associated to the updates. If there are new posts, I open the previous metadata, append the new metadata, and then save. However, I find that quite often (can't figure out when it does and does not work), these updates produce an empty file, and I lose all the metadata.

if new_post_count > 0:
        file_name = 'all_posts' + user

        previous_posts = pickle.load(open(file_name, 'rb'))

        current_posts = get_posts(client, user, start_from_posts=0, total_posts=new_post_count)

        all_posts = previous_posts.extend(current_posts)

        f = open(file_name, 'wb')
        pickle.dump(all_posts, f)
        f.close() 

Looking at the forums, it might make more sense to use something else to save my data besides pickle (any suggestions?). Even if that's the case, I'd still like to know what I'm doing wrong here so I don't create the same mistake later.

Answer 1

The problem is not with the pickle module but the line that reads:

 all_posts = previous_posts.extend(current_posts)

What actually happens is that the extend method is called, extending previous_posts in place, which once it has successfully completed returns the keyword None .

You then have this keyword being assigned to all_posts instead of the contents of previous_posts, and then it gets written to the file.

Try modifying it as follows:

if new_post_count > 0:
    file_name = 'all_posts' + user

    previous_posts = pickle.load(open(file_name, 'rb'))

    current_posts = get_posts(client, user, start_from_posts=0, total_posts=new_post_count)

    previous_posts.extend(current_posts)

    f = open(file_name, 'wb')
    pickle.dump(previous_posts, f)
    f.close() 

Even better would be to include Jean-Francois' suggestion:

if new_post_count > 0:
    file_name = 'all_posts' + user
    with open(file_name, 'rb') as f:
        previous_posts = pickle.load(f)

    current_posts = get_posts(client, user, start_from_posts=0, total_posts=new_post_count)

    previous_posts.extend(current_posts)

    with open(file_name, 'wb') as f:
        pickle.dump(previous_posts, f)