Why am I getting "[Errno 13] Permission denied:" when making consecutive open and write calls?

Question

Update

It's throwing the error with a direct call now, and in the relations docs. I got rid of write_el() altogether and just do this:

...
if el["doc_type"] == "node":
    with codecs.open((fo_pre+"_nodes.json"), mode) as fo:
        fo.write(json.dumps(el, indent=2)+"\n")
...

Also, it should be noted that the XML document (OSM) has all the node elements first, followed by the way elements, then the relation elements.

Original post

I'm writing to multiple JSON files from within Python xml.etree.ElemenTree.iterparse , using codecs.open and json.dumps . I call a separate function to write to the file.

This works for some of the elements/documents, but not all. It writes only so many then stops with PermissionError: [Errno 13] Permission denied: <file name> . The last call to the file write method returns 207, but so do many of the previous calls. And, the next element looks normal:

<!--Last element written to JSON file.-->
  <node id="7898832843" lat="48.7888301" lon="-122.5067978" version="1" timestamp="2020-09-11T22:37:30Z" changeset="90779671" uid="10612244" user="mapstuffs"/>

<!--Next element, not written to JSON file.-->
  <node id="7898832844" lat="48.7888177" lon="-122.5058429" version="1" timestamp="2020-09-11T22:37:30Z" changeset="90779671" uid="10612244" user="mapstuffs"/>

Plus, it throws the error at a different element each time I try. And, it sometimes doesn't throw the error.

Stripped down Python:

import xml.etree.ElementTree as ET
import codecs
import json

def write_el(el, file_out, mode = "a"):
    with codecs.open(file_out, mode) as fo:
        fo.write(json.dumps(el, indent=2)+"\n")
    return

def process_map(file_in, fo_pre, mode = "a"):
    
    for _, element in ET.iterparse(file_in):
        # shape_element() formats XML elements into JSON-compatible Python
        # dictionaries and lists.
        el = shape_element(element)
        if el:
            if el["doc_type"] == "node":
# Calling open/write directly works.
#                 with codecs.open(fo_pre+"_nodes.json", mode) as fo:
#                     fo.write(json.dumps(el, indent=2)+"\n")
# But, calling write_el for this doc_type throws permission error
# halfway through the document. The element following the last written looks
# just fine.
                write_el(el=el, file_out=fo_pre+"_nodes.json", mode=mode)
# Calling write_el works fine for the other doc_types, if error not thrown
# from previous block first.
            elif el["doc_type"] == "way":
                write_el(el=el, file_out=fo_pre+"_ways.json", mode=mode)
            elif el["doc_type"] == "relation":
                write_el(el=el, file_out=(fo_pre+"_relations.json"),
                         mode=mode, write=write, pretty=pretty)
                
def test():
    process_map(file_in=filename, fo_pre="test_bham", write=True)

    return

test()

Returns

PermissionError: [Errno 13] Permission denied: 'test_bham_nodes.json'

Answer 1

Thanks to some commentors, I've found that nesting with open() outside of iterparse() does the trick.

My antivirus program was likely the culprit, accessing the file each time it was closed, which occasionally caused an access conflict on the next file open operation.

I didn't like opening and closing the file for each element because of the operation cost, and I originally had parsed the XML from within the file open when I was writing to a single file. But, when I decided to write to multiple files, I didn't want to parse the XML over again for each JSON file I wanted to write from it.

Seems like a no-brainer now, but nesting file opens works and saves compute. It might build up the indents if you want to write a lot of files, but c'est la vie.

def process_map(file_in, fo_pre, mode = "a"):
    
    with codecs.open(fo_pre+"_nodes.json", mode) as nd_fo, \
    codecs.open(fo_pre+"_ways.json", mode) as wy_fo, \
    codecs.open(fo_pre+"_relations.json", mode) as rl_fo:

        for _, element in ET.iterparse(file_in):
                el = shape_element(element)
                if el:
                    if el["doc_type"] == "node":
                        write_el(el=el, file_out=fo_pre+"_nodes.json",
                                 mode=mode)
                    elif el["doc_type"] == "way":
                        write_el(el=el, file_out=fo_pre+"_ways.json",
                                 mode=mode)
                    elif el["doc_type"] == "relation":
                        write_el(el=el, file_out=fo_pre+"_relations.json",
                                 mode=mode)

    return