Python-是否可以在不打开UTF8 BOM的情况下将其添加到文件中？

Question

How to add an utf8-bom to a text file without open() it?

Theoretically, we just need to add utf8-bom to the beginning of the file, we don't need to read-in 'all' the content?

Answer 1

You need to read in the data because you need to move all the data to make room for the BOM. Files can't just prepend arbitrary data. Doing it in place is harder than just writing a new file with the BOM followed by the original data, then replacing the original file, so the easiest solution is usually something like:

import os
import shutil

from os.path import dirname, realpath
from tempfile import NamedTemporaryFile

infile = ...

# Open original file as UTF-8 and tempfile in same directory to add sig
indir = dirname(realpath(infile))
with NamedTemporaryFile(dir=indir, mode='w', encoding='utf-8-sig') as tf:
    with open(infile, encoding='utf-8') as f:
        # Copy from one file to the other by blocks 
        # (avoids memory use of slurping whole file at once)
        shutil.copyfileobj(f, tf)

    # Optional: Replicate metadata of original file
    tf.flush()
    shutil.copystat(f.name, tf.name) # Replicate permissions of original file

    # Atomically replace original file with BOM marked file
    os.replace(tf.name, f.name)

    # Don't try to delete temp file if everything worked
    tf.delete = False

This also verifies that the input file was in fact UTF-8 by side-effect, and the original file never exists in an inconsistent state; it's either the old or the new data, not the intermediate working copy.

If your files are large and your disk space is limited (so you can't have two copies on disk at once), then in-place mutation might be acceptable. The easiest way to do this is the mmap module which simplifies the process of moving the data around considerably vs. using in-place file object operations:

import codecs
import mmap

# Open file for read and write and then immediately map the whole file for write
with open(infile, 'r+b') as f, mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_WRITE) as mm:
    origsize = mm.size()
    bomlen = len(codecs.BOM_UTF8)
    # Allocate additional space for BOM
    mm.resize(origsize+bomlen)

    # Copy file contents down to make room for BOM
    # This reads and writes the whole file, and is unavoidable
    mm.move(bomlen, 0, origsize)

    # Insert the BOM before the shifted data
    mm[:bomlen] = codecs.BOM_UTF8

Answer 2

If you need in-place update, something like

def add_bom(fname, bom=None, buf_size=None):
    bom = bom or BOM
    buf_size = buf_size or max(resource.getpagesize(), len(bom))
    buf = bytearray(buf_size)
    with open(fname, 'rb', 0) as in_fd, open(fname, 'rb+', 0) as out_fd:
        # we cannot just just read until eof, because we
        # will be writing to that very same file, extending it.
        out_fd.seek(0, 2)
        nbytes = out_fd.tell()
        out_fd.seek(0)
        # Actually, we want to pass buf[0:n_bytes], but 
        # that doesn't result in in-place updates.
        in_bytes = in_fd.readinto(buf)
        if in_bytes < len(bom) or not buf.startswith(bom):
            # don't write the BOM if it's already there
            out_fd.write(bom)
        while nbytes > 0:
            # if we still need to write data, do so.
            # but only write as much data as we need
            out_fd.write(buffer(buf, 0, min(in_bytes, nbytes)))
            nbytes -= in_bytes
            in_bytes = in_fd.readinto(buf)

should do the trick.

As you can see, in-place updates are a little finnicky, because you are

1. Writing data to the place you've just read from. The read must always stay ahead of the write, otherwise you are overwriting not-yet-processed data.
2. Extending the file you are reading, so reading till EOF doesn't work.

In addition, this may leave the file in an inconsistent state. The copy to temporary -> move temporary to original method is much preferred if possible.