在系统级别擦除文件中已读取的数据

Question

Suppose you have a data file that some application is writing to. Once the data is read, it has no use so it can be deleted to save space. Here my design for handling this:

1) Once the file reaches a certain size, acquire a lock on it 2) Look at the index (pointer representing the byte within the file you're currently on) of a read call in the vnode table for this file 3) Delete all file data that comes before the index 4) Update the index of the read call to the new beginning of the file 5) Unlock the file so that reading/writing can resume

I don't have much systems experience myself, but I assume that if this is done at the systems level then it's language independent (ie if an application is using a java call or a python call to read/write there is no problem).

The data file is on a unix v6 file system. Monitoring the size of a file and deleting data is no problem, but I can't find a system call to 1) access other entries in the vnode table to see where they are at in reading the file and 2) Update the read pointers of these system calls.

Answer 1

A Unix v6 file system doesn't make any sense. I'm sure you don't mean the v6 Unix system released by Bell Labs for the PDP-11 in 1975; for one thing, I'm quite sure it never supported Java or Python! So you probably need to be a bit more specific what system you mean. It can't be OpenBSD (which hasn't released a version 6 yet), or FreeBSD (which is up to version 10 and version 6 would be quite obsolete). Maybe NetBSD?

In any case, I wouldn't recommend trying to make changes in the kernel to support this, as it would be very non-portable, and far more difficult than you think to do things right. In particular, renumbering the mapping between logical to physical blocks is tricky . There are operating systems and file systems (such as Linux's ext4 and xfs) that support the PUNCH HOLE operation, which will deallocate blocks between specified starting and ending offset, so long as those offsets are multiples of the file system block size. You can even use a COLLAPSE RANGE operation that will "delete" bytes from the beginning or middle of the file --- but again, they have to be multiples of the file system block size, and they are not going to affect the file offsets of any open file descriptors. COLLAPSE RANGE is in general a really bad idea, since it requires eliminating all of the cached pages for the file in order to maintain page cache consistency. The times when it is useful are extremely rare. The main use case seems to be people who are manipulating really large video files. But architecturally, this is almost certainly not what you want to do.

What I would recommend is to do this in userspace. Yes, it means you will need to implement support in Java and Python, but trust me, this will be easier than trying to do kernel and file system level hacking. If you really insist, you can create a C library and then create SWIG interfaces that can be called from Java and Python, but it's actually probably easier to reimplement the logic twice in idiomatic and native Java and Python code.

What I would do is to have the writer write in chunks that are approximately 1 megabyte in size or so, once a chunk reaches a certain size, start a new chunk in a new file. Name the chunks numerically --- ie, data0001, data0002, etc. The reader can just simply read the chunk, and when it is done with it, delete the chunk file, and move on.

Really simple, and it won't get you lost in the weeds of trying to do kernel-level hacking.