Java：使用状态的ASCII随机行文件访问

Question

Is there a better [pre-existing optional Java 1.6] solution than creating a streaming file reader class that will meet the following criteria?

• Given an ASCII file of arbitrary large size where each line is terminated by a \\n
• For each invocation of some method readLine() read a random line from the file
• And for the life of the file handle no call to readLine() should return the same line twice

Update:

• All lines must eventually be read

Context: the file's contents are created from Unix shell commands to get a directory listing of all paths contained within a given directory; there are between millions to a billion files (which yields millions to a billion lines in the target file). If there is some way to randomly distribute the paths into a file during creation time that is an acceptable solution as well.

Answer 1

In order to avoid reading in the whole file, which may not be possible in your case, you may want to use a RandomAccessFile instead of a standard java FileInputStream . With RandomAccessFile , you can use the seek(long position) method to skip to an arbitrary place in the file and start reading there. The code would look something like this.

RandomAccessFile raf = new RandomAccessFile("path-to-file","rw");
HashMap<Integer,String> sampledLines = new HashMap<Integer,String>();
for(int i = 0; i < numberOfRandomSamples; i++)
{
    //seek to a random point in the file
    raf.seek((long)(Math.random()*raf.length()));

    //skip from the random location to the beginning of the next line
    int nextByte = raf.read();
    while(((char)nextByte) != '\n')
    {
        if(nextByte == -1) raf.seek(0);//wrap around to the beginning of the file if you reach the end
        nextByte = raf.read();
    }

    //read the line into a buffer
    StringBuffer lineBuffer = new StringBuffer();
    nextByte = raf.read();
    while(nextByte != -1 && (((char)nextByte) != '\n'))
        lineBuffer.append((char)nextByte);

    //ensure uniqueness
    String line = lineBuffer.toString();
    if(sampledLines.get(line.hashCode()) != null)
        i--;
    else
       sampledLines.put(line.hashCode(),line);
}

Here, sampledLines should hold your randomly selected lines at the end. You may need to check that you haven't randomly skipped to the end of the file as well to avoid an error in that case.

EDIT: I made it wrap to the beginning of the file in case you reach the end. It was a pretty simple check.

EDIT 2: I made it verify uniqueness of lines by using a HashMap .

Answer 2

Pre-process the input file and remember the offset of each new line. Use a BitSet to keep track of used lines. If you want to save some memory, then remember the offset of every 16th line; it is still easy to jump into the file and do a sequential lookup within a block of 16 lines.

Answer 3

Since you can pad the lines, I would do something along those lines, and you should also note that even then, there may exist a limitation with regards to what a List can actually hold.

Using a random number each time you want to read the line and adding it to a Set would also do, however this ensures that the file is completely read:

public class VeryLargeFileReading
    implements Iterator<String>, Closeable
{
    private static Random RND = new Random();
    // List of all indices
    final List<Long> indices = new ArrayList<Long>();
    final RandomAccessFile fd;

    public VeryLargeFileReading(String fileName, long lineSize)
    {
        fd = new RandomAccessFile(fileName);
        long nrLines = fd.length() / lineSize;
        for (long i = 0; i < nrLines; i++)
            indices.add(i * lineSize);
        Collections.shuffle(indices);
    }

    // Iterator methods
    @Override
    public boolean hasNext()
    {
        return !indices.isEmpty();
    }

    @Override
    public void remove()
    {
        // Nope
        throw new IllegalStateException();
    }

    @Override
    public String next()
    {
        final long offset = indices.remove(0);
        fd.seek(offset);
        return fd.readLine().trim();
    }

    @Override
    public void close() throws IOException
    {
        fd.close();
    }
}

Answer 4

If the number of files is truly arbitrary it seems like there could be an associated issue with tracking processed files in terms of memory usage (or IO time if tracking in files instead of a list or set). Solutions that keep a growing list of selected lines also run in to timing-related issues.

I'd consider something along the lines of the following:

1. Create n "bucket" files. n could be determined based on something that takes in to account the number of files and system memory. (If n is large, you could generate a subset of n to keep open file handles down.)
2. Each file's name is hashed, and goes into an appropriate bucket file, "sharding" the directory based on arbitrary criteria.
3. Read in the bucket file contents (just filenames) and process as-is (randomness provided by hashing mechanism), or pick rnd(n) and remove as you go, providing a bit more randomosity.
4. Alternatively, you could pad and use the random access idea, removing indices/offsets from a list as they're picked.