使用RandomAccessFile通过Java中的多线程下载不正确的文件

Question

Hello My Respected Seniors :)

My Goal: Download a URL Resource, given a URL, by using Multi-Threading in Java, ie download a single file into multiple pieces (much like how IDM does) & at the end of download, combine all of them to 1 final file. For instance, if an Image is of 260KB, I want to download it into 2 threads for which, 1st Thread should download from 0KB to 130KB & 2nd Thread should download from 131KB to 260KB .

Technology Using: Java, RandomAccessFile, MultiThreading, InputStreams

Problem: This code works for 1 thread, ie If I run a single thread that starts from 0 to 260KB . But when I try to download it in chunks, following errors occur:

1. It will download extra garbage KBs, ie for a 260KB file, 300+KB will be downloaded.

   OR

2. Sometimes exact 260KB are downloaded, but file is corrupt, I can't open the Image.

Please help me. I've tried alot since the whole week & I can't seem to understand the problem.

INITIAL CODE

void InitiaeDownload
{


        HttpURLConnection uc = (HttpURLConnection) url.openConnection();
        uc.connect();
        long fileSize = uc.getContentLengthLong();
        System.out.println("File Size = "+ fileSize );
        uc.disconnect();

//------------------

        long chunkSize = (long) Math.ceil(fileSize/2);

        long startFrom = 0;
        long endRange = (startFrom + chunkSize) - 1;

        System.out.println("Chunk Size = " + chunkSize);
        System.out.println("Part 1 :: Start = " + startFrom + "\tEnd To = " + endRange);

        Thread t1 = new MyThread(url, file, startFrom, endRange);
        t1.start();

        startFrom += chunkSize;
        long temp = endRange + chunkSize;
        endRange = temp + (fileSize-temp);  //also add remaining bytes

        System.out.println("Part 2 :: Start = " + startFrom + "\tEnd To = " + endRange );

        Thread t2 = new MyThread(url, file, startFrom, endRange);
        t2.start();
}

& Now,

THREAD CLASS

class MyThread extends Thread {

    private URL url;
    private long startFrom;
    private long range;
    private InputStream inStream;
    private RandomAccessFile file;


    public MyThread(URL url, RandomAccessFile file, long startFrom,  long range)    //parameterized constructor
    {

        this.url = url;
        this.file = file;
        this.startFrom = startFrom;
        this.range = range;

    }

    public void run() {

        System.out.println("Thread Running..");

        Thread.currentThread().setPriority(MAX_PRIORITY);

        System.setProperty("http.proxyHost", "192.168.10.50");
        System.setProperty("http.proxyPort", "8080");   

        HttpURLConnection uc = null;
        try  {

            uc = (HttpURLConnection) url.openConnection();

            uc.setRequestProperty("Range", "bytes="+startFrom+"-"+range);
            uc.connect();

            inStream = uc.getInputStream();
            System.out.println("Starting Download");

            int bytesRead = 0;
            byte[] buffer = new byte[ (int) (range-startFrom) ];


            file.seek(startFrom);   //adjusted start of file

            while( (bytesRead = inStream.read(buffer) ) != -1 ) {

                file.write(buffer, 0, bytesRead);
            }

            System.err.println("Download Completed!");
            uc.disconnect();
        }
        catch(IOException e) {
            System.err.println("Exception in " + Thread.currentThread().getName() + "\t Exception = " + e );
        }           
    }  ///END OF run()  
} ////END OF MyThread Class

Answer 1

You are attempting to write to the same file object from multiple threads. When you seek() or write() it has only one view of where this is being performed.

If you want to write in different places to the same at once, each thread must have it's own RandomAccessFile object even if they all point to the same underlying file.

BTW: Changing the thread priority of an IO bound process is likely to do very little (and nothing unless you are an administrator)

Answer 2

First: use an ExecutorService instead of raw threads; you will have an easier time managing your work.

Second: why file size / 2 each time? Better (imho) would be to use a fixed chunk size.

Third: use a FileChannel and map your file in chunks (use FileChannel.open() to obtain the channel).

Four: use java.nio and try-with-resources since you use Java 7 ;)

Here is how you would write from offset 1000 (included) to offset 2000 (excluded) in a file; note that the file size is expanded as needed!

// "channel" is your FileChannel.
// Argument to the worker: the channel, the offset, the size to write

final ByteBuffer buf = channel.map(FileChannel.MapMode.READ_WRITE, 1000L, 1000L);

try (
    final InputStream in = uc.getInputStream();    
    final ReadableByteChannel inChannel = Channels.newChannel(in);
) {
    inChannel.read(buf); // returns the number of bytes read
    channel.force(false);
}

That is basically the content of one of your workers.

Answer 3

To seak withing a file, the file must be at least the length of you speak position. So you have to first create the file with the resulting size and then override the data based on you seek positions.