How to Use Dropbox Upload Session For Files Larger than 150mb?

Question

I want to upload a file larger than 150mb.

In the Dropbox API V2 docs, it says you should start an upload session .

The docs say you can't send a POST with more than 150mb of data, but I'm unsure how to achieve that with the upload_session API.

Answer 1

While any individual request shouldn't be larger than 150 MB (and typically you should use a significantly smaller chunk size), you can upload files larger than that by using multiple requests.

There's an example of using upload sessions below. That example uses the Python SDK, but the JavaScript SDK, but it should serve as a useful reference, as the logic is the same. (They both use the same underlying API.)

This uses the Dropbox Python SDK to upload a file to the Dropbox API from the local file as specified by file_path to the remote path as specified by dest_path . It also chooses whether or not to use an upload session based on the size of the file:

f = open(file_path)
file_size = os.path.getsize(file_path)

CHUNK_SIZE = 4 * 1024 * 1024

if file_size <= CHUNK_SIZE:

    print dbx.files_upload(f.read(), dest_path)

else:

    upload_session_start_result = dbx.files_upload_session_start(f.read(CHUNK_SIZE))
    cursor = dropbox.files.UploadSessionCursor(session_id=upload_session_start_result.session_id,
                                               offset=f.tell())
    commit = dropbox.files.CommitInfo(path=dest_path)

    while f.tell() < file_size:
        if ((file_size - f.tell()) <= CHUNK_SIZE):
            print dbx.files_upload_session_finish(f.read(CHUNK_SIZE),
                                            cursor,
                                            commit)
        else:
            dbx.files_upload_session_append(f.read(CHUNK_SIZE),
                                            cursor.session_id,
                                            cursor.offset)
            cursor.offset = f.tell()

f.close()

Answer 2

You can quickly upload file chunks using files/upload_session/start , files/upload_session/append_v2 and files/upload_session/finish API endpoints. Here is an example which uses my tiny dropbox v2 api wrapper ( dropbox-v2-api ):

const CHUNK_LENGTH = 100;
//create read streams, which generates set of 100 (CHUNK_LENGTH) characters of values: 1 and 2
const firstUploadChunkStream = () => utils.createMockedReadStream('1', CHUNK_LENGTH); 
const secondUploadChunkStream = () => utils.createMockedReadStream('2', CHUNK_LENGTH);

sessionStart((sessionId) => {
    sessionAppend(sessionId, () => {
        sessionFinish(sessionId);
    });
});

function sessionStart(cb) {
    dropbox({
        resource: 'files/upload_session/start',
        parameters: {
            close: false
        },
        readStream: firstUploadChunkStream()
    }, (err, response) => {
        if (err) { return console.log('sessionStart error: ', err) }
        console.log('sessionStart response:', response);
        cb(response.session_id);
    });
}


function sessionAppend(sessionId, cb) {
    dropbox({
        resource: 'files/upload_session/append_v2',
        parameters: {
            cursor: {
                session_id: sessionId,
                offset: CHUNK_LENGTH
            },
            close: false,
        },
        readStream: secondUploadChunkStream()
    }, (err, response) => {
        if(err){ return console.log('sessionAppend error: ', err) }
        console.log('sessionAppend response:', response);
        cb();
    });
}

function sessionFinish(sessionId) {
    dropbox({
        resource: 'files/upload_session/finish',
        parameters: {
            cursor: {
                session_id: sessionId,
                offset: CHUNK_LENGTH * 2
            },
            commit: {
                path: "/result.txt",
                mode: "add",
                autorename: true,
                mute: false
            }
        }
    }, (err, response) => {
        if (err) { return console.log('sessionFinish error: ', err) }
        console.log('sessionFinish response:', response);
    });
}

Answer 3

I have an example!

testFile1Data = "test file data 1";
dbx.filesUploadSessionStart({
  contents: testFile1Data,
  close: true,
})
.then(function (response) {
  file1Start = response;
})
.catch(function (err) {
  console.log(err);
});

testFile2Data = "test file data 2";
dbx.filesUploadSessionStart({
  contents: testFile2Data,
  close: true,
})
.then(function (response) {
  file2Start = response;
})
.catch(function (err) {
  console.log(err);
});

dbx.filesUploadSessionFinishBatch({entries: [
    {cursor: {session_id: file1Start.session_id, offset: testFile1Data.length}, commit: {path: "/testFile1.txt"}},
    {cursor: {session_id: file2Start.session_id, offset: testFile2Data.length}, commit: {path: "/testFile2.txt"}},
  ]})
.then(function (response) {
  finishBatch = response;
})
.catch(function (err) {
  console.log(err);
});

dbx.filesUploadSessionFinishBatchCheck({async_job_id: finishBatch.async_job_id})
.then(function (response) {
  finishBatch = response
})
.catch(function (err) {
  console.log(err);
});

I got the example from an issue thread on github - https://github.com/dropbox/dropbox-sdk-js/issues/80#issuecomment-283189888