如何在服务器端拆分JSON，然后在客户端加入并流式传输

Question

Currently i have a big JSON file download link with the size of 1GB. What i can do is download it using android app then parse contents by json streaming through gson.

But i noticed a problem when users try to download such big file worth 1GB.

1. What if the user's internet broke down in the middle of download?
2. What if the user's device suddenly ran out of space?

Based on the above scenarios, the download should start again at the beginning.

To fix this, I thought of a possible solution which is breaking the JSON into parts. So the app can check if a part of is already downloaded and it just simple means the app is resuming it's download from the last downloaded part thus saving data and time.

My question is, how can i split the big json file? it's just a very big json file nothing special. I'm using PHP to generate the JSON file from mysql with the codes:

$fp = fopen($target_dir.'/json.json', 'w');
 fwrite($fp,json_encode(array('reading'=>$r,'book'=>$b,'article'=>$a,'month'=>$m, 'status'=>$status)));
        fclose($fp);

Then producing the link via:

$file = $target_dir.'/json.json';

    if (file_exists($file)) {
        header('Content-Description: File Transfer');
        header('Content-Type: application/octet-stream');
        header('Content-Disposition: attachment; filename="'.basename($file).'"');
        header('Expires: 540');
        header('Cache-Control: must-revalidate');
        header('Pragma: public');
        header('Content-Length: ' . filesize($file));
        readfile($file);
        exit;
    }

Furthermore, if i'm able to split the JSON file and create download links for the app to use, how would i join them via android app? If joining is not necessary, can i parse them from chunks?

I don't know if it's the best approach for this kind of problem, but i noticed 3D mobile games like real racing and GTA san andreas download big files or maybe a bunch of medium sized files that take a while to finish.

Answer 1

One way to create chunk of files is using fread() PHP function. Using fread we can specify number of bytes we want to read from a file. So what you can do is create chunks of files after/while creating file from database.

This is how you can create chunks.

$fp = fopen("jsonfile.txt",'r');
$i=1;
while(!feof($fp)) {
    $json_contents = fread($fp,4000);
    file_put_contents('json_chunk_'.$i.'.txt',$json_contents);
    $i++;
}

Now in android app keep a counter of which chunk was downloaded last and merge all the chunks at the end.

This might solve the problem if user's internet connection went down during downloading of data.

Answer 2

That depends on data. If data is like rows in DB with ID, client can download IDs one by one, so you won't need a chunk of data, just a row. When client loses connection on one row they will retry that one until there's no more data.