将大型JavaScript数组发送到Web API

Question

I am struggling with this issue for 2 days...

I have a JavaScript array (20,000K rows and 41 columns). It was originally received in javaScript through an ajax call as shown below,

var dataArray = [];
var dataRequest = {};
            dataRequest.SearchCondition = 'some value'; 
            $.ajax({
                type: "POST",
                url: "api/GetData/ProcessRequest",              
                dataType: 'json',
                cache: false,
                async: true,
                crossDomain: false,
                data: dataRequest ,
                success: function (response) {
                  dataArray = response; 
                },
                error: function (xhr, textStatus, errorThrown) {
                   dataArray = null;
                }
            });

In the application, the user will verify the data and send it back to Web API method.

I am trying to send the same data back (dataArray) to web api method but, it fails. Please see the code below,

Option 1: (failed - the request did not hit web api method)

 var dataArrayJsonStr = JSON.stringify(dataArray);                     

    $.ajax({
        type: "POST",
        url: "api/SendData/ProcessRequest",
        dataType: 'json',
        data: {'dataValue':dataArrayJsonStr },
        success: function (response) {
            alert('success');
        },
        error: function (xhr, textStatus, errorThrown) {
            alert(errorThrown)
        }
    });

In IE 8, I am getting 'out of memory' exception popup. (most of our application users still have IE 8)

In Chrome, it crashes.

Option 2 tried: (don't know how to read the value)

I tried to send the same value to web api through XmllHttpRequest

var dataArrayJsonStr = JSON.stringify(dataArr);
            var xmlRequest;
            if (window.XMLHttpRequest) {
                xmlRequest = new XMLHttpRequest();                
            }
            xmlRequest.open("POST", "api/SendData/ProcessRequest", false);
            xmlRequest.setRequestHeader('Content-Type', 'application/text');
            xmlRequest.send("dataValue=" + dataArrayJsonStr);

Using Chrome, I am able to post the data successfully to Web API, I am seeing the content-length as '128180309'. But, I don't see the values. How do i get the values in Web API?

Please suggest me how to send large data back to web api from javascript.

Thanks, Vim

Answer 1

I think you create overhead, maybe I wrong, you can edit me. Did you really need send back all datas back or you just need send modified data? Because in real life hard to imagine that user will review 20.000 of rows.

Good example is ExtJS stores, you can see example here Key thing of stores that they send back to the server only modified or deleted data, it save browser, network and server resources.

Answer 2

Try to add more memory for API or more time excecution, also you can try return data in more small parts. Defining the number of parts to send.

Answer 3

Did you try to send the data by chunks? I mean, you need to split it in small pieces and perform multiple number of requests.

For example, it can be like:

--HELLO SERVER. STARTING TRANSMITION FOR DATA SET #177151--  
PIECE 1/13  
PIECE 2/13  
...  
PIECE 13/13  
--BUE SERVER-- 

So, it will take some time, but you can send any amounts of data without memory problems. If you're struggling with it for 2 days, I think you got some time to code it :)

UPD1 : Client code example.

Here's an example of client code. This is a simple chunking algorithm. Have to say I didn't test it, because it would take a lot of time to represent your situation.

So, you should read it and get the point. You have a simple function, that takes you whole data set and callbacks for each response (to update your progress bar, eg), for successful finish and for error.

Hope, it will help you to make some problems. Also, I can help you to build architecture on the server-side, but I need to know what technologies do you use.

function sendData(data, onEach, onFinish, onError) {
    var CHUNK_SIZE = 1000;
    var isFailed = false;
    var chunkNum = 0;
    var chunk, chunkStart, chunkEnd;

    while(data.length + CHUNK_SIZE > chunkNum * CHUNK_SIZE) {
        if(isFailed) {
            return;
        }
        chunkStart = chunkNum * CHUNK_SIZE;
        chunkEnd   = chunkStart + CHUNK_SIZE + 1;

        chunk = {
            num: chunkNum,
            data: data.slice(chunkStart, chunkEnd)
        };

        ajaxCall(chunk);
        chunkNum++;
    }

    function ajaxCall(data) {
        $.ajax({
            type: "POST",
            url: "api/GetData/ProcessRequest",              
            dataType: 'json',
            async: true,
            data: dataRequest ,
            success: function (response) {
                onEach(data, response);
            },
            error: function (xhr, textStatus, errorThrown) {
                isFailed = true;
                onError(arguments);
            }
        });
    }


}