How to handle large amount of data in webservice call?

Question

I am calling a drupal RESET webservice call in c# and this webservice return large amount of data (more then 30 mb).I am taking this data as a XML format but its not handling that much amount of data . If i limit my query for limited amount of rows then my code works fine. I would just like to know how to handle large amount of data which i received through webservice call in c#.

Answer 1

It's too wide question. I mean there is a lot of possible places where to solve this problem. What is a part which you want to solve?

In common, you can do following things:

1. You zip compression on http layer. This can reduce throughput of your solution.
2. You can choose another serialization, eg json or eg protobuf, kryo
3. You can provide paging. You probably already did it. This solution provides reasonable page sizes for all possible queries.
4. You can provide streaming processing of incoming data. I don't think that REST is designed to provide large xml output (tens of mbs) but you can process the data in stream approach on the client. This can reduce memory requirements of your solution.