Why REST API (Odata) update is so heavy

Question

really want to hear your opinion about data upload using Odata

I access REST API (Odata) that provides data about 350k rows (only 7 attributes). I am using Power BI to download this data. So when I am downloading the whole day set it downloads 67mb!

Can someone explain to me why file with just 350k rows is so heavy? ps when I am saving it in CSV it is just 3.7mb.

Answer 1

Lets do the math.

67MB / 350K rows = 191 bytes per row.

Each row will likely have at least a { , a } and a , , so that's 188 bytes per row.

188 / 7 = around 27 bytes per attribute. Each attribute probably also has at least 2 quotes ( " ), or 4 if they are all strings, and a : and , .

So that leaves around 22 bytes for a property name and value. That is also only true if you don't format your JSON with whitespace. If you use whitespace for a prettier output a significant portion is going to be that whitespace.

22 bytes is not a ton of space for both of these. I assume that your rows will at least have a bunch of strings in there. This paragraph for example is 162 bytes.

CSV is going to be shorter because column names only appear once, and often there's only a comma between fields. That's a significant drop in per-row overhead.

If you need this to be smaller, the first thing you should do is gzip your response. Compression is going to be very effective if there's repetition.

Or your API can just return CSV. There's no rule that says that everything has to be JSON. Maybe in OData there is, but it's fine to make exceptions for outliers.

3.7 MB for your CSV seems too low, unless most of fields are empty. If you have 7 fields per row, then there will be at least 6 commas and a "\r\n" line ending, so that only leaves 3 bytes for all your data. Perhaps it's 37MB?