AGOL hosted feature service returns more features than maxRecordCount

Question

I'm doing some experiments with ArcGIS Online hosted feature services in R using the sf package.

I cannot figure out why the response of this request contains more than 60k features: https://services.arcgis.com/nSZVuSZjHpEZZbRo/ArcGIS/rest/services/Rijksmonumenten/FeatureServer/0/query?where=1%3D1&outFields=%2A&returnCountOnly=false&f=geojson

According to the service parameters maxRecordCount is 10000: https://services.arcgis.com/nSZVuSZjHpEZZbRo/ArcGIS/rest/services/Rijksmonumenten/FeatureServer/0?f=json

I thought I had to use pagination, but that does not seem to be the case. Thus I'm supprised. Can somebody please explain.

This is a link to my gist containing the R code: https://gist.github.com/FrieseWoudloper/47f4dcf68be39bff9dfaf2cd57610db3

Regards, Willy

Answer 1

Using the wireshark network packet inspector, and changing the protocol scheme to http from https , you can see that st_read is doing the pagination for you, and doing 7 requests to the server to get all the data. Each request has a different request offset and then the resulting sets of 10,000 records plus the last few are concatenated.

The GeoJSON driver of OGR is doing this so sf gets it for free. I suspect its controlled by the FEATURE_SERVER_PAGING option documented here:

https://www.gdal.org/drv_geojson.html#

(Note I switched to http since its clearer to see what's going on with an unencrypted channel, the same thing happens with https)