How to use OData to do pagination on azure table storage APIs?

Question

Well my requirement is to do client side paging. ie return a set of records based on the values($top, $skip) given by client. But based on my below code, am able to use only filter keyword and top or skip.

[HttpGet]        
public PageResult<PersistedUser> GetAllUsers(ODataQueryOptions options)
{
    TableServiceContext serviceContext = tableClient.GetDataServiceContext();
    serviceContext.IgnoreResourceNotFoundException = true;

    CloudTableQuery<PersistedUser> users = serviceContext
        .CreateQuery<PersistedUser>(TableNames.User)
        .AsTableServiceQuery();    

    IQueryable<PersistedUser> results = options
        .ApplyTo(users.AsQueryable()) as IQueryable<PersistedUser>;

    // manipulate results. Add some calculated variables to the collection etc

    return new PageResult<PersistedUser>(results, null, 0);
}

I am not really sure if this is the correct way to do it as well. But my basic requirement is that I have a huge db, but i just need to return a small set of entities at a time in an efficient time. I would really appreciate if someone could provide some code snippets.

Answer 1

I'm using the same way and it's working fine.

Few differences:

I have a service layer that expose my entites. In my services I return IQueryable and apply the O Data filter.

    [AuthKey]
    [GET("api/brands/")]
    public PageResult<BrandViewModel> GetBrands(ODataQueryOptions<Brand> options)
    {
        var brands = (IQueryable<Brand>)options.ApplyTo(_brandServices.FindBrands());

        return new PageResult<BrandViewModel>(BrandViewModel.ToViewModel(brands), Request.GetNextPageLink(), Request.GetInlineCount());
    }

Answer 2

Here's an updated version of your code that uses the generic version of ODataQueryOptions and applies the $top and $skip options for paging.

[HttpGet]        
public PageResult<PersistedUser> GetAllUsers(
    ODataQueryOptions<PersistedUser> options)
{
    TableServiceContext serviceContext = tableClient.GetDataServiceContext();
    serviceContext.IgnoreResourceNotFoundException = true;

    CloudTableQuery<PersistedUser> users = serviceContext
        .CreateQuery<PersistedUser>(TableNames.User)
        .AsTableServiceQuery();    

    IQueryable<PersistedUser> results = options.ApplyTo(users);

    int skip = options.Skip == null ? 0 : options.Skip.Value;
    int take = options.Top == null ? 25 : options.Top.Value;

    return new PageResult<PersistedUser>(
        results.Skip(skip).Take(take).ToList(), 
        Request.GetNextPageLink(),
        null);
}

Answer 3

When planning your OData model, separate it from the underlying storage model. In some domains it may be possible to exposed groups and then use navigation properties to access the members of the group.

For example, suppose you have a booking system. You may store your bookings in a long, long table by date-time.

But you could potentially expose the OData model by grouping into collections of year and week.

http://service.net/odata/year(2013)/week(22)/bookings

In your controller, you'd compose a Table Storage range query from the temporal parameters supplied.

If there were >1,000 bookings, but not a huge number more, you could page through them server-side, exhaust the set, and deliver all the bookings back to the OData client, or sort them and allow IQueryable over this set. See note, bottom.

This would afford the OData consumer a natural mechanism for filtering data while keeping result set sizes low. Where there are many bookings per week, it could be sub-grouped further by day of week and hour.

This is completely theoretical, but I think OData v4 and its containment feature would allow such URLs to be routed and the relationship described so as to produce correct metadata for OData consumers like Excel.

http://www.asp.net/web-api/overview/odata-support-in-aspnet-web-api/odata-v4/odata-containment-in-web-api-22

Note in the above sample code that they create an arbitrary route/URL for counting the contained items, so it seems flexible.

If nested containment is not allowed, then alternatively, consider a BookingRange entity in the OData EDM with Year and Week properties to allow:

http://service.net/odata/bookingrange(2013,22)/bookings

Another idea I've considered is calculating a page number on insert. That is, to have a PageNumber on the TS entity itself and use an algorithm to assign a page number to it. This is essentially the same as generating a good partition key but with many pages/partitions.

A table expecting to hold 200m rows could have 1m pages (generate a large psuedo random number and Mod it by 1m), each with 200 items. At first, most pages would be empty, so you'd need to write a page mapper algo that alters as the row count grows. "Page" 1 maps to page range 0000001 - 0000100, etc.

As you can see, this is getting complex, but its essentially the same system that Azure uses itself to auto-balance your data across partitions and spread those partitions across its nodes.

Ultimately, this would again need a way for the "Page" number to be specified in the URL. Finally, each page would contain varying numbers of items, depending on the distribution of the algorithm used.

Note - I think the reason that TS does not support top and skip, or skip, is that the order of the rows returned cannot be guaranteed and there is no ordering mechanism within TS (which would be a big burden). Therefore, pages collated from top and skip would contain a "random" bag each time.

This means that my suggestion above to offer paging over a subset/group of data, requires that the entire subset is brought into the service tier and a sort order applied to them, before applying the top and skip, though it could be argued that the client should understand that top/skip without an order is meaningless and the onus is on them to send the right options.