Can EF Core return IQueryable from Stored Procedure / Views / Table Valued Function?

Question

We're in need of passing ODATA-V4 query search, order by clauses to Database directly.

Here is the case:

1. There are joins among tables and we invoke (inline) table valued functions using SQL to get desired records.
2. ODATA where clauses needs to be applied on the result-set, then we apply pagination Skip, Take and Order By.

We started with Dapper, however Dapper supports only IEnumerable, Thus Dapper would bring entire records from DB then only OData (Query Options ApplyTo) pagination will apply, spoiling the performance gain :-(

        [ODataRoute("LAOData")]
        [HttpGet]
        public IQueryable<LAC> GetLAOData(ODataQueryOptions<LAC> queryOptions)
        {
            using (IDbConnection connection = new SqlConnection(RoutingConstants.CascadeConnectionString))
            {
                var sql = "<giant sql query";
                IQueryable<LAC> iqLac = null;
                IEnumerable<LAC> sqlRes = connection.Query<LAC>(sql, commandTimeout: 300);
                **IQueryable<LAC> iq = sqlRes.AsQueryable();
                iqLac = queryOptions.ApplyTo(iq) as IQueryable<LAC>;
                return iqLac;**
            }
        }

Most of the example we see on Stored procedure, Views support apparently returns List. https://hackernoon.com/execute-a-stored-procedure-that-gets-data-from-multiple-tables-in-ef-core-1638a7f010c

Can we configure EF Core 2.2 to return IQueryable so that ODATA could further filter out and then yield only desired counts say 10.?

Answer 1

Well, yes and no. You can certainly return an IQueryable , and you're already doing so, it seems. And, you can certainly further query via LINQ on that IQueryble , in memory .

I think what you're really asking, is if you can further query at the database-level, such only the ultimate result set you're after is returned from the database. The answer to that is a hard no. The stored procedure must be evaluated first. Once you've done that, all the result have been returned from the database. You can further filter in memory, but it's already too late for the database.

That said, you should understand that OData is fundamentally incompatible with the idea of using something like a stored procedure. The entire point is to describe the query via URL parameters - the entire query . You could use a view instead, but stored procedures should not be used along-side OData.

Answer 2

EF cannot return IQueryable from a Stored Procedure because the database engine itself doesn't provide a mechanism for selectively querying or manipulating execution of the script, you can't for instance do the following in SQL:

SELECT Field1, Field2
EXEC dbo.SearchForData_SP()
WHERE Field2 IS NOT NULL
ORDER BY Field3

The Stored Procedure is a black box to the engine, and because of this there are certain types of expressions and operations that you can use in SPs that you cannot use in normal set based SQL queries or expressions. For instance you can execute other stored procedures. SPs must be executed in their entirety, before you can process the results.

If the database engine itself cannot do anything to optimise the execution of Stored Procedures, its going to be hard for your ORM framework to do so. This is why most documentation and examples around executing SPs via EF returns a List as that makes it clear that the entire contents of that list is in memory, casting that List to IQueryable with .AsQueryable() doesn't change the fact that the data is maintained within that List object.

1. There are joins among tables and we invoke (inline) table valued functions using SQL to get desired records.

What you are describing here is similar to what OData and EF try to offer you, mechanisms for composing complex queries. To take full advantage of OData and EF your should consider replicating or replacing your TVFs with linq statements. EF is RDBMS agnostic so it tries to use and enforce generic standards that can be applied to many database engines, not just SQLSERVER. When it comes to CTEs, TVFs and SPs the implementation and syntax in each database engine becomes a lot more specific even to specific versions in some cases. Rather than trying to be everything to everyone, the EF team has to enforce some limits so they can maintain quality of the services they have offered for us.

There is a happy medium that can be achieved however, where you can leverage the power of the two engines:

1. Design your SPs so that the filtering variables are passed through as parameters and restrict dependence on Stored Procedures to scenarios where the structure of the output is as efficient as you would normally need. You can then expose the SP as an Action endpoint in OData and the called can pass the parameter values directly through to the SP.

   • You can still wrap the response in an IQueryable<T> and decorate this action with the EnableQuery attribute, this will perform in memory $select, $expand and simple $filter operations but the service will still load the entire recordset into memory before constructing the response payload. This mechanism can still reduce bandwidth between the server and the client, just not between the database and the service layer.
   • Make different versions of your SP if you need to have different result structures for different use cases.

2. Use TVFs or Views only when the query is too complex to express easily using linq or you need to use Table Hints , CTEs , Recursive CTEs or Window Functions that cannot be easily replicated in Linq.

   • In many cases where CTEs (non recursive) are used, the expression can be easier to construct in Linq.

   • To squeeze the most performance from indexes you can use Table Hints in SQL, because we don't have tight control over how our Linq expressions will be composed into SQL it can take a lot of work to construct some queries in a way that the database can optimise them for us. In many scenarios, as with CTEs above, going through the process of rewriting your query in Linq can help avoid scenarios where you would traditionally have used table hints.

     • There are limits, when you want or need to take control, using specialised SQL Server concepts that EF doesn't support, you are making a conscious decision to have one and not the other.

I do not agree that OData and Stored Procedures are fundamentally incompatible there are many use cases where the two get along really well, you have to find the balance though. If you feel the need to pass through query options such as $select, $expand $filter, $top, $skip... to your Stored Procedure, either change your implementation to be constructed purely in Linq (so no SP) or change the client implementation so that you pass through formal parameters that can be handled directly in the SP.