Slow paging with SQL query (stored procedure) in SQL Server

Question

I've got the following SQL statement that's generated in a dynamic stored procedure based on various parameters:

SELECT [MetadataId], [DocumentType], InvoiceNumber FROM 
( 
  SELECT [MetadataId], [DocumentType], InvoiceNumber, ROW_NUMBER() 
  OVER (ORDER BY [MetadataId] Asc) 
  AS [Row_ID] 
  FROM [Metadata] 
  WHERE ([DocumentType] = 'Invoice')
) Wrapper 
WHERE Row_ID BETWEEN 999980 AND 1000000

where the Row_ID changes based on the my grid's current page.

The above query works great when I navigate initially navigate from page 1 to page 2,3,4,5, and so forth but I can't say the same if I immediately navigate from page 1 to page 50,000 which is the last page in my test database which contains 1 million random randomly generated invoices where my page size is 20.

It takes around 29/30 seconds to load up and the RAM used by my SQL Server instance goes from around 400MB to 1.61GB.

Once the initial delay has elapsed, going to page 49999, 49998, 49997, etc... is instant or navigating back and forth between page 1 to 50000 is also instant.

I can only assume that the entire data set is somehow loaded in memory.

Additional Notes:

1. MetadataId is set as a primary key.
2. Other searchable columns such as DocumentType , InvoiceNumber , etc... are also indexed but not unique.
3. I need to keep using the dynamic stored procedure for various reasons but the main one being that while the field requirements change from client to client, the result being used by our app remains the same.
4. Using SQL Server 2014 Developers edition for my tests.

So my questions are:

1. Can someone explain to me what's actually happening? Is the all data being loaded in memory?

2. Is there a way to improve this? Note that I need to Row_ID generate by 'ROW_NUMBER() OVER' as the WHERE clause part of my SQL statement can change quite drastically depending on what parameter the user is searching under.

Thank you.

UPDATE-1

Here's the execution plan:

Answer 1

Dynamic sql is good,hope you are using sp_executesql.

Row_ID BETWEEN 1 AND 20, then 21 to 40 then 41 to 60, the results are instant

Because when first query is executed then plan is cache which is reuse by subsequent query between row_id BETWEEN 1 AND 20, then 21 to 40 then 41 to 60.

Row_ID BETWEEN 999981 AND 1000000, it takes 28/29 secs for to load

I guess Query optimiser create new plan for these range so it take longer to execute for first time then plan is cache.

next time row_id 999980 to 999960 execute faster because it reuse the plan.

I think it has Parameter Sniffing problem.

I think your query has scope to optimise but can't say without looking at it.

OFFSET / FETCH may improve your query as it will reduce one Select statement, not sure it depend entirely on main query.

That much plan is not enough.Clustered Index Scan is not always bad.

Answer 2

I believe there should be a non clustered composite index for the columns used in the query that way all table columns in the query will be sorted together and pointing to the metedata_Id pk; or a view created using a filtered index on the where clause of the query. Disk space consumptiion for storage is expected. I have a concern that there is a order by clause using row_id that is not part of the select query columns but a renamed column..thought there should have been a error.