Azure SQL stored procedure ridiculously slow called from C#

Question

Summary :

We have two identical databases, one on a local server, one on Azure.

We have a C# system that accesses these databases, calling stored procedures.

The stored procedures are running very, very slowly when called from the C# system to the Azure database. They're running fine from C# to the local server, and from SSMS to both the Azure and the local databases.

As an example, calling the stored procedure 'usp_DevelopmentSearch_Select'

Local database, SSMS: 1 second

Local database, C#: 1 second

Azure database, SSMS: 1 second

Azure database, C#: 17 minutes

This is happening on multiple stored procedures, I'm just using usp_DevelopmentSearch_Select as an example, to test solutions and to trace the execution plan.

I've ruled out ARITHABORT (the usual suspect), and it seems that running usp_DevelopmentSearch_Select in SSMS and from the C# system generate a functionally identical execution plan.

Details :

We write a very large C# system, which accesses SQL Server databases.

Currently all our clients host their own databases locally on their own servers, however we are looking into the option of hosting the databases on Azure. So I set up some small Azure test databases, ironed out the kinks, and got an Azure-hosted system going.

Then I copied one of our client's databases up, to compare performance hosted locally vs hosted on Azure.

The actual client database is performing unusably badly on Azure!

The first screen calls a stored procedure 'usp_DevelopmentSearch_Select'

Connection to the database on their server:-

In SSMS, calling the stored procedure (below) returns the values in about 1 second

EXEC usp_DevelopmentSearch_Select @MaxRecord = 100, @SearchType = 'CUR'

In our C# program, calling the stored procedure returns the values in about 1 second

Connection to the database on Azure:-

In SSMS, calling the stored procedure returns the values in about 1 second

In our C# program, calling the stored procedure returns the values in about 17 minutes !

Fast in SSMS and slow from C# usually means ARITHABORT, so I turned it on at the start of the stored procedure:

SET ARITHABORT ON; 

That didn't make any difference, so I updated it to convert the passed parameters to local variables.

ALTER PROCEDURE [dbo].[usp_DevelopmentSearch_Select]
     (@MAXRECORD INT,
      @SEARCHTYPE VARCHAR(3))
AS
BEGIN
    SET ARITHABORT ON; 

    DECLARE @MAXRECORD_Var INT = @MAXRECORD
    DECLARE @SEARCHTYPE_Var VARCHAR(3) = @SEARCHTYPE

    ... (Updated all references to @MAXRECORD and @SEARCHTYPE to @MAXRECORD_Var and @SEARCHTYPE_Var)

END

Still no joy, so I got the Execution Plan details for both:-

select o.object_id, s.plan_handle, h.query_plan 
from sys.objects o 
inner join sys.dm_exec_procedure_stats s on o.object_id = s.object_id
cross apply sys.dm_exec_query_plan(s.plan_handle) h
where o.object_id = object_id('usp_DevelopmentSearch_Select')

And just to check, I reloaded the screen in the C# program, and checked the running query:-

SELECT sqltext.TEXT,
req.session_id,
req.status,
req.command,
req.cpu_time,
req.total_elapsed_time,
req.plan_handle
FROM sys.dm_exec_requests req
CROSS APPLY sys.dm_exec_sql_text(sql_handle) AS sqltext

It is definitely using one of the two execution plans returned above.

So, check the settings for the Execution Plans

SELECT * FROM sys.dm_exec_plan_attributes (0x05002D00D1A1EA5510E66E783602000001);
SELECT * FROM sys.dm_exec_plan_attributes (0x05002D00D1A1EA55E0FC6E783602000001);

Set_Options is 4345 for both, so they're definitely both using ARITHABORT.

The only differences are the localisation bits: Language and Date Format. The Azure database is stuck in American, can't seem to change that, while the C# program forces it to British.

I tried the C# program without forcing it to British, and still got the same issue. It also used exactly the same Execution Plan, so clearly localisation doesn't affect that.

So, I called up the info on the Execution Plans:-

SELECT * FROM sys.dm_exec_query_plan (0x05002D00D1A1EA5510E66E783602000001);
SELECT * FROM sys.dm_exec_query_plan (0x05002D00D1A1EA55E0FC6E783602000001);

Saved them both, and compared the results:-

The two columns far left show the overall comparison: yellow being different, white being the same. As you can see, the two Execution Plans are almost identical, just with a handful of differences at the top.

The first differences can be seen in the above screenshot: the 'StatementCompId' is one higher in the SSMS (left) pane than the C# (right) pane. Google doesn't want to tell me what StatementCompId is, but given they're in sequence I'm guessing it's the order to do them in, and the SSMS is one higher because the EXEC command that called the SP counts as one.

For ease, I've compiled all the remaining differences into a single screenshot:-

Compile times and CPU usages, free memory, and a couple more 'StatementCompId'

So, the two Execution Plans are functionally identical, with identical settings (except localisation which doesn't seem to have an effect).

So why does it take around 17 minutes calling the Azure SP from C# compared to around 1 second calling the Azure SP from SSMS or the local SP from the locally-hosted database either way?

The Stored Procedure itself is just a SELECT FROM, with a few LEFT JOINs to other tables, nothing fancy and it's never given us any trouble on locally-hosted databases.

SELECT TOP (@MAXRECORD_Var) <FieldList>
FROM (
    SELECT DISTINCT <FieldList>
    FROM <TableName> WITH (NOLOCK)
    LEFT JOIN <TableName> WITH (NOLOCK) ON <Link>
    LEFT JOIN <TableName> WITH (NOLOCK) ON <Link>
    LEFT JOIN <TableName> WITH (NOLOCK) ON <Link>
    LEFT JOIN <TableName> WITH (NOLOCK) ON <Link>
    LEFT JOIN <TableName> WITH (NOLOCK) ON <Link>
    WHERE (
        <Conditions>
    ) AS Base
ORDER BY <FieldName>

Edit: Some Progress

I tried several things that came up from Googling:-

1) WITH RECOMPILE

I tried adding this to the Stored Procedure, didn't make any difference

2) OPTION (OPTIMIZE FOR (@MAXRECORD_Var UNKNOWN, @SEARCHTYPE_Var UNKNOWN))

I tried adding this to the Stored Procedure, didn't make any difference

3) Explicitly setting all options

This one made a noticeable (but still far too small) difference!

I wrote a query to tell me the current options

DECLARE @options INT
SELECT @options = @@OPTIONS
PRINT @options
PRINT 'SET DISABLE_DEF_CNST_CHK ' + CASE WHEN ( (1 & @options) = 1 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET IMPLICIT_TRANSACTIONS ' + CASE WHEN ( (2 & @options) = 2 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET CURSOR_CLOSE_ON_COMMIT ' + CASE WHEN ( (4 & @options) = 4 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ANSI_WARNINGS ' + CASE WHEN ( (8 & @options) = 8 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ANSI_PADDING ' + CASE WHEN ( (16 & @options) = 16 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ANSI_NULLS ' + CASE WHEN ( (32 & @options) = 32 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ARITHABORT ' + CASE WHEN ( (64 & @options) = 64 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ARITHIGNORE ' + CASE WHEN ( (128 & @options) = 128 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET QUOTED_IDENTIFIER ' + CASE WHEN ( (256 & @options) = 256 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET NOCOUNT ' + CASE WHEN ( (512 & @options) = 512 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ANSI_NULL_DFLT_ON ' + CASE WHEN ( (1024 & @options) = 1024 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ANSI_NULL_DFLT_OFF ' + CASE WHEN ( (2048 & @options) = 2048 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET CONCAT_NULL_YIELDS_NULL ' + CASE WHEN ( (4096 & @options) = 4096 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET NUMERIC_ROUNDABORT ' + CASE WHEN ( (8192 & @options) = 8192 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET XACT_ABORT ' + CASE WHEN ( (16384 & @options) = 16384 ) THEN 'ON' ELSE 'OFF' END + ';'

This produced a set of SET statements, and the current Options value

5496
SET DISABLE_DEF_CNST_CHK OFF;
SET IMPLICIT_TRANSACTIONS OFF;
SET CURSOR_CLOSE_ON_COMMIT OFF;
SET ANSI_WARNINGS ON;
SET ANSI_PADDING ON;
SET ANSI_NULLS ON;
SET ARITHABORT ON;
SET ARITHIGNORE OFF;
SET QUOTED_IDENTIFIER ON;
SET NOCOUNT OFF;
SET ANSI_NULL_DFLT_ON ON;
SET ANSI_NULL_DFLT_OFF OFF;
SET CONCAT_NULL_YIELDS_NULL ON;
SET NUMERIC_ROUNDABORT OFF;
SET XACT_ABORT OFF;

Note : Running SET DISABLE_DEF_CNST_CHK OFF; throws an error, so I commented that one out.

'DISABLE_DEF_CNST_CHK' is not a recognized SET option.

Adding this to the start of the Stored Procedure brought the time down from 17 minutes to 40 seconds .

Still far more than 1 second it takes to run in SSMS, and still not enough to be usable, but progress none the less.

However, I noticed that the Options value it returned ( 5496 ) was different to the value I got from the Execution Plan details above ( 4345 ), and also some of the settings where different from the settings for that database.

So, I re-ran the query hard-coded to 4345

DECLARE @options INT
SELECT @options = 4345 --@@OPTIONS
PRINT @options
PRINT 'SET DISABLE_DEF_CNST_CHK ' + CASE WHEN ( (1 & @options) = 1 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET IMPLICIT_TRANSACTIONS ' + CASE WHEN ( (2 & @options) = 2 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET CURSOR_CLOSE_ON_COMMIT ' + CASE WHEN ( (4 & @options) = 4 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ANSI_WARNINGS ' + CASE WHEN ( (8 & @options) = 8 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ANSI_PADDING ' + CASE WHEN ( (16 & @options) = 16 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ANSI_NULLS ' + CASE WHEN ( (32 & @options) = 32 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ARITHABORT ' + CASE WHEN ( (64 & @options) = 64 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ARITHIGNORE ' + CASE WHEN ( (128 & @options) = 128 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET QUOTED_IDENTIFIER ' + CASE WHEN ( (256 & @options) = 256 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET NOCOUNT ' + CASE WHEN ( (512 & @options) = 512 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ANSI_NULL_DFLT_ON ' + CASE WHEN ( (1024 & @options) = 1024 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET ANSI_NULL_DFLT_OFF ' + CASE WHEN ( (2048 & @options) = 2048 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET CONCAT_NULL_YIELDS_NULL ' + CASE WHEN ( (4096 & @options) = 4096 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET NUMERIC_ROUNDABORT ' + CASE WHEN ( (8192 & @options) = 8192 ) THEN 'ON' ELSE 'OFF' END + ';'
PRINT 'SET XACT_ABORT ' + CASE WHEN ( (16384 & @options) = 16384 ) THEN 'ON' ELSE 'OFF' END + ';'

This returned

4345
SET DISABLE_DEF_CNST_CHK ON;
SET IMPLICIT_TRANSACTIONS OFF;
SET CURSOR_CLOSE_ON_COMMIT OFF;
SET ANSI_WARNINGS ON;
SET ANSI_PADDING ON;
SET ANSI_NULLS ON;
SET ARITHABORT ON;
SET ARITHIGNORE ON;
SET QUOTED_IDENTIFIER OFF;
SET NOCOUNT OFF;
SET ANSI_NULL_DFLT_ON OFF;
SET ANSI_NULL_DFLT_OFF OFF;
SET CONCAT_NULL_YIELDS_NULL ON;
SET NUMERIC_ROUNDABORT OFF;
SET XACT_ABORT OFF;

Again, the line SET DISABLE_DEF_CNST_CHK ON; says it's not an option you can set, so I commented it out.

Updated the Stored Procedure with those SET values, and tried again.

It still takes 40 seconds, so no further progress.

Running it in SSMS still takes 1 second, so at least it didn't break that, not that it's any help but nice to know!

Edit #2 : Or not...

Seems yesterday's apparent progress was a blip: it's back to taking 17 minutes again! (With nothing changed)

Tried combining all three options: WITH RECOMPILE, OPTION OPTIMIZE and explicitly setting the SET OPTIONS. Still takes 17 minutes.

Edit 3 : Parameter Sniffing Setting

In SQL Azure, you can turn off Parameter Sniffing from the database options screen.

And check them using

SELECT * FROM sys.database_scoped_configurations

Tried SSMS and C# twice each after setting this to OFF.

As before, SSMS takes 1 second, C# still takes 15+ minutes.

Of course, given C# forces a load of parameters to a specific state when it connects, it's entirely possible that it's overriding it.

So, just to say I tried it, I added turning it off to the Stored Procedure

ALTER DATABASE SCOPED CONFIGURATION SET PARAMETER_SNIFFING = OFF;

Still 15+ minutes.

Ah well, was worth a try!

Plus, lots of new parameters to look up and test.

Edit #4 : Azure Staging Pool Configurations and Automatic Tuning

I tried out several different configurations on the Staging Pool, to see if that made a difference. I didn't try the worst query, as it was costing us money to up the eDTUs, but I tried several others, twice each (going down the list each time, so not the same one twice straight away).

Going from 50 eDTUs to 100 eDTUs made a bit of a difference, so I guess on our Test Elastic Pool we use all of the 50, but after that it didn't make any difference. Oddly, the Premium gave worse performance than Standard in places.

I then posted this on the Azure MSDN site (when they finally got round to Verifying my account), and they suggested going through all the Performance options on the Azure Portal and see if that recommends anything.

It suggested a couple of indexes, which I enabled, but that was all.

Then I flipped the Automatic Tuning over from 'Server' to 'Azure Defaults'

I re-ran most of the same timing tests, just to see what difference it had made.

The query that had been taking 17 minutes now generally took 13 seconds, a massive improvement! Yay!

The rest were a mixed bag. C was generally quicker, most still took around the same time, and E now takes nearly twice as long (26s up from 14s).

There results also seemed to have a lot more variance than they did before, although it's possible that changing the eDTU size resets the tunings. The second run was usually better than the first, often noticeably so.

Still all a lot slower than running the same system against a database on a local server, but a huge improvement for the slowest Stored Procedure at least.

Answer 1

Have you ruled out environmental issues? Check your allocation of memory, disk space, cpu, and network bandwidth for your azure server. SQL is a hog for both memory and disk space.

But you probably already checked all that. My next guess would be the connection string. You'll definitely need to use different syntax in the connection string for connecting to an Azure Sql instance as opposed to a local/network hosted one. See connectionstrings.com for details on the azure specific connection string syntax. (Remember that Azure Sql Server is different software than regular Sql Server. They try to look the same and pretend to behave the same, but they really are different under the hood.) It's possible that something like authentication or encryption is the bottleneck. Especially authentication is likely to be different between your SSMS connection and your code connection string.

Answer 2

When SQL runs in 1 second in SSMS and 17 minutes in C#, check your connection settings.

The default SSMS connection settings are:

ANSI_NULLS

ANSI_NULL_DFLT_ON

ANSI_PADDING

ANSI_WARNINGS

CONCAT_NULL_YIELDS_NULL

QUOTED_IDENTIFIER

And all sample code on Microsoft websites are testing with these settings

Code Example Conventions Unless stated otherwise, the examples provided in the Transact-SQL Reference were tested by using SQL Server Management Studio and its default settings for the following options: https://learn.microsoft.com/en-us/sql/t-sql/language-elements/transact-sql-syntax-conventions-transact-sql?view=sql-server-ver15

Here are samples of fixing issues using the connection settings

SQL Email ERROR: transaction not longer valid

https://learn.microsoft.com/en-us/troubleshoot/sql/admin/database-mail-fails-send-message

FIX: SET NUMERIC_ROUNDABORT OFF

SET ARITHABORT must be ON when you're creating or changing indexes on computed columns or indexed views. If SET ARITHABORT is OFF, CREATE, UPDATE, INSERT, and DELETE statements on tables with indexes on computed columns or indexed views fail.

https://learn.microsoft.com/en-us/sql/t-sql/statements/set-arithabort-transact-sql?view=sql-server-ver15

Answer 3

When you call your SP from C# you should include the name of the database:

[YourDatabaseName].[dbo].[usp_DevelopmentSearch_Select]

In SSMS , your database will most likely be active. Hence, the server will know which database you are querying. When running on your local server, you most likely have very few databases (perhaps only one?). Hence, your local server will know which database you are querying.

But, in Azure you may very well have multiple databases, so it might need to scan through multiple databases. That will explain the delay you are seeing.