EF Core - 在 PK 上没有标识的情况下在 DB 中按顺序插入记录

Question

Disclaimer: This question is regarding "modernization" of almost 17 years old system. ORM that was used 17 years ago required that PK's don't use Identity . I know how bad it is, and I can't change anything regarding this.

So I have (simplified) following table in database:

CREATE TABLE [dbo].[KlantDocument](
    [OID] [bigint] NOT NULL,
    [Datum] [datetime] NOT NULL,
    [Naam] [varchar](150) NOT NULL,
    [Uniek] [varchar](50) NOT NULL,
    [DocumentSoort] [varchar](50) NULL,
 CONSTRAINT [PK_KlantDocument] PRIMARY KEY CLUSTERED 
(
    [OID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90, OPTIMIZE_FOR_SEQUENTIAL_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[KlantDocument] CHECK CONSTRAINT [FK_KlantDocument_Klant]
GO

As you can see, table doesn't have Identity set on PK, so it has to be inserted manually.

Project is being rebuilt in Web Api .Net Core 5 , that is going to do all of the CRUD operations. It is using EF Core as ORM, and it is agreed that Unit of Work pattern is going to be used here ( please do keep reading, UoW is not issue here ).

For those curious or for what it's worth, you can take a look at it here ( https://pastebin.com/58bSDkUZ ) ( this is by no means full UoW, just partially and without comments).

UPDATE: Caller Controller Action:

[HttpPost]
        [Route("UploadClientDocuments")]
        public async Task<IActionResult> UploadClientDocuments([FromForm] ClientDocumentViewModel model)
        {
            if (!ModelState.IsValid)
                return BadRequest(ModelStateExtensions.GetErrorMessage(ModelState));

            var dto = new ClientDocumentUploadDto
            {
                DocumentTypeId = model.DocumentTypeId,
                ClientId = model.ClientId,
                FileData = await model.File.GetBytes(),
                FileName = model.File.FileName, // I need this property as well, since I have only byte array in my "services" project
            };

            var result = await _documentsService.AddClientDocument(dto);
            return result ? Ok() : StatusCode(500);
        } 

When I am inserting record I am doing it like so:

// Called by POST method multiple times at once
public async Task<bool> AddClientDocument(ClientDocumentUploadDto dto)
{
    try
    {
        var doc = new KlantDocument
        {
        /* All of the logic below to fetch next id will be executed before any of the saves happen, thus we get "Duplicate key" exception. */
            // 1st try to fetch next id
            // Oid = _uow.Query<KlantDocument>().OrderByDescending(s => s.Oid).First().Oid + 1,
            // 2nd try to fetch next id
            // Oid = _uow.Query<KlantDocument>().OrderBy(s => s.Oid).Last().Oid + 1,
            // 3rd try to fetch next id
            // Oid = await _uow.Query<KlantDocument>().OrderBy(s => s.Oid).AsNoTracking().Select(s => s.Oid).LastAsync() + 1,
            // 4th try to fetch next id
            // Oid = _uow.Query<KlantDocument>().OrderBy(s => s.Oid).AsNoTracking().Select(s => s.Oid).Last() + 1,
            // 5th try to fetch next id
            // Oid = (_uow.Query<KlantDocument>().OrderBy(s => s.Oid).AsNoTracking().Max(s => s.Oid) + 1),
            Naam = dto.FileName,
            DocumentSoort = dto.DocumentTypeId,
            Datum = DateTime.Now,
            Uniek = Guid.NewGuid() + "." + dto.FileName.GetExtension()
        };

        _uow.Context.Set<KlantDocument>().Add(doc); // Does not work
        _uow.Commit();
    }
    catch (Exception e)
    {
        _logger.Error(e);
        return false;
    }
}

I get "Duplicate key" exception because 2 of the records are overlapping when inserting.

I have tried to wrap it into the transaction like so:

_uow.ExecuteInTransaction(() => { 
        var doc = new KlantDocument
        {
        /* All of the logic below to fetch next id will be executed before any of the saves happen, thus we get "Duplicate key" exception. */
            // 1st try to fetch next id
            // Oid = _uow.Query<KlantDocument>().OrderByDescending(s => s.Oid).First().Oid + 1,
            // 2nd try to fetch next id
            // Oid = _uow.Query<KlantDocument>().OrderBy(s => s.Oid).Last().Oid + 1,
            // 3rd try to fetch next id
            // Oid = await _uow.Query<KlantDocument>().OrderBy(s => s.Oid).AsNoTracking().Select(s => s.Oid).LastAsync() + 1,
            // 4th try to fetch next id
            // Oid = _uow.Query<KlantDocument>().OrderBy(s => s.Oid).AsNoTracking().Select(s => s.Oid).Last() + 1,
            // 5th try to fetch next id
            // Oid = (_uow.Query<KlantDocument>().OrderBy(s => s.Oid).AsNoTracking().Max(s => s.Oid) + 1),
            Naam = dto.FileName,
            DocumentSoort = dto.DocumentTypeId,
            Datum = DateTime.Now,
            Uniek = Guid.NewGuid() + "." + dto.FileName.GetExtension()
        };

        _uow.Context.Set<KlantDocument>().Add(doc); // Does not work
        _uow.Commit();
});

and it does not work. I still get "Duplicate key" exception.

From what I know, shouldn't EF by default lock database until transaction is complete?

I tried to manually write insert SQL like so:

using (var context = _uow.Context)
{
    using (var dbContextTransaction = context.Database.BeginTransaction())
    {
        try
        {
            // VALUES ((SELECT MAX(OID) + 1 FROM KlantDocument), -- tried this also, but was not yielding results
            var commandText = @"INSERT INTO KlantDocument
            (
             OID
            ,Datum
            ,Naam
            ,Uniek
            ,DocumentSoort
            )
            VALUES (
                    (SELECT TOP(1) (OID + 1) FROM KlantDocument ORDER BY OID DESC),
                    @Datum,
                    @Naam,
                    @Uniek,
                    @DocumentSoort
            )";
            var datum = new SqlParameter("@Datum", DateTime.Now);
            var naam = new SqlParameter("@Naam", dto.FileName);
            var uniek = new SqlParameter("@Uniek", Guid.NewGuid() + "." + dto.FileName.GetExtension());
            var documentSoort = new SqlParameter("@DocumentSoort", dto.DocumentTypeId ?? "OrderContents");

            context.Database.ExecuteSqlRaw(commandText, datum, naam, uniek, documentSoort);
            dbContextTransaction.Commit();
        }
        catch (Exception e)
        {
            dbContextTransaction.Rollback();
            return false;
        }
    }
}

Same thing.

I did a lot research to try to tackle this issue:

1. .NET Core - ExecuteSqlRaw - last inserted id? - solutions either don't work, or work only with identity columns
2. I got idea to use SQL OUTPUT variable, but issue is that it's really hard or even impossible to achieve, feels hacky and overall if managed, there is no guarantee that it will work. As seen here: SQL Server Output Clause into a scalar variable and here How can I assign inserted output value to a variable in sql server?

As noted above, ORM at the time did some magic where it did sequentially insert records in database and I would like to keep it that way.

Is there any way that I can achieve sequential insert when dealing with given scenario?

Answer 1

Ok, I have figured this out finally.

What I tried that did not work:

1.) IsolationLevel as proposed by @bluedot:

  public enum IsolationLevel
  {
    Unspecified = -1, // 0xFFFFFFFF
    Chaos = 16, // 0x00000010
    ReadUncommitted = 256, // 0x00000100
    ReadCommitted = 4096, // 0x00001000
    RepeatableRead = 65536, // 0x00010000
    Serializable = 1048576, // 0x00100000
    Snapshot = 16777216, // 0x01000000
  }

• Unspecified // Failed half of them Violation of PRIMARY KEY constraint 'PK_KlantDocument'. Cannot insert duplicate key in object 'dbo.KlantDocument'. The duplicate key value is (1652).

• Chaos // Failed all of them Exception thrown: 'System.ArgumentOutOfRangeException' in System.Private.CoreLib.dll

• ReadUncommitted // Failed half of them Violation of PRIMARY KEY constraint 'PK_KlantDocument'. Cannot insert duplicate key in object 'dbo.KlantDocument'. The duplicate key value is (1659).

• ReadCommitted // Failed half of them Violation of PRIMARY KEY constraint 'PK_KlantDocument'. Cannot insert duplicate key in object 'dbo.KlantDocument'. The duplicate key value is (1664).

• RepeatableRead // Failed half of them Violation of PRIMARY KEY constraint 'PK_KlantDocument'. Cannot insert duplicate key in object 'dbo.KlantDocument'. The duplicate key value is (1626).

• Serializable // Failed 3 of them Transaction (Process ID 66) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

• Snapshot // Failed all of them Snapshot isolation transaction failed accessing database 'Test' because snapshot isolation is not allowed in this database. Use ALTER DATABASE to allow snapshot isolation.

2.) Sequence in DbContext:

Thanks to a really awesome human being from EF Core team, Roji . I asked this on GitHub as well ( https://github.com/dotnet/efcore/issues/26480 ) and he actually pushed me in right direction.

I followed this documentation: https://docs.microsoft.com/en-us/ef/core/modeling/sequences , I was ecstatic to find this could actually be " automated " (read, I could just add them to context and commit changes using my UoW) and I won't be having the need to write the SQL queries when inserting data all the time. Thus I created following code to setup the sequence in DbContext and to run it using migrations.

My OnModelCreating method changes looked like so:

        modelBuilder.HasSequence<long>("KlantDocumentSeq")
            .StartsAt(2000)
            .IncrementsBy(1)
            .HasMin(2000);
            
        modelBuilder.Entity<KlantDocument>()
            .Property(o => o.Oid)
            .HasDefaultValueSql("NEXT VALUE FOR KlantDocumentSeq");

But since my Oid (PK) is not nullable , whenever I tried to insert my document like so:

            var doc = new KlantDocument
            {
                Naam = dto.FileName,
                DocumentSoort = dto.DocumentTypeId,
                Datum = DateTime.Now,
                Uniek = Guid.NewGuid() + "." + dto.FileName.GetExtension()
            };
            _uow.Context.Set<KlantDocument>().Add(doc);
            _uow.Commit();
            
            

It produced this SQL (beautified for visibility purposes):

SET NOCOUNT ON;
INSERT INTO [KlantDocument] ([OID], [Datum], [DocumentSoort], [Naam], [Uniek])
VALUES (0, '2021-10-29 14:34:06.603', NULL, 'sample 1 - Copy (19).pdf', '56311d00-4d7c-497c-a53e-92330f9f78d4.pdf');

And naturally I got exception:

InnerException = {"Violation of PRIMARY KEY constraint 'PK_KlantDocument'. Cannot insert duplicate key in object 'dbo.KlantDocument'. The duplicate key value is (0).\r\nThe statement has been terminated."}

Since it is default value of long (non-nullable) OID value to 0 , when creating doc variable, as it is not nullable .

NOTE: Be very very cautios if you are using this approach even if you can. It will alter all of your Stored procedures where this OID is used. In my case it created all of these:

 SELECT * FROM sys.default_constraints
 WHERE 
    name like '%DF__ArtikelDocu__OID__34E9A0B9%' OR
    name like '%DF__KlantDocume__OID__33F57C80%' OR
    name like '%DF__OrderDocume__OID__33015847%'

and I had to manually ALTER each table to drop these constrains .

What worked:

Again, thanks to a really awesome human being from EF Core team, Roji, I managed to implement this to work. Same as above, I created change in DbContext , OnModelCreating method:

        modelBuilder.HasSequence<long>("KlantDocumentSeq")
            .StartsAt(2000)
            .IncrementsBy(1)
            .HasMin(2000);

Added migration that produced this code:

    protected override void Up(MigrationBuilder migrationBuilder)
    {
        migrationBuilder.CreateSequence(
            name: "KlantDocumentSeq",
            startValue: 2000L,
            minValue: 2000L);
    }

    protected override void Down(MigrationBuilder migrationBuilder)
    {
        migrationBuilder.DropSequence(
            name: "KlantDocumentSeq");
    }

Updated database, and for DB work that was it. Final change came into my method where I had to replace notorious code block ((SELECT MAX(OID) + 1 FROM KlantDocument) or (SELECT TOP(1) (OID + 1) FROM KlantDocument ORDER BY OID DESC)

with

NEXT VALUE FOR KlantDocumentSeq

and it worked.

Full code of my method is here:

using (var context = _uow.Context)
{
    using (var dbContextTransaction = context.Database.BeginTransaction())
    {
        try
        {
            var commandText = @"INSERT INTO KlantDocument
            (
             OID
            ,Datum
            ,Naam
            ,Uniek
            ,DocumentSoort
            )
            VALUES (
                    NEXT VALUE FOR KlantDocumentSeq,
                    @Datum,
                    @Naam,
                    @Uniek,
                    @DocumentSoort
            )";
            var datum = new SqlParameter("@Datum", DateTime.Now);
            var naam = new SqlParameter("@Naam", dto.FileName);
            var uniek = new SqlParameter("@Uniek", Guid.NewGuid() + "." + dto.FileName.GetExtension());
            var documentSoort = new SqlParameter("@DocumentSoort", dto.DocumentTypeId ?? "OrderContents");

            context.Database.ExecuteSqlRaw(commandText, datum, naam, uniek, documentSoort);
            dbContextTransaction.Commit();
        }
        catch (Exception e)
        {
            dbContextTransaction.Rollback();
            return false;
        }
    }
}

and SQL it generates is:

INSERT INTO KlantDocument 
(
 OID
,Datum
,Naam
,Uniek
,DocumentSoort
)
VALUES (
        NEXT VALUE FOR KlantDocumentSeq,
        '2021-10-29 15:34:43.943',
        'sample 1 - Copy (13).pdf',
        'd4419c6c-dff9-431c-a7ed-6db54407051d.pdf',
        'OrderContents'
)