SQL Server 2016批量插入csv并从特定列生成sha1并插入到列中

Question

I am fairly new to SQL Server and I am running the following command from inside my c# application:

DECLARE @SQLString nvarchar(4000);
SET @SQLString = N'BULK INSERT events FROM '+ QUOTENAME(@p0) +' WITH ( BATCHSIZE = 50000, CODEPAGE = ''65001'', FIELDTERMINATOR = ''|'', ROWTERMINATOR =''\n'' )'
EXECUTE sp_executesql @SQLString

With @p0 being the path to the .csv file.

Right now I am generating a sha1 HashCode as a BigInteger from a combination of 3 columns in my C# code and write it into the csv file into a new column (which is the primary key).

Now I saw it is possible to generate the sha1 hash inside the sql server. Is this possible while bulk inserting?

eg Bulk insert csv file. For each row take column X,Y,Z and generate sha1 hash. Convert it to BigInteger and insert it in column P?

EDIT: I am trying the answer from @Nick.McDermaid: But I can't seem to get it working:

CREATE TABLE [dbo].[test] (
[User] [nvarchar](185) NOT NULL,
[Stat] [nvarchar](25) NOT NULL,
[Name] [nvarchar](max) NOT NULL,
[HashByte] AS (convert(bigint, HASHBYTES('SHA1',CONVERT(nvarchar(max),[User]+[Stat]+‌​[Name])))),

CONSTRAINT [PK_dbo.test] PRIMARY KEY ([HashByte]))

I get an Error for incorrect syntax.

Answer 1

I suggest you take a step back here: firstly: are you saying that if there is a one character difference in your varchar(max) field (2Gb), then the record is a unique record? What is the purpose of defining unique records here and what happens when a "duplicate" appears?

In this situation I recommend you follow this very commonly used staging pattern, which ends up being used for most data import processes

1. BULK INSERT into a staging table that has no PK
2. Use INSERT to only insert unique records into your real table
3. Your real table has a simple int identity PK and is guaranteed to be unique on the required columns due to step 2
4. You can identify records with issues in your staging table

From my experiments, it's not possible to create any kind of unique constraint/unique index/PK on this particular calculated field

Some sample code for step 1&2 would be:

-- Bulk insert into staging table
BULK INSERT staging.events FROM.....

-- Only insert records from staging that aren't already there
INSERT INTO dbo.events (User,Stat,Name)
SELECT User,Stat,Name 
FROM staging.events S
WHERE NOT EXISTS (
   SELECT * FROM dbo.events E
   WHERE E.User = S.User
   AND E.Stat = S.Stat
   AND E.Name = S.Name
)

Now if you like you can write another update back to the staging table that identifies duplicates.

It really comes back to the meaning of 'duplicate'. If you have one character difference in your Name column, is that a duplicate?

Answer 2

I got it working with

CREATE TABLE dbo.test ( 
[User] nvarchar(185) NOT NULL, 
[Stat] nvarchar(25) NOT NULL, 
[Name] nvarchar(max) NOT NULL, 
[HashByte] AS CAST(HASHBYTES('SHA1', CONCAT([User],[Stat],​[Name])) AS BIGINT)
PERSISTED, CONSTRAINT [PK_dbo.test] PRIMARY KEY ([HashByte]) ) 

And then use no_dup_keys!