如何识别由于.txt文件中隐藏的＃而导致列中数据丢失的行

Question

I have a below .txt files exported from the source system. Due to the # in one field in source system while exporting the .txt file some of the data after # fields do not have any data in the .txt file when exported.

For example below..

LINE|PANO| INOW|DEL|EASLN|EBSAP|LIM1IT|NOMIT|VALUE|KTE1|
1|7870|1000000||40500369|10|25624.0||0.00|SERVI TORNG|33277|
2|294|1000000||500324|10|590.84 ||0.00|REFUDIAL GATNGWAM|30448|
3|9410|1000000||200500325|10|5905.61||0.00|SUPLIVER EXTRACNS|37478|
4|573|1000000||600004075|10||||||||
5|739|1000000||700500290|10|40917.37|||||||
6|741|1000000||50500289|10|2782.53 ||0.00|SECUERVIC LUWE|29161|
7|948|1000000||||||||||||
8|996|1000000||960050035|10|7497.3||0.00|SCOUOUT URBISH IDM647 |38271|
9|1320|1000000||800500319|10|1395.93||0.00|TUATO AIRS|36427|
10|12054|1000000||9000287|10|458.42||0.00|SECURICE GOLA|||||

In the above example line 4, 5, 7 and 10 data is missing after certain fields due to the # in the source system field. But there is data in the source system for these line items.

How to recognize these line items as the missing information / records issue, if I have a large volume of .txt file for 10 Million-line items.

Please kindly share the SQL query/ any other way to identify these line items with the missing data.

another example

LINE|PANO| INOW|DEL|EASLN|EBSAP|LIM1IT|NOMIT|VALUE|KTE1|
1|7870|1000000||40500369|10|25624.0||0.00|SERVI TORNG|33277|
2|294|1000000||500324|10|590.84 ||0.00|REFUDIAL GATNGWAM|30448|
3|9410|1000000||200500325|10|5905.61||0.00|SUPLIVER EXTRACNS|37478|
4|573|1000000||600004075|10
5|739|1000000||700500290|10|40917.37
6|741|1000000||50500289|10|2782.53 ||0.00|SECUERVIC LUWE|29161|
7|948|1000000
8|996|1000000||960050035|10|7497.3||0.00|SCOUOUT URBISH IDM647 |38271|
9|1320|1000000||800500319|10|1395.93||0.00|TUATO AIRS|36427|
10|12054|1000000||9000287|10|458.42||0.00|SECURICE GOLA

data truncated if # exists.

Answer 1

Would the following do what you require?

I created a temporary table #HiddenHash and populated it with some of your example data, you will obviously have the data from a BULK INSERT or whatever mechanism you are using.

CREATE TABLE 
#HiddenHash
(
LINE VARCHAR (2)
,PANO VARCHAR (25) 
,INOW VARCHAR (25)
,DEL VARCHAR (25)
,EASLN VARCHAR (25)
,EBSAP VARCHAR (25)
,LIM1IT VARCHAR (25)
,NOMIT VARCHAR (25)
,VALUE VARCHAR (25)   
,KTE1 VARCHAR (25)
)

INSERT INTO #HiddenHash
VALUES
('1','7870','1000000','','40500369','10','25624.0','0.00','SERVI TORNG','33277')
,('2','294','1000000','',' 500324','10','590.84 ','0.00','REFUDIAL GATNGWAM','30448')
,('3','9410','1000000','','200500325','10','5905.61','0.00','SUPLIVER EXTRACNS','37478')
,('4','573','1000000','','600004075','10','','','','')
,('5','739','1000000','','700500290','10','40917.37','','','')
,('6','741','1000000','','50500289','10','2782.53 ','0.00','SECUERVIC LUWE','29161')
,('7','948','1000000','','','','','','','')
,('8','996','1000000','','960050035','10','7497.3','0.00','SCOUOUT URBISH IDM647 ','38271')
,('9','1320','1000000','','800500319','10','1395.93','0.00','TUATO AIRS','36427')
,('10','12054','1000000','','9000287','10','458.42','0.00','SECURICE GOLA','')

Then I count how many columns there are in the table.

    DECLARE @CountColumns INT


    SET @CountColumns = (SELECT COUNT (*) 
                        FROM TEMPDB.SYS.COLUMNS
                        WHERE NAME <> 'DEL' AND
                         object_id = object_id('tempdb.dbo.#HiddenHash')
                         )

Then count those rows where the columns are blank and show those where they do not match the number of columns contained in the variable.

    SELECT LINE,PANO,INOW,EASLN,EBSAP,LIM1IT,NOMIT,VALUE,KTE1 
       FROM (
            SELECT 
            LINE,PANO,INOW,EASLN,EBSAP,LIM1IT,NOMIT,VALUE,KTE1, 
            (
            SELECT COUNT(*) 
            FROM (VALUES (LINE),(PANO),(INOW),(EASLN),(EBSAP),(LIM1IT),(NOMIT), 
           (VALUE),(KTE1)) AS Cnt(col) 
        WHERE Cnt.Col <> ''
        ) AS NotBlank
    FROM #HiddenHash)cc
    WHERE cc.NotBlank <> @CountColumns

Which gives the following result

LINE    PANO    INOW    EASLN       EBSAP   LIM1IT   NOMIT  VALUE       KTE1
4       573     1000000 600004075   10              
5       739     1000000 700500290   10      40917.37            
7       948     1000000                     
10      12054   1000000 9000287     10       458.42  0.00   SECURICE GOLA