如何仅搜索文本文件中具有特定长度的行

Question

I have a large delimited txt file with two columns and about 17 million lines. I have imported it to the database, mistakenly one column in table had shorter size than the text from the file. ie varchar (4000) instead of varchar (7000)

about 48 thousands records with longer text has been cut into 40k chars.

How would I replace these without re-importing the files again?

I am thinking if I could be able to filter from txt file only lines with certain length, and remove them, and try to insert update the longer lines.

But How do I select all lines with certain length in a text file? or which program can do that.

I am using MySQL DB and emEditor for large files text editing.

Thanks.

Answer 1

Depending a bit on how this is connected to other infrastructure, my guess would be that the easiest and safest way of handeling it is just to drop and reimport the table...

If that for some reason not is an option, I would write a script that goes through the text file, and either just updates the large text field unconditionally or checks if it is of a length that makes an update neccesary (ie > 4000 chars)

If there might have been any changes to the table since the data was imported, it is important to check what will be overwritten and that the record really is the one you want to update (depending on how the table is indexed)

Hope this gave you some starting points.

You have my sympathy, been there done it...

Answer 2

You could also build a query that returns every record where that column contains 4000 bytes. Which are possibly the ones that were cut when you imported the file. With that set of records in hand, you could try to find them on the file if you have a line reference on the database table ofc. If there is too many, a simple script could do the trick.