使用C＃程序区分具有相同内容但格式不同的两个文本文件

Question

I have two text files - they both contain the same information but are available in two different formats.

Format 1 is having line breaks and looks well formatted. Format 2 "appears" to be continuous but in reality it also has line breaks but the line break is being represented in a very weird way.

https://www.dropbox.com/sh/ljlqen94a5cwza2/AAAOcuYU_EDnSLiNPRP_CDbga?dl=0

Please refer to the attachements (LineBreak.dat and NoLineBreak.dat) In the latter file, there are line breaks but not visible - looks like some kind of transformation on the data has changed the representation. If you start counting from the first position (start counting from i=0) by using the right cursor on the keyboard then at i=19 you will find that the cursor gets stuck for one press - you have to press twice to navigate to next position. This happens at many places in the document - I figured these are the places there were line breaks that have now been corrupted.

In my business case scenario, the latter type of file is to be regarded as invalid. So I need to be able to write a C# program to detect the type of file - if its in Format1 or Format2 and need help with this.

I tried to see if the encoding on them is different by reading BOM but its the same on both files. I got the following BOM readings : [0]: 57 [1]: 57 [2]: 48 [3]: 54

I am using the following program to detect encoding :

public static void GetEncoding(string pFilePath,out Encoding pFileEncoding)
{
    // Read the BOM
    var bom = new byte[4];
    using (var file = new FileStream(pFilePath, FileMode.Open, FileAccess.Read))
    {
        file.Read(bom, 0, 4);
    }

    // Analyze the BOM
    if (bom[0] == 0x2b && bom[1] == 0x2f && bom[2] == 0x76) pFileEncoding = Encoding.UTF7;
    if (bom[0] == 0xef && bom[1] == 0xbb && bom[2] == 0xbf) pFileEncoding= Encoding.UTF8;
    if (bom[0] == 0xff && bom[1] == 0xfe) pFileEncoding =Encoding.Unicode; //UTF-16LE
    if (bom[0] == 0xfe && bom[1] == 0xff) pFileEncoding= Encoding.BigEndianUnicode; //UTF-16BE
    if (bom[0] == 0 && bom[1] == 0 && bom[2] == 0xfe && bom[3] == 0xff) pFileEncoding= Encoding.UTF32;
    pFileEncoding= Encoding.ASCII;//or Encoding.Default
}

Answer 1

The two files have different style Linebreaks - You can use a string replace in one of the files to make this identical. Try to look at https://superuser.com/questions/545461/replace-carriage-return-and-line-feed-in-notepad For a way to do it manual, but you can do this in you C# code as well just replace \\n with \\r\\n.

If you want to be sure it will work everywhere you can replace \\n AND \\r\\n with Environment.NewLine

Hope it helps :)

Answer 2

The Format2 file isn't corrupt; it just has unix-style line breaks (just a linefeed or \\n ) at the end of each line. The other file has windows-format line breaks (carriage return followed by linefeed or \\r\\n ).

You can easily fix the latter files by checking for the existence of \\r and if none exist in the file, doing a string.Replace("\\n", "\\r\\n") across the whole file.

Answer 3

If you open your text file in a "potent" text editor like Notepad++ you are able to see every single byte in your file, even if it is "whitespace" ie not displayed in "normal" text editors.

In your case you'll find out that the line breaks are "Linefeed" characters ('\\n', Dec 10, Hex 0x0A). This is the usual way to represent "New Line" in Unix systems.

If you want to flag such files as "invalid" just search for Carriage Return ('\\r', dec 13 Hex 0x0D) character and "Linefeed" characters.

In windows text files you'll find 0x0D/0x0A Pairs

In Unix Files 0x0A only

In Apple Files 0x0D only

(All this has nothing to do with encodings)