Linux如何从行尾^ A知道嵌入式^ A

Question

I'm having a data issue with embedded ^A characters, which i can fully reproduce with this small file:

Observe that I have embedded ^A characters. I put them there using vi with the ^V technique.

Now, notice I also put a line break after the "p,q," string on the third line. That was done with the Enter key, but it just puts in a ^A, we can see here:

[ ~/hack ] cat t.csv
a,b,c,d,e
f,g,,i,j
k,l,,n,o
p,q,
,s,t
u,v,w,x,y
[ ~/hack ] xxd < t.csv > u.csv
[ ~/hack ] cat u.csv
0000000: 612c 622c 632c 642c 650a 662c 672c 012c  a,b,c,d,e.f,g,.,
0000010: 692c 6a0a 6b2c 6c2c 012c 6e2c 6f0a 702c  i,j.k,l,.,n,o.p,
0000020: 712c 0a2c 732c 740a 752c 762c 772c 782c  q,.,s,t.u,v,w,x,
0000030: 790a                                     y.
[ ~/hack ]

Note that for the "cat" listing, the double comma has the ^A in it, it just doesn't print to the screen with cat.

But notice also, the normal end-of-line is also a ^A. This is where it gets tricky...how does Linux differentiate between a ^A that is an embedded character, and one that is the end of line?

Note in the hex dump, after the "e", is an 0a, as expected. But there is an 0a between the two commas between 'l' and 'n' too. Yet my manually broken line between 'q' and 's' shows an actual line break--but it's just a 0a like any other!!!

My ultimate need is I need to programmatically find all broken lines like the p,q,.,s,t one, and get rid of those line breaks. But sed can't see that as a line break. That is, if I replace ^A, it would see the ones on the 'f' and 'k' lines, but it can't find the ones on the 'p' line.

So, 1) As a matter of conceptual understanding, can someone explain how on Earth Linux knows the difference between the 0a character that is embedded and one that is an end of line, and 2) What is the piece of code that would find the artificial line breaks and mend the line?

Thanks!

Answer 1

^A is not 0a. ^A (control-A) is ASCII character 1 (01), while the newline/linefeed character (0a, ASCII 10) is ^J (control-J).