Sed with UTF8 input

Question

I need help with my work to school. I have got (from stackoverflow of course) this script which capitalizing first character of string.

sed -r "s/(^|\.\s+)./\U&/g" <$temp_file_2

But output of this is in ANSI encoding or what is that. file -bi shows unknown-8bit encoding format.

Is it any change to get output in utf-8 to file?

PS: This sed command is used for capitalizing firs character of line. (with support of special Slovak characters like ščťžýáíéď etc) PS: File have to be UTF-8 because content is inserted to mysql database. Converting file causing loosing information.

Answer 1

The problem is that sed might have trouble dealing with non-ASCII characters, especially when the system locale is not UTF8.

$ bash -c "echo 'abc,ščťžýáíéď' | LANG= LC_CTYPE= sed -E --debug 's/./\U&/g'"
SED PROGRAM:
  s/./\U&/g
INPUT:   'STDIN' line 1
PATTERN: abc,\o37777777705\o37777777641\o37777777704\o37777777615\o37777777705\o37777777645\o37777777705\o37777777676\o37777777703\o37777777675\o37777777703\o37777777641\o37777777703\o37777777655\o37777777703\o37777777651\o37777777704\o37777777617
COMMAND: s/./\U&/g
MATCHED REGEX REGISTERS
  regex[0] = 0-1 'a'
PATTERN: ABC,\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777\o37777777777
END-OF-CYCLE:
ABC,

As you can see, sed view each non-ASCII character as several separated bytes, so it incorrectly uppercased them. One solution is setting LANG and LC_CTYPE to a UTF8 compatible locale.

$ bash -c "echo 'abc,ščťžýáíéď' | LANG=C.UTF8 LC_CTYPE=C.UTF8 sed -E --debug 's/./\U&/g'"
SED PROGRAM:
  s/./\U&/g
INPUT:   'STDIN' line 1
PATTERN: abc,\o37777777705\o37777777641\o37777777704\o37777777615\o37777777705\o37777777645\o37777777705\o37777777676\o37777777703\o37777777675\o37777777703\o37777777641\o37777777703\o37777777655\o37777777703\o37777777651\o37777777704\o37777777617
COMMAND: s/./\U&/g
MATCHED REGEX REGISTERS
  regex[0] = 0-1 'a'
PATTERN: ABC,\o37777777705\o37777777640\o37777777704\o37777777614\o37777777705\o37777777644\o37777777705\o37777777675\o37777777703\o37777777635\o37777777703\o37777777601\o37777777703\o37777777615\o37777777703\o37777777611\o37777777704\o37777777616
END-OF-CYCLE:
ABC,ŠČŤŽÝÁÍÉĎ

References:

• https://superuser.com/questions/1347826/how-to-avoid-sed-changing-the-file-format
• Why does sed fail with International characters and how to fix?

Answer 2

Try this

  cat <src> | iconv -f <srcenc> | sed .... | iconv -t <targetenc> > target

To see list of encodings:

  iconv -l

To see if you guessed encoding of your input file correctly check

cat <src> | iconv -f <srcenc>