Sed无法使用特殊字符

Question

I have a sample text file with some numbers encoded as Non Ascii characters. I Have the character map used to encode the file but when I am using sed to replace each of these characters, I am getting unexpected results.

like these

 ¤»¤ ¡  1 3

3ô1ô ôôôôô1ô
ôôôô
                       ôôôôô¤ôôôôô»ôôôôô¤ôôôôôô ô¡ ô 1 3ô

The commands which I have tried are these

sed -r 's/`echo ô`/5/g' new.txt
sed -r 's/\ô/5/g' new.txt

also perl

perl -pe 's/\ô/5/g' < new.txt

I need help for this please. Thanks.

Answer 1

I think the way to solve this would be to first get the characters (in both files) in an unambiguous form. Then iterate through the mapping file, adding each unambiguous character to a hash with it's said value. Finally, loop through the unambiguous sample characters (the size of an unambiguous character has a length of 16), replacing each one with it's hashed value. This can be broken if the sample file were to contain ASCII characters (ie where the length of it's unambiguous form is not 16). You may need to fix this depending on your input but if your sample text is indicative of your actual file, you shouldn't have any problems. Please let me know if the results are not what you were expecting.

Run like:

./translate.pl CharMap.txt sample.txt

Contents of translate.pl :

#!/usr/bin/perl
use strict;
use warnings;

# open the files up for reading.
# ARGV[0] points to the first file listed, 'CharMap.txt'
# ARGV[1] points to the second file listed, 'sample.txt'
open CHARMAP, $ARGV[0] or die;
open SAMPLE, $ARGV[1] or die;

# execute `sed -n 'l0'` on each file and capture output into two arrays
# the '-n' flag suppresses printing of pattern space
# the 'l0' command simply means print the pattern space in an unambiguous form
my @charmap = `sed -n 'l0' $ARGV[0]`;
my @sample = `sed -n 'l0' $ARGV[1]`;

# declare a hash
my %charhash;

# loop through the array of character mappings
for (@charmap) {
    # use a subroutine to sanitize each element
    $_ = sanitize($_);
    # add each unambiguous character to a hash with its mapping pair
    $charhash{ substr $_, 2 } = substr $_, 0, 1;
}

# now loop through the unambiguous sample data
# in your sample file there is only a single element so the loop is unnecessary
for (@sample) {
    # use a subroutine to sanitize each element
    $_ = sanitize($_);
    # so each unambiguous character is 16 readable characters longs.
    # so we need to loop through 16 chars at a time. These can be stored in $1. 
    # then we ask the hash 'what is the value of the element $1?
    # we then print this value.
    print $charhash{$1} while $_ =~ /(.{16})/g;

    # print a newline char to replace the chomped input
    print "\n";
}

close CHARMAP;
close SAMPLE;

sub sanitize {

    # read in the element passed to the subroutine
    my $line = shift;

    # remove newline endings
    chomp $line;

    # for some reason your files contained this transparent 12 digit unreadable
    # unambiguous character right at the start of the two files. I do not know
    # what it is or what it looks like, but for convenience, I simply remove it
    # from every line, even if I only found on the first line.
    $line =~ s/^\\357\\273\\277//;

    # trim off a trailing line ending
    $line =~ s/\$$//;

    # trim off a trailing newline ending
    $line =~ s/\\r$//;

    return $line;
}

Result:

3177191281013,997,094

Some more info can be found about sed l0 in the sed manual