Uniqing基于字段子集的分隔文件

Question

I have data such as below:

1493992429103289,207.55,207.5
1493992429103559,207.55,207.5
1493992429104353,207.55,207.5
1493992429104491,207.6,207.55
1493992429110551,207.55,207.5

Due to the nature of the last two columns, their values change throughout the day and their values are repeated regularly. By grouping the way outlined in my desired output (below), I am able to view each time there was a change in their values (with the enoch time in the first column). Is there a way to achieve the desired output shown below:

1493992429103289,207.55,207.5
1493992429104491,207.6,207.55
1493992429110551,207.55,207.5

So I consolidate the data by the second two columns. However, the consolidation is not completely unique (as can be seen by 207.55, 207.5 being repeated)

I have tried:

uniq -f 1

However the output gives only the first line and does not go on through the list

The awk solution below does not allow the occurrence which happened previously to be outputted again and so gives the output (below the awk code):

awk '!x[$2 $3]++'

1493992429103289,207.55,207.5
1493992429104491,207.6,207.55

I do not wish to sort the data by the second two columns. However, since the first is epoch time, it may be sorted by the first column.

Answer 1

You can use an Awk statement as below,

awk 'BEGIN{FS=OFS=","} s != $2 && t != $3 {print} {s=$2;t=$3}' file

which produces the output as you need.

1493992429103289,207.55,207.5
1493992429104491,207.6,207.55
1493992429110551,207.55,207.5

The idea is to store the second and third column values in variables s and t respectively and print the line contents only if the current line is unique .

Answer 2

You can't set delimiters with uniq , it has to be white space. With the help of tr you can

tr ',' ' ' <file | uniq -f1 | tr ' ' ','

1493992429103289,207.55,207.5
1493992429104491,207.6,207.55
1493992429110551,207.55,207.5 

Answer 3

I found an answer which is not as elegant as Inian but satisfies my purpose. Since my first column is always enoch time in microseconds and does not increase or decrease in characters, I can use the following uniq command:

uniq -s 17

Answer 4

You can try to manually (with a loop) compare current line with previous line.

previous_line=""
# start at first line
i=1

# suppress first column, that don't need to compare
sed 's@^[0-9][0-9]*,@@' ./data_file > ./transform_data_file

# for all line within file without first column
for current_line in $(cat ./transform_data_file)
do 
  # if previous record line are same than current line
  if [ "x$prev_line" == "x$current_line" ]
  then
    # record line number to supress after
    echo $i >> ./line_to_be_suppress
  fi

  # record current line as previous line
  prev_line=$current_line

  # increment current number line
  i=$(( i + 1 ))
done

# suppress lines
for line_to_suppress in $(tac ./line_to_be_suppress) ; do sed -i $line_to_suppress'd' ./data_file ; done

rm line_to_be_suppress
rm transform_data_file

Answer 5

Since your first field seems to have a fixed length of 18 characters (including the , delimiter), you could use the -s option of uniq , which would be more optimal for larger files:

uniq -s 18 file

Gives this output:

1493992429103289,207.55,207.5
1493992429104491,207.6,207.55
1493992429110551,207.55,207.5

From man uniq :

-f num

Ignore the first num fields in each input line when doing comparisons. A field is a string of non-blank characters separated from adjacent fields by blanks. Field numbers are one based, ie, the first field is field one.

-s chars

Ignore the first chars characters in each input line when doing comparisons. If specified in conjunction with the -f option, the first chars characters after the first num fields will be ignored. Character numbers are one based, ie, the first character is character one.