Hadoop面试查询-Mapreduce-Pig-Hive

Question

This was a question asked for me in my hadoop interview. I have the table data like below.

I have taken a new bike and on the 1st day the distance I have travelled 20 km 2nd day the reading on the meter was 50(day 1 + day 2) 3rd day the reading on the meter was 60(day 1+ day 2+ day 3)

Day Distance
1    20
2    50
3    60

Now the question is, I want the output to be like below

Day  Distance
1    20
2    30
3    10

ie I want the distance travelled only on the 1st day, 2nd day and 3rd day.

Answer can be in Hive/Pig/MapReduce.

Thanks

Answer 1

This is a running totals like problem, you can resolve it by this Hive query

with b as (
select 0 as d, 0 as dst
union all 
select d, dst from mytable
)
SELECT a.d, a.km-b.km new_dst from mytable a, b 
where a.d-b.d==1

Answer 2

You can use Hive's in-built windowing and analytics functions to get the desired resutls.

Here is one way.

SELECT day, NVL(CAST(distance-LAG(distance) OVER (ORDER BY day) AS INT),20) 
FROM table;

Answer 3

I tried in map reduce. package hadoop;

public class distance {

public static class disMapper extends Mapper<LongWritable,Text,IntWritable,IntWritable>
{
    //1 20
    int pValue=0;
    IntWritable outkey=new IntWritable();
    IntWritable outvalue=new IntWritable();
    public void map(LongWritable key,Text values,Context context) throws IOException, InterruptedException
    {
        String cols[]=values.toString().split("\t");
        int dis=Integer.parseInt(cols[1])-pValue;
        outkey.set(Integer.parseInt(cols[0]));
        outvalue.set(dis);
        pValue=Integer.parseInt(cols[1]);
        context.write(outkey, outvalue);
    }
}

public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
    // TODO Auto-generated method stub
    Configuration conf=new Configuration();
    Job job =new Job(conf,"dfdeff");
    job.setJarByClass(distance.class);
    job.setMapperClass(disMapper.class);

    job.setMapOutputKeyClass(IntWritable.class);
    job.setMapOutputValueClass(IntWritable.class);

    job.setNumReduceTasks(0);

    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    System.exit(job.waitForCompletion(true)?1:0);

}

}

Answer 4

Below query also works fine

select day, CASE WHEN day = 1 THEN distance ELSE (distance - LAG(distance) over(ORDER BY day)) END AS dailyReading from table;