Difference between hive,pig,map-reduce use cases

Question

Difference between map-reduce ,hive ,pig

pig : its a data flow language, it can work on any data basically used to convert semi structure ,unstructured data to structure so that can be used in hive advance analytics using windowing function etc.

Hive : Work on structure data and provide sql type query language .

I know at back end both pig and hive uses map -reduces .

I know map-reduce can be good tool for programmer ,hive or pig for sql guy

I just want to know is there any specific use cases where we go for hive,pig and map-reduce

basically we decide that we have to use pig here hive here or we must use map -reduce .

Answer 1

Map-Reduce: Has better performance than pig or hive but requires more development time.

PIg: Less development time but poor performance when compared to map-reduce.

Hve: SQL type language with some good features like partitioning and bucketing to improve performance reads.Also, hive enforces schema on read.

Answer 2

Pig is used to format your unstructured/semi structure data format.Lets say you have a timestamp in your data which is not as per Hive timestamp format.You can convert same using pigUDF and format your data.This is just a example to explain.You can do many more things using Pig.

Hive is basically used for structured data .This maynot work well with unstructured data.This takes more time to execute as it converts into Mapreduce job.I suggest you to use impala which is much faster than hive.

Answer 3

Pig is a data flow language. This means that you can not use if statements or loops. If you need to do a lot of repetition, it would be preferable to learn mapreduce.

You are able to get around this by embedding pig into a python script but this would take even longer since it would have to load all the jar files with every iteration of the loop.

Basically it boils down to how much time you spend prototyping vs. how much production work you have. If you are a data scientist or an analyst, most of your work is new projects that require a lot of prototyping. This means that you care about getting results fast. Then you would prefer Pig or Hive. If you are in a development team, you want to build robust code based on agreed upon methodology that does not need to be tested and then you would prefer mapreduce.

There are companies like Cloudera that provide a package of Pig, Hive, and other Hadoop tools so you wouldn't have to choose between the two.

Answer 4

Map Reduce is a inner component of hadoop, other Pig and hive are hadoop eco systems it means run on the top of hadoop. The purpose of both mapreduce, pig and hive purpose is process the vast amount of data in different manner.

Mapreduce : apache implemented it. highly recommendable to process entire data, it's time consume and required program skills like java (highly recommendable), pyghon, ruby and other programming languages. total data aggregate and sort by using mapper and reducer functions. Hadoop use it by default.

Hive : Facebook implemented it. most of the analysts especially bigdata analysts use this tool to analyze the data especially structure data. Backend this hive tool use mapreduce to be processed. Internally Hive use special language called HQL, It's subset of SQL language. Who is wellever in SQL, they can goes with Hive. It's highly recommended to the Datawarehouse oriented projects. Much difficult to process un structured especially schema-less data.

Pig: Pig is a scripting language, implemented by Yahoo. The main difference between pig and Hive is pig can process any type of data, either structured or unstructured data. It means it's highly recommendable for streaming data like satellite generated data, live events, schema-less data etc. Pig first load the data later programmer write a program depends on data to make it structured. Who is expert in programming languages they will choose this Hadoop ecosystems.