How to Store Tweets in HDFS?
Question
How to Store particular website tweets in HDFS ?
Suppose one website www.abcd.com and I want to collect all user's tweet for this website and stored into HDFS or Hive.
Flume and sqoop also helpful for storing data.
so anyone please suggest me how flume and sqoop work in storing tweets in HDFS?
2 answers
solution1
1 2013-09-19 09:09:36
Sqoop was not made for this purpose. Flume is used for these kind of needs. You can write your custom Flume source that will pull the tweets and dump them into your HDFS. See this for example. It shows how to use Flume to collect data from the Twitter Streaming API, and forward it to HDFS.
You can find more in the official documentation .
solution2
0 2014-01-28 10:16:48
I referred this link and i could easily stream data from twitter. This might help you . http://www.thecloudavenue.com/2013/03/analyse-tweets-using-flume-hadoop-and.html
download site : http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.5.0/bk_installing_manually_book/content/rpm-chap13.html
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.