简体繁体 English

如何在HDFS中存储推文？

[英]How to Store Tweets in HDFS?

原文 2013-09-19 09:01:07 8 2 twitter/ hadoop/ hdfs/ sqoop/ flume

How to Store particular website tweets in HDFS ? 如何在HDFS中存储特定的网站推文？

Suppose one website www.abcd.com and I want to collect all user's tweet for this website and stored into HDFS or Hive. 假设一个网站www.abcd.com，我想收集该网站的所有用户推文并存储到HDFS或Hive中。

Flume and sqoop also helpful for storing data. Flume和Squoop还有助于存储数据。

so anyone please suggest me how flume and sqoop work in storing tweets in HDFS? 所以有人请建议我，在HDFS中存储鸣叫时，水槽和浓汤是如何工作的？

2 个解决方案

Sqoop was not made for this purpose. Sqoop并非出于此目的。 Flume is used for these kind of needs. Flume用于此类需求。 You can write your custom Flume source that will pull the tweets and dump them into your HDFS. 您可以编写您的自定义Flume源代码，该源代码将拉动这些推文并将其转储到HDFS中。 See this for example. 参见此示例。 It shows how to use Flume to collect data from the Twitter Streaming API, and forward it to HDFS. 它显示了如何使用Flume从Twitter Streaming API收集数据，并将其转发到HDFS。

You can find more in the official documentation . 您可以在官方文档中找到更多信息。

I referred this link and i could easily stream data from twitter. 我引用了此链接，我可以轻松地从Twitter流数据。 This might help you . 这可能对您有帮助。 http://www.thecloudavenue.com/2013/03/analyse-tweets-using-flume-hadoop-and.html http://www.thecloudavenue.com/2013/03/analyse-tweets-using-flume-hadoop-and.html

download site : http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.5.0/bk_installing_manually_book/content/rpm-chap13.html 下载站点： http : //docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.5.0/bk_installing_manually_book/content/rpm-chap13.html

如何使用 Spark Streaming Java API 将推特推文写入 HDFS - How to write twitter tweets to HDFS using Spark Streaming Java API

如何在单个字典中存储推文流？ - How to store stream of tweets in a single dict?

Python：如何搜索推文并存储在数据库中？ - Python: How to search tweets and store in database?

无法将推文存储在CouchDB中 - Not able to store tweets in CouchDB

我如何流式传输和存储Twitter的公共时间线，特定用户的推文和标记的推文？ - how can i stream and store twitter's public timeline, particular user's tweets and tagged tweets?

我们如何将Twitter HomeTimeline推文存储到MSSQL数据库中？ - How can we store twitter HomeTimeline tweets into MSSQL database?

如何存储Twitter搜索API推文并转发 - How to store twitter search API tweets and retweet them

如何轻松存储推文列表（状态）？ - How can I store loacally a list of tweets (status)?

流推文并将其存储在外部驱动器上：如何插入路径文件 - Streaming tweets and store them on external drive: how to insert path file

无法在SQL数据库中存储推文 - Unable to store tweets in an sql database

暂无

暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 如何使用 Spark Streaming Java API 将推特推文写入 HDFS - How to write twitter tweets to HDFS using Spark Streaming Java API 如何在单个字典中存储推文流？ - How to store stream of tweets in a single dict? Python：如何搜索推文并存储在数据库中？ - Python: How to search tweets and store in database? 无法将推文存储在CouchDB中 - Not able to store tweets in CouchDB 我如何流式传输和存储Twitter的公共时间线，特定用户的推文和标记的推文？ - how can i stream and store twitter's public timeline, particular user's tweets and tagged tweets? 我们如何将Twitter HomeTimeline推文存储到MSSQL数据库中？ - How can we store twitter HomeTimeline tweets into MSSQL database? 如何存储Twitter搜索API推文并转发 - How to store twitter search API tweets and retweet them 如何轻松存储推文列表（状态）？ - How can I store loacally a list of tweets (status)? 流推文并将其存储在外部驱动器上：如何插入路径文件 - Streaming tweets and store them on external drive: how to insert path file 无法在SQL数据库中存储推文 - Unable to store tweets in an sql database

相关标签

粤ICP备18138465号 © 2020-2024 STACKOOM.COM