在Spark上使用Twitter Streaming API無法獲得Tweets流

Question

我正在嘗試使用Twitter Streaming API對推文進行一些分析。

我首先想從流中打印狀態消息，然后從那里開始。

我的代碼如下所示：

public static void main(String[] args) {
  SparkConf conf = new SparkConf().setAppName("TwitterStreamPrinter").setMaster("local");

  Configuration twitterConf = new ConfigurationBuilder()
      .setOAuthConsumerKey(consumerKey)
      .setOAuthConsumerSecret(consumerSecret)
      .setOAuthAccessToken(accessToken)
      .setOAuthAccessTokenSecret(accessTokenSecret).build();
  OAuth2Authorization auth = new OAuth2Authorization(twitterConf);
  JavaReceiverInputDStream<Status> twitterStream = TwitterUtils.createStream(ssc, auth);

  JavaDStream<String> statuses = twitterStream.map(new Function<Status, String>() {
    public String call(Status status) throws Exception {
      return status.getText();
    }
  });
  statuses.print();

除了Spark日志外，它不會打印其他任何內容。 我最初以為這是因為授權，所以我嘗試了各種不同的方式來傳遞授權，但也許不是授權。

我看了我可以從網絡上找到的每個示例（盡管數量不多），並且該代碼看起來像是獲取Twitter狀態的標准代碼，但是為什么不打印任何內容？ 我也嘗試了System.out.println ，但是沒有用。

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/03/19 12:02:23 INFO SparkContext: Running Spark version 1.6.1
16/03/19 12:02:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/03/19 12:02:24 INFO SecurityManager: Changing view acls to: abcd
16/03/19 12:02:24 INFO SecurityManager: Changing modify acls to: abcd
16/03/19 12:02:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(abcd); users with modify permissions: Set(abcd)
16/03/19 12:02:24 INFO Utils: Successfully started service 'sparkDriver' on port 50995.
16/03/19 12:02:24 INFO Slf4jLogger: Slf4jLogger started
16/03/19 12:02:25 INFO Remoting: Starting remoting
16/03/19 12:02:25 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.0.0.12:51003]
16/03/19 12:02:25 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 51003.
16/03/19 12:02:25 INFO SparkEnv: Registering MapOutputTracker
16/03/19 12:02:25 INFO SparkEnv: Registering BlockManagerMaster
16/03/19 12:02:25 INFO DiskBlockManager: Created local directory at /private/var/folders/3b/wzflbsn146qgwdglbm_6ms3m0000hl/T/blockmgr-e3de07a6-0c62-47cf-9940-da18382c9241
16/03/19 12:02:25 INFO MemoryStore: MemoryStore started with capacity 2.4 GB
16/03/19 12:02:25 INFO SparkEnv: Registering OutputCommitCoordinator
16/03/19 12:02:25 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/03/19 12:02:25 INFO SparkUI: Started SparkUI at http://10.0.0.12:4040
16/03/19 12:02:25 INFO Executor: Starting executor ID driver on host localhost
16/03/19 12:02:25 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51016.
16/03/19 12:02:25 INFO NettyBlockTransferService: Server created on 51016
16/03/19 12:02:25 INFO BlockManagerMaster: Trying to register BlockManager
16/03/19 12:02:25 INFO BlockManagerMasterEndpoint: Registering block manager localhost:51016 with 2.4 GB RAM, BlockManagerId(driver, localhost, 51016)
16/03/19 12:02:25 INFO BlockManagerMaster: Registered BlockManager
16/03/19 12:02:25 WARN StreamingContext: spark.master should be set as local[n], n > 1 in local mode if you have receivers to get data, otherwise Spark jobs will not get resources to process the received data.
16/03/19 12:02:26 INFO SparkContext: Invoking stop() from shutdown hook
16/03/19 12:02:26 INFO SparkUI: Stopped Spark web UI at http://10.0.0.12:4040
16/03/19 12:02:26 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/03/19 12:02:26 INFO MemoryStore: MemoryStore cleared
16/03/19 12:02:26 INFO BlockManager: BlockManager stopped
16/03/19 12:02:26 INFO BlockManagerMaster: BlockManagerMaster stopped
16/03/19 12:02:26 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/03/19 12:02:26 INFO SparkContext: Successfully stopped SparkContext
16/03/19 12:02:26 INFO ShutdownHookManager: Shutdown hook called
16/03/19 12:02:26 INFO ShutdownHookManager: Deleting directory /private/var/folders/3b/.....

Answer 1

您的日志中有所有內容：

19/6/03 12:02:25 WARN StreamingContext：如果您有接收器來獲取數據，則在本地模式下將spark.master設置為local [n]，n> 1，否則Spark作業將不會獲得資源來處理接收到的數據數據。

因此答案設置為master為本地[*]

另外，您忘了開始嗎？

jssc.start（）; //開始計算

jssc.awaitTermination（）;