Get All User timeline tweets using twitter4j and java

Question

I have a problem if anyone can help, I'm trying to get tweets done by a specific user, here's my code:

    Paging pg = new Paging();
    String userName = "Obama";
    pg.setCount(200);
    ConfigurationBuilder cb = new ConfigurationBuilder();
  cb.setOAuthConsumerKey("");
  cb.setOAuthConsumerSecret("");
  cb.setOAuthAccessToken("");
  cb.setOAuthAccessTokenSecret("");


  Twitter twitter = new TwitterFactory(cb.build()).getInstance();
  int numberOfTweets = 1000000;
  long lastID = Long.MAX_VALUE;
  ArrayList<Status> tweets = new ArrayList<Status>();
  while (tweets.size () < numberOfTweets) {


  tweets.addAll(twitter.getUserTimeline(userName,pg));
  //System.out.println("Gathered " + tweets.size() + " tweets");
  for (Status t: tweets) {
    System.out.println(t.getUser().getName() + ": " + t.getText()+ " " );


}; 
    pg.setMaxId(lastID-1);
  }
        System.out.println(tweets.size());

    }

The problem is that the result is only the same results, the algorithm takes only the first few tweets from the timeline and makes them X time, and the profile has a million of tweets. Can someone tell me how can I solve this problem please? Thanks

Answer 1

Here is a way to do :

ArrayList<Status> statuses = new ArrayList<>();
int pageno = 1;
while(true) {
    try {
        System.out.println("getting tweets");
        int size = statuses.size(); // actual tweets count we got
        Paging page = new Paging(pageno, 200);
        statuses.addAll(twitter.getUserTimeline(screenName, page));
        System.out.println("total got : " + statuses.size());
        if (statuses.size() == size) { break; } // we did not get new tweets so we have done the job
        pageno++;
        sleep(1000); // 900 rqt / 15 mn <=> 1 rqt/s
        }
    catch (TwitterException e) {
        System.out.println(e.getErrorMessage());
        }
    } // while(true)

And you need a sleep function to respect rate limit :

static void sleep(long ms) {
    try { Thread.sleep(ms); }
    catch(InterruptedException ex) { Thread.currentThread().interrupt(); }
    }

Reference : https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html