Parse Raw Json Strings

Question

I have used Twitter4j to crawl millions of tweets, but in my surprise all the tweets are stored in raw json format. Here is an example of a formatted line:

StatusJSONImpl{
  createdAt=TueNov0119: 00: 04CET2016,
  id=793512948027326464,
  text='RT @DylanYamaha_: Et profitez vraiment des personnes qui sont près de vous, parce que sa arrive très très vite un malheur..',
  source='<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>',
  isTruncated=false,
  inReplyToStatusId=-1,
  inReplyToUserId=-1,
  isFavorited=false,
  isRetweeted=false,
  favoriteCount=0,
  inReplyToScreenName='null',
  geoLocation=null,
  place=null,
  retweetCount=0,
  isPossiblySensitive=false,
  lang='fr',
  contributorsIDs=[

  ],
  retweetedStatus=StatusJSONImpl{
    createdAt=TueNov0118: 38: 05CET2016,
    id=793507418244313088,
    text='Et profitez vraiment des personnes qui sont près de vous, parce que sa arrive très très vite un malheur..',
    source='<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>',
    isTruncated=false,
    inReplyToStatusId=-1,
    inReplyToUserId=-1,
    isFavorited=false,
    isRetweeted=false,
    favoriteCount=56,
    inReplyToScreenName='null',
    geoLocation=null,
    place=null,
    retweetCount=175,
    isPossiblySensitive=false,
    lang='fr',
    contributorsIDs=[

    ],
    retweetedStatus=null,
    userMentionEntities=[

    ],
    urlEntities=[

    ],
    hashtagEntities=[

    ],
    mediaEntities=[

    ],
    symbolEntities=[

    ],
    currentUserRetweetId=-1,
    user=UserJSONImpl{
      id=2242998313,
      name='_',
      screenName='DylanYamaha_',
      location='Marseille, France',
      description='null',
      isContributorsEnabled=false,
      profileImageUrl='http://pbs.twimg.com/profile_images/793110090727424002/9bLOivem_normal.jpg',
      profileImageUrlHttps='https://pbs.twimg.com/profile_images/793110090727424002/9bLOivem_normal.jpg',
      isDefaultProfileImage=false,
      url='null',
      isProtected=false,
      followersCount=12357,
      status=null,
      profileBackgroundColor='ABB8C2',
      profileTextColor='333333',
      profileLinkColor='89C9FA',
      profileSidebarFillColor='DDEEF6',
      profileSidebarBorderColor='FFFFFF',
      profileUseBackgroundImage=false,
      isDefaultProfile=false,
      showAllInlineMedia=false,
      friendsCount=87,
      createdAt=ThuDec1223: 15: 28CET2013,
      favouritesCount=6007,
      utcOffset=3600,
      timeZone='Amsterdam',
      profileBackgroundImageUrl='http://abs.twimg.com/images/themes/theme1/bg.png',
      profileBackgroundImageUrlHttps='https://abs.twimg.com/images/themes/theme1/bg.png',
      profileBackgroundTiled=false,
      lang='fr',
      statusesCount=2049,
      isGeoEnabled=false,
      isVerified=false,
      translator=false,
      listedCount=43,
      isFollowRequestSent=false,
      withheldInCountries=null
    },
    withHeldInCountries=null,
    quotedStatusId=-1,
    quotedStatus=null
  },
  userMentionEntities=[
    UserMentionEntityJSONImpl{
      name='_',
      screenName='DylanYamaha_',
      id=2242998313
    }
  ],
  urlEntities=[

  ],
  hashtagEntities=[

  ],
  mediaEntities=[

  ],
  symbolEntities=[

  ],
  currentUserRetweetId=-1,
  user=UserJSONImpl{
    id=393519159,
    name='Tiphaine.',
    screenName='LehmannTiphaine',
    location='France',
    description='Snapchat : tiphainelehmann | IG : tiphainelmn',
    isContributorsEnabled=false,
    profileImageUrl='http://pbs.twimg.com/profile_images/777174096958332928/yoz2aPp2_normal.jpg',
    profileImageUrlHttps='https://pbs.twimg.com/profile_images/777174096958332928/yoz2aPp2_normal.jpg',
    isDefaultProfileImage=false,
    url='null',
    isProtected=false,
    followersCount=145,
    status=null,
    profileBackgroundColor='000000',
    profileTextColor='333333',
    profileLinkColor='000000',
    profileSidebarFillColor='F3F3F3',
    profileSidebarBorderColor='000000',
    profileUseBackgroundImage=true,
    isDefaultProfile=false,
    showAllInlineMedia=false,
    friendsCount=200,
    createdAt=TueOct1819: 17: 04CEST2011,
    favouritesCount=3202,
    utcOffset=3600,
    timeZone='Paris',
    profileBackgroundImageUrl='http://pbs.twimg.com/profile_background_images/753348262/9d241c29a193586d5dc519838bded4c9.jpeg',
    profileBackgroundImageUrlHttps='https://pbs.twimg.com/profile_background_images/753348262/9d241c29a193586d5dc519838bded4c9.jpeg',
    profileBackgroundTiled=true,
    lang='fr',
    statusesCount=3462,
    isGeoEnabled=true,
    isVerified=false,
    translator=false,
    listedCount=3,
    isFollowRequestSent=false,
    withheldInCountries=null
  },
  withHeldInCountries=null,
  quotedStatusId=-1,
  quotedStatus=null
  }

I have had many problems in parsing these JSON strings. The first issue is that some tweets comes in more than one line. I resolved this by concatenating the lines of the same tweet. Then I have to replace all StatusJSONImpl and UserJSONImpl occurrences by "" to use the JSONObject constructor. The main issue is how to get all the tweet attributes. I used :

JSONObject jsonObj = new JSONObject(jsonline); //from twitter4j.JSONObject;

but whenever the tweet text contains characters like " ' ", I have:

twitter4j.JSONException: Expected a ',' or '}' at 117 [character 118 line 1]

Even when the text does not contain "'", I still unable to extract the time creation (createdAt) of the tweet. So I created a String:

String statusFromRaw = "{\"filter_level\": \"low\",";
    statusFromRaw+= "\"retweeted\":"+jsonObj.get("isRetweeted")+",";
.....

in order to recreate a cleaner status, and use the:

Status status = TwitterObjectFactory.createStatus(statusFromRaw);

But I still have problems in parsing many attributes that are null in most of the cases. Is there any suggestions ?

Answer 1

Don't use Twitter4j. Just hit Twitter's API yourself and use Jackson to map it to objects.

You shouldn't be trying to solve around a framework IMO.