通过正则表达式或至少在没有外部库的情况下从 Java 中的 html 内容中获取推文

Question

How can I get the latest tweet from html content through either regex or without any external libraries. I am happy to use external libraries I would just prefer not to. I just wanted to know how it would be possible. I have written the html download part in Java and if anyone wants I will post it here. So I'll do a pit of pseudo code so that I'm not only targeting Java developers This is how my program looks so far.

1.)Load site("www.twitter.com/user123")
2.)Get initial string and write it to variable->buffer
3.)Loop start
4.)    Append string->buffer
5.)    If there is no more ->break
6.)print buffer

Obviously the variable buffer will now have raw html content. How can I sort this out to get the tweet. I have found a way but this is too inconsistent. The way I managed it was to find the string which held the tweets and get the content surrounded by the code. However there were too many changes in this section. What I mean is some content inside of it changes, like the font size. I could write multiple if statements but is there a neater solution?

Answer 1

Let me just start off by saying that jsoup is an amazing lightweight HTML parsing library. You can use things like CSS selectors and whatnot. If you ever decide to use a library jsoup will make your life a lot easier.

You can just query for the element with the class of TweetTextSize , then get the text content. This will give you all text, hashtags, and links. (The downside being pictures are also given in links)

Otherwise, you'll need to manually traverse the DOM. For example, use regex to find the beginning of the first TweetTextSize , and then just keep all text which is not between a < and a > .

Unfortunately, this second solution is volatile and may not work in the future, and you'll end up with a big glob of code which is overly complex and hard to debug.

Answer 2

Simple answer if you want a regex and not a sophisticated third party library.

<p[^>]+js-tweet-text[^>]*>(.*)</p>

Try the above on the "view-source" of https://twitter.com/a

Thanks.

EDIT: Source Code:

import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.util.regex.Matcher;
import java.util.regex.Pattern;


public class TweetSucker {

    public static void main(String[] args) throws Exception {
        URLConnection urlConnection = new URL("https://twitter.com/a").openConnection();
        InputStream inputStream = urlConnection.getInputStream();
        String encoding = urlConnection.getContentEncoding();

        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();

        byte[] buffer = new byte[8192];
        int len = 0;
        while ((len = inputStream.read(buffer)) != -1) {
            byteArrayOutputStream.write(buffer, 0, len);
        }
        String htmlContent = null;
        if (encoding != null) {
            htmlContent = new String(byteArrayOutputStream.toByteArray(), encoding);
        } else {
            htmlContent = new String(byteArrayOutputStream.toByteArray());
        }
        Pattern TWEET_PATTERN = Pattern.compile("(<p[^>]+js-tweet-text[^>]*>(.*)</p>)", Pattern.CASE_INSENSITIVE);
        Matcher matcher = TWEET_PATTERN.matcher(htmlContent);
        while (matcher.find()) {
            System.out.println("Tweet Found: " + matcher.group(2));
        }
    }
}

Answer 3

I know that you don't want any libraries but if you want something really quick this is working code in C#:

    using (IE browser = new IE())
    {
        browser.GoTo("https://twitter.com/user");
        List tweets = browser.List(Find.ById("stream-items-id"));
        if (tweets != null)
        {
            foreach (var tweet in tweets.ListItems)
            {
                var tweetText = tweet.Paras.FirstOrDefault();
                if (tweetText != null)
                {
                    MessageBox.Show(tweetText.Text);
                }
            }
        }
    }

This program uses a library called WatiN (if you use Visual Studio go to Tools Menu, select "NuGet Package Manager" then select "Manage Nuget Packages for Solution" and then select "Browse" and then type "Watin" on the search box, after you find the library hit "Install", after it is installed you just add a reference in your code and then a using statement:

using WatiN.Core;

You can just copy and paste the code I wrote above in a button handler and it'll work, you need to change the twitter.com/XXXXXX user name to list all their tweets. Modify code accordingly to meet your needs.