用Java解析文本文件（大数据集）

Question

I have a text file, each line looks like this : (a movie reviews database)

product/productId: B00004CK40   review/userId: A39IIHQF18YGZA   review/profileName: C. A. M. Salas  review/helpfulness: 0/0 review/score: 4.0   review/time: 1175817600 review/summary: Reliable comedy review/text: Nice script, well acted comedy, and a young Nicolette Sheridan. Cusak is in top form.

I want to parse this file in order to retrieve :

• product/productId
• review/userId
• review/profileName
• review/helpfulness
• review/score
• review/time
• review/summary
• review/text

This information will be later encapsulated using MovieReview & Movie class.

public class MovieReview {

    private Movie movie;
    private String userId;
    private String profileName;
    private String helpfulness;
    private Date timestamp;
    private String summary;
    private String review;
...

Can anyone offer a proper & efficient way to parse this file (large dataset) ?

Thanks.

Answer 1

If it's a large dataset, you'll want to avoid loading the entire list into memory at once. I'd probably solve this with a handler for each row

public interface MovieReviewHandler {
    void handle(MovieReview revies);
}

Then you could parse as follows:

public class MovieReviewParser {
    public void parse(BufferedReader reader, MovieReviewHandler handler) {
        Pattern regex = Pattern.compile("product/productId:(.*)review/userId:(.*)review/profileName:(.*)"); // add other fields

        String line;
        while ((line = reader.readLine()) != null) {
            Matcher matcher = regex.matcher(line);
            if (!matcher.matches()) throw new RuntimeException();
            MovieReview review = new MovieReview();
            review.productId = matcher.group(1);
            review.userId = matcher.group(2);
            review.profileName = matcher.group(3);
            // etc

            handler.handle(review);
        }
    }
}