java-全文反向索引定义一个词

Question

I am working on a simple full text inverted index trying to build an index of words that I extract from PDF files. I am using PDFBox library to achieve this.

However, I would like to know how does one define a definition of word to index.The way my indexing works is define every word with a space is a word token. For example,

This string, is a code.

In this case: the index table would contain

This
string,
is
a
code.

The flaw here is for like string, , it comes with a comma where I think string would just be sufficient enough because nobody searches string, or code.

Back to my question, is there a specific rule there I could use to define my word token in a way to prevent this kind of issue with what I have ?

Code:

File folder = new File("D:\\PDF1");
File[] listOfFiles = folder.listFiles();

for (File file : listOfFiles) {
   if (file.isFile()) {
      HashSet<String> uniqueWords = new HashSet<>();
      String path = "D:\\PDF1\\" + file.getName();
      try (PDDocument document = PDDocument.load(new File(path))) {    
          if (!document.isEncrypted()) {    
             PDFTextStripper tStripper = new PDFTextStripper();
             String pdfFileInText = tStripper.getText(document);
             String lines[] = pdfFileInText.split("\\r?\\n");
             for(String line : lines) {
                String[] words = line.split(" ");    
                for (String word : words) {
                    uniqueWords.add(word);   
                }

             }                            
          }
       } catch (IOException e) {
         System.err.println("Exception while trying to read pdf document - " + e);
       }
   }
}

Answer 1

If you wanted to remove all punctuation you could do:

for(String word : words) {
    uniqueWords.add(word.replaceAll("[.,!?]", ""));
}

Which will replace all periods, commas, exclamation marks, and question marks.

---

If you also want to get rid of quotes you can do:

uniqueWords.add(word.replaceAll("[.,?!\"]", "")

Answer 2

Yes. You can use replaceAll method to get rid of non-word characters like this:

uniqueWords.add(word.replaceAll("([\\W]+$)|(^[\\W]+)", ""));