Remove word after line from txt file is read

Question

I have this code which is used to read lines from a file and insert it into Postgre:

try {
            BufferedReader reader;
            try {
                reader = new BufferedReader(new FileReader(
                        "C:\\in_progress\\test.txt"));
                String line = reader.readLine();
                while (line != null) {
                    System.out.println(line);

                    Thread.sleep(100);
                    Optional<ProcessedWords> isFound = processedWordsService.findByKeyword(line);

                    if(!isFound.isPresent()){
                        ProcessedWords obj = ProcessedWords.builder()
                                .keyword(line)
                                .createdAt(LocalDateTime.now())
                                .build();
                        processedWordsService.save(obj);
                    }

                    // read next line
                    line = reader.readLine();
                }
                reader.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        catch (Exception e) {
            e.printStackTrace();
        }

How I can remove a line from the file after the line is inserted into SQL database?

Answer 1

For Reference

import java.io.*;

public class RemoveLinesFromAfterProcessed {
    public static void main(String[] args) throws Exception {
        String fileName = "TestFile.txt";
        String tempFileName = "tempFile";

        File mainFile = new File(fileName);
        File tempFile = new File(tempFileName);

        try (BufferedReader br = new BufferedReader(new FileReader(mainFile));
             PrintWriter pw = new PrintWriter(new FileWriter(tempFile))
        ) {
            String line;
            while ((line = br.readLine()) != null) {
                if (toProcess(line)) {  // #1
                    // process the code and add it to DB
                    // ignore the line (i.e, not add to temp file)
                } else {
                    // add to temp file.
                    pw.write(line + "\n");  // #2
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        }

        // delete the old file
        boolean hasDeleted = mainFile.delete();  // #3
        if (!hasDeleted) {
            throw new Exception("Can't delete file!");
        }
        boolean hasRenamed = tempFile.renameTo(mainFile);  // #4
        if (!hasRenamed) {
            throw new Exception("Can't rename file!");
        }

        System.out.println("Done!");
    }

    private static boolean toProcess(String line) {
        // any condition
        // sample condition for example
        return line.contains("aa");
    }
}

Read the file.
1: The condition to decide whether to delete the line or to retain it.
2: Write those line which you don't want to delete into the temporary file.
3: Delete the original file.
4: Rename the temporary file to original file name.

Answer 2

The basic idea is the same as what @Shiva Rahul said in his answer.

---

However another approach can be , store all the line numbers you want to delete in a list . After you have all the required line numbers that you want to delete you can use LineNumberReader to check and duplicate your main file.

Mostly I have used this technique in batch-insert where I was unsure how many lines may have a particular file plus before removal of lines had to do lot of processing. It may not be suitable for your case ,just posting the suggestion here if any one bumps to this thread.

private void deleteLines(String inputFilePath,String outputDirectory,List<Integer> lineNumbers) throws IOException{
    File tempFile = new File("temp.txt");
    File inputFile = new File(inputFilePath);

    // using LineNumberReader we can fetch the line numbers of each line
    LineNumberReader lineReader = new LineNumberReader(new FileReader(inputFile));

    //writter for writing the lines into new file
    BufferedWriter bufferedWriter = new BufferedWriter(new FileWriter(tempFile));
    String currentLine;
    while((currentLine = lineReader.readLine()) != null){

        //if current line number is present in removeList then put empty line in new file
        if(lineNumbers.contains(lineReader.getLineNumber())){
            currentLine="";
        }
        bufferedWriter.write(currentLine + System.getProperty("line.separator"));
    }
    //closing statements
    bufferedWriter.close();
    lineReader.close();

    //delete the main file and rename the tempfile to original file Name
    boolean delete = inputFile.delete();
    //boolean b = tempFile.renameTo(inputFile); // use this to save the temp file in same directory;
    boolean b = tempFile.renameTo(new File(outputDirectory+inputFile.getName()));
}

To use this function all you have to do is gather all the required line numbers. inputFilePath is the path of the source file and outputDirectory is where I want store the file after processing.

Answer 3

The issues with the current code:

• Adhere to the Single responsibility principle . Your code is doing too many things: reads from a file, performs findByKeyword() call, prepares the data and hands it out to store in the database. It's hardly can be thoroughly tested , and it's very difficult to maintain .
• Always use try-with-recourses to get your recourses closed at any circumstances.
• Don't catch the general Exception type - your code should only catch thous exceptions, which are more or less expected and for which there's a clear scenario on how to handle them. But don't catch all the exceptions.

How I can remove a line from the file after the line is inserted into SQL database?

It is not possible to remove a line from a file in the literal sense. You can override the contents of the file or replace it with another file.

My advice would be to file data in memory, process it, and then write the lines which should be retained into the same file ( ie override the file contents ).

You can argue that the file is huge and dumping it into memory would result in an OutOfMemoryError . And you want to read a line from a file, process it somehow, then store the processed data into the database and then write the line into a file... So that everything is done line by line, all actions in one go for a single line, and as a consequence all the code is crammed in one method. I hope that's not the case because otherwise it's a clear XY-problem .

Firstly, File System isn't a reliable mean of storing data, and it's not very fast. If the file is massive, then reading and writing it will a take a considerable amount of time, and it's done just it in order to use a tinny bit of information then this approach is wrong - this information should be stored and structured differently (ie consider placing into a DB) so that it would be possible to retrieve the required data, and there would be no problem with removing entries that are no longer needed.

But if the file is lean, and it doesn't contain critical data. Then it's totally fine, I will proceed assuming that it's the case.

The overall approach is to generate a map Map<String, Optional<ProcessedWords>> based on the file contents, process the non-empty optionals and prepare a list of lines to override the previous file content.

The code below is based on the NIO2 file system API.

public void readProcessAndRemove(ProcessedWordsService service, Path path) {
    
    Map<String, Optional<ProcessedWords>> result;
    
    try (var lines = Files.lines(path)) {
        result = processLines(service, lines);
    } catch (IOException e) {
        result = Collections.emptyMap();
        logger.log();
        e.printStackTrace();
    }
    
    List<String> linesToRetain = prepareAndSave(service, result);
    writeToFile(linesToRetain, path);
}

Processing the stream of lines from a file returned Files.lines() :

private static Map<String, Optional<ProcessedWords>> processLines(ProcessedWordsService service,
                                                                  Stream<String> lines) {
    return lines.collect(Collectors.toMap(
        Function.identity(),
        service::findByKeyword
    ));
}

Saving the words for which findByKeyword() returned an empty optional:

private static List<String> prepareAndSave(ProcessedWordsService service,
                                           Map<String, Optional<ProcessedWords>> wordByLine) {
    wordByLine.forEach((k, v) -> {
        if (v.isEmpty()) saveWord(service, k);
    });
    
    return getLinesToRetain(wordByLine);
}

private static void saveWord(ProcessedWordsService service, String line) {
    
    ProcessedWords obj = ProcessedWords.builder()
        .keyword(line)
        .createdAt(LocalDateTime.now())
        .build();
    service.save(obj);
}

Generating a list of lines to retain:

private static List<String> getLinesToRetain(Map<String, Optional<ProcessedWords>> wordByLine) {
    
    return wordByLine.entrySet().stream()
        .filter(entry -> entry.getValue().isPresent())
        .map(Map.Entry::getKey)
        .collect(Collectors.toList());
}

Overriding the file contents using Files.write() . Note: since varargs OpenOption isn't provided with any arguments, this call would be treated as if the CREATE , TRUNCATE_EXISTING , and WRITE options are present.

private static void writeToFile(List<String> lines, Path path) {
    try {
        Files.write(path, lines);
    } catch (IOException e) {
        logger.log();
        e.printStackTrace();
    }
}