MongoDB 4.0和SpringData中的事务

Question

I'm developing an application that reads a file that contains just text data. Some times, this files are small (around 500kb~1mb) but some times the file size can be 5bm~20mb. I need to make sure that all the data in the files is stored in the database, so the transactions come in.

Each line of the file represents information that will be validated and stored in that database. If an error occurs in this process, the transaction is aborted.

My file import process needs to be as follows:

Import Function

@Transactional
public void importFile(MultipartFile multipartFile) throws IOException, IllegalAccessException, NoSuchFieldException {
    try (BufferedReader file = new BufferedReader(new InputStreamReader(multipartFile.getInputStream()))) {
        this.validateFile(file);
    }

    try (BufferedReader arquivo = new BufferedReader(new InputStreamReader(multipartFile.getInputStream()))) {
        String line = file.readLine();

        ValidHeader validHeader = this.validHeaderService.toParse(line.split(","));
        this.validHeaderService.save(validHeader);

        while ((line = file.readLine()) != null)
            this.importFileContent(linha.split(",")); // -> This will validade and save others file content.
    }
}

ValidHeader.java

@Document(collection = "ValidHeader")
public class ValidHeader {

    @Id
    private ObjectId id;

    @Enumerated(EnumType.STRING)
    private TipoRegistro tipoRegistro;

    private int pontoVenda;

    //Constructor & getters & setters

}

For small files the transactions works fine, but my problem is for large files.

org.springframework.data.mongodb.UncategorizedMongoDbException: Command failed with error 251 (NoSuchTransaction): 'Transaction 1 has been aborted.' on server localhost:27017.

But debugging I realized that the main problem was

com.mongodb.MongoCommandException: Command failed with error 257 (TransactionTooLarge): 'Total size of all transaction operations must be less than 16793600. Actual size is 16794367' on server localhost:27017

From what I understand, Mongo interprets the entire transaction scope as a single document, why does it do this? and how can I solve these problems?

--ADTIONAL INFORMATION--

SpringDataMongo: 2.1.0.M3
MongoJavaDriver: 3.8
HibernateOgmMongoDb: 5.3.1.Final

Answer 1

I do not understand of MongoDB but why you persist each line from the file? Can you first read the whole file and after that persist it? Now you make transaction for each line and this is most probably not right. Yes you will have 20 mb in the memory at one time but that why we are using computers with gigabytes of ram. :) If you get out of memory for example you have 20 GB files then you can split them on chunks and persist them one by one.

Just to see how this will impact your code first make StringBuilder. Then in your while look append all of the lines. After the while then your your method for persisting into the database