Document disappears in destination after ETL in RavenDB

Question

I have to migrate and transform some data between two RavenDB instances. Basically there are two versions of the same app and the client wants to move to the new version, but to keep the data from the old one. I defined the transformations scripts and tested on the database level successfully. Now that I am testing on QA environment a specific document that holds information about the user (which is part of the collection I want to move) for the app is disappears after the task is completed.

In order to start small I have put a condition in the ETL to only migrate a specific document. This document is moved properly and the transformation is working as expected, but the app crashes as it is missing the current user data. When I put it in manually it works again.

Do you have any ideas what might be causing the issue?

Here is a sample of my transformation script:

if (this.MDMId == "dac50220-7c14-eb11-a813-000d3aacb01f"){
    this.CompanySectorCode = "";
    this.CompanySubSectorCode = "";
...
    
    loadToProfiles(this);
}

With this I am expecting that the only modified document will be the one for which the condition evaluates to true and I confirmed that the document that is disappearing has a different MDMId.

Any rip is appreciated, Thank you!

Answer 1

RavenDB uses delete and put to handle document modifications. You can change the deletion behavior by define deleteDocumentsOf<CollectionName>Behavior . https://ravendb.net/docs/article-page/4.2/csharp/server/ongoing-tasks/etl/raven#deletions

Note that there is a little difference if you ETL to the same collection (from users to users for example) or you ETL to a different collection (from user to people for example). For the same collection, RavenDB uses the same id and the deletion command executes for this specific id. For a different collection, the source uses the id as a prefix and the actual id generated on the target so deletions execute by the prefix.

In addition, RavenDB assumes that it is the only writer to the target database.