How to query a graph of documents of different types at once in Marklogic?

Question

Background

I'm using NoSql database supporting graphs for the first time. It is a huge medical application handling thousands of patients. It is a greenfield project and we as a team are struggling with our persistence layer. We don't know how relationships should be represented and if we should use Triples to handle queries involving huge amount of data. We are using Java API .

Data structure

Imagine that there are 3 types of JSON documents in our Marklogic database: Patient, Event, File Evidence.

• There are thousands of patients in the application
• One patient can have multiple events associated with this patient (admitted, discharged, transferred, prescribed medications, added note, changed internal status etc.)
• each event can have multiple files attached to it as an evidence

Assume there are hundreds of thousands of patients, events and files.

Question

Is it possible to query patients with events and files at once ? Is using semantics (possible triples: 'patient has event', 'event has file') recommended in our case?

Our approach

We try to use triples to provide relationships between our documents, add them to one graph, use combination query to fetch IRI first and then in the second call fetch documents by IRI. We tried self-paced trainings and exploring https://github.com/marklogic/marklogic-samplestack but with no luck. Help of someone who has done that in the past and would like to share his experience would be great.

Answer 1

I your situation, keep in mind that you can also store the triples in each of the documents themselves (with the inferred subject being the document itself). Then in your example, you could be combining cts:triple-range-query with standard cts:search.

Example: If I had events and embedded a triple such as [this event-> ownedByPatient -> [iri/for/patiens#12345]

Then I could query: search for events filtered by fragments where the cts:triple-range-query states that the events are owned by patient 12345

This approach is a combination of semantics and MarkLogic search - using triples to link the appropriate types.

As for different types of documents, triples do not care what they are pointing at - an IRI of a person, event, etc. Its just about how you model you data itself and the ontology used to describe the relationships. So, you can also approach this as managed triples (not embedded) and treat it all as a graph database pointing at your content (like the approach you are describing)

Once you get further along, you may also decide to force restrictions on the types of relationships using RDF rules.

Answer 2

You've given us very little information to work with to answer such broad questions. Nevertheless, I'll do my best with what you gave.

One option is organize the data however is most intuitive to you, and use server-side Javascript (SJS) to combine the documents at query time into whatever you need for a particular query. That SJS could be in the form of a resource extension or search response transform . A resource extension has the advantage that it could do multiple queries across different document types and piece them together to form an answer. A search response transform, on the other hand will be given the results of only one query but could do additional queries as needed to bring in more data. Since you only have hundreds of thousands of records, you may not need to stress too much about raw speed.

If you plan to scale to millions of documents and want raw speed, you could keep everything you want to query about one patient in the patient record. That would allow you to find a patient by full-text search through all their records plus field-match on patient-specific data.

That assumes the only search results you ever want are patients. If you want something else, you'll need to let us know what other search results you might want.

When you say "attachment" I think of binary documents with scanned images, no metadata, and no full-text to search. Those would obviously be stored as separate binary documents. If they have metadata or full-text, you'll have to decide whether any of that should be in the big patient record for fast queries or in separate documents. All "attachment" documents that are separate JSON files could have a field that points to the patient by id.

I'd avoid triples at first. As David Ennis pointed out, you can combine triples and search, but it's a bit of a ninja move. One big JSON document per patient is much easier for most developers to understand.