Use spark RDD as a source of data in a REST API

Question

There is a graph that computes on Spark and stores to Cassandra.
Also there is a REST API which has endpoint to get graph node with edges and edges of edges.
This second degree graph may include up to 70000 nodes.
Currently uses Cassandra as the database, but to extract a lot of data by key from Cassandra takes much time and resources.
We tried TitanDB, Neo4j and OriendDB to improve performance but Cassandra showed the best results.

Now there is another idea. Persist RDD (or may be GrapgX object) in the API service and on API call filter necessary data from persisted RDD.
I guess that it will work fast while RDD fits in memory, but in the case that it caches to disk it will work like a full scan (eg full scan parquet file). Also I expect that we will face to these issues:

• memory leak in spark;
• updating this RDD (unpersist previous, read new and persist new one) will require stop API;
• concurrent using this RDD will require manually manage CPU resources.

Do anybody have such experience?

Answer 1

Spark is NOT a storage engine. Unless you will process big amount of data each time, you should consider:

• In-memory data grids - Hazelcast, Apache Ignite, Coherence, GigaSpaces, etc.
• Cassandra in-memory - https://docs.datastax.com/en/datastax_enterprise/4.5/datastax_enterprise/inMemory.html
• search for "in-memory" option in other framework/database