Memory graph query engine with persisted storage
Various examples of improving an in-memory graph query engine using a persisted storage component are provided. The method includes updating data stored in an in-memory graph query engine and, based on updating the data, converting the data to a plain text form that may be more efficiently stored in the persistent storage component. The method further includes updates to additional in-memory graph query engines from the persistent storage component such that in-memory data stored in the graph query engines is synchronized.
1. A computer-implemented method, comprising:
receiving, at a first in-memory graph query engine, an electronic request;
processing, by a first in-memory graph query engine, the received electronic request resulting in modified in-memory data, wherein processing the electronic request comprises:
based on the electronic request being a request to add data to the first in-memory graph query engine, adding a new node and a new edge between nodes as a key-value pair, the key representing a vertex identifier and a schema element identifier that represents a property name of a node;
based on the electronic request being a request to update data stored in the first in-memory graph query engine, updating a key-value pair in the first in-memory graph query engine;
based on the electronic request being a request to remove data stored in the first in-memory graph query engine, deleting a key-value pair from the first in-memory graph query engine;
based on processing the received electronic request, converting, by a data converter of the first in-memory graph query engine, the modified in-memory data to a document format, the data converter being configured to convert key-value pairs stored in the first in-memory graph query engine, to a document format with native types of a persistent storage; and
outputting the modified in-memory data in the document format to the persistent storage.
2. The computer-implemented method of claim 1 , further comprising reading or writing, by a secondary data processor, documents stored in the persistent storage that are agnostic of the first in-memory graph query engine.
3. The computer-implemented method of claim 1 , wherein the persistent storage contains data of all vertex and edge states of the first in-memory graph query engine in the document format.
4. The computer-implemented method of claim 1 , wherein the electronic request is an online transaction processing (OLTP) request.
5. The computer-implemented method of claim 1 , wherein converting the key-value pair to a document format occurs by a single atomic write.
6. The computer-implemented method of claim 1 , further comprising:
executing, by a load balancer, a machine learning algorithm to balance the electronic request to the first in-memory graph query engine of a plurality of in-memory graph query engines, the load balancer being configured to distribute requests across the plurality of in-memory graph query engines such that no particular graph query engine receives a number of requests that overloads that graph query engine; and
distributing, by the load balancer, the electronic request to the first in-memory graph query engine.
7. The computer-implemented method of claim 6 , further comprising:
maintaining, by the persistent storage, a change feed, wherein the change feed is a record of changes made to data stored in the persistent storage;
detecting a new change to the data stored in the persistent storage;
updating, by the persistent storage, the change feed based on the converted in-memory data from the first in-memory graph query engine; and
outputting the updated change feed to a second in-memory graph query engine of the plurality of in-memory graph query engines.
8. A computer-implemented method, comprising:
causing a first in-memory graph query engine to:
receive an electronic request;
process the received electronic request resulting in modified in-memory data, wherein processing the electronic request includes:
based on the electronic request being a request to add data to the first in-memory graph query engine, adding a new node and a new edge between nodes as a key-value pair, the key representing a vertex identifier and a schema element identifier that represents a property name of a node;
based on the electronic request being a request to update data stored in the first in-memory graph query engine, updating a key-value pair in the first in-memory graph query engine;
based on the electronic request being a request to remove data stored in the first in-memory graph query engine, deleting a key-value pair from the first in-memory graph query engine;
convert, by a data converter, the modified in-memory data to a document format, the data converter being configured to convert key-value pairs stored in the first in-memory graph query engine, to a document format with native types of a persistent storage;
output the converted in-memory data in the document format to the persistent storage; and
updating persistent storage data at the persistent storage with the converted in-memory data in the document format.
9. The computer-implemented method of claim 8 , further comprising:
causing a secondary data processor to read or write documents stored in the persistent storage that are agnostic of the first in-memory graph query engine.
10. The computer-implemented method of claim 8 , wherein the persistent storage contains data of all vertex and edge states of the first in-memory graph query engine in the document format.
11. The computer-implemented method of claim 8 , wherein the electronic request is an online transaction processing (OLTP) request.
12. The computer-implemented method of claim 8 , wherein converting the key-value pair to a document format occurs by a single atomic write.
13. The computer-implemented method of claim 8 , further comprising:
causing a load balancer to:
execute a machine learning algorithm to balance the electronic request to the first in-memory graph query engine of a plurality of in-memory graph query engines, the load balancer being configured to distribute requests across the plurality of in-memory graph query engines such that no particular graph query engine receives a number of requests that overloads that graph query engine; and
distribute the electronic request to the first in-memory graph query engine.
14. The computer-implemented method of claim 13 , further comprising:
maintaining, by the persistent storage, a change feed, wherein the change feed is a record of changes made to the persistent storage data;
detecting a new change to the persistent storage data at the persistent storage;
updating, by the persistent storage, the change feed based on the converted in-memory data from the first in-memory graph query engine; and
outputting the updated change feed to a second in-memory graph query engine of the plurality of in-memory graph query engines.
15. A computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to:
receive, at a first in-memory graph query engine, an electronic request;
process, by a first in-memory graph query engine, the received electronic request resulting in modified in-memory data, wherein processing the electronic request comprises:
based on the electronic request being a request to add data to the first in-memory graph query engine, adding a new node and a new edge between nodes as a key-value pair, the key representing a vertex identifier and a schema element identifier that represents a property name of a node;
based on the electronic request being a request to update data stored in the first in-memory graph query engine, updating a key-value pair in the first in-memory graph query engine;
based on the electronic request being a request to remove data stored in the first in-memory graph query engine, deleting a key-value pair from the first in-memory graph query engine;
based on processing the received electronic request, convert, by a data converter of the first in-memory graph query engine, the modified in-memory data to a document format, the data converter being configured to convert key-value pairs stored in the first in-memory graph query engine, to a document format with native types of a persistent storage; and
output the modified in-memory data in the document format to the persistent storage.
16. The computer-readable storage medium of claim 15 , further storing instructions to read or write, by a secondary data processor, documents stored in the persistent storage that are agnostic of the first in-memory graph query engine.
17. The computer-readable storage medium of claim 15 , wherein the persistent storage contains data of all vertex and edge states of the first in-memory graph query engine in the document format.
18. The computer-readable storage medium of claim 15 , wherein converting the key-value pair to a document format occurs by a single atomic write.
19. The computer-readable storage medium of claim 15 , further storing instructions that, when executed by the processor, further cause the processor to:
execute, by a load balancer, a machine learning algorithm to balance the electronic request to a first in-memory graph query engine of a plurality of in-memory graph query engines, the load balancer being configured to distribute requests across the plurality of in-memory graph query engines such that no particular graph query engine receives a number of requests that overloads that graph query engine; and
distribute, by the load balancer, the electronic request to the first in-memory graph query engine.
20. The computer-readable storage medium of claim 19 , further storing instructions that, when executed by the processor, further cause the processor to:
maintain, by the persistent storage, a change feed, wherein the change feed is a record of changes made to data stored in the persistent storage;
detect a new change to the data stored in the persistent storage;
update, by the persistent storage, the change feed based on the converted in-memory data from the first in-memory graph query engine; and
output the updated change feed to a second in-memory graph query engine of the plurality of in-memory graph query engines.