In-memory graph query engine with persisted storage
Various examples of improving an in-memory graph query engine using a persisted storage component are provided. The method includes updating data stored in an in-memory graph query engine and, based on updating the data, converting the data to a plain text form that may be more efficiently stored in the persistent storage component. The method further includes updates to additional in-memory graph query engines from the persistent storage component such that in-memory data stored in the graph query engines is synchronized.
1. A computer-implemented method, comprising:
receiving, at a first in-memory graph query engine, an electronic request;
processing, by the first in-memory graph query engine, the received electronic request, wherein processing the electronic request includes updating in-memory data stored in the first in-memory graph query engine;
converting, by the first in-memory graph query engine, the updated in-memory data to a document format;
outputting, by the first in-memory graph query engine, the converted in-memory data in the document format to a persistent storage;
causing the persistent storage to:
update persistent storage data at the persistent storage with the converted in-memory data in the document format;
update, by the persistent storage, a change feed based on the updated persistent storage data, wherein the change feed is a record of changes made to the persistent storage data;
output the updated change feed to a second in-memory graph query engine; and
causing a second in-memory graph query engine to, in response to receiving, from the persistent storage, the change feed, updating in-memory data stored in the second in-memory graph query engine based on the record of changes in the change feed, such that the in-memory data stored in the second in-memory graph query engine is the same as the in-memory data stored in the first in-memory graph query engine.
2. The computer-implemented method of claim 1 , wherein a graph query compute includes the first in-memory graph query engine and the second in-memory graph query engine.
3. The computer-implemented method of claim 1 , wherein updating the in-memory data stored in the first in-memory graph query engine includes one or more of adding a key-value pair to the first in-memory graph query engine, updating a key-value pair in the first in-memory graph query engine, and deleting a key-value pair from the first in-memory graph query engine.
4. The computer-implemented method of claim 1 , wherein converting the updated in-memory data to the document format further comprises converting, by a data converter implemented on the first in-memory graph query engine, the in-memory data stored in the first in-memory graph query engine from a binary form to a plain text form.
5. The computer-implemented method of claim 1 , wherein outputting the updated in-memory data to the persistent storage includes outputting the updated in-memory data to at least two partitions in the persistent storage.
6. The computer-implemented method of claim 1 , wherein outputting the updated change feed to the second in-memory graph query engine occurs at a regular interval of time.
7. The computer-implemented method of claim 1 , wherein outputting the updated change feed to the second in-memory graph query engine occurs upon a threshold number of changes to the change feed being reached.
8. A computer-implemented method, comprising:
causing a first in-memory graph query engine to:
receive an electronic request;
process the received electronic request, wherein processing the electronic request includes updating in-memory data stored in the first in-memory graph query engine;
convert the updated in-memory data to a document format; and
output the converted in-memory data in the document format to a persistent storage;
updating persistent storage data at the persistent storage with the converted in-memory data in the document format;
updating, by the persistent storage, a change feed based on the updated persistent storage data, wherein the change feed is a record of changes made to the persistent storage data;
outputting the updated change feed to a second in-memory graph query engine; and
causing a second in-memory graph query engine to, in response to receiving, from the persistent storage, the change feed, update in-memory data stored in the second in-memory graph query engine based on the record of changes in the change feed, such that the in-memory data stored in the second in-memory graph query engine is the same as the in-memory data stored in the first in-memory graph query engine.
9. The computer-implemented method of claim 8 , wherein, to update the in-memory data stored in the first in-memory graph query engine, the first in-memory graph query engine is further configured to perform one or more of adding a key-value pair to the first in-memory graph query engine, updating a key-value pair in the first in-memory graph query engine, and deleting a key-value pair from the first in-memory graph query engine.
10. The computer-implemented method of claim 8 , wherein the first in-memory graph query engine further includes a data converter configured to convert the in-memory data stored in the first in-memory graph query engine from a binary form to a plain text form.
11. The computer-implemented method of claim 8 , wherein, to output the in-memory data to the persistent storage, the first in-memory graph query engine outputs the updated in-memory data to at least two partitions in the persistent storage.
12. The computer-implemented method of claim 8 , wherein the persistent storage is further configured to output the updated change feed to the second in-memory graph query engine at a regular interval of time.
13. The computer-implemented method of claim 8 , wherein the persistent storage is further configured to output the updated change feed to the second in-memory graph query engine upon a threshold number of changes to the change feed being reached.
14. The computer-implemented method of claim 8 , wherein the persistent storage is further configured to output the updated change feed to the second in-memory graph query engine each time the change feed is changed.
15. One or more computer-readable storage media storing instructions that, when
executed by a processor, cause the processor to:
receive, at a first in-memory graph query engine, an electronic request;
process, by the first in-memory graph query engine, the received electronic request, wherein processing the electronic request includes updating in-memory data stored in the first in-memory graph query engine;
convert, by the first in-memory graph query engine, the updated in-memory data to a document format;
output, by the first in-memory graph query engine, the converted in-memory data in the document format to a persistent storage;
update persistent storage data at the persistent storage with the converted in-memory data in the document format;
update, by the persistent storage, a change feed based on the updated persistent storage data, wherein the change feed is a record of changes made to the persistent storage data;
output the updated change feed to a second in-memory graph query engine; and
in response to receiving, at the second in-memory graph query engine, from the persistent storage, the change feed, update in-memory data stored in the second in-memory graph query engine based on the record of changes in the change feed, such that the in-memory data stored in the second in-memory graph query engine is the same as the in-memory data stored in the first in-memory graph query engine.
16. The one or more computer-readable storage media of claim 15 , further storing instructions to update the in-memory data stored in the first in-memory graph query engine that, when executed by the processor, further cause the processor to add a key-value pair to the first in-memory graph query engine, update a key-value pair in the first in-memory graph query engine, or delete a key-value pair from the first in-memory graph query engine.
17. The one or more computer-readable storage media of claim 15 , further storing instructions to convert the updated in-memory data to the document format that, when executed by the processor, further cause the processor to convert the in-memory data stored in the first in-memory graph query engine from a binary form to a plain text form.
18. The one or more computer-readable storage media of claim 15 , further storing instructions to output the updated in-memory data to the persistent storage that, when executed by the processor, further cause the processor to output the updated in-memory data to at least two partitions in the persistent storage.
19. The one or more computer-readable storage media of claim 15 , further storing instructions that, when executed by the processor, further cause the processor to output the updated change feed to the second in-memory graph query engine at a regular interval of time.
20. The one or more computer-readable storage media of claim 15 , further storing instructions that, when executed by the processor, further cause the processor to output the updated change feed to the second in-memory graph query engine occurs upon a threshold number of changes to the change feed being reached.