Reflected distributed data access
The present disclosure is directed to systems and methods for securely reflecting data from a local datastore. In one example embodiment, a user may connect local datastores to edge nodes that may facilitate communication between a third-party data management hub and the local datastores. A user operating the third-party data management hub may request to read/write certain data at the local datastores (that the user owns). Data from the local datastore may be generated in response to the request. The responsive data may be transmitted over an encrypted data session to the data management hub, where the responsive data is reflected to the user on a user interface. The responsive data is stored in a volatile memory store and not moved from the persistent memory at the local datastore. Upon termination of the encrypted data session, the volatile memory store is erased to preserve the privacy/security of the underlying data.
1 . A system for reflecting data from at least one datastore, comprising:
a memory configured to store non-transitory computer readable instructions; and
a processor communicatively coupled to the memory, wherein the processor, when executing the non-transitory computer readable instructions, is configured to:
receive at least one user instruction, wherein the at least one user instruction targets the at least one datastore;
transmit the at least one user instruction to a queue at an edge gateway,
wherein the queue is associated with an edge node,
wherein the edge node lacks a network path to the edge gateway,
wherein the edge node is associated with the at least one datastore, and
wherein the edge node polls the queue to receive the at least one user instruction;
initiate an encrypted data session with the edge node;
receive reflected data retrieved from the at least one datastore over the encrypted data session while the encrypted data session is uninterrupted,
wherein the reflected data is stored only in at least one volatile memory location associated with the encrypted data session; and
while the encrypted data session is in progress, display the reflected data.
2 . The system of claim 1 , the processor further configured to: when the encrypted data session is disconnected, erase the reflected data from the at least one volatile memory location.
3 . The system of claim 1 , the processor further configured to: receive at least one data-quality result from a metastore.
4 . The system of claim 3 , the processor further configured to: based on the at least one data-quality result from the metastore, transmit a second user instruction to the edge node.
5 . The system of claim 3 , wherein the at least one data-quality result is an error notice associated with the at least one datastore.
6 . The system of claim 1 , wherein the at least one user instruction is deserialized at the edge node prior to execution at the at least one datastore.
7 . The system of claim 1 , wherein the reflected data is encrypted prior to transmission over the encrypted data session.
8 . The system of claim 1 , wherein the at least one user instruction is at least one of: a read command and a write command.
9 . The system of claim 1 , the processor further configured to: generate a metastore.
10 . The system of claim 9 , wherein the metastore comprises at least one scoping requirement associated with underlying data stored in the at least one datastore.
11 . The system of claim 9 , wherein the metastore is associated with a second datastore.
12 . The system of claim 9 , wherein the metastore comprises at least one of: schema, user instructions, DQ analysis results, and platform data.
13 . The system of claim 1 , wherein the at least one user instruction is a view command to view information associated with a plurality of datastores in a unified user interface.
14 . The system of claim 13 , the processor further configured to: based on the view command, display platform data associated with the plurality of datastores in the unified user interface.
15 . A method for reflecting data from a datastore, comprising:
connecting at least one edge node to the datastore;
transmitting a first user instruction to process the data stored in the datastore for data quality,
wherein the first user instruction is transmitted to a queue at an edge gateway,
wherein the queue is associated with the at least one edge node,
wherein the at least one edge node lacks a network path to the edge gateway, and
wherein the at least one edge node polls the queue to receive the at least one user instruction;
receiving at least one data quality result from the at least one edge node;
based on the at least one data quality result received from the at least one edge node, transmitting a second user instruction to read a portion of the data stored in the datastore;
initiating an encrypted data session;
receiving reflected data from the at least one edge node over the encrypted data session while the encrypted data session is uninterrupted,
wherein the reflected data is stored only in at least one volatile memory location associated with the encrypted data session; and
displaying the reflected data on a user interface.
16 . The method of claim 15 , further comprising erasing the reflected data from the at least one volatile memory location upon disconnection of the encrypted data session.
17 . The method of claim 15 , further comprising erasing the reflected data from the at least one volatile memory location upon disconnection of the user interface.
18 . A non-transitory computer-readable media storing computer executable instructions that when executed cause a computing system to perform the steps for reflecting data from a datastore, comprising:
connecting at least one edge node to the datastore;
transmitting a first user instruction to process the data stored in the datastore for data quality,
wherein the first user instruction is transmitted to a queue at an edge gateway,
wherein the queue is associated with the at least one edge node,
wherein the at least one edge node lacks a network path to the edge gateway, and
wherein the at least one edge node polls the queue to receive the at least one user instruction;
receiving at least one data quality result from the at least one edge node;
based on the at least one data quality result received from the at least one edge node, transmitting a second user instruction to read a portion of the data stored in the datastore;
initiating an encrypted data session;
receiving reflected data from the at least one edge node over the encrypted data session while the encrypted data session is uninterrupted,
wherein the reflected data is stored only in at least one volatile memory location associated with the encrypted data session; and
displaying the reflected data on a user interface.
19 . The non-transitory computer-readable medium of claim 18 , wherein the at least one data quality result from the at least one edge node is an error notice.