Managing data objects for graph-based data structures
Various embodiments provide methods, systems, apparatus, computer program products, and/or the like for managing, ingesting, monitoring, updating, and/or extracting/retrieving information/data associated with an electronic record (ER) stored in an ER data store and/or accessing information/data from the ER data store, wherein the ERs are generated, updated/modified, and/or accessed via a graph-based domain ontology.
1. A method for data processing by a data processing module, the method comprising:
receiving, by one or more processors, a plurality of messages via one or more communications interfaces, wherein each message corresponds to an entity;
for each message of the plurality of messages:
(a) generating, by the one or more processors, an application programming interface (API) request to an identity matching service, wherein the API request comprises a first attribute and a second attribute,
(b) receiving, by the one or more processors, an API response comprising an entity identifier corresponding to the entity, and
(c) generating, by the one or more processors, a data artifact packet data object; and
responsive to at least one of (a) an incoming message volume satisfying an incoming message volume threshold or (b) a resource availability satisfying a resource availability threshold, dynamically changing, by the one or more processors, the data processing used by the data processing module from real-time data processing or near real-time data processing to deferred data processing, wherein the deferred data processing comprises:
(i) storing, by the one or more processors, each data artifact packet data object generated for each message of the plurality of messages in a first datastore for deferred data processing, wherein each data artifact packet data object is (1) identifiable by the entity identifier, and (2) not provided to the data processing module for data processing until a user interaction event associated with the entity identifier is detected,
(ii) detecting, by the one or more processors, a user interaction event,
(iii) responsive to detecting the user interaction event, retrieving, by the one or more processors, each data artifact packet data object from the first datastore associated with the entity identifier, and
(iv) providing, by the one or more processors, each data artifact packet data object associated with the entity identifier to the data processing module for data processing, wherein, after processing by the data processing module, data from each data artifact packet data object is stored in a second datastore and in association with the entity identifier.
2. The method of claim 1 , wherein (a) the user interaction event authenticates a user via an interface through which the user has access to data stored in association with a user profile corresponding to the entity identifier and (b) the user is an owner corresponding to the entity identifier.
3. The method of claim 1 , wherein (a) the user interaction event authenticates a user via an interface through which the user has access to data stored in association with a user profile corresponding to the entity identifier and (b) the user has a relationship defined within a graph-based domain ontology with an entity that is an owner corresponding to the entity identifier.
4. The method of claim 3 , wherein detecting the user interaction event comprises:
determining that the user has been authenticated via the interface;
identifying one or more relationships of which the user is a participant based at least in part on the entity identifier corresponding to the user; and
determining, based at least in part on the one or more relationships, that the user has access to data corresponding to the entity identifier.
5. The method of claim 4 , wherein the one or more relationships are identified based at least in part on a graph data structure of the graph-based domain ontology.
6. The method of claim 1 , wherein the user interaction event is detected based at least in part on historical behavior of one or more users having access to data corresponding to the entity identifier.
7. The method of claim 1 , further comprising, responsive to at least one of (a) the incoming message volume no longer satisfying the incoming message volume threshold or (b) the resource availability no longer satisfying the resource availability threshold, dynamically changing the data processing used by the data processing module from deferred data processing to real-time data processing or near real-time data processing.
8. The method of claim 1 further comprising, for each of the plurality of data artifact packet data objects:
generating a container tree data structure comprising a data artifact packet container node as a root node, wherein (a) the container tree data structure is generated based at least in part on the data artifact packet data object, (b) the container tree data structure comprises a plurality of container nodes that are descendants of the root node based at least in part on the data artifact packet data object, and (c) each container node of the plurality of container nodes corresponds to one pair of a plurality of observable-value pairs;
traversing the container tree data structure in a depth-first traversal to generate a data transfer object for each of container node of the plurality of container nodes, wherein each data transfer object corresponds to one pair of the plurality of observable-value pairs; and
providing at least one of the plurality of data transfer objects for use in performing a database update function.
9. The method of claim 8 , wherein generating the container tree data structure comprises:
determining a type of container node for an observable-value pair of the plurality of observable-value pairs based at least in part on a graph-based domain ontology;
determining whether a container node having the determined type is present in the container tree data structure; and
responsive to determining that a container node having the determined type is present in the container tree data structure, storing the observable-value pair in the container node.
10. The method of claim 8 , wherein generating the container tree data structure comprises:
determining a type of container node for an observable-value pair of the plurality of observable-value pairs based at least in part on a graph-based domain ontology;
determining whether a container node having the determined type is present in the container tree data structure; and
responsive to determining that a container node having the determined type is not present in the container tree data structure:
(a) constructing the container node having the determine type,
(b) inserting the container node into an appropriate position in the container tree data structure, wherein the appropriate position in the container tree data structure is determined based at least in part on the graph-based domain ontology, and
(c) storing the observable-value pair in the container node.
11. A system for data processing by a data processing module, the system comprising one or more processors, one or more memory storage areas comprising program code, the one or more memory storage areas and the program code configured to, with the one or more processors, cause the system to at least:
receive a plurality of messages via one or more communications interfaces, wherein each message corresponds to an entity;
for each message of the plurality of messages:
(a) generate an application programming interface (API) request to an identity matching service, wherein the API request comprises a first attribute and a second attribute,
(b) receive an API response comprising an entity identifier corresponding to the entity, and
(c) generate a data artifact packet data object; and
responsive to at least one of (a) an incoming message volume satisfying an incoming message volume threshold or (b) a resource availability satisfying a resource availability threshold, dynamically change the data processing used by the data processing module from real-time data processing or near real-time data processing to deferred data processing, wherein the deferred data processing comprises:
(i) store each data artifact packet data object generated for each message of the plurality of messages in a first datastore for deferred data processing, wherein each data artifact packet data object is (1) identifiable by the entity identifier, and (2) is not provided to the data processing module for data processing until a user interaction event associated with the entity identifier is detected,
(ii) detect a user interaction event,
(iii) responsive to detecting the user interaction event, retrieve each data artifact packet data object from the first datastore associated with the entity identifier, and
(iv) each data artifact packet data object associated with the entity identifier to the data processing module for data processing, wherein, after processing by the data processing module, data from each data artifact packet data object is stored in a second datastore and in association with the entity identifier.
12. The system of claim 11 , wherein (a) the user interaction event authenticates a user via an interface through which the user has access to data stored in association with a user profile corresponding to the entity identifier and (b) the user is an owner corresponding to the entity identifier.
13. The system of claim 11 , wherein (a) the user interaction event authenticates a user via an interface through which the user has access to data stored in association with a user profile corresponding to the entity identifier and (b) the user has a relationship defined within a graph-based domain ontology with an entity that is an owner corresponding to the entity identifier.
14. The system of claim 13 , wherein detecting the user interaction event comprises:
determining that the user has been authenticated via the interface;
identifying one or more relationships of which the user is a participant based at least in part on the entity identifier corresponding to the user; and
determining, based at least in part on the one or more relationships, that the user has access to data corresponding to the entity identifier.
15. The system of claim 14 , wherein the one or more relationships are identified based at least in part on a graph data structure of the graph-based domain ontology.
16. The system of claim 11 , wherein the user interaction event is detected based at least in part on historical behavior of one or more users having access to data corresponding to the entity identifier.
17. The system of claim 11 , wherein the one or more memory storage areas and the program code configured to, with the one or more processors, further cause the system to, responsive to at least one of (a) the incoming message volume no longer satisfying the incoming message volume threshold or (b) the resource availability no longer satisfying the resource availability threshold, dynamically changing the data processing used by the data processing module from deferred data processing to real-time data processing or near real-time data processing.
18. The system of claim 11 , wherein the one or more memory storage areas and the program code configured to, with the one or more processors, further cause the system to, for each of the plurality of data artifact packet data objects:
generate a container tree data structure comprising a data artifact packet container node as a root node, wherein (a) the container tree data structure is generated based at least in part on the data artifact packet data object, (b) the container tree data structure comprises a plurality of container nodes that are descendants of the root node based at least in part on the data artifact packet data object, and (c) each container node of the plurality of container nodes corresponds to one pair of a plurality of observable-value pairs;
traverse the container tree data structure in a depth-first traversal to generate a data transfer object for each of container node of the plurality of container nodes, wherein each data transfer object corresponds to one pair of the plurality of observable-value pairs; and
provide at least one of the plurality of data transfer objects for use in performing a database update function.
19. The system of claim 18 , wherein generating the container tree data structure comprises:
determining a type of container node for an observable-value pair of the plurality of observable-value pairs based at least in part on a graph-based domain ontology;
determining whether a container node having the determined type is present in the container tree data structure; and
responsive to determining that a container node having the determined type is present in the container tree data structure, storing the observable-value pair in the container node.
20. The system of claim 18 , wherein generating the container tree data structure comprises:
determining a type of container node for an observable-value pair of the plurality of observable-value pairs based at least in part on a graph-based domain ontology;
determining whether a container node having the determined type is present in the container tree data structure; and
responsive to determining that a container node having the determined type is not present in the container tree data structure:
(a) constructing the container node having the determine type,
(b) inserting the container node into an appropriate position in the container tree data structure, wherein the appropriate position in the container tree data structure is determined based at least in part on the graph-based domain ontology, and
(c) storing the observable-value pair in the container node.
21. A computer program product for data processing by a data processing module, the computer program product comprising at least one non-transitory computer-readable storage medium having computer-readable program code stored therein, the computer-readable program code configured to at least:
receive a plurality of messages via one or more communications interfaces, wherein each message corresponds to an entity;
for each message of the plurality of messages:
(a) generate an application programming interface (API) request to an identity matching service, wherein the API request comprises a first attribute and a second attribute,
(b) receive an API response comprising an entity identifier corresponding to the entity, and
(c) generate a data artifact packet data object; and
responsive to at least one of (a) an incoming message volume satisfying an incoming message volume threshold or (b) a resource availability satisfying a resource availability threshold, dynamically change the data processing used by the data processing module from real-time data processing or near real-time data processing to deferred data processing, wherein the deferred data processing comprises:
(i) store each data artifact packet data object generated for each message of the plurality of messages in a first datastore for deferred data processing, wherein each data artifact packet data object is (1) identifiable by the entity identifier, and (2) not provided to the data processing module for data processing until a user interaction event associated with the entity identifier is detected,
(ii) detect a user interaction event,
(iii) responsive to detecting the user interaction event, retrieve each data artifact packet data object from the first datastore associated with the entity identifier, and
(iv) each data artifact packet data object associated with the entity identifier to the data processing module for data processing, wherein, after processing by the data processing module, data from each data artifact packet data object is stored in a second datastore and in association with the entity identifier.
22. The computer program product of claim 21 , wherein (a) the user interaction event authenticates a user via an interface through which the user has access to data stored in association with a user profile corresponding to the entity identifier and (b) the user is an owner corresponding to the entity identifier.
23. The computer program product of claim 21 , wherein (a) the user interaction event authenticates a user via an interface through which the user has access to data stored in association with a user profile corresponding to the entity identifier and (b) the user has a relationship defined within a graph-based domain ontology with an entity that is an owner corresponding to the entity identifier.
24. The computer program product of claim 23 , wherein detecting the user interaction event comprises:
determining that the user has been authenticated via the interface;
identifying one or more relationships of which the user is a participant based at least in part on the entity identifier corresponding to the user; and
determining, based at least in part on the one or more relationships, that the user has access to data corresponding to the entity identifier.
25. The computer program product of claim 24 , wherein the one or more relationships are identified based at least in part on a graph data structure of the graph-based domain ontology.
26. The computer program product of claim 21 , wherein the user interaction event is detected based at least in part on historical behavior of one or more users having access to data corresponding to the entity identifier.
27. The computer program product of claim 21 , wherein the computer-readable program code is further configured to, responsive to at least one of (a) the incoming message volume no longer satisfying the incoming message volume threshold or (b) the resource availability no longer satisfying the resource availability threshold, dynamically change the data processing used by the data processing module to real-time data processing or near real-time data processing.
28. The computer program product of claim 21 , wherein the computer-readable program code is further configured to, for each of the plurality of data artifact packet data objects:
generate a container tree data structure comprising a data artifact packet container node as a root node, wherein (a) the container tree data structure is generated based at least in part on the data artifact packet data object, (b) the container tree data structure comprises a plurality of container nodes that are descendants of the root node based at least in part on the data artifact packet data object, and (c) each container node of the plurality of container nodes corresponds to one pair of a plurality of observable-value pairs;
traverse the container tree data structure in a depth-first traversal to generate a data transfer object for each of container node of the plurality of container nodes, wherein each data transfer object corresponds to one pair of the plurality of observable-value pairs; and
provide at least one of the plurality of data transfer objects for use in performing a database update function.
29. The computer program product of claim 28 , wherein generating the container tree data structure comprises:
determining a type of container node for an observable-value pair of the plurality of observable-value pairs based at least in part on a graph-based domain ontology;
determining whether a container node having the determined type is present in the container tree data structure; and
responsive to determining that a container node having the determined type is present in the container tree data structure, storing the observable-value pair in the container node.
30. The computer program product of claim 28 , wherein the computer-readable program code is further configured to, wherein generating the container tree data structure comprises:
determining a type of container node for an observable-value pair of the plurality of observable-value pairs based at least in part on a graph-based domain ontology;
determining whether a container node having the determined type is present in the container tree data structure; and
responsive to determining that a container node having the determined type is not present in the container tree data structure:
(a) constructing the container node having the determine type,
(b) inserting the container node into an appropriate position in the container tree data structure, wherein the appropriate position in the container tree data structure is determined based at least in part on the graph-based domain ontology, and
(c) storing the observable-value pair in the container node.