Systems and methods for streaming data pipelines
View Patent ↗A method for streaming data pipelines includes receiving, at a first update application programming interface, a global unique identifier corresponding an entity and information associated with the entity. The method also includes determining whether the global unique identifier exists in an entity database, the entity data being configured to store information associated with one or more entities and, in response to a determination that the global unique identifier does not exist in the entity database, performing a write command to write the global unique identifier to a registry. The method also includes determining whether the write command was successful, and, in response to a determination that the write command was unsuccessful: retrieving the global unique identifier written to the registry by a second update application programming interface; and updating the entity database with information associated with the global unique identifier.
1 . A system for synchronizing streaming data pipelines, the system comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to:
receive, at a first update application programming interface of a first data pipeline, a global unique identifier corresponding to an entity and information associated with the entity;
determine whether the global unique identifier exists in an entity database, the entity database being configured to store information associated with one or more entities;
in response to a determination that the global unique identifier does not exist in the entity database, perform a write command to write the global unique identifier to a registry;
determine whether the write command was successful; and
in response to a determination that the write command was unsuccessful:
retrieve the global unique identifier written to the registry by a second update application programming interface of a second data pipeline; and
update the entity database with information associated with the global unique identifier, wherein the information associated with the global unique identifier corresponds to a change data capture event, wherein the second data pipeline is configured to update another entity database with the information associated with the global unique identifier and using the global unique identifier, wherein the first data pipeline and the second data pipeline are run in parallel and the entity is represented in the entity database and the other entity database, and wherein only one of the first update application programming interface and the second update application programing interface can successfully perform the write command to the registry for the global unique identifier.
2 . The system of claim 1 , wherein the instructions further cause the processor to, in response to a determination that the global unique identifier exists in the entity database, update the entity database with the information associated with the global unique identifier.
3 . The system of claim 2 , wherein, in response to the determination that the global unique identifier exists in the entity database, updating the entity database with the information associated with the global unique identifier includes correlating the information associated with the global unique identifier with information associated with the global unique identifier previously stored in the entity database.
4 . The system of claim 1 , wherein, in response to the determination that the write command was unsuccessful, updating the entity database with the information associated with the global unique identifier includes creating an entry in the entity database for the global unique identifier and storing the information associated with the global unique identifier.
5 . The system of claim 1 , wherein the instructions further cause the processor to, in response to a determination that the write command was successful, update the entity database with the information associated with the global unique identifier.
6 . The system of claim 5 , wherein, in response to the determination that the write command was successful, updating the entity database with the information associated with the global unique identifier includes creating an entry in the entity database for the global unique identifier and storing the information associated with the global unique identifier.
7 . The system of claim 1 , wherein the write command includes a conditional write command.
8 . The system of claim 1 , wherein a determination that the write command was successful indicates that the global unique identifier was not in the registry.
9 . The system of claim 1 , wherein the first update application programming interface is associated with a first computing device associated with the processor and the second update application programming interface is associated with a second computing device.
10 . The system of claim 1 , wherein the first update application programming interface is executed in parallel to the second update application programming interface.
11 . The system of claim 1 , wherein the one or more entities are associated with at least one of patient medical records, pharmacy claims, and insurance claims.
12 . A method for synchronizing streaming data pipelines, the method comprising:
receiving, at a first update application programming interface of a first data pipeline, a global unique identifier corresponding an entity and information associated with the entity;
determining whether the global unique identifier exists in an entity database, the entity database being configured to store information associated with one or more entities;
in response to a determination that the global unique identifier does not exist in the entity database, performing a write command to write the global unique identifier to a registry;
determining whether the write command was successful; and
in response to a determination that the write command was unsuccessful:
retrieving the global unique identifier written to the registry by a second update application programming interface of a second data pipeline; and
updating the entity database with information associated with the global unique identifier, wherein the information associated with the global unique identifier corresponds to a change data capture event, wherein the second data pipeline is configured to update another entity database with the information associated with the global unique identifier and using the global unique identifier, and wherein the first data pipeline and the second data pipeline are run in parallel and the entity is represented in the entity database and the other entity database, and wherein only one of the first update application programming interface and the second update application programing interface can successfully perform the write command to the registry for the global unique identifier.
13 . The method of claim 12 , further comprising, in response to a determination that the global unique identifier exists in the entity database, updating the entity database with the information associated with the global unique identifier.
14 . The method of claim 13 , wherein, in response to the determination that the global unique identifier exists in the entity database, updating the entity database with the information associated with the global unique identifier includes correlating the information associated with the global unique identifier with information associated with the global unique identifier previously stored in the entity database.
15 . The method of claim 12 , wherein, in response to the determination that the write command was unsuccessful, updating the entity database with the information associated with the global unique identifier includes creating an entry in the entity database for the global unique identifier and storing the information associated with the global unique identifier.
16 . The method of claim 12 , further comprising, in response to a determination that the write command was successful, updating the entity database with the information associated with the global unique identifier.
17 . The method of claim 16 , wherein, in response to the determination that the write command was successful, updating the entity database with the information associated with the global unique identifier includes creating an entry in the entity database for the global unique identifier and storing the information associated with the global unique identifier.
18 . The method of claim 12 , wherein the write command includes a conditional write command.
19 . The method of claim 12 , wherein a determination that the write command was successful indicates that the global unique identifier was not in the registry.
20 . A tangible, non-transitory computer-readable medium storing instructions that, when executed, cause a first computing device to:
receive, at a first update application programming interface of a first data pipeline associated with the first computing device, a global unique identifier corresponding an entity and information associated with the entity;
determine whether the global unique identifier exists in an entity database, the entity database being configured to store information associated with one or more entities;
in response to a determination that the global unique identifier exists in the entity database, update the entity database with information associated with the global unique identifier by correlating the information associated with the global unique identifier with information associated with the global unique identifier previously stored in the entity database;
in response to a determination that the global unique identifier does not exist in the entity database, perform a write command to write the global unique identifier to a registry;
determine whether the write command was successful;
in response to a determination that the write command was successful:
create an entry in the entity database for the global unique identifier; and
store the information associated with the global unique identifier; and
in response to a determination that the write command was unsuccessful:
retrieve the global unique identifier written to the registry by a second update application programming interface of a second data pipeline associated with a second computing device and being executed in parallel to the first update application programming interface;
create an entry in the entity database for the global unique identifier; and
store the information associated with the global unique identifier,
wherein the information associated with the global unique identifier corresponds to a change data capture event, wherein the second data pipeline is configured to store the information associated with the global unique identifier in another entity database using the global unique identifier, and wherein the first data pipeline and the second data pipeline are run in parallel and the entity is represented in the entity database and the other entity database, and wherein only one of the first update application programming interface and the second update application programing interface can successfully perform the write command to the registry for the global unique identifier.