Methods and systems for implementing a data reconciliation framework
Some aspects are directed to data reconciliation frameworks. An example framework is configured to receive core data, the system model comprising a plurality of data records associated with at least two assets, receive a system model, the system model comprising context data indicating, execute a configuration operation of a data validation process based on the system model, execute the data validation process to identify at least one inconsistent or incomplete record among the plurality of data records, determine at least one data reconciliation technique from a plurality of data reconciliation techniques based on the system model, and apply the at least one data reconciliation technique to the core data to reconcile the at least one inconsistent or incomplete record.
1. A computer system implementing a data reconciliation framework, the computer system configured to:
receive core data, the core data comprising a plurality of data records associated with at least two assets;
receive a system model, the system model comprising context data indicating at least one characteristic of the at least two assets;
execute a configuration operation of a data validation process based on the system model, wherein the configuration operation comprises selecting at least one heuristic included in the system model, the selection based on the at least one characteristic;
execute the data validation process to identify at least one missing, inconsistent, or incomplete record among the plurality of data records;
determine at least one data reconciliation technique from a plurality of data reconciliation techniques based on the system model; and
generate a consistent data set by applying the at least one data reconciliation technique to the core data to impute the at least one missing data record or remove the at least one inconsistent or the at least one incomplete record.
2. The computer system of claim 1 , wherein:
the core data comprises repair maintenance data (RMD) for a plurality of aircraft engines, wherein the system model comprises information identifying a relationship between the plurality of aircraft engines, one or more flights, and one or more aircraft, and wherein the data validation process identifies the at least one inconsistent or incomplete record based on determining that the core data is missing at least one entry for at least one of the one or more flights.
3. The computer system of claim 1 , wherein the system model further comprises data configuring at least one external data access process, and wherein the data validation process further comprises retrieving external data from at least one external datastore according to the data configuring the at least one external data access process.
4. The computer system of claim 3 , further configured to determine whether at least one of the plurality of data records corresponds to the external data from the at least one external datastore as part of the data validation process.
5. The computer system of claim 4 , wherein the external data is airport flight information.
6. The computer system of claim 1 , wherein the context data comprises system context data, and wherein the at least one characteristic indicates a grouping of assets among the plurality of assets.
7. The computer system of claim 6 , wherein the grouping indicates a hierarchical relationship.
8. The computer system of claim 7 , wherein the grouping is a grouping by aircraft and wherein the assets are aircraft engines.
9. The computer system of claim 1 , wherein the context data comprises asset context data, and wherein the at least one characteristic is associated with a particular asset type of the at least two assets.
10. A method for implementing a data reconciliation framework, the method comprising:
receiving core data, the core data comprising a plurality of data records associated with at least two assets;
receiving a system model, the system model comprising context data indicating at least one characteristic of the at least two assets;
executing a configuration operation of a data validation process based on the system model, wherein the configuration operation comprises selecting at least one heuristic included in the system model, the selection based on the at least one characteristic;
executing the data validation process to identify at least one missing, inconsistent, or incomplete record among the plurality of data records;
determining at least one data reconciliation technique from a plurality of data reconciliation techniques based on the system model; and
generating a consistent data set by applying the at least one data reconciliation technique to the core data to impute the at least one missing data record or remove the at least one inconsistent or the at least one incomplete record.
11. The method of claim 10 , wherein:
the core data comprises repair maintenance data (RMD) for a plurality of aircraft engines, wherein the system model comprises information identifying a relationship between the plurality of aircraft engines, one or more flights, and one or more aircraft, and wherein the data validation process identifies the at least one inconsistent or incomplete record based on determining that the core data is missing at least one entry for at least one of the one or more flights.
12. The method of claim 10 , wherein the system model further comprises data configuring at least one external data access process, and wherein the data validation process further comprises retrieving external data from at least one external datastore according to the data configuring the at least one external data access process.
13. The method of claim 12 , further comprising determining whether at least one of the plurality of data records corresponds to the external data from the at least one external datastore as part of the data validation process.
14. The method of claim 13 , wherein the external data is airport flight information.
15. The method of claim 10 , wherein the context data comprises system context data, and wherein the at least one characteristic indicates a grouping of assets among the plurality of assets.
16. The method of claim 15 , wherein the grouping indicates a hierarchical relationship.
17. The method of claim 16 , wherein the grouping is a grouping by aircraft and wherein the assets are aircraft engines.
18. The method of claim 10 , wherein the context data comprises asset context data, and wherein the at least one characteristic is associated with a particular asset type of the at least two assets.
19. A non-transitory computer readable storage medium comprising program instructions for implementing a data reconciliation framework, the instructions, when executed by a processor, cause the processor to:
receive core data, the core data comprising a plurality of data records associated with at least two assets;
receive a system model, the system model comprising context data indicating at least one characteristic of the at least two assets;
execute a configuration operation of a data validation process based on the system model, wherein the configuration operation comprises selecting at least one heuristic included in the system model, the selection based on the at least one characteristic;
execute the data validation process to identify at least one missing, inconsistent, or incomplete record among the plurality of data records;
determine at least one data reconciliation technique from a plurality of data reconciliation techniques based on the system model; and
generate a consistent data set by applying the at least one data reconciliation technique to the core data to impute the at least one missing data record or remove the at least one inconsistent or the at least one incomplete record.
20. The computer readable storage medium of claim 19 , wherein:
the core data comprises repair maintenance data (RMD) for a plurality of aircraft engines, wherein the system model comprises information identifying a relationship between the plurality of aircraft engines, one or more flights, and one or more aircraft, and wherein the data validation process identifies the at least one inconsistent or incomplete record based on determining that the core data is missing at least one entry for at least one of the one or more flights.