Granted Patent B2

US 12,277,135 · App. 18/492,425 · Granted Apr 15, 2025

Inventors: Andrew Blom (Cambridge, MA); Darren Miller (Westford, MA); Marshall A. Isman (Newton, MA)

Assignee: Ab Initio Technology LLC

G06F16/254G06F8/34G06F16/258G06F16/9024H04L67/565

Abstract

Claims (45)

1. A method for automatically generating an executable application to transform and load data into a data store, the method, comprising:

receiving at least one metadata file that specifies one or more logical rules for mapping a source field of a data record to a target field and transforming the data record into transformed data for storing in the data store;

validating that the one or more logical rules are not inconsistent for transforming the data record;

generating, based on the validating, data rules that specify one or more standards for storing the transformed data in the data store in accordance with the one or more logical rules; and

generating, based on the data rules that specify the one or more standards, an executable data processing application for a runtime environment, the executable data processing application configurable to:

receive source data comprising a data record from one or more data sources;

determine, based on the one or more logical rules in the metadata file, (i) a source field of the data record that includes at least one source key value of the data record and (ii) a format for at least one transformed key value in a target field of the transformed data in the data store; and

transform the source data into the transformed data, the at least one source key value of the source field being transformed to have the format of the at least one transformed key value of the target field.

2. The method of claim 1 , wherein the executable data processing application includes a dataflow graph, a dataflow subgraph, or a plurality of dataflow graphs.

3. The method of claim 1 , wherein the data store includes a data warehouse.

4. The method of claim 1 , wherein the metadata file specifies an identifier of the target field for the at least one transformed key value, the at least one source key value of the source field being transformed based on the identifier.

5. The method of claim 1 , wherein the one or more logical rules include a data historization rule for the transformed data.

6. The method of claim 5 , wherein the data historization rule includes a time stamp indicative of when the transformed data is updated.

7. The method of claim 1 , wherein receiving the metadata file includes parsing a header row of the metadata file to determine which of the one or more logical rules are represented in the metadata file.

8. The method of claim 1 , wherein transforming the source data into the transformed data includes:

determining that at least two different portions of the source data specify identical key values; and

specifying a new key value for at least one of the two different portions of the source data, the new key value being different from the identical key values and based on a key sequence of the transformed data.

9. The method of claim 1 , wherein the one or more logical rules include one or more semantic rules that specify a semantic label for interpreting values of the transformed data.

10. The method of claim 1 , wherein the one or more logical rules include a data quality rule to determine acceptable data values for including in the transformed data.

11. The method of claim 1 , wherein the at least one metadata file that specifies a key mapping scheme for the transformed data.

12. The method of claim 1 , wherein the at least one metadata file that specifies whether the transformed data are configured to be read-optimized or write-optimized.

13. The method of claim 1 , wherein the one or more logical include a key surrogation rule.

14. The method of claim 1 , wherein the executable data processing application is further configured to load the transformed data into the data store.

15. A system for automatically generating an executable application to transform and load data into a data store, the system comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, causes the one or more processors to perform operations comprising:

receiving at least one metadata file that specifies one or more logical rules for mapping a source field of a data record to a target field and transforming the data record into transformed data for storing in the data store;

validating that the one or more logical rules are not inconsistent for transforming the data record;

generating, based on the validating, data rules that specify one or more standards for storing the transformed data in the data store in accordance with the one or more logical rules; and

generating, based on the data rules that specify the one or more standards, an executable data processing application for a runtime environment, the executable data processing application configurable to:

receive source data comprising a data record from one or more data sources;

determine, based on the one or more logical rules in the metadata file, (i) a source field of the data record that includes at least one source key value of the data record and (ii) a format for at least one transformed key value in a target field of the transformed data in the data store; and

transform the source data into the transformed data, the at least one source key value of the source field being transformed to have the format of the at least one transformed key value of the target field.

16. The system of claim 15 , wherein the at least one metadata file that specifies a key mapping scheme for the transformed data.

17. The system of claim 15 , wherein the one or more logical include a key surrogation rule.

18. One or more non-transitory computer readable media storing instructions that are executable by one or more processors configured to perform operations for automatically generating an executable application to transform and load data into a data store, the operations including:

receiving at least one metadata file that specifies one or more logical rules for mapping a source field of a data record to a target field and transforming the data record into transformed data for storing in the data store;

validating that the one or more logical rules are not inconsistent for transforming the data record;

generating, based on the validating, data rules that specify one or more standards for storing the transformed data in the data store in accordance with the one or more logical rules; and

generating, based on the data rules that specify the one or more standards, an executable data processing application for a runtime environment, the executable data processing application configurable to:

receive source data comprising a data record from one or more data sources;

determine, based on the one or more logical rules in the metadata file, (i) a source field of the data record that includes at least one source key value of the data record and (ii) a format for at least one transformed key value in a target field of the transformed data in the data store; and

transform the source data into the transformed data, the at least one source key value of the source field being transformed to have the format of the at least one transformed key value of the target field.

19. The one or more non-transitory computer readable media of claim 18 , wherein the at least one metadata file that specifies a key mapping scheme for the transformed data.

20. The one or more non-transitory computer readable media of claim 18 , wherein the one or more logical include a key surrogation rule.

Assignments (3)

Continuity (3)

References Cited (39)