System and method for automatic generation of extract, transform, load (ETL) asserts

1. A system for use with an analytic applications environment, for automatic generation of asserts comprising:

a computer including one or more processors, that provides access by an analytic applications environment to a data warehouse for storage of data by a plurality of tenants, wherein the data warehouse is associated with an analytic applications schema;

wherein each tenant of the plurality of tenants is associated with a customer tenancy, and a customer schema for use by the tenant in populating a data warehouse instance, wherein data associated with a particular tenant is provisioned in the data warehouse instance associated with, and accessible to, the particular tenant, in accordance with the analytic applications schema and the customer schema associated with the particular tenant;

wherein during loading of the data to the data warehouse by a data pipeline, one or more asserts associated with warehouse data are automatically generated, based on an evaluation of the data received from the enterprise application or database environment, wherein the data pipeline comprises a shared data pipeline having one or more common transformation maps or repositories, and the shared data pipeline comprises data values across multiple customer data warehouses associated with a data set including a rule and that are compared to confirm a correctness of the rule and automatically generate the one or more asserts using the one or more common transformation maps or repositories in response to confirming the correctness of the rule; and

wherein one or more of validation and/or updates to generated asserts are performed, upon receipt of data from the enterprise application or database environment.

2. The system of claim 1 , wherein the evaluation of the data received from the enterprise application or database environment includes an evaluation that columns of the received data include integer values and are one of strictly sequential, cyclic, or non-negative values.

3. The system of claim 1 , wherein the evaluation of the data received from the enterprise application or database environment, and automatic generation of asserts associated with the warehouse data, includes identifying asserts for received data having floating point values.

4. The system of claim 1 , wherein the generating of asserts associated with warehouse data, based on an evaluation of the data received from the enterprise application or database environment, includes a marking of one or more asserts as mutable or immutable.

5. The system of claim 1 , wherein the generating of asserts associated with warehouse data, based on an evaluation of the data received from the enterprise application or database environment, includes associating and updating confidence values with one or more of the asserts.

6. The system of claim 1 , wherein the continuing to perform one or more of validation and/or updates to generated asserts includes validating the asserts in response to receipt of additional or changed data from the enterprise application or database environment.

7. The system of claim 1 , wherein the continuing to perform one or more of validation and/or updates to generated asserts includes updating asserts in response to receipt of additional or changed data from the enterprise application or database environment.

8. The system of claim 1 , comprising:

modeling probabilities of historic values of data stored in the data warehouse;

determining a value in the data received from the enterprise application or database environment that is not present within the historic values of the data; and

determining, using kernel density estimation (KDE) or other means, a probability that a received value is one of lower than, or exceeds, a predetermined threshold for the data, for use in generating an assert associated with the received value.

9. The system of claim 1 , wherein the computer environment maintains, for a plurality of tenants (customers) of the environment:

a data warehouse instance associated with each tenant, including a first data warehouse instance associated with a first tenant, and a second data warehouse instance associated with a second tenant; and

an analytics schema associated with each data warehouse instance, that enables data to be loaded automatically, by the data pipeline or other processing component, to a particular data warehouse instance in accordance with the analytics schema, to pre-populate the data warehouse instance with business intelligence or analytics data retrieved from an associated tenant enterprise application or database environment.

10. The system of claim 1 , wherein each tenant and data warehouse instance is additionally associated with a customer schema, including:

a first customer schema associated with the first tenant and first data warehouse instance; and

a second customer schema associated with the second tenant and second data warehouse instance;

wherein the contents of the particular data warehouse instance are controlled by the data pipeline or other processing component operating automatically in accordance with the analytics schema, and by the customer schema associated with the particular data warehouse instance.

11. A method for use with an analytic applications environment, for automatic generation of asserts comprising:

providing, at a computer including one or more processors, access by an analytic applications environment to a data warehouse for storage of data by a plurality of tenants, wherein the data warehouse is associated with an analytic applications schema;

wherein each tenant of the plurality of tenants is associated with a customer tenancy, and a customer schema for use by the tenant in populating a data warehouse instance, wherein data associated with a particular tenant is provisioned in the data warehouse instance associated with, and accessible to, the particular tenant, in accordance with the analytic applications schema and the customer schema associated with the particular tenant; and

during loading of the data to the data warehouse by a data pipeline, automatically generating one or more asserts associated with warehouse data, based on an evaluation of the data received from the enterprise application or database environment, wherein the data pipeline comprises a shared data pipeline having one or more common transformation maps or repositories, wherein the shared data pipeline comprises data values across multiple customer data warehouses associated with a data set including a rule and that are compared to confirm a correctness of the rule and automatically generate the one or more asserts using the one or more common transformation maps or repositories in response to confirming the correctness of the rule; and

performing one or more of validation and/or updates to generated asserts, upon receipt of data from the enterprise application or database environment.

12. The method of claim 11 , wherein the evaluation of the data received from the enterprise application or database environment includes an evaluation that columns of the received data include integer values and are one of strictly sequential, cyclic, or non-negative values; or identifying asserts for received data having floating point values.

13. The method of claim 11 , wherein the generating of asserts associated with warehouse data, based on an evaluation of the data received from the enterprise application or database environment, includes a marking of one or more asserts as mutable or immutable.

14. The method of claim 11 , wherein the generating of asserts associated with warehouse data, based on an evaluation of the data received from the enterprise application or database environment, includes associating and updating confidence values with one or more of the asserts.

15. The method of claim 11 , wherein the continuing to perform one or more of validation and/or updates to generated asserts includes validating the asserts in response to receipt of additional or changed data from the enterprise application or database environment.

16. The method of claim 11 , wherein the continuing to perform one or more of validation and/or updates to generated asserts includes updating asserts in response to receipt of additional or changed data from the enterprise application or database environment.

17. The method of claim 11 , comprising:

modeling probabilities of historic values of data stored in the data warehouse;

determining a value in the data received from the enterprise application or database environment that is not present within the historic values of the data; and

determining, using kernel density estimation (KDE) or other means, a probability that a received value is one of lower than, or exceeds, a predetermined threshold for the data, for use in generating an assert associated with the received value.

18. The method of claim 11 , wherein the computer environment maintains, for a plurality of tenants (customers) of the environment:

a data warehouse instance associated with each tenant, including a first data warehouse instance associated with a first tenant, and a second data warehouse instance associated with a second tenant; and

an analytics schema associated with each data warehouse instance, that enables data to be loaded automatically, by the data pipeline or other processing component, to a particular data warehouse instance in accordance with the analytics schema, to pre-populate the data warehouse instance with business intelligence or analytics data retrieved from an associated tenant enterprise application or database environment.

19. The method of claim 11 , wherein each tenant and data warehouse instance is additionally associated with a customer schema, including:

a first customer schema associated with the first tenant and first data warehouse instance; and

a second customer schema associated with the second tenant and second data warehouse instance;

wherein the contents of the particular data warehouse instance are controlled by the data pipeline or other processing component operating automatically in accordance with the analytics schema, and by the customer schema associated with the particular data warehouse instance.

20. A non-transitory computer readable storage medium having instructions thereon, which when read and executed by a computer including one or more processors cause the computer to perform a method comprising:

providing access by an analytic applications environment to a data warehouse for storage of data by a plurality of tenants, wherein the data warehouse is associated with an analytic applications schema;

wherein each tenant of the plurality of tenants is associated with a customer tenancy, and a customer schema for use by the tenant in populating a data warehouse instance, wherein data associated with a particular tenant is provisioned in the data warehouse instance associated with, and accessible to, the particular tenant, in accordance with the analytic applications schema and the customer schema associated with the particular tenant; and

during loading of the data to the data warehouse by a data pipeline, automatically generating one or more asserts associated with warehouse data, based on an evaluation of the data received from the enterprise application or database environment, wherein the data pipeline comprises a shared data pipeline having one or more common transformation maps or repositories, and the shared data pipeline comprises data values across multiple customer data warehouses associated with a data set including a rule and that are compared to confirm a correctness of the rule and automatically generate the one or more asserts using the one or more common transformation maps or repositories in response to confirming the correctness of the rule; and

performing one or more of validation and/or updates to generated asserts, upon receipt of data from the enterprise application or database environment.