System and method for automatic generation of BI models using data introspection and curation
In accordance with an embodiment, described herein are systems and methods for automatic generation of business intelligence (BI) data models using data introspection and curation, as may be used, for example, with enterprise resource planning (ERP) or other enterprise computing or data analytics environments. The described approach uses a combination of manually-curated artifacts, and automatic generation of a model through data introspection, of a source data environment, to derive a target BI data model. For example, a pipeline generator framework can evaluate the dimensionality of a transaction type, degenerate attributes, and application measures; and use the output of this process to create an output target model and pipeline or load plan. The systems and methods described herein provide a technical improvement in the building of new subject areas or a BI data model within much shorter periods of time.
1. A system for automatic generation of data models using data introspection and curation, comprising:
a computer including one or more processors, that provides access by an analytic applications environment to a data warehouse for storage of data by a plurality of tenants, wherein data associated with a tenant is provisioned in a data warehouse instance associated with the tenant in accordance with an analytic applications schema shared by the plurality of tenants, and a customer schema associated with the tenant, wherein the data warehouse instance associated with the tenant is populated with data received from the tenant's source data environment, as defined by a combination of the analytic applications schema and their customer schema;
wherein the analytic applications environment provides a data transformation layer that is used by the analytic applications environment to process a received transactional data associated with the tenant and load a transformed data into the data warehouse;
wherein the analytic applications environment provides a semantic layer that includes data defining a semantic model of the tenant's data including:
a physical layer that maps to a physical data model;
a logical layer that operates as a mapping and transformation layer where calculations can be defined; and
a presentation layer that enables access to the data as content;
wherein the system provides a generator framework and semantic model extension process operable to generate automatically one or more data maps associated with the tenant's source data environment, by reference to combination of:
a seed repository that includes curated artifacts including basic dimensions associated with the source data environment, and
automatically-determined or interpreted variables provided by a data introspection of the source data environment and an associated source model;
said process comprising:
generating or updating the semantic model of the tenant's data for transaction types associated with the tenant's source data environment, including determining, based on the introspection of the source data environment, dimensions and facts associated with the source data to include in the semantic model; and
overlaying the generated semantic model with security artifacts corresponding to those described in the source model that control data visibility.
2. The system of claim 1 , wherein the system performs an extract, transform, load data pipeline or process in accordance with the analytic applications schema and the customer schema associated with the tenant, to receive data from the tenant's enterprise software application or source data environment, for loading into the data warehouse instance associated with the tenant.
3. The system of claim 1 , wherein generation of one or more extract, transform, load (ETL) maps includes receiving from the seed repository the curated artifacts, including basic dimensions associated with the source data environment; and
wherein additional transaction dimensions, columns, or security artifacts, are then automatically generated by the generator framework.
4. The system of claim 1 , wherein the semantic model as generated is stored as a business intelligence (BI) Repository (RPD) file.
5. The system of claim 1 , wherein the source data environment is one of a NetSuite, business intelligence (BI), enterprise resource planning (ERP), cloud computing, enterprise computing, or other computing environment.
6. The system of claim 1 , wherein the data warehouse is associated with an analytic applications schema;
wherein each tenant instance of the data warehouse is populated with data received from an enterprise software application or source data environment, wherein data associated with a particular tenant of the analytic applications environment is provisioned in the data warehouse instance associated with, and accessible to, the particular tenant, in accordance with the analytic applications schema and the customer schema associated with the particular tenant;
wherein a first customer tenancy for a first tenant comprises a first data warehouse instance, and a second customer tenancy for a second tenant comprises a second data warehouse instance.
7. The system of claim 1 , wherein the curated artifacts are provided within a curated data model and first data pipeline or process that publishes a customer data to the analytic applications schema, while an external or custom data is onboarded to the customer schema using a second data pipeline or process.
8. A method for automatic generation of data models using data introspection and curation, comprising:
providing, by a computer including one or more processors, access by an analytic applications environment to a data warehouse for storage of data by a plurality of tenants, wherein data associated with a tenant is provisioned in a data warehouse instance associated with the tenant in accordance with an analytic applications schema shared by the plurality of tenants, and a customer schema associated with the tenant, wherein the data warehouse instance associated with the tenant is populated with data received from the tenant's source data environment, as defined by a combination of the analytic applications schema and their customer schema;
wherein the analytic applications environment provides a data transformation layer that is used by the analytic applications environment to process a received transactional data associated with the tenant and load a transformed data into the data warehouse;
wherein the analytic applications environment provides a semantic layer that includes data defining a semantic model of the tenant's data including:
a physical layer that maps to a physical data model;
a logical layer that operates as a mapping and transformation layer where calculations can be defined; and
a presentation layer that enables access to the data as content;
generating automatically, by a semantic model extension process, one or more data maps associated with the tenant's source data environment, by reference to a combination of:
a seed repository that includes curated artifacts including basic dimensions associated with the source data environment, and
automatically-determined or interpreted variables provided by a data introspection of the source data environment and an associated source model;
said process comprising:
generating or updating the semantic model of the tenant's data for transaction types associated with the tenant's source data environment, including determining, based on the introspection of the source data environment, dimensions and facts associated with the source data to include in the semantic model; and
overlaying the generated semantic model with security artifacts corresponding to those described in the source model that control data visibility.
9. The method of claim 8 , further comprising performing an extract, transform, load data pipeline or process in accordance with the analytic applications schema and the customer schema associated with the tenant, to receive data from the tenant's enterprise software application or source data environment, for loading into the data warehouse instance associated with the tenant.
10. The method of claim 8 , wherein generation of one or more extract, transform, load (ETL) maps includes receiving from the seed repository the curated artifacts, including basic dimensions associated with the source data environment; and
wherein additional transaction dimensions, columns, or security artifacts, are then automatically generated by the generator framework.
11. The method of claim 8 , wherein the semantic model as generated is stored as a business intelligence (BI) Repository (RPD) file.
12. The method of claim 8 , wherein the source data environment is one of a NetSuite, business intelligence (BI), enterprise resource planning (ERP), cloud computing, enterprise computing, or other computing environment.
13. The method of claim 8 , wherein the data warehouse is associated with an analytic applications schema;
wherein each tenant instance of the data warehouse is populated with data received from an enterprise software application or source data environment, wherein data associated with a particular tenant of the analytic applications environment is provisioned in the data warehouse instance associated with, and accessible to, the particular tenant, in accordance with the analytic applications schema and the customer schema associated with the particular tenant;
wherein a first customer tenancy for a first tenant comprises a first data warehouse instance, and a second customer tenancy for a second tenant comprises a second data warehouse instance.
14. The method of claim 8 , wherein the curated artifacts are provided within a curated data model and first data pipeline or process that publishes a customer data to the analytic applications schema, while an external or custom data is onboarded to the customer schema using a second data pipeline or process.
15. A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by one or more computers cause the one or more computers to perform a method comprising:
providing, by a computer including one or more processors, access by an analytic applications environment to a data warehouse for storage of data by a plurality of tenants, wherein data associated with a tenant is provisioned in a data warehouse instance associated with the tenant in accordance with an analytic applications schema shared by the plurality of tenants, and a customer schema associated with the tenant, wherein the data warehouse instance associated with the tenant is populated with data received from the tenant's source data environment, as defined by a combination of the analytic applications schema and their customer schema;
wherein the analytic applications environment provides a data transformation layer that is used by the analytic applications environment to process a received transactional data associated with the tenant and load a transformed data into the data warehouse;
wherein the analytic applications environment provides a semantic layer that includes data defining a semantic model of the tenant's data including:
a physical layer that maps to a physical data model;
a logical layer that operates as a mapping and transformation layer where calculations can be defined; and
a presentation layer that enables access to the data as content;
generating automatically, by a semantic model extension process, one or more data maps associated with the tenant's source data environment, by reference to a combination of:
a seed repository that includes curated artifacts including basic dimensions associated with the source data environment, and
automatically-determined or interpreted variables provided by a data introspection of the source data environment and an associated source model;
said process comprising:
generating or updating the semantic model of the tenant's data for transaction types associated with the tenant's source data environment, including determining, based on the introspection of the source data environment, dimensions and facts associated with the source data to include in the semantic model; and
overlaying the generated semantic model with security artifacts corresponding to those described in the source model that control data visibility.
16. The non-transitory computer readable storage medium of claim 15 , further comprising performing an extract, transform, load data pipeline or process in accordance with the analytic applications schema and the customer schema associated with the tenant, to receive data from the tenant's enterprise software application or source data environment, for loading into the data warehouse instance associated with the tenant.
17. The non-transitory computer readable storage medium of claim 15 , wherein generation of one or more extract, transform, load (ETL) maps includes receiving from the seed repository the curated artifacts, including basic dimensions associated with the source data environment; and
wherein additional transaction dimensions, columns, or security artifacts, are then automatically generated by the generator framework.
18. The non-transitory computer readable storage medium of claim 15 , wherein the semantic model as generated is stored as a business intelligence (BI) Repository (RPD) file.
19. The non-transitory computer readable storage medium of claim 15 , wherein the source data environment is one of a NetSuite, business intelligence (BI), enterprise resource planning (ERP), cloud computing, enterprise computing, or other computing environment.
20. The non-transitory computer readable storage medium of claim 15 , wherein the data warehouse is associated with an analytic applications schema;
wherein each tenant instance of the data warehouse is populated with data received from an enterprise software application or source data environment, wherein data associated with a particular tenant of the analytic applications environment is provisioned in the data warehouse instance associated with, and accessible to, the particular tenant, in accordance with the analytic applications schema and the customer schema associated with the particular tenant;
wherein a first customer tenancy for a first tenant comprises a first data warehouse instance, and a second customer tenancy for a second tenant comprises a second data warehouse instance.