Semantic layer for data platform
An illustrative method for querying multiple datasets may include generating, based on models each associated with and defining attributes of a different datasets stored in a plurality of data stores, a semantic layer defining relationships between the models and that provides a centralized application programming interface (API) for exposing the datasets by way of a common query language, receiving, by way of the centralized API, a query request for information that depends on data included in multiple datasets included in the plurality of datasets, querying, based on the query request and the relationships between the models defined by the semantic layer, the multiple datasets, and presenting, based on the querying, a query result representative of the information that depends on the data included in the multiple datasets.
1 . A method comprising:
generating, by a data platform, a semantic layer defining relationships between a first model and a second model to provide a centralized application programming interface (API), wherein the first model defines first attributes of a first dataset having a first configuration associated with a first data platform client and the second model defines second attributes of a second dataset having a second configuration associated with a second data platform client, and wherein the API exposes the first and second datasets by way of a common query language;
receiving, via the centralized API, a query request using the common query language for data included in the first and second datasets;
querying, by the data platform, the first and second data sets based on the relationships between the first and second models defined by the semantic layer; and
presenting, by the data platform, a query result representative of the information that depends on the data included in the first and second datasets.
2 . The method of claim 1 , wherein the attributes of a dataset included in the first and second datasets include one or more of a type of data included in the dataset, a configuration associated with the data included in the dataset, a time associated with the data included in the dataset, a key associated with the data included in the dataset, a customer identification associated with the data included in the dataset, an identity of a user associated with the data included in the dataset, a service associated with the data included in the dataset, a size associated with the data include in the dataset, a process execution history associated with the data included in the dataset, security-related data associated with the data included in the dataset, a grouping associated with the data included in the dataset, an aggregation associated with the data included in the dataset, a filter associated with the data included in the dataset, or a type of data store associated with the dataset.
3 . The method of claim 1 , wherein the first and second attributes defined in the first and second models are based on a user input designating the attributes.
4 . The method of claim 1 , wherein first and second datasets are generated by an agent configuration configured to collect runtime workload data associated with a workload deployed within a compute environment.
5 . The method of claim 4 , further comprising:
constructing, by the data platform and based on the runtime workload data, a graph comprising a plurality of nodes connected by a plurality of edges, wherein each node of the plurality of nodes represents a logical entity associated with the runtime workload data and each edge of the plurality of edges represents a behavioral relationship between nodes connected by the edge;
wherein the first and second datasets are generated further based on the graph.
6 . The method of claim 1 , wherein the first and second datasets are generated by an agentless workload scanning configuration configured to collect non-runtime workload data associated with a workload deployed within a compute environment.
7 . The method of claim 1 , wherein the first and second datasets are generated based on a user input.
8 . The method of claim 7 , wherein the user input includes designating one or more of a configuration for ingesting data included in the first and second datasets into the data platform, a type of data store associated with the one or more datasets, or a model associated with the one or more datasets.
9 . The method of claim 7 , wherein the user input is provided by a user within an organization that is a customer of the data platform.
10 . The method of claim 1 , wherein the defining the relationships between the first and second models within the semantic layer is further based on a user input.
11 . The method of claim 1 , further comprising generating one or more snapshots of the first and second datasets, wherein the querying the multiple datasets includes tracking data included in the first and second datasets from the one or more snapshots.
12 . The method of claim 1 , further comprising modifying data included in the first and second datasets without updating the models associated with the one or more datasets.
13 . The method of claim 1 , wherein the presenting the query result comprises presenting a visual chart associated with the information that depends on the data included in the first and second datasets.
14 . The method of claim 13 , wherein the presenting the visual chart includes presenting information that depends on the data included in the first and second datasets in hierarchical sets.
15 . The method of claim 13 , wherein one or more objects of the visual chart are presented based on a user input.
16 . The method of claim 1 , wherein the information that depends on the data included in the first and second datasets includes one or more vulnerabilities.
17 . A system comprising:
a memory storing instructions; and
one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising:
generating a semantic layer defining relationships between a first model and a second model to provide a centralized application programming interface (API), wherein the first model defines first attributes of a first dataset having a first configuration associated with a first data platform client and the second model defines second attributes of a second dataset having a second configuration associated with a second data platform client, and wherein the API exposes the first and second datasets by way of a common query language:
receiving, via the centralized API, a query request using the common query language for information that depends on data included in the first and second datasets;
querying the first and second data sets based on the relationships between the first and second models defined by the semantic layer; and
presenting a query result representative of the information that depends on the data included in the multiple datasets.
18 . The system of claim 17 , wherein the attributes of a dataset included in the first and second datasets include one or more of a type of data included in the dataset, a configuration associated with the data included in the dataset, a time associated with the data included in the dataset, a key associated with the data included in the dataset, a customer identification associated with the data included in the dataset, an identity of a user associated with the data included in the dataset, a service associated with the data included in the dataset, a size associated with the data included in the dataset, a process execution history associated with the data included in the dataset, security-related data associated with the data included in the dataset, a grouping associated with the data included in the dataset, an aggregation associated with the data included in the dataset, a filter associated with the data included in the dataset, or a type of data store associated with the dataset.
19 . The system of claim 17 , wherein the attributes defined in the first and second models are based on a user input designating the attributes.
20 . A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
generating a semantic layer defining relationships between a first model and a second model to provide a centralized application programming interface (API), wherein the first model defines first attributes of a first dataset having a first configuration associated with a first data platform client and the second model defines second attributes of a second dataset having a second configuration associated with a second data platform client, and wherein the API exposes the first and second datasets by way of a common query language;
receiving, via the centralized API, a query request using the common query language for information that depends on data included in the first and second datasets;
querying the first and second data sets-based on the relationships between the first and second models defined by the semantic layer; and
presenting a query result representative of the information that depends on the data included in the multiple datasets.