Data serving layer for data platform
An illustrative method for querying multiple datasets may include accessing a plurality of datasets including data associated with monitoring one or more compute environments and ingested at varying time intervals, receiving a query request for information that depends on the data included in multiple datasets of the plurality of datasets, querying, based on the query request, the multiple datasets, prioritizing, based on the varying time intervals, information that depends on the data included in the multiple datasets and presenting, based on the prioritizing, a query result representative of the information that depends on the data included in the multiple datasets.
1 . A method comprising:
generating, by a data platform, a semantic layer defining relationships between a first model associated with a first dataset and a second model associated with a second dataset to provide a centralized application programming interface (API), wherein the first model defines first attributes of the first dataset having a first configuration associated with a first data compute environment and the second model defines second attributes of the second dataset having a second configuration associated with a second compute environment, and wherein the API exposes the first and second datasets by way of a common query language;
querying, by the data platform the first and second datasets based on a query request received via the centralized API;
prioritizing, by the data platform, information that depends on the data included in the first and second datasets based on first and second time intervals; and
presenting, by the data platform, a query result based on the prioritizing.
2 . The method of claim 1 , further comprising:
accessing the first and second datasets, including generating a comprehensive dataset based on data included in the first and second datasets; and
wherein querying the first and second datasets includes querying the comprehensive dataset.
3 . The method of claim 1 , further comprising:
accessing the first and second datasets, including accessing the first and second models defining the first and second attributes and the querying the first and second datasets based on the relationships between the first and second models defined by the semantic layer.
4 . The method of claim 3 , wherein the first and second attributes are based on a user input designating the first and second attributes.
5 . The method of claim 3 , wherein the prioritizing the information is further based on the relationships between the first and second models defined by the semantic layer.
6 . The method of claim 1 , wherein the querying the first and second datasets includes querying prioritized information that depends on the data included in the first and second datasets.
7 . The method of claim 1 , wherein the prioritizing the information is further based on a user input designating the prioritizing.
8 . The method of claim 1 , wherein the prioritizing the information is further based on the query request.
9 . The method of claim 1 , wherein the first dataset includes data in a first stage of a data ingestion pipeline of the data platform, the second dataset includes data in a second stage of a data ingestion pipeline of the data platform, and a third dataset includes data in a third stage of a data ingestion pipeline of the data platform.
10 . The method of claim 9 , wherein the data in the first stage includes data received in the data ingestion pipeline, the data in the second stage includes data bundled in the data ingestion pipeline, and the data in the third stage includes data loaded to a data store.
11 . The method of claim 10 , wherein the first stage includes a data streaming platform configured to receive the data in the data ingestion pipeline.
12 . The method of claim 1 , wherein the querying the first and second datasets includes authenticating user access to the data included in the first and second datasets.
13 . The method of claim 12 , wherein the authenticating the user access is based on one or more resource groups associated with a user.
14 . The method of claim 12 , wherein the authenticating the user access is based on a token associated with the query request.
15 . The method of claim 12 , wherein the querying the first and second datasets further includes filtering data included in the first and second datasets based on the authenticating user access.
16 . The method of claim 1 , wherein the first and second datasets are generated by an agent configuration configured to collect runtime workload data associated with a workload deployed within the first and second compute environments.
17 . The method of claim 16 , further comprising:
constructing, by the data platform and based on the runtime workload data, a graph comprising a plurality of nodes connected by a plurality of edges, wherein each node of the plurality of nodes represents a logical entity associated with the runtime workload data and each edge of the plurality of edges represents a behavioral relationship between nodes connected by the edge;
wherein the first and second datasets are generated further based on the graph.
18 . The method of claim 1 , wherein one or more datasets included in the first and second datasets are generated by an agentless workload scanning configuration configured to collect non-runtime workload data associated with a workload deployed within one or more compute environments.
19 . A system comprising:
a memory storing instructions; and
one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising:
generating a semantic layer defining relationships between a first model associated with a first dataset and a second model associated with a second dataset to provide a centralized application programming interface (API), wherein the first model defines first attributes of the first dataset having a first configuration associated with a first data compute environment and the second model defines second attributes of the second dataset having a second configuration associated with a second compute environment, and wherein the API exposes the first and second datasets by way of a common query language;
querying the first and second datasets based on a query request received via the centralized API;
prioritizing information that depends on the data included in the first and second datasets based on first and second time intervals; and
presenting a query based on the prioritizing.
20 . A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
generating a semantic layer defining relationships between a first model associated with a first dataset and a second model associated with a second dataset to provide a centralized application programming interface (API), wherein the first model defines first attributes of the first dataset having a first configuration associated with a first data compute environment and the second model defines second attributes of the second dataset having a second configuration associated with a second compute environment, and wherein the API exposes the first and second datasets by way of a common query language;
querying the first and second datasets based on a query request received via the centralized API;
prioritizing information that depends on the data included in the first and second datasets based on first and second time intervals; and
presenting a query result based on the prioritizing.