Quality evaluation and augmentation of data provided by a federated query system
Various embodiments of the present disclosure provide federated query processing techniques for quality evaluation and augmentation of data provided by a federated query system. The techniques include receiving an execution plan for a federated query. The techniques also include receiving one or more data segments from a plurality of third-party data sources and receiving quality evaluation data using a set of data accessing tasks for the execution plan. The techniques also include generating a result set for the federated query using a set of data processing tasks for the execution plan. Additionally, the techniques include generating quality metrics data for the one or more data segments based on the quality evaluation data and generating an augmented result set for the federated query based on the result set and the quality metrics data.
1. A computer-implemented method comprising:
receiving, by one or more processors, a federated query that references one or more data segments from a plurality of third-party data sources;
receiving, by the one or more processors, an execution plan for the federated query, wherein the execution plan comprises a plurality of executable tasks for resolving the federated query, and the plurality of executable tasks comprises (i) a data accessing task for accessing the one or more data segments from the plurality of third-party data sources and (ii) a data processing task for processing the one or more data segments to generate a result set for the federated query;
receiving, by the one or more processors and using the data accessing task, the one or more data segments from the plurality of third-party data sources;
generating, by the one or more processors and using the data processing task, the result set for the federated query based on the one or more data segments;
receiving, by the one or more processors and based on one or more federated query attributes of the federated query, quality evaluation data indicative of one or more predefined quality attributes for the one or more data segments;
generating, by the one or more processors and based on the quality evaluation data, quality metrics data indicative of one or more quality metrics for the one or more data segments;
generating, by the one or more processors, an augmented result set for the federated query by augmenting the result set with the quality metrics data; and
generating, by the one or more processors, a query response that comprises the augmented result set to visually indicate the one or more quality metrics for the one or more data segments.
2. The computer-implemented method of claim 1 , wherein receiving the quality evaluation data comprises:
receiving the quality evaluation data at least approximately in parallel to receiving the one or more data segments.
3. The computer-implemented method of claim 1 , wherein generating the quality metrics data comprises:
generating the quality metrics data at least approximately in parallel to processing the one or more data segments.
4. The computer-implemented method of claim 1 , wherein receiving the federated query comprises:
receiving the federated query via an application programming interface (API) gateway of a federated query system communicatively coupled to the plurality of third-party data sources.
5. The computer-implemented method of claim 1 , wherein the quality evaluation data comprises timestamp data for one or more previous updates to the one or more data segments, and wherein generating the quality metrics data comprises:
comparing a federated query timestamp for the federated query to the timestamp data.
6. The computer-implemented method of claim 1 , wherein determining the quality evaluation data comprises:
determining a type of quality metric for the one or more data segments.
7. The computer-implemented method of claim 1 , further comprising:
providing the query response that comprises the augmented result set to a computing entity associated with the federated query to render visual data associated with the augmented result set via a user interface of the computing entity.
8. The computer-implemented method of claim 1 , further comprising:
updating one or more portions of a metadata store for the one or more data segments based on the quality metrics data.
9. The computer-implemented method of claim 8 , wherein the federated query is a first federated query, the augmented result set is a first augmented result set, and the computer-implemented method further comprises:
receiving a second federated query that references the one or more data segments from the plurality of third-party data sources; and
generating a second augmented result set for the second federated query based on metadata associated with the quality metrics data.
10. A system comprising:
one or more processors; and
one or more memories storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a federated query that references one or more data segments from a plurality of third-party data sources;
receiving an execution plan for the federated query, wherein the execution plan comprises a plurality of executable tasks for resolving the federated query, and the plurality of executable tasks comprises (i) a data accessing task for accessing the one or more data segments from the plurality of third-party data sources and (ii) a data processing task for processing the one or more data segments to generate a result set for the federated query;
receiving, using the data accessing task, the one or more data segments from the plurality of third-party data sources;
generating, using the data processing task, the result set for the federated query based on the one or more data segments;
receiving, based on one or more federated query attributes of the federated query, quality evaluation data indicative of one or more predefined quality attributes for the one or more data segments;
generating, based on the quality evaluation data, quality metrics data indicative of one or more quality metrics for the one or more data segments;
generating an augmented result set for the federated query by augmenting the result set with the quality metrics data; and
generating a query response that comprises the augmented result set to visually indicate the one or more quality metrics for the one or more data segments.
11. The system of claim 10 , wherein receiving the quality evaluation data comprises:
receiving the quality evaluation data at least approximately in parallel to receiving the one or more data segments.
12. The system of claim 10 , wherein generating the quality metrics data comprises:
generating the quality metrics data at least approximately in parallel to processing the one or more data segments.
13. The system of claim 10 , wherein the processor-executable instructions, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:
receiving the federated query via an application programming interface (API) gateway of a federated query system communicatively coupled to the plurality of third-party data sources.
14. The system of claim 10 , wherein the quality evaluation data comprises timestamp data for one or more previous updates to the one or more data segments, and wherein generating the quality metrics data comprises:
comparing a federated query timestamp for the federated query to the timestamp data.
15. The system of claim 10 , wherein determining the quality evaluation data comprises:
determining a type of quality metric for the one or more data segments.
16. The system of claim 10 , wherein the processor-executable instructions, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:
providing the query response that comprises the augmented result set to a computing entity associated with the federated query to render visual data associated with the augmented result set via a user interface of the computing entity.
17. The system of claim 10 , wherein the processor-executable instructions, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:
updating one or more portions of a metadata store for the one or more data segments based on the quality metrics data.
18. The system of claim 17 , wherein the federated query is a first federated query, the augmented result set is a first augmented result set, and the processor-executable instructions, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:
receiving a second federated query that references the one or more data segments from the plurality of third-party data sources; and
generating a second augmented result set for the second federated query based on metadata associated with the quality metrics data.
19. One or more non-transitory computer-readable storage media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a federated query that references one or more data segments from a plurality of third-party data sources;
receiving an execution plan for the federated query, wherein the execution plan comprises a plurality of executable tasks for resolving the federated query, and the plurality of executable tasks comprises (i) a data accessing task for accessing the one or more data segments from the plurality of third-party data sources and (ii) a data processing task for processing the one or more data segments to generate a result set for the federated query;
receiving, using the data accessing task, the one or more data segments from the plurality of third-party data sources;
generating, using the data processing task, the result set for the federated query based on the one or more data segments;
receiving, based on one or more federated query attributes of the federated query, quality evaluation data indicative of one or more predefined quality attributes for the one or more data segments;
generating, based on the quality evaluation data, quality metrics data indicative of one or more quality metrics for the one or more data segments;
generating an augmented result set for the federated query by augmenting the result set with the quality metrics data; and
generating a query response that comprises the augmented result set to visually indicate the one or more quality metrics for the one or more data segments.
20. The one or more non-transitory computer-readable storage media of claim 19 , wherein the processor-executable instructions, when executed to by the one or more processors, further cause the one or more processors to perform operations comprising:
providing the query response that comprises the augmented result set to a computing entity associated with the federated query to render visual data associated with the augmented result set via a user interface of the computing entity.