Systems and methods of metadata monitoring and analysis
A system and method of generating platform-dependent queries from a platform-agnostic query are disclosed. A data pipeline comprising a plurality of events is implemented. Each event in the plurality of events has a set of platform-dependent metadata associated therewith and each of the plurality of events is processed by one of a plurality of ingestion platforms. Metadata associated with each of the plurality of events is stored in a combined metadata repository. The combined metadata repository stores metadata extracted from two or more platforms in a first repository. A platform-agnostic query configured to obtain one or more metadata search results from the platform-dependent metadata is received and deployed to the first repository within the combined metadata repository. The platform-agnostic query is configured to return a result set including metadata obtained from each of the two or more platforms.
1. A system for generating item recommendations, comprising:
a memory having instructions stored thereon, and a processor configured to read the instructions to:
implement a data pipeline comprising a plurality of events, wherein each event in the plurality of events has a set of platform-dependent metadata associated therewith, and wherein each of the plurality of events is processed by one of a plurality of ingestion platforms;
store metadata associated with each of the plurality of events in a combined metadata repository, wherein the combined metadata repository stores metadata extracted from two or more platforms in a first repository;
receive a platform-agnostic query configured to obtain one or more metadata search results from the platform-dependent metadata; and
deploy the platform-agnostic query to the first repository within the combined metadata repository, wherein the platform-agnostic query is configured to return a result set including metadata obtained from each of the two or more platforms.
2. The system of claim 1 , wherein the combined metadata repository comprises a schema repository.
3. The system of claim 2 , wherein the processor is configured to read the instructions to:
extract a first set of metadata from the data pipeline, wherein the first set of metadata is associated with a first platform;
generate a metadata schema for the first set of metadata; and
compare the generated metadata schema to a plurality of stored metadata schemas maintained by the combined metadata repository.
4. The system of claim 1 , wherein the processor is configured to read the instructions to generate a first platform-dependent query from the platform-agnostic query, wherein the first platform-dependent query is configured to be deployed to the first repository.
5. The system of claim 4 , wherein the first platform-dependent query is generated by applying a first set of conversion rules to the platform-agnostic query.
6. The system of claim 1 , wherein the processor is configured to read the instructions to:
receive one or more user query parameters; and
generate the platform-agnostic query in a predetermined platform-agnostic language.
7. The system of claim 6 , wherein the platform-agnostic language is SQL.
8. A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by a processor cause a device to perform operations comprising:
implementing a data pipeline comprising a plurality of events, wherein each event in the plurality of events has a set of platform-dependent metadata associated therewith, and wherein each of the plurality of events is processed by one of a plurality of ingestion platforms;
storing metadata associated with each of the plurality of events in a combined metadata repository, wherein the combined metadata repository stores metadata extracted from two or more platforms in a first repository;
receiving a platform-agnostic query configured to obtain one or more metadata search results from the platform-dependent metadata; and
deploying the platform-agnostic query to the first repository within the combined metadata repository, wherein the platform-agnostic query is configured to return a result set including metadata obtained from each of the two or more platforms.
9. The non-transitory computer readable medium of claim 8 , wherein the combined metadata repository comprises a schema repository.
10. The non-transitory computer readable medium of claim 9 , wherein the instructions when executed by the processor cause the device to perform further operations comprising:
extracting a first set of metadata from the data pipeline, wherein the first set of metadata is associated with a first platform;
generating a metadata schema for the first set of metadata; and
comparing the generated metadata schema to a plurality of stored metadata schemas maintained by the combined metadata repository.
11. The non-transitory computer readable medium of claim 8 , wherein the instructions when executed by the processor cause the device to perform further operations comprising generating a first platform-dependent query from the platform-agnostic query, wherein the first platform-dependent query is configured to be deployed to the first repository.
12. The non-transitory computer readable medium of claim 11 , wherein the first platform-dependent query is generated by applying a first set of conversion rules to the platform-agnostic query.
13. The non-transitory computer readable medium of claim 8 , wherein the instructions when executed by the processor cause the device to perform further operations comprising:
receiving one or more user query parameters; and
generating the platform-agnostic query in a predetermined platform-agnostic language.
14. The non-transitory computer readable medium of claim 13 , wherein the platform-agnostic language is SQL.
15. A method, comprising:
implementing a data pipeline comprising a plurality of events, wherein each event in the plurality of events has a set of platform-dependent metadata associated therewith, and wherein each of the plurality of events is processed by one of a plurality of ingestion platforms;
storing metadata associated with each of the plurality of events in a combined metadata repository, wherein the combined metadata repository stores metadata extracted from two or more platforms in a first repository;
receiving a platform-agnostic query configured to obtain one or more metadata search results from the platform-dependent metadata; and
deploying the platform-agnostic query to the first repository within the combined metadata repository, wherein the platform-agnostic query is configured to return a result set including metadata obtained from each of the two or more platforms.
16. The method of claim 15 , wherein the combined metadata repository comprises a schema repository.
17. The method of claim 16 , comprising:
extracting a first set of metadata from the data pipeline, wherein the first set of metadata is associated with a first platform;
generating a metadata schema for the first set of metadata; and
comparing the generated metadata schema to a plurality of stored metadata schemas maintained by the combined metadata repository.
18. The method of claim 15 , comprising generating a first platform-dependent query from the platform-agnostic query, wherein the first platform-dependent query is configured to be deployed to the first repository.
19. The method of claim 18 , wherein the first platform-dependent query is generated by applying a first set of conversion rules to the platform-agnostic query.
20. The method of claim 15 , comprising:
receiving one or more user query parameters; and
generating the platform-agnostic query in a predetermined platform-agnostic language.