Managing provenance information for data processing pipelines
A method for managing provenance information associated to one or more interconnected provenance entities in a provenance system for data processing pipelines in a distributed cloud environment over a network interface, wherein each of the data processing pipelines is configured to read in data, transform the data, and output transformed data is disclosed. The method comprises steps being performed by a configuration component of obtaining at least one declarative intent representing a configuration indicative of requirements and levels of priority for storage of provenance information for each of the data processing pipelines, deriving the requirements and levels of priority for storage of provenance information for each of the data processing pipelines based on the obtained at least one declarative intent, wherein one of the levels of priority—first level of priority—is higher than the other levels of priority—second levels of priority, estimating storage capacity for storage of provenance information in the provenance system based on the derived requirements and levels of priority, storing the provenance information according to the derived requirements and levels of priority for storage of provenance information and for each of the data processing pipelines, and when actual storage consumption for storage of provenance information in the provenance system meets a threshold of storage capacity set based on the estimated storage capacity: reducing a data amount for storage of provenance information of the second levels of priority in the provenance system. Corresponding computer program product, arrangement, configuration component, and system are also disclosed.
1. A method for managing provenance information associated with one or more interconnected provenance entities in a provenance system, the method being performed by a configuration component, the method comprising:
obtaining a declarative intent for storage of provenance information for a processing pipeline, wherein the processing pipeline is part of a distributed cloud environment and the processing pipeline is configured to read in data, transform the data, and output transformed data;
identifying a category of provenance information based on the declarative intent;
deriving a first level of priority associated with the category of provenance information based on the declarative intent;
estimating an amount of storage based on the category of provenance information and the first level of priority;
storing the provenance information in a storage of provenance information;
determining a data storage level associated with the provenance system is greater than a threshold, wherein the threshold is based on the estimated amount of storage; and
reducing an amount of provenance information stored in the storage of provenance information as a result of determining the data storage level is greater than the threshold, wherein reducing the amount of provenance information stored in the storage of provenance information is based on a second level of priority.
2. The method of claim 1 , wherein reducing the amount of provenance information stored in the storage of provenance information comprises transmitting a signal for causing:
compressing the provenance information in the storage,
removing indexing of the provenance information in the storage, and/or
moving the provenance information in the storage to remote storage in the distributed cloud environment, wherein the provenance information is compressed or non-compressed.
3. The method of claim 1 , wherein deriving the first level of priority is based on:
an age of the processing pipeline,
specific tags of the processing pipeline,
specific clients/groups of clients associated with the processing pipeline, and/or
specific data sources for the processing pipeline.
4. The method of claim 1 , wherein the declarative intent is indicative of a time interval requirement, wherein the time interval requirement identifies a time interval during which the provenance information associated with the one or more interconnected provenance entities should be stored in the provenance system.
5. The method of claim 1 , wherein the declarative intent is indicative of a continuation requirement for continuously storing provenance information associated with the one or more interconnected provenance entities in the provenance system.
6. The method of claim 1 , further comprising:
notifying a client associated with the declarative intent that the stored provenance information is available for provision.
7. The method of claim 6 , further comprising:
providing the client with access to the stored provenance information over a network interface.
8. The method of claim 6 , further comprising:
providing the client with data streams of the stored provenance information over a network interface.
9. The method of claim 8 , wherein the provided data streams of the stored provenance information to the client serves as data for:
debugging,
analysis queries,
scheduling, and/or
optimizations.
10. The method of claim 1 , wherein the provenance information comprises:
information identifying a data collection location,
information identifying a data collection purpose,
information identifying context data,
information identifying a person associated with collected data,
information identifying a person with access to the collected data,
attributes of the collected data,
updates to the collected data,
creation of the collected data, and/or
lineage of collected data, depending on data type and an environment of collection.
11. The method of claim 1 , wherein the one or more interconnected provenance entities in the provenance system represent one or more interconnected nodes of a graph structure and the interconnections represent edges and relationships between the nodes of a subgraph of the graph structure.
12. The method of claim 11 , wherein the graph structure is a graph database.
13. The method of claim 1 , wherein the configuration component comprises a plurality of sub-components each acting as a separate configuration component for a data processing pipeline.
14. The method of claim 1 , wherein the stored provenance information relates to:
industrial robotics,
sensors in a building, and/or
production machines.
15. A computer program product comprising a non-transitory computer readable medium storing a computer program comprising instructions which when executed by processing circuitry causes the processing circuitry to perform the method of claim 1 .
16. An arrangement for a configuration component adapted for managing provenance information associated with one or more interconnected provenance entities in a provenance system the arrangement comprising a controller configured to:
obtain a declarative intent for storage of provenance information for a processing pipeline, wherein the processing pipeline is part of a distributed cloud environment and the processing pipeline is configured to read in data, transform the data, and output transformed data;
identify a category of provenance information based on the declarative intent;
derive a first level of priority associated with the category of provenance information based on the declarative intent;
estimate an amount of storage based on the category of provenance information and the first level of priority;
store the provenance information in a storage of provenance information;
determine a data storage level associated with the provenance system is greater than a threshold, wherein the threshold is based on the estimated amount of storage; and
reduce an amount of provenance information stored in the storage of provenance information as a result of determining the data storage level is greater than the threshold, wherein reducing the amount of provenance information stored in the storage of provenance information is based on a second level of priority.
17. A configuration component comprising the arrangement according to claim 16 .
18. A system comprising the arrangement according to claim 16 .
19. The system according to claim 18 , further comprising at least one client, a local storage, and a remote storage.