Data valuation at content ingest
A data set is ingested by a data storage system. A valuation is calculated for the data set at the time of ingestion by the data storage system. The calculated valuation is stored. In one illustrative example, the data set is ingested by the data storage system as a data stream such that the valuation calculation step is performed on the data set as the data set streams into the data storage system.
1. A method comprising:
obtaining a given data set as the given data set is ingested by a data storage system;
calculating a valuation for the given data set at the time of ingestion by the data storage system with at least one valuation processing node of a valuation framework to obtain a first valuation for the given data set;
determining whether the given data set correlates with one other ingested data set and calculating, at the time of ingestion and with at least one correlation processing node of the valuation framework, a correlation value representative of a correlation between the given data set and the one other ingested data set, the correlation value being independent of the first valuation;
updating, at the time of ingestion, the first valuation for the given data set based upon the correlation value and a valuation for the one other ingested data set to obtain an updated valuation for the given data set; and
storing at least one of the first valuation, the updated valuation and the correlation value in at least one valuation database;
wherein the obtaining, calculating, determining, updating and storing steps are performed by one or more processing devices, each processing device comprising a processor and a memory.
2. The method of claim 1 , further comprising accessing the valuation database via an application programming interface.
3. The method of claim 1 , further comprising assigning a valuation handle to the given data set upon ingestion by the data storage system.
4. The method of claim 1 , wherein the given data set is ingested by the data storage system as a data stream.
5. The method of claim 4 , wherein calculating a valuation is performed on the given data set as the data set streams into the data storage system.
6. The method of claim 5 , wherein calculating a valuation is performed on the content of the streaming data set.
7. The method of claim 1 , further comprising:
receiving a query corresponding to the given data set; and
returning the calculated valuation for the given data set in response to the query.
8. The method of claim 1 , further comprising updating the valuation of the one other ingested data set based on the first valuation of the given data set and the correlation value.
9. The method of claim 8 , wherein updating the valuation of the one other ingested data set is performed on the other ingested data set as the data set streams into the data storage system.
10. The method of claim 1 , wherein calculating a valuation comprises machine learning based valuation processes.
11. The method of claim 1 , wherein calculating a valuation is performed on the given data set prior to transformation of the given data set.
12. The method of claim 11 , wherein the storing step includes flagging the first valuation in the at least one valuation database as a calculation performed prior to transformation of the given data set.
13. The method of claim 1 , wherein calculating a valuation is based on a given data policy.
14. The method of claim 1 , further comprising:
monitoring fluctuations in calculated valuations for data sets ingested by the data storage system over a given time period; and
triggering one or more actions to be performed based on the monitored fluctuations.
15. The method of claim 1 , further comprising determining placement of the given data set in the data storage system based on at least one of the first valuation and the updated valuation.
16. The method of claim 1 , wherein calculating a valuation is performed by one or more processing nodes interconnected to form a directed acyclic graph.
17. An article of manufacture comprising a processor-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by one or more processing devices implement steps of:
obtaining a given data set as the given data set is ingested by a data storage system;
calculating a valuation for the given data set at the time of ingestion by the data storage system with at least one valuation processing node of a valuation framework to obtain a first valuation for the given data set;
determining whether the data set correlates with one other ingested data set and calculating, at the time of ingestion and with at least one correlation processing node of the valuation framework, a correlation value representative of a correlation between the given data set and the one other ingested data set, the correlation value being independent of the first valuation;
updating the first valuation for the given data set based upon the correlation value and a valuation for the one other ingested data set to obtain an updated valuation for the given data set; and
storing at least one of the first valuation, the updated valuation and the correlation value in at least one valuation database.
18. The article of manufacture of claim 17 wherein the one or more software programs when executed by one or more processing devices implement steps of:
as the data set streams into the data storage system, updating the valuation of the one other ingested data set based on the first valuation of the given data set and the correlation value.
19. A system comprising:
one or more processors operatively coupled to one or more memories configured to:
obtain a given data set as the given data set is ingested by a data storage system;
calculate a valuation for the given data set at the time of ingestion by the data storage system with at least one valuation processing node of a valuation framework to obtain a first valuation for the given data set;
determine whether the given data set correlates with one other ingested data set and calculate, at the time of ingestion and with at least one correlation processing node of the valuation framework, a correlation value representative of a correlation between the given data set and the one other ingested data set, the correlation value being independent of the first valuation;
update the first valuation for the given data set based upon the correlation valuation and a valuation for the one other ingested data set to obtain an updated valuation for the given data set; and
store at least one of the first valuation, updated valuation and the correlation value in at least one valuation database.
20. The system of claim 19 wherein the one or more processors are configured to:
update the valuation of the one other ingested data set based on the first valuation of the given data set and the correlation value.