Dynamic ingestion of data in a massive data ingestion system
A method for execution by one or more computing devices of a massive data ingestion system, the method includes retrieving a data segment from a cache memory of the massive data ingestion system. The method further includes error encoding, in accordance with error encoding parameters, the data segment to produce a set of encoded data slices, where the error encoding parameters include a pillar width number and a decode threshold number, and where the decode threshold number is a minimum number of encoded data slices needed to reconstruct the data segment. The method further includes storing the set of encoded data slices in a set of storage units of the massive data ingestion system.
1 . A method for execution by one or more computing devices of a massive data ingestion system, the method comprising:
retrieving a data segment from a cache memory of the massive data ingestion system;
error encoding, in accordance with error encoding parameters, the data segment to produce a set of encoded data slices, wherein the error encoding parameters include a pillar width number and a decode threshold number, and wherein the decode threshold number is a minimum number of encoded data slices needed to reconstruct the data segment; and
storing the set of encoded data slices in a set of storage units of the massive data ingestion system.
2 . The method of claim 1 further comprises:
determining the set of encoded data slices is successfully stored in the set of storage units; and
deleting the data segment from the cache memory.
3 . The method of claim 2 further comprises:
updating a directory to disassociate a data identifier of the data segment with the cache memory.
4 . The method of claim 3 , wherein the updating the directory comprises being based on receiving a confirmation message.
5 . The method of claim 3 , wherein the updating the directory comprises being based on a query.
6 . The method of claim 3 , wherein the updating the directory comprises being based on receiving at least a write threshold number of commit acknowledgments associated with the set of encoded data slices.
7 . The method of claim 1 , wherein the retrieving is based on one or more retrieval parameters.
8 . The method of claim 7 , wherein a retrieval parameter of the one or more retrieval parameters comprises a schedule.
9 . The method of claim 7 , wherein a retrieval parameter of the one or more retrieval parameters comprises a fixed time.
10 . The method of claim 7 , wherein a retrieval parameter of the one or more retrieval parameters comprises a time that the data was stored in the cache memory.
11 . The method of claim 7 , wherein a retrieval parameter of the one or more retrieval parameters comprises a data identifier associated with the data segment.
12 . The method of claim 7 , wherein a retrieval parameter of the one or more retrieval parameters comprises an error message.
13 . The method of claim 7 , wherein a retrieval parameter of the one or more retrieval parameters comprises a cache memory capacity indicator.
14 . The method of claim 7 , wherein a retrieval parameter of the one or more retrieval parameters comprises a cache memory capacity threshold.
15 . The method of claim 1 further comprises:
receiving data for storage in the massive data ingestion system; and
storing the data segment of the data in the cache memory.
16 . A non-transitory computer readable storage device comprises:
at least one memory section for storing operational instructions that, when executed by one or more computing devices of a storage network, cause the one or more computing devices to:
retrieve a data segment from a cache memory of a massive data ingestion system;
error encode, in accordance with error encoding parameters, the data segment to produce a set of encoded data slices, wherein the error encoding parameters include a pillar width number and a decode threshold number, and wherein the decode threshold number is a minimum number of encoded data slices needed to reconstruct the data segment; and
store the set of encoded data slices in a set of storage units of the massive data ingestion system.
17 . The non-transitory computer readable storage device of claim 16 , wherein the at least one memory section stores further operational instructions that, when executed by the one or more computing devices, cause the one or more computing devices to retrieve the data segment from the cache memory based on one or more retrieval parameters.
18 . The non-transitory computer readable storage device of claim 17 , wherein the at least one memory section stores further operational instructions that, when executed by the one or more computing devices, cause the one or more computing devices to determine a retrieval parameter of the one or more retrieval parameters is a time when the data was stored in the cache memory.
19 . The non-transitory computer readable storage device of claim 17 , wherein the at least one memory section stores further operational instructions that, when executed by the one or more computing devices, cause the one or more computing devices to determine a retrieval parameter of the one or more retrieval parameters is a schedule.
20 . The non-transitory computer readable storage device of claim 17 , wherein the at least one memory section stores further operational instructions that, when executed by the one or more computing devices, cause the one or more computing devices to determine a retrieval parameter of the one or more retrieval parameters is a cache memory capacity indicator.