Ingesting and storing data in a massive data ingestion system
A method for execution by one or more computing devices of a massive data ingestion system includes receiving data for storage in the massive data ingestion system. The method further includes storing a data segment of the data in a cache memory of the massive data ingestion system. The method further includes retrieving, based on one or more retrieval parameters, the data segment from the cache memory. The method further includes error encoding, in accordance with error encoding parameters, the data segment to produce a set of encoded data slices, wherein the error encoding parameters include a pillar width number and a decode threshold number, and wherein the decode threshold number is a minimum number of encoded data slices needed to reconstruct the data segment. The method further includes storing the set of encoded data slices in a set of storage units of the massive data ingestion system.
1. A method for execution by one or more computing devices of a massive data ingestion system, the method comprising:
receiving data for storage in the massive data ingestion system;
storing a data segment of the data in a cache memory of the massive data ingestion system;
retrieving, based on one or more retrieval parameters, the data segment from the cache memory;
error encoding, in accordance with error encoding parameters, the data segment to produce a set of encoded data slices, wherein the error encoding parameters include a pillar width number and a decode threshold number, and wherein the decode threshold number is a minimum number of encoded data slices needed to reconstruct the data segment; and
storing the set of encoded data slices in a set of storage units of the massive data ingestion system.
2. The method of claim 1 , wherein a retrieval parameter of the one or more retrieval parameters comprises a schedule.
3. The method of claim 1 , wherein a retrieval parameter of the one or more retrieval parameters comprises a fixed time.
4. The method of claim 1 , wherein a retrieval parameter of the one or more retrieval parameters comprises a time that the data was stored in the cache memory.
5. The method of claim 1 , wherein a retrieval parameter of the one or more retrieval parameters comprises a data identifier associated with the data segment.
6. The method of claim 1 , wherein a retrieval parameter of the one or more retrieval parameters comprises an error message.
7. The method of claim 1 , wherein a retrieval parameter of the one or more retrieval parameters comprises a cache memory capacity indicator.
8. The method of claim 1 , wherein a retrieval parameter of the one or more retrieval parameters comprises a cache memory capacity threshold.
9. The method of claim 1 further comprises:
determining the set of encoded data slices is successfully stored in the set of storage units; and
deleting the data segment from the cache memory.
10. The method of claim 9 further comprises:
updating a directory to disassociate a data identifier of the data segment with the cache memory.
11. A computing device of a massive data ingestion system, the computing device comprises:
memory;
an interface; and
a processing module operably coupled to the memory and the interface, wherein the processing module is operable to:
receive, via the interface, data for storage in the massive data ingestion system;
store a data segment of the data in a cache memory of the massive data ingestion system;
retrieve, via the interface and based on one or more retrieval parameters, the data segment from the cache memory;
error encode, in accordance with error encoding parameters, the data segment to produce a set of encoded data slices, wherein the error encoding parameters include a pillar width number and a decode threshold number, and wherein the decode threshold number is a minimum number of encoded data slices needed to reconstruct the data segment; and
store the set of encoded data slices in a set of storage units of the massive data ingestion system.
12. The computing device of claim 11 , wherein the processing module is further operable to determine a retrieval parameter of the one or more retrieval parameters is a schedule.
13. The computing device of claim 11 , wherein the processing module is further operable to determine a retrieval parameter of the one or more retrieval parameters is a fixed time.
14. The computing device of claim 11 , wherein the processing module is further operable to determine a retrieval parameter of the one or more retrieval parameters is a time that the data was stored in the cache memory.
15. The computing device of claim 11 , wherein the processing module is further operable to determine a retrieval parameter of the one or more retrieval parameters comprises a data identifier associated with the data segment.
16. The computing device of claim 11 , wherein the processing module is further operable to determine a retrieval parameter of the one or more retrieval parameters comprises an error message.
17. The computing device of claim 11 , wherein the processing module is further operable to determine a retrieval parameter of the one or more retrieval parameters comprises a cache memory capacity indicator.
18. The computing device of claim 11 , wherein the processing module is further operable to determine a retrieval parameter of the one or more retrieval parameters comprises a cache memory capacity threshold.
19. The computing device of claim 11 , wherein the processing module is further operable to:
determine the set of encoded data slices is successfully stored in the set of storage units; and
delete the data segment from the cache memory.
20. The computing device of claim 19 , wherein the processing module is further operable to:
update a directory to disassociate a data identifier of the data segment with the cache memory.