Dynamically processing data in a vast data ingestion system
A method begins with a first group of computing devices of a plurality of computing devices of a storage network receiving data objects having a first data type identifier of a plurality of data type identifiers, where the plurality of data type identifiers correspond to a plurality of data types associated with the data objects. The method continues with the first group of computing devices interpreting the data objects having the first data type identifier to sort, based on sorting criteria the data objects into a first processing category and a second processing category. The method continues with the first group of computing devices error encoding the data objects in the second processing category based on short term storage error encoding parameters to produce pluralities of sets of encoded data slices and sending the pluralities of sets of encoded data slices to storage and execution units for storage therein.
1. A method for execution in a storage network, the method comprises:
determining, by one or more storage and execution units of the storage network, data objects of interest from data objects that are associated with a particular processing category, wherein the determining the data objects of interest is based on interest criteria, wherein the data objects are associated with data type identifiers of a plurality of data type identifiers, and wherein the plurality of data type identifiers correspond to a plurality of data types associated with a plurality of data objects that include the data objects;
error encoding, by the one or more of storage and execution units, the data objects of interest based on long term storage error encoding parameters to produce pluralities of sets of encoded data slices; and
sending, by the one or more of storage and execution units, the pluralities of sets of encoded data slices to a second set of storage and execution units for storage therein.
2. The method of claim 1 , wherein a data type of the plurality of data types comprises one or more of:
a data file;
an email;
a blog;
a white paper;
a social media post; and
graphics.
3. The method of claim 1 further comprises:
receiving the plurality of data objects; and
assigning a data type identifier of the plurality of data type identifiers to each data object of the plurality of data objects based on a data type of the each data object.
4. The method of claim 1 , wherein the long term storage error encoding parameters comprise a pillar width to decode threshold difference greater than a long term difference threshold.
5. The method of claim 1 , wherein the long term storage error encoding parameters comprise a pillar width number greater than a long term pillar width number threshold.
6. The method of claim 1 , wherein the long term storage error encoding parameters comprise a decode threshold number greater than a long term decode number threshold.
7. The method of claim 1 , wherein the interest criteria comprises:
a data type of the plurality of data types and a result from sorting the plurality of data objects into the particular processing category based on sorting criteria.
8. The method of claim 7 , wherein the sorting criteria comprises one or more of:
a word search;
an image search;
destination information;
facial recognition;
an encryption type; and
source information.
9. A computer readable memory device of a storage network comprises:
at least one memory element for storing operational instructions that, when executed by at least one of one or more storage and execution units of the storage network and one or more computing devices of the storage network, causes the at least one of the one or more storage and execution units of the storage network and the one or more computing devices to:
determine data objects of interest from data objects that are associated with a processing category, wherein the determining the data objects of interest is based on interest criteria, wherein the data objects are associated with data type identifiers of a plurality of data type identifiers, and wherein the plurality of data type identifiers correspond to a plurality of data types associated with a plurality of data objects that include the data objects;
error encode the data objects of interest based on long term storage error encoding parameters to produce pluralities of sets of encoded data slices; and
send the pluralities of sets of encoded data slices to a second set of storage and execution units for storage.
10. The computer readable memory device of claim 9 , wherein a data type of the plurality of data types comprises one or more of:
a data file;
an email;
a blog;
a white paper;
a social media post; and
graphics.
11. The computer readable memory device of claim 9 , wherein the at least one memory element stores further operational instructions that, when executed by the at least one of the one or more storage and execution units and the one or more computing devices, causes the at least one of the one or more storage and execution units and the one or more computing devices to:
receive the plurality of data objects; and
assign a data type identifier of the plurality of data type identifiers to each data object of the plurality of data objects based on a data type of the each data object.
12. The computer readable memory device of claim 9 , wherein the at least one memory element stores further operational instructions that, when executed by the at least one of the one or more storage and execution units and the one or more computing devices of the plurality of computing devices, causes the at least one of the one or more storage and execution units and the one or more computing devices to:
determine the long term storage error encoding parameters include a pillar width to decode threshold difference greater than a long term difference threshold.
13. The computer readable memory device of claim 9 , wherein the at least one memory element stores further operational instructions that, when executed by the at least one of the one or more storage and execution units and the one or more computing devices, causes the at least one of the one or more storage and execution units and the one or more computing devices to:
determine the long term storage error encoding parameters include a decode threshold number greater than a long term decode number threshold.
14. The computer readable memory device of claim 9 , wherein the at least one memory element stores further operational instructions that, when executed by the at least one of the one or more storage and execution units and the one or more computing devices, causes the at least one of the one or more storage and execution units and the one or more computing devices to:
determine the long term storage error encoding parameters include a pillar width number greater than a long term pillar width number threshold.
15. The computer readable memory device of claim 1 , wherein the interest criteria comprises:
a data type of the plurality of data types and a result from sorting the plurality of data objects into the particular processing category based on sorting criteria.
16. The computer readable memory device of claim 15 , wherein the sorting criteria comprises one or more of:
a word search;
an image search;
destination information;
facial recognition;
an encryption type; and
source information.