Dispersed storage network with metadata generation and methods for use therewith
View Patent ↗A technique by a processing module of a computing device of a dispersed storage network (DSN) to generate metadata associated with a stored data object. Once the data object is identified, a data segment of the data object is determined and a number of data slices corresponding to the data segment is retrieved. Then, the data segment is regenerated from the retrieved data slices. Once the data segment is regenerated, metadata that is associated with the data segment is generated and stored.
1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:
a) identifying a data object stored in the DSN, in which the data object is segmented into a plurality of data segments and respective data segments are error encoded to generate a plurality of data slices and the plurality of data slices are stored in a plurality of storage devices of the DSN, and in which a threshold number of data slices of a particular data segment are needed to reconstruct the particular data segment, the threshold number of data slices being less than the plurality of data slices generated for the particular data segment;
b) determining a data segment of the data object stored in the plurality of storage devices of the DSN;
c) retrieving the threshold number of data slices corresponding to the data segment;
d) regenerating the data segment from the threshold number of data slices retrieved corresponding to the data segment;
e) generating metadata from the regenerated data segment;
f) storing the metadata associated with the data segment;
g) comparing an amount of the stored metadata to a completeness threshold to determine if additional metadata needs to be generated and, when additional metadata is required, determining that the metadata does not include data in at least one of a plurality of categories;
h) determining another data segment of the data object when the amount of the stored metadata does not meet the completeness threshold; and
i) repeating c), d), e), f), g) and h) for one or more additional data segment or segments until a determination is made that the stored metadata meets the completeness threshold.
2. The method of claim 1 wherein the metadata corresponding to the data segment includes at least one of: a type of data, a keyword, a phrase, a pattern, a priority indicator, a relationship to other data objects, a security indicator, a user identifier, or a timestamp.
3. The method of claim 1 further comprising:
determining when additional metadata is no longer required.
4. A dispersed storage (DS) processing unit, having a hardware processor and memory, for use in a dispersed storage network (DSN) comprises:
at least one module, operable with the hardware processor, causes the hardware processor to:
a) identify a data object stored in the DSN, in which the data object is segmented into a plurality of data segments and respective data segments are error encoded to generate a plurality of data slices and the plurality of data slices are stored in a plurality of storage devices of the DSN, and in which a threshold number of data slices of a particular data segment are needed to reconstruct the particular data segment, the threshold number of data slices being less than the plurality of data slices generated for the particular data segment;
b) determine a data segment of the data object stored in the plurality of storage devices of the DSN;
c) retrieve the threshold number of data slices corresponding to the data segment;
d) regenerate the data segment from the threshold number of data slices retrieved corresponding to the data segment;
e) generate metadata from the regenerated data segment;
f) store the metadata associated with the data segment;
g) compare an amount of the stored metadata to a completeness threshold to determine if additional metadata needs to be generated and, when additional metadata is required, determine that the metadata does not include data in at least one of a plurality of categories;
h) determine another data segment of the data object when the amount of the stored metadata does not meet the completeness threshold; and
i) repeat c), d), e), f), g) and h) for one or more additional data segment or segments until a determination is made that the stored metadata meets the completeness threshold.
5. The DS processing unit of claim 4 wherein the metadata corresponding to the data segment includes at least one of: a type of data, a keyword, a phrase, a pattern, a priority indicator, a relationship to other data objects, a security indicator, a user identifier, or a timestamp.
6. The DS processing unit of claim 4 wherein the at least one module, operable with the hardware processor, further causes the hardware processor to:
determine when additional metadata is no longer required.
7. A non-transitory computer readable storage medium comprises:
at least one memory section that stores operational instructions that, when executed by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), causes the one or more computing devices to:
a) identify a data object previously stored in the DSN, in which the data object is segmented into a plurality of data segments and respective data segments are error encoded to generate a plurality of data slices and the plurality of data slices are stored in a plurality of storage devices of the DSN, and in which a threshold number of data slices of a particular data segment are needed to reconstruct the particular data segment, the threshold number of data slices being less than the plurality of data slices generated for the particular data segment;
b) determine a data segment of the data object stored in the plurality of storage devices of the DSN;
c) retrieve the threshold number of data slices corresponding to the data segment;
d) regenerate the data segment from the threshold number of data slices retrieved corresponding to the data segment;
e) generate metadata from the regenerated data segment;
f) store the metadata associated with the data segment;
g) compare an amount of the stored metadata to a completeness threshold to determine if additional metadata needs to be generated and, when additional metadata is required, determine that the metadata does not include data in at least one of a plurality of categories;
h) determine another data segment of the data object when the amount of the stored metadata does not meet the completeness threshold; and
i) repeat c), d), e), f), g) and h) for one or more additional data segment or segments until a determination is made that the stored metadata meets the completeness threshold.
8. The non-transitory computer readable storage medium of claim 7 wherein the metadata corresponding to the data segment includes at least one of: a type of data, a keyword, a phrase, a pattern, a priority indicator, a relationship to other data objects, a security indicator, a user identifier, or a timestamp.
9. The non-transitory computer readable storage medium of claim 7 wherein the operational instructions, when executed by the one or more processing modules of the one or more computing devices of the dispersed storage network (DSN), further causes the one or more computing devices to:
determine when additional metadata is no longer required.
10. The non-transitory computer readable storage medium of claim 7 wherein the operational instructions, when executed by the one or more processing modules of the one or more computing devices of the dispersed storage network (DSN), further causes the one or more computing devices to:
determine if the metadata does not include data in at least one of a plurality of categories.