Segmenting data for storage in a dispersed storage network
View Patent ↗A method begins by a processing module receiving data of a file for storage in a dispersed storage network (DSN) memory and determining a segmentation scheme for storing the data. The method continues with the processing module determining how to store the data in accordance with the segmentation scheme to produce information for storing the data and generating an entry within a segment allocation table associated with the file, wherein the entry includes the information for storing the data and the segmentation scheme. The method continues with the processing module facilitating storage of the segment allocation table in the DSN memory. The method continues with the processing module segmenting the data in accordance with the segmentation scheme to produce a plurality of data segments and facilitating storage of the plurality of data segments in the DSN memory in accordance with the information for storing the data.
1. A method comprises:
receiving data of a file for storage in a dispersed storage network (DSN) memory;
determining a segmentation scheme for storing the data;
determining how to store the data in accordance with the segmentation scheme to produce information for storing the data;
generating an entry within a segment allocation table associated with the file, wherein the entry includes the information for storing the data and the segmentation scheme;
facilitating storage of the segment allocation table in the DSN memory by:
obtaining a segment allocation table vault source name;
dispersed storage error encoding the segment allocation table to produce encoded table slices; and
outputting the encoded table slices to the DSN memory for storage therein utilizing the segment allocation table vault source name;
segmenting the data in accordance with the segmentation scheme to produce a plurality of data segments; and
facilitating storage of the plurality of data segments in the DSN memory in accordance with the information for storing the data.
2. The method of claim 1 , wherein the determining the segmentation scheme is based on at least one of:
a previous segmentation scheme;
a data size indicator;
a data type indicator;
a storage requirement;
a vault identifier (ID);
a lookup;
a message; and
a query.
3. The method of claim 1 further comprises:
the information for storing the data including:
a start segment vault source name; and
a total length of the data; and
the segmenting scheme including:
a segment size; and
a segmentation approach.
4. The method of claim 3 , wherein the segmenting scheme comprises:
a fixed segmentation approach and a corresponding fixed segment size such that each of the plurality of data segments has a size no greater than the fixed segment size.
5. The method of claim 3 , wherein the segmenting scheme comprises:
a varying segmentation approach and an initial segment size such that a data segment of the plurality of data segments has a size corresponding to the initial segment size and remaining data segments of the plurality of data segments have a size based on the initial segment size and the varying segmentation approach.
6. The method of claim 1 , wherein the facilitating storage of the plurality of data segments in the DSN memory comprises:
for each data segment of the plurality of data segments:
dispersed storage error encoding the data segment to produce a set of encoded data slices;
generating a set of slice names corresponding to the set of encoded data slices based on a start segment vault source name of the segment allocation table, wherein a common segment number of the set of slice names includes a sequentially increasing segment number; and
sending the set of encoded data slices and the set of slice names to the DSN memory.
7. The method of claim 1 , wherein the facilitating storage of the plurality of data segments in the DSN memory further comprises:
for each data segment of the plurality of data segments:
dispersed storage error encoding the data segment to produce a set of encoded data slices;
generating a set of slice names corresponding to the set of encoded data slices based on a start segment vault source name of the segment allocation table, wherein an append marker field of the set of slice names includes a sequentially increasing append marker; and
sending the set of encoded data slices and the set of slice names to the DSN memory.
8. A dispersed storage (DS) processing unit comprises:
a first module for receiving data of a file for storage in a dispersed storage network (DSN) memory;
a second module for:
determining a segmentation scheme for storing the data;
determining how to store the data in accordance with the segmentation scheme to produce information for storing the data; and
generating an entry within a segment allocation table associated with the file, wherein the entry includes the information for storing the data and the segmentation scheme;
a third module for facilitating storage of the segment allocation table in the DSN memory, wherein the third module facilitates the storage of the segment allocation table by:
obtaining a segment allocation table vault source name;
dispersed storage error encoding the segment allocation table to produce encoded table slices; and
outputting the encoded table slices to the DSN memory for storage therein utilizing the segment allocation table vault source name;
a fourth module for segmenting the data in accordance with the segmentation scheme to produce a plurality of data segments; and
a fifth module for facilitating storage of the plurality of data segments in the DSN memory in accordance with the information for storing the data.
9. The DS processing unit of claim 8 , wherein the second module determines the segmentation scheme based on at least one of:
a previous segmentation scheme;
a data size indicator;
a data type indicator;
a storage requirement;
a vault identifier (ID);
a lookup;
a message; and
a query.
10. The DS processing unit of claim 8 further comprises:
the information for storing the data including:
a start segment vault source name; and
a total length of the data; and
the segmenting scheme including:
a segment size; and
a segmentation approach.
11. The DS processing unit of claim 10 , wherein the segmenting scheme comprises:
a fixed segmentation approach and a corresponding fixed segment size such that each of the plurality of data segments has a size no greater than the fixed segment size.
12. The DS processing unit of claim 10 , wherein the segmenting scheme comprises:
a varying segmentation approach and an initial segment size such that a data segment of the plurality of data segments has a size corresponding to the initial segment size and remaining data segments of the plurality of data segments have a size based on the initial segment size and the varying segmentation approach.
13. The DS processing unit of claim 8 , wherein the facilitating storage of the plurality of data segments in the DSN memory comprises:
for each data segment of the plurality of data segments:
dispersed storage error encoding the data segment to produce a set of encoded data slices;
generating a set of slice names corresponding to the set of encoded data slices based on a start segment vault source name of the segment allocation table, wherein a common segment number of the set of slice names includes a sequentially increasing segment number; and
sending the set of encoded data slices and the set of slice names to the DSN memory.
14. The DS processing unit of claim 8 , wherein the fifth module further facilitates storage of the plurality of data segments in the DSN memory by:
for each data segment of the plurality of data segments:
dispersed storage error encoding the data segment to produce a set of encoded data slices;
generating a set of slice names corresponding to the set of encoded data slices based on a start segment vault source name of the segment allocation table, wherein an append marker field of the set of slice names includes a sequentially increasing append marker; and
sending the set of encoded data slices and the set of slice names to the DSN memory.