Large object parallel writing
A method includes partitioning data into first and second partitions and dispersed storage (DS) error encoding the first and second partition into first and second plurality of sets of encoded data slices (EDSs). The method further includes generating first SAT regarding storage of the first plurality of sets of EDSs and second SAT regarding storage of the second plurality of sets of EDSs. The method further includes DS error encoding the first and the second SAT to produce first and second sets of SAT slices, sending the first plurality of sets of EDSs and the first set of SAT slices to the first set of storage units, and sending the second plurality of sets of EDSs and the second set of SAT slices to the second set of storage units. The method further includes generating a third SAT regarding storage of the first and second sets of SAT slices.
1. A method comprises:
partitioning, by a computing device of a dispersed storage network (DSN), a data object into a first partition and a second partition;
dispersed storage error encoding, by the computing device, the first partition into a first plurality of sets of encoded data slices and the second partition into a second plurality of sets of encoded data slices;
generating, by the computing device, a first segment allocation table (SAT) regarding storage of the first plurality of sets of encoded data slices in a first set of storage units of the DSN and a second SAT regarding storage of the second plurality of sets of encoded data slices in a second set of storage units of the DSN;
dispersed storage error encoding, by the computing device, the first SAT to produce a first set of SAT slices and the second SAT to produce a second set of SAT slices;
sending, by the computing device, the first plurality of sets of encoded data slices and the first set of SAT slices to the first set of storage units;
sending, by the computing device, the second plurality of sets of encoded data slices and the second set of SAT slices to the second set of storage units; and
generating, by the computing device, a third SAT regarding storage of the first and second sets of SAT slices in the first and second set of storage units.
2. The method of claim 1 further comprises:
determining, by the computing device, to partition the data object based on one of:
a data object attribute;
a partitioning scheme lookup; and
receiving the partitioning scheme.
3. The method of claim 1 , wherein the first SAT comprises:
a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the first plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the first partition is dispersed error encoded into the first set of encoded data slices;
a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the first partition; and
a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the first partition.
4. The method of claim 1 , wherein the second SAT comprises:
a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the second plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the second partition is dispersed error encoded into the first set of encoded data slices;
a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the second partition; and
a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the second partition.
5. The method of claim 1 , wherein the third SAT comprises:
a first data entry region including:
a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the second plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the second partition is dispersed error encoded into the first set of encoded data slices;
a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the second partition; and
a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the second partition; and
a second data entry region including:
a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the second plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the second partition is dispersed error encoded into the first set of encoded data slices;
a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the second partition; and
a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the second partition.
6. The method of claim 1 further comprises:
dispersed storage error encoding, by the computing device, the third SAT to produce a third set of SAT slices; and
sending, by the computing device, the third set of SAT slices to the first set of storage units or the second set of storage units.
7. The method of claim 1 further comprises:
updating, by the computing device, a directory with a first SAT vault source name, a second SAT vault source name, and a third SAT vault source name.
8. The method of claim 1 further comprises:
further partitioning, by the computing device, the data object into a third partition;
dispersed storage error encoding, by the computing device, the third partition into a third plurality of sets of encoded data slices;
generating, by the computing device, a fourth SAT regarding storage of the third plurality of sets of encoded data slices in a third set of storage units of the DSN;
dispersed storage error encoding, by the computing device, the fourth SAT to produce a fourth set of SAT slices;
sending, by the computing device, the third plurality of sets of encoded data slices and the fourth set of SAT slices to the third set of storage units; and
generating, by the computing device, a fifth SAT regarding storage of the first, second, and third sets of SAT slices in the first, second, and third set of storage units.
9. The method of claim 1 further comprises:
dispersed storage error encoding the first partition into the first plurality of sets of encoded data slices and the dispersed storage error encoding of the second partition into the second plurality of sets of encoded data slices substantially concurrently.
10. A computing device of a dispersed storage network (DSN), the computing device comprises:
an interface;
memory; and
a processing module operably coupled to the memory and the interface, wherein the processing module is operable to:
partition a data object into a first partition and a second partition;
dispersed storage error encode the first partition into a first plurality of sets of encoded data slices and the second partition into a second plurality of sets of encoded data slices;
generate a first segment allocation table (SAT) regarding storage of the first plurality of sets of encoded data slices in a first set of storage units of the DSN and a second SAT regarding storage of the second plurality of sets of encoded data slices in a second set of storage units of the DSN;
dispersed storage error encode the first SAT to produce a first set of SAT slices and the second SAT to produce a second set of SAT slices;
send the first plurality of sets of encoded data slices and the first set of SAT slices to the first set of storage units;
send the second plurality of sets of encoded data slices and the second set of SAT slices to the second set of storage units; and
generate a third SAT regarding storage of the first and second sets of SAT slices in the first and second set of storage units.
11. The computing device of claim 10 , wherein the processing module is operable to:
determine to partition the data object based on one of:
a data object attribute;
a partitioning scheme lookup; and
receiving the partitioning scheme.
12. The computing device of claim 10 , wherein the first SAT comprises:
a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the first plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the first partition is dispersed error encoded into the first set of encoded data slices;
a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the first partition; and
a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the first partition.
13. The computing device of claim 10 , wherein the second SAT comprises:
a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the second plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the second partition is dispersed error encoded into the first set of encoded data slices;
a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the second partition; and
a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the second partition.
14. The computing device of claim 10 , wherein the third SAT comprises:
a first data entry region including:
a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the second plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the second partition is dispersed error encoded into the first set of encoded data slices;
a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the second partition; and
a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the second partition; and
a second data entry region including:
a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the second plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the second partition is dispersed error encoded into the first set of encoded data slices;
a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the second partition; and
a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the second partition.
15. The computing device of claim 10 , wherein the processing module is operable to:
dispersed storage error encode the third SAT to produce a third set of SAT slices; and
send the third set of SAT slices to the first set of storage units or the second set of storage units.
16. The computing device of claim 10 , wherein the processing module is operable to:
update a directory with a first SAT vault source name, a second SAT vault source name, and a third SAT vault source name.
17. The computing device of claim 10 , wherein the processing module is operable to:
further partition the data object into a third partition;
dispersed storage error encode the third partition into a third plurality of sets of encoded data slices; generate a fourth SAT regarding storage of the third plurality of sets of encoded data slices in a third set of storage units of the DSN;
dispersed storage error encode the fourth SAT to produce a fourth set of SAT slices;
send the third plurality of sets of encoded data slices and the fourth set of SAT slices to the third set of storage units; and
generate a fifth SAT regarding storage of the first, second, and third sets of SAT slices in the first, second, and third set of storage units.
18. The computing device of claim 10 , wherein the processing module is operable to:
dispersed storage error encode the first partition into the first plurality of sets of encoded data slices and the second partition into the second plurality of sets of encoded data slices substantially concurrently.