Data index for deduplication storage system

1. A storage system comprising:

a processor;

a memory; and

a machine-readable storage storing instructions, the instructions executable by the processor to:

receive a data stream to be stored in persistent storage of a deduplication storage system;

store data units of the data stream in a container entity group object according to arrival time, wherein the container entity group object is one of a plurality of container entity group objects;

record, in a plurality of container indexes, metadata for the data units stored in the container entity group object;

record, in a data index associated with the container entity group object, a list of the plurality of container indexes that record the metadata for the data units stored in the container entity group object, wherein the data index is one of a plurality of data indexes, wherein each data index of the plurality of data indexes is associated with a different container entity group object of the plurality of container entity group objects, and wherein each data index of the plurality of data indexes is to list only container indexes that reference data units stored in the container entity group object associated with that data index; and

in response to a determination that the total size of the container entity group object exceeds a first threshold size, transfer the container entity group object from memory to the persistent storage.

2. The storage system of claim 1 , including instructions executable by the processor to:

in response to the determination that the total size of the container entity group object exceeds the first threshold size:

initialize a new container entity group object in the memory; and

initialize a new data index in the memory.

3. The storage system of claim 2 , including instructions executable by the processor to:

include the data index and the new data index in a data index group; and

in response to a determination that the data index group exceeds a second threshold size:

write the data index group from memory to the persistent storage; and

initialize a new data index group in the memory.

4. The storage system of claim 3 , wherein the first threshold size and the second threshold size are configuration settings of the storage system.

5. The storage system of claim 1 , including instructions executable by the processor to:

access a plurality of reference counts included in a first container index, wherein each of the plurality of reference counts corresponds to a different data unit stored in the container entity group object;

determine whether each of the plurality of reference counts is equal to zero; and

in response to a determination that each of the plurality of reference counts is equal to zero, delete an identifier of the first container index from the data index.

6. The storage system of claim 1 , including instructions executable by the processor to:

determine whether the data index is empty; and

in response to a determination that the data index is empty, delete the data index and the container entity group object.

7. The storage system of claim 1 , wherein the persistent storage is a network-based storage service, and wherein the storage system is coupled to the network-based storage service via a network connection.

8. A method comprising:

receiving, by a storage controller of a deduplication storage system, a data stream to be stored in persistent storage of a deduplication storage system;

storing, by the storage controller, data units of the data stream in a container entity group object according to arrival time, wherein the container entity group object is one of a plurality of container entity group objects;

recording, by the storage controller, in a plurality of container indexes, metadata for the data units stored in the container entity group object;

recording, by the storage controller, in a data index associated with the container entity group object, a list of the plurality of container indexes that record the metadata for the data units stored in the container entity group object, wherein the data index is one of a plurality of data indexes, wherein each data index of the plurality of data indexes is associated with a different container entity group object of the plurality of container entity group objects, and wherein each data index of the plurality of data indexes is to list only container indexes that reference data units stored in the container entity group object associated with that data index;

determining, by the storage controller, whether a total size of the container entity group object exceeds a threshold size; and

in response to a determination that the total size of the container entity group object exceeds the threshold size, writing, by the storage controller, the container entity group object from memory to the persistent storage.

9. The method of claim 8 , further comprising:

in response to the determination that the total size of the container entity group object exceeds the first threshold size:

initializing a new container entity group object in the memory; and

initializing a new data index in the memory.

10. The method of claim 9 , further comprising:

including the data index and the new data index in a data index group;

determining whether the data index group exceeds a second threshold size; and

in response to a determination that the data index group exceeds the second threshold size:

writing the data index group from memory to the persistent storage; and

initializing a new data index group in the memory.

11. The method of claim 10 , wherein the first threshold size and the second threshold size are configuration settings of the storage system.

12. The method of claim 8 , further comprising:

accessing a plurality of reference counts included in a first container index, wherein each of the plurality of reference counts corresponds to a different data unit stored in the container entity group object;

determining whether each of the plurality of reference counts is equal to zero; and

in response to a determination that each of the plurality of reference counts is equal to zero, deleting an identifier of the first container index from the data index.

13. The method of claim 8 , further comprising:

determining whether the data index is empty; and

in response to a determination that the data index is empty, deleting the data index and the container entity group object.

14. The method of claim 8 , wherein the persistent storage is a network-based storage service, and wherein the storage system is coupled to the network-based storage service via a network connection.

15. A non-transitory machine-readable medium storing instructions that upon execution cause a processor to:

receive a data stream to be stored in persistent storage of a deduplication storage system;

store data units of the data stream in a container entity group object according to arrival time, wherein the container entity group object is one of a plurality of container entity group objects;

record, in a plurality of container indexes, metadata for the data units stored in the container entity group object;

record, in a data index associated with the container entity group object, a list of the plurality of container indexes that record the metadata for the data units stored in the container entity group object, wherein the data index is one of a plurality of data indexes, wherein each data index of the plurality of data indexes is associated with a different container entity group object of the plurality of container entity group objects, and wherein each data index of the plurality of data indexes is to list only container indexes that reference data units stored in the container entity group object associated with that data index; and

in response to a determination that the total size of the container entity group object exceeds the threshold size, transfer the container entity group object from memory to the persistent storage.

16. The non-transitory machine-readable medium of claim 15 , including instructions that upon execution cause the processor to:

in response to the determination that the total size of the container entity group object exceeds the first threshold size:

initialize a new container entity group object in the memory; and

initialize a new data index in the memory.

17. The non-transitory machine-readable medium of claim 16 , including instructions that upon execution cause the processor to:

include the data index and the new data index in a data index group; and

in response to a determination that the data index group exceeds a second threshold size:

write the data index group from memory to the persistent storage; and

initialize a new data index group in the memory.

18. The non-transitory machine-readable medium of claim 17 , wherein the first threshold size and the second threshold size are configuration settings of the storage system.

19. The non-transitory machine-readable medium of claim 15 ,

including instructions that upon execution cause the processor to:

access a plurality of reference counts included in a first container index, wherein each of the plurality of reference counts corresponds to a different data unit stored in the container entity group object;

determine whether each of the plurality of reference counts is equal to zero; and

in response to a determination that each of the plurality of reference counts is equal to zero, delete an identifier of the first container index from the data index.

20. The non-transitory machine-readable medium of claim 15 , including instructions that upon execution cause the processor to:

determine whether the data index is empty; and

in response to a determination that the data index is empty, delete the data index and the container entity group object.