Representing and managing sampled data in storage systems

1. A method comprising:

generating, by a system comprising a hardware processor, a database for a set of entities associated with a plurality of sampled data units of a storage system, wherein the set of entities comprises a parent entity, and at least one of a snapshot of the parent entity, a clone of the parent entity, a snapshot of the clone of the parent entity, or a clone of the snapshot of the parent entity, and wherein the database comprises one or more data structures;

in response to an input/output (I/O) operation, accessing, by the system, a first data structure in the database that maps, in a bit vector of the first data structure, an entity in the set of entities with respective sampled signatures of a set of sampled signatures, wherein each respective entry of the bit vector indicates whether the entity refers to a respective sampled signature of the set of sampled signatures, and wherein corresponding sampled signatures of the set of sampled signatures are computed based on applying a function on corresponding sampled data units of the plurality of sampled data units;

comparing, by the system, the bit vector of the first data structure with a prior version of the bit vector;

controlling, by the system, a backup of the entity based on the comparing; and

receiving, by the system, a query to determine a rate of change for the entity, wherein the comparing of the bit vector to the prior version of the bit vector is in response to the query;

identifying, by the system, changes between the bit vector and the prior version of the bit vector; and

determining the rate of change for the entity based on the identified changes between the bit vector and the prior version of the bit vector.

2. The method of claim 1 , further comprising:

computing, by the system, a reference count for a first sampled signature of the set of sampled signatures, wherein the reference count is computed by performing a bit vector summation of the bit vector with at least a second bit vector of the first data structure, the second bit vector mapping a second entity to respective sampled signatures of the set of sampled signatures, and wherein the reference count specifies a quantity of entities referring to the first sampled signature.

3. The method of claim 1 , further comprising:

detecting, by the system, a generation of a new entity in the storage system; and

updating, by the system, the first data structure with a further bit vector mapping the new entity to respective sampled signatures of the set of sampled signatures.

4. The method of claim 1 , wherein the parent entity is one of a file, a filesystem, or a volume.

5. The method of claim 1 , further comprising:

employing, by the system, a second data structure in the database to track an intra-entity reference count of a first sampled signature of the set of sampled signatures, wherein the intra-entity reference count specifies a quantity of references to the first sampled signature by a first entity of the set of entities.

6. The method of claim 5 , wherein the tracking of the intra-entity reference count comprises:

detecting a new sampled signature for a sampled data unit associated with the first entity;

determining if the new sampled signature is present in the second data structure for the first entity; and

incrementing the intra-entity reference count in response to determining that the new sampled signature is present in the second data structure for the first entity.

7. The method of claim 1 , further comprising:

creating pointers for respective entities of the set of entities, wherein each pointer of the pointers is a link to access a respective bit vector stored in the database for a corresponding entity of the set of entities.

8. The method of claim 1 , further comprising:

receiving, by the system, a query to determine a unique space in the storage system occupied by a subset of entities of the set of entities;

performing a bit vector summation of bit vectors in the first data structure associated with the subset of entities to compute first reference counts for corresponding sampled signatures of the set of sampled signatures;

accessing, by the system, second reference counts from a second data structure, each second reference count of the second reference counts representing a quantity of entities referring to a respective sampled signature of the set of sampled signatures;

comparing, by the system, for each corresponding sampled signature of the set of sampled signatures, a corresponding first reference count of the first reference counts to a corresponding second reference count of the second reference counts;

identifying, by the system, a subset of sampled signatures of the set of sampled signatures for which corresponding first reference counts computed from the first data structure match corresponding second reference counts in the second data structure; and

determining, by the system, the unique space occupied by the subset of entities in the storage system based on the identified subset of sampled signatures.

9. The method of claim 8 , further comprising:

adjusting, by the system, a second reference count in the second data structure in response to I/O operations that add or delete data units with respect to the set of entities in the storage system.

10. The method of claim 1 , further comprising:

storing, by the system, bit vectors in the first data structure in compressed format.

11. The method of claim 1 , wherein the controlling of the backup of the entity is based on determined rate of change.

12. A system comprising:

a processor; and

a non-transitory machine-readable medium storing instructions that, when executed on the processor, cause the system to:

generate a database for a set of entities associated with a plurality of sampled data units of a storage system, wherein the set of entities comprises a parent entity, and at least one of a snapshot of the parent entity, a clone of the parent entity, a snapshot of the clone of the parent entity, or a clone of the snapshot of the parent entity, and wherein the database comprises one or more data structures;

in response to an input/output (I/O) operation, access a first data structure in the database that maps, in a bit vector of the first data structure, an entity in the set of entities with respective sampled signatures of a set of sampled signatures, wherein each respective entry of the bit vector indicates whether the entity refers to a respective sampled signature of the set of sampled signatures, and wherein corresponding sampled signatures of the set of sampled signatures are computed based on applying a hash function on corresponding sampled data units of the plurality of sampled data units;

compare the bit vector of the first data structure with a prior version of the bit vector;

control a backup of the entity based on the comparing; and

compute a reference count for a first sampled signature of the set of sampled signatures, wherein the reference count is computed by performing a bit vector summation of the bit vector with at least a second bit vector of the first data structure, the second bit vector mapping a second entity to respective sampled signatures of the set of sampled signatures, and wherein the reference count specifies a quantity of entities referring to the first sampled signature.

13. The system of claim 12 , wherein the instructions when executed on the processor cause the system to:

detect a generation of a new entity in the storage system; and

update the first data structure with a further bit vector mapping the new entity to respective sampled signatures of the set of sampled signatures.

14. The system of claim 12 , wherein the instructions, when executed on the processor cause the system to:

receive a request to determine a unique space in the storage system occupied by a subset of entities of the set of entities;

perform a bit vector summation of bit vectors in the first data structure associated with the subset of entities to compute first reference counts for corresponding sampled signatures of the set of sampled signatures;

access second reference counts from a second data structure, each second reference count of the second reference counts representing a quantity of entities referring to a respective sampled signature of the set of sampled signatures;

compare, for each corresponding sampled signature of the set of sampled signatures, a corresponding first reference count of the first reference counts to a corresponding second reference count of the second reference counts;

identify a subset of sampled signatures of the set of sampled signatures for which corresponding first reference counts computed from the first data structure match corresponding second reference counts in the second data structure; and

determine the unique space occupied by the subset of entities in the storage system based on the identified subset of sampled signatures.

15. The system of claim 12 , wherein the instructions, when executed on the processor cause the system to:

receive a request to determine a rate of change for the entity, wherein the comparing of the bit vector to the prior version of the bit vector is in response to the request;

identify changes between the bit vector and the prior version of the bit vector; and

determine the rate of change for the entity based on the identified changes between the bit vector and the prior version of the bit vector.

16. The system of claim 15 , wherein the controlling of the backup of the entity is based on determined rate of change.

17. A non-transitory machine-readable storage medium comprising instructions that upon execution cause a system to:

generate a database for a set of entities associated with a plurality of sampled data units of a storage system, wherein the set of entities comprises a parent entity, and at least one of a snapshot of the parent entity, a clone of the parent entity, a snapshot of the clone of the parent entity, or a clone of the snapshot of the parent entity, and wherein the database comprises one or more data structures;

in response to an input/output (I/O) operation, access a first data structure in the database that maps, in a bit vector of the first data structure, an entity in the set of entities with respective sampled signatures of a set of sampled signatures, wherein each respective entry of the bit vector indicates whether the entity refers to a respective sampled signature of the set of sampled signatures, and wherein corresponding sampled signatures of the set of sampled signatures are computed based on applying a hash function on corresponding sampled data units of the plurality of sampled data units;

compare the bit vector of the first data structure with a prior version of the bit vector; and

control a backup of the entity based on the comparing;

receive a request to determine a rate of change for the entity, wherein the comparing of the bit vector to the prior version of the bit vector is in response to the request;

identify changes between the bit vector and the prior version of the bit vector; and

determine the rate of change for the entity based on the identified changes between the bit vector and the prior version of the bit vector,

wherein the controlling of the backup of the entity is based on determined rate of change.