Methods and systems for affinity-aware prefetching
Disclosed are techniques that provide for deduplication in an efficient and effective manner. For example, such methods, computer program products, and computer systems can include retrieving container information for a first one or more containers of a plurality of containers of one or more backup images (where the one or more backup images were produced under an existing backup policy), generating pre-processed container information (where the generating the pre-processed container information comprises performing data pre-processing on the container information), determining a plurality of container ranges for the first one or more containers, generating container range affinity information for the one or more backup images (where the generating the container range affinity information comprises performing a container range operation using the plurality of container ranges, and storing the container range affinity information in a container range data structure.
1. A method comprising:
retrieving container information for a first one or more containers of a plurality of containers of one or more backup images, wherein the one or more backup images were produced under an existing backup policy;
generating pre-processed container information, wherein
the generating the pre-processed container information comprises
performing data pre-processing on the container information;
determining a plurality of container ranges for the first one or more containers,
wherein
the determining the plurality of container ranges comprises
generating feature information, wherein
the generating the feature information comprises
performing feature extraction on the pre-processed container information, and
generating the plurality of container ranges from the feature information, wherein
the generating the plurality of container ranges comprises performing a container range calculation on the feature information;
generating container range affinity information for the one or more backup images, wherein
the generating the container range affinity information comprises
performing a container range operation using the plurality of container ranges;
generating new container range affinity information for a second one or more containers of another backup image; and
determining a container range affinity between at least one of the first one or more containers and at least one of a second one or more containers of another backup image, using the container range affinity information and the new container range affinity information.
2. The method of claim 1 , wherein
the container range affinity information is stored in a container range data structure; and
the another backup image was produced using another backup policy.
3. The method of claim 2 , wherein
the container range data structure is a container range affinity list,
the another backup image is a result of a backup operation of data, and
the another backup image is an initial backup of the data using the another backup policy.
4. The method of claim 2 , further comprising:
identifying an identified container of the first one or more containers based, at least in part, on the container range affinity; and
prefetching the identified container.
5. The method of claim 4 , further comprising:
performing a deduplication operation on a unit of data of the another backup image using the identified container.
6. The method of claim 4 , further comprising:
performing the container range calculation using existing feature information for the one or more existing backup images, wherein
the existing feature information was generated as an output of a machine learning process that received the first one or more containers as another input.
7. The method of claim 6 , wherein
performing the container range calculation using new feature information for the another backup image, wherein
the new feature information was generated as an output of the machine learning process that received the second one or more containers as another input.
8. The method of claim 4 , further comprising:
producing a sample of data of the another backup image, wherein
the sample of the data of the another backup image is a portion of the data of the another backup image; and
generating new pre-processed container information for the sample of the data of the another backup image, wherein
the generating the new pre-processed container information comprises
performing the data pre-processing on new container information, and
the new container information is for the sample of the data of the another backup image.
9. The method of claim 8 , the data pre-processing comprises at least one of:
performing an outlier removal operation, or
performing a data clustering operation.
10. The method of claim 2 , wherein
the container range affinity information comprises a container range and an affinity score, and
the container range comprises a lower bound, a mean, and an upper bound.
11. A method comprising:
generating new feature information for a new backup image, wherein
the new backup image comprises data and metadata,
the new feature information comprises one or more patterns of container usage by virtue of the generating comprising performing an analysis of a sample of the metadata to produce container range information, and
the container range information is produced by a container range operation;
obtaining corresponding feature information for each of a plurality of existing backup images, wherein
the corresponding feature information comprises a corresponding one or more patterns of container usage for the each of the plurality of existing backup images, and
each of the corresponding one or more patterns of container usage was generated by performing the analysis on corresponding metadata of the each of the plurality of existing backup images;
determining a corresponding distance for the each of the plurality of existing backup images, of a plurality of corresponding distances between the new backup image and the each of the plurality of existing backup images,
wherein
the plurality of corresponding distances are determined by performing a machine learning process; and
prefetching a number of the plurality of existing backup images based on the corresponding distances of the plurality of corresponding distances.
12. The method of claim 11 , wherein
each corresponding distance of the plurality of corresponding distances is determined by calculating a relative entropy between a probability density function of the one or more patterns and the corresponding one or more patterns, and
the method further comprises
performing at least one of
an outlier removal operation, or
a grouping operation,
the analysis comprises performing at least one of
a statistical analysis,
a clustering operation, or
a kernel density estimation.
13. A non-transitory computer-readable storage medium, comprising program instructions, which, when executed by one or more processors of a computing system, perform a method comprising:
retrieving container information for a first one or more containers of a plurality of containers of one or more backup images, wherein
the one or more backup images were produced under an existing backup policy;
generating pre-processed container information, wherein
the generating the pre-processed container information comprises
performing data pre-processing on the container information;
determining a plurality of container ranges for the first one or more containers, wherein
the determining the plurality of container ranges comprises
generating feature information, wherein
the generating the feature information comprises performing feature extraction on the pre-processed container information, and
generating the plurality of container ranges from the feature information,
the generating the plurality of container ranges comprises
performing a container range calculation on the feature information;
generating container range affinity information for the one or more backup images, wherein
the generating the container range affinity information comprises
performing a container range operation using the plurality of container ranges;
generating new container range affinity information for a second one or more containers of another backup image; and
determining a container range affinity between at least one of the first one or more containers and at least one of a second one or more containers of another backup image, using the container range affinity information and the new container range affinity information.
14. The non-transitory computer-readable storage medium of claim 13 , wherein
the container range affinity information is stored in a container range data structure, and
the another backup image was produced using another backup policy.
15. The non-transitory computer-readable storage medium of claim 14 , wherein
the container range data structure is a container range affinity list,
the another backup image is a result of a backup operation of data, and
the another backup image is an initial backup of the data using the another backup policy.
16. The non-transitory computer-readable storage medium of claim 14 , the method further comprising:
identifying an identified container of the first one or more containers based, at least in part, on the container range affinity; and
prefetching the identified container.
17. The non-transitory computer-readable storage medium of claim 16 , the method further comprising:
performing a deduplication operation on a unit of data of the another backup image using the identified container.
18. The non-transitory computer-readable storage medium of claim 16 , the method further comprising:
performing the container range calculation using existing feature information for the one or more existing backup images, wherein
the existing feature information was generated as an output of a machine learning process that received the first one or more containers as another input.
19. The non-transitory computer-readable storage medium of claim 18 , wherein
performing the container range calculation using new feature information for the another backup image, wherein
the new feature information was generated as an output of the machine learning process that received the second one or more containers as another input.
20. The non-transitory computer-readable storage medium of claim 16 , the method further comprising:
producing a sample of data of the another backup image, wherein
the sample of the data of the another backup image is a portion of the data of the another backup image; and
generating new pre-processed container information for the sample of the data of the another backup image, wherein
the generating the new pre-processed container information comprises
performing the data pre-processing on new container information, and
the new container information is for the sample of the data of the another backup image.
21. The non-transitory computer-readable storage medium of claim 20 , the method further comprising:
performing an outlier removal operation, or
performing a data clustering operation.
22. The method of claim 11 , wherein
the new backup image comprises one or more containers,
each of the one or more containers comprise container data and container metadata,
the data is comprised in the container data of the one or more containers,
the metadata is comprised in the container metadata of the one or more containers,
the metadata comprises one or more container identifiers and size information for each container of the one or more containers, and
the corresponding metadata comprises one or more corresponding container identifiers and corresponding size information for each container of one or more containers of the each of the plurality of existing backup images.