Methods and systems for improved deduplication performance using prefetched backup information
Methods, computer program products, computer systems, and the like are disclosed that provide for improved deduplication performance using prefetched backup information. For example, such methods, computer program products, and computer systems can include generating new feature information for a new backup image, (for each existing backup image in a plurality of existing backup images) comparing the new feature information with existing feature information for the each existing backup image, identifying one or more existing backup images of the plurality of existing backup images, and prefetching the one or more existing backup images. The feature information is generated as an output of a machine learning process. The machine learning process receives at least a portion of data of the new backup image as an input. The one or more existing backup images are identified based, at least in part, on a result of the comparing.
1. A method comprising:
generating new feature information for a portion of data, wherein
the portion of the data is a sample of data of a new backup image,
the new backup image is a full image,
the sample of the data of the new backup image is the data of the new backup image that is within a sample window of the data of the new backup image,
the new feature information comprises one or more generative models for a machine learning process,
the new feature information of the sample of the data is representative of one or more features of the data of the new backup image by virtue of the one or more generative models being representative of one or more features of the data of the new backup image,
the one or more features are representative of container usage of the new backup image,
the container usage of the new backup image comprises a logical size of each of one or more fingerprints stored in a data container of the new backup image and a data container identifier of the data container of the new backup image, and
the machine learning process receives the portion of the data of the new backup image as an input;
for each existing backup image in a plurality of existing backup images,
comparing the new feature information with existing feature information, wherein
the each existing backup image is another full image,
the each existing backup image comprises existing data,
the existing feature information is representative of one or more existing features of the data of the each existing backup image,
the one or more existing features are representative of container usage of the each existing backup image,
the container usage of the each existing backup image comprises another logical size of each of one or more fingerprints stored in a data container of the each existing backup image and a data container identifier of the data container of the each existing backup image,
the existing feature information comprises one or more other generative models for the machine learning process,
the comparing comprises
performing a distance determination operation,
the distance determination operation determines a distance between a feature of the new feature information and a feature of the existing feature information based on the logical size of the each of the one or more fingerprints stored in the data container of the new backup image, the data container identifier of the data container of the new backup image, the another logical size of the each of the one or more fingerprints stored in the data container of the each existing backup image, and the data container identifier of the data container of the each existing backup image, and
the comparing results in corresponding distance information for the each existing backup image, with respect to the new feature information;
in response to the comparing, selecting one or more of the plurality of existing backup images as selected existing backup images, based, at least in part, on the corresponding distance information for each of the plurality of existing backup images; and
prefetching the selected existing backup images.
2. The method of claim 1 , wherein the generating comprises:
producing pre-processed data by performing one or more data pre-processing operations on the portion of the data; and
producing the new feature information by performing a feature extraction operation on the pre-processed data, wherein
the feature extraction operation is comprised in the machine learning process.
3. The method of claim 2 , wherein
the one or more data pre-processing operations comprise at least one of
an outlier removal operation,
a data clustering operation, or
a kernel density estimation operation.
4. The method of claim 2 , wherein the generating further comprises:
retrieving the portion of the data, wherein
each of the one or more features of the data of the new backup image and each of the one or more existing features of the data of the each existing backup image comprises a lower bound of a data container identifier range, a mean of the data container identifier range, and an upper bound of the data container identifier range, and
the logical size is a size of data at the mean of the data container identifier range.
5. The method of claim 1 , wherein the comparing comprises:
comparing one or more features of the new feature information and one or more features of the existing feature information.
6. The method of claim 2 , wherein
the one or more data pre-processing operations comprise a data clustering operation,
the data clustering operation comprises a density-based clustering algorithm, and
the distance determination operation comprises
calculating a relative entropy between a first probability density function and a second probability density function.
7. The method of claim 6 , wherein
the density-based clustering algorithm is a density-based spatial clustering of applications with noise (DBSCAN) algorithm,
the new feature information is represented by the first probability density function calculated using the DBSCAN algorithm,
the existing feature information is represented by the second probability density function calculated using the DBSCAN algorithm, and
the distance is defined by a relative entropy between the first probability density function and the second probability density function.
8. The method of claim 1 , wherein the selecting comprises:
identifying the selected existing backup images, wherein
the identifying comprises
determining a number of the selected existing backup images, and
determining a backup identifier for each of the selected existing backup images.
9. The method of claim 1 , wherein
the one or more existing backup images are prefetched prior to a deduplication operation being performed on the new backup image.
10. The method of claim 1 , wherein
the new feature information comprises
a mean value for a first feature,
a covariance for the first feature, and
sampled data for the first feature, and
the existing feature information comprises
a mean value for a second feature,
a covariance for the second feature, and
sampled data for the second feature.
11. A non-transitory computer-readable storage medium, comprising program instructions, which, when executed by one or more processors of a computing system, perform a method comprising:
generating new feature information for a portion of data, wherein
the portion of the data is a sample of data of a new backup image,
the new backup image is a full image,
the sample of the data of the new backup image is the data of the new backup image that is within a sample window of the data of the new backup image,
the new feature information comprises one or more generative models for a machine learning process,
the new feature information of the sample of the data is representative of one or more features of the data of the new backup image by virtue of the one or more generative models being representative of one or more features of the data of the new backup image,
the one or more features are representative of container usage of the new backup image,
the container usage of the new backup image comprises a logical size of each of one or more fingerprints stored in a data container of the new backup image and a data container identifier of the data container of the new backup image, and
the machine learning process receives the portion of the data of the new backup image as an input;
for each existing backup image in a plurality of existing backup images,
comparing the new feature information with existing feature information, wherein
the each existing backup image is another full image,
the each existing backup image comprises existing data,
the existing feature information is representative of one or more existing features of the data of the each existing backup image,
the one or more existing features are representative of container usage of the each existing backup image,
the container usage of the each existing backup image comprises another logical size of each of one or more fingerprints stored in a data container of the each existing backup image and a data container identifier of the data container of the each existing backup image,
the existing feature information comprises one or more other generative models for the machine learning process,
the comparing comprises
performing a distance determination operation,
the distance determination operation determines a distance between a feature of the new feature information and a feature of the existing feature information based on the logical size of the each of the one or more fingerprints stored in the data container of the new backup image, the data container identifier of the data container of the new backup image, the another logical size of the each of the one or more fingerprints stored in the data container of the each existing backup image, and the data container identifier of the data container of the each existing backup image, and
the comparing results in corresponding distance information for the each existing backup image, with respect to the new feature information;
in response to the comparing, selecting one or more of the plurality of existing backup images as selected existing backup images, based, at least in part, on the corresponding distance information for each of the plurality of existing backup images; and
prefetching the selected existing backup images.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the generating comprises:
producing pre-processed data by performing one or more data pre-processing operations on the portion of the data; and
producing the new feature information by performing a feature extraction operation on the pre-processed data, wherein
the feature extraction operation is comprised in the machine learning process.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the generating further comprises:
retrieving the portion of the data, wherein
the one or more data pre-processing operations comprise at least one of
an outlier removal operation,
a data clustering operation, or
a kernel density estimation operation,
each of the one or more features of the data of the new backup image and each of the one or more existing features of the data of the each existing backup image comprises a lower bound of a data container identifier range, a mean of the data container identifier range, and an upper bound of the data container identifier range, and
the logical size is a size of data at the mean of the data container identifier range.
14. The non-transitory computer-readable storage medium of claim 12 , wherein
the one or more data pre-processing operations comprise a data clustering operation,
the data clustering operation comprises a density-based clustering algorithm, the distance determination operation comprises
determining a distance between each feature of the new feature information and a feature of the existing feature information by calculating a relative entropy between a first probability density function and a second probability density function,
the density-based clustering algorithm is a density-based spatial clustering of applications with noise (DBSCAN) algorithm,
the new feature information is represented by the first probability density function calculated using the DBSCAN algorithm,
the existing feature information is represented by the second probability density function calculated using the DBSCAN algorithm, and
the distance is defined by a relative entropy between the first probability density function and the second probability density function.
15. The non-transitory computer-readable storage medium of claim 11 , wherein the selecting comprises:
identifying the selected existing backup images, wherein
the identifying comprises
determining a number of the selected existing backup images, and
determining a backup identifier for each of the selected existing backup images.
16. A computing system comprising:
one or more processors; and
a non-transitory computer-readable storage medium coupled to the one or more processors, comprising program instructions, which, when executed by the one or more processors, perform a method comprising
generating new feature information for a portion of data, wherein
the portion of the data is a sample of data of a new backup image,
the new backup image is a full image,
the sample of the data of the new backup image is the data of the new backup image that is within a sample window of the data of the new backup image,
the new feature information comprises one or more generative models for a machine learning process,
the new feature information of the sample of the data is representative of one or more features of the data of the new backup image by virtue of the one or more generative models being representative of one or more features of the data of the new backup image,
the one or more features are representative of container usage of the new backup image,
the container usage of the new backup image comprises a logical size of each of one or more fingerprints stored in a data container of the new backup image and a data container identifier of the data container of the new backup image, and
the machine learning process receives the portion of the data of the new backup image as an input,
for each existing backup image in a plurality of existing backup images,
comparing the new feature information with existing feature information, wherein
the each existing backup image is another full image,
the each existing backup image comprises existing data,
the one or more existing features are representative of container usage of the each existing backup image,
the container usage of the each existing backup image comprises another logical size of each of one or more fingerprints stored in a data container of the each existing backup image and a data container identifier of the data container of the each existing backup image,
the existing feature information comprises one or more other generative models for the machine learning process,
the comparing comprises
performing a distance determination operation,
the distance determination operation determines a distance between a feature of the new feature information and a feature of the existing feature information based on the logical size of the each of the one or more fingerprints stored in the data container of the new backup image, the data container identifier of the data container of the new backup image, the another logical size of the each of the one or more fingerprints stored in the data container of the each existing backup image, and the data container identifier of the data container of the each existing backup image, and
the comparing results in corresponding distance information for the each existing backup image, with respect to the new feature information,
in response to the comparing, selecting one or more of the plurality of existing backup images as selected existing backup images, based, at least in part, on the corresponding distance information for each of the plurality of existing backup images, and
prefetching the selected existing backup images.
17. The computing system of claim 16 , wherein the generating comprises:
producing pre-processed data by performing one or more data pre-processing operations on the portion of the data; and
producing the new feature information by performing a feature extraction operation on the pre-processed data, wherein
the feature extraction operation is comprised in the machine learning process.
18. The computing system of claim 17 , wherein the generating further comprises:
retrieving the portion of the data,
the one or more data pre-processing operations comprise at least one of
an outlier removal operation,
a data clustering operation, or
a kernel density estimation operation,
each of the one or more features of the data of the new backup image and each of the one or more existing features of the data of the each existing backup image comprises a lower bound of a data container identifier range, a mean of the data container identifier range, and an upper bound of the data container identifier range, and
the logical size is a size of data at the mean of the data container identifier range.
19. The computing system of claim 17 , wherein
the one or more data pre-processing operations comprise a data clustering operation,
the data clustering operation comprises a density-based clustering algorithm, the distance determination operation comprises
determining a distance between each feature of the new feature information and a feature of the existing feature information by calculating a relative entropy between a first probability density function and a second probability density function,
the density-based clustering algorithm is a density-based spatial clustering of applications with noise (DBSCAN) algorithm,
the new feature information is represented by the first probability density function calculated using the DBSCAN algorithm,
the existing feature information is represented by the second probability density function calculated using the DBSCAN algorithm, and
the distance is defined by a relative entropy between the first probability density function and the second probability density function.
20. The computing system of claim 16 , wherein the selecting comprises:
identifying the selected existing backup images, wherein
the identifying comprises
determining a number of the selected existing backup images, and
determining a backup identifier for each of the selected existing backup images.