Efficient method to find changed data between indexed data and new backup
An apparatus, method, and system for updating a file index in a search engine in a data backup system to reflect file changes introduced in a new backup is disclosed. The operations comprise: generating a first external file, the first external file comprising file hashes for files already indexed in a file index in a search engine of a data backup storage system that are not associated with a deleted status; generating a second external file, the second external file comprising file hashes for files in a new backup; determining one or more file changes introduced in the new backup based on a comparison between the first external file and the second external file; and updating the file index in the search engine to reflect the one or more file changes introduced in the new backup.
1. A computer-implemented method, comprising:
generating a first external file, the first external file comprising file hashes for files already indexed in a file index in a search engine of a data backup storage system that are not associated with a deleted status;
generating a second external file, the second external file comprising file hashes for files in a new backup;
determining one or more file changes introduced in the new backup based on a comparison between the file hashes in the first external file and the file hashes in the second external file; and
updating the file index in the search engine to reflect the one or more file changes introduced in the new backup.
2. The method of claim 1 , wherein the new backup is associated with a backup target indicative of a source of the new backup, and only file hashes for already-indexed files that are associated with the same backup target are included in the first external file.
3. The method of claim 2 , wherein the backup target comprises a directory name or a virtual machine (VM) name.
4. The method of claim 1 , wherein a file hash for each file is calculated based on a combination of a backup server identifier, a backup identifier, a file full path, and a time of last modification associated with the file.
5. The method of claim 1 , wherein the one or more file changes introduced in the new backup comprise one or more files newly added in the new backup, one or more files that have been deleted in the new backup, or a combination thereof.
6. The method of claim 5 , wherein updating the file index in the search engine to reflect the one or more file changes comprises performing at least one of adding one entry to the file index for each of the one or more files newly added in the new backup or associating a respective entry for each of the one or more files that have been deleted in the new backup with a deleted status.
7. The method of claim 1 , wherein the file hashes in the first and second external files are sorted based on their values before determining the one or more file changes.
8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform data indexing operations, the operations comprising:
generating a first external file, the first external file comprising file hashes for files already indexed in a file index in a search engine of a data backup storage system that are not associated with a deleted status;
generating a second external file, the second external file comprising file hashes for files in a new backup;
determining one or more file changes introduced in the new backup based on a comparison between the file hashes in the first external file and the file hashes in the second external file; and
updating the file index in the search engine to reflect the one or more file changes introduced in the new backup.
9. The non-transitory machine-readable medium of claim 8 , wherein the new backup is associated with a backup target indicative of a source of the new backup, and only file hashes for already-indexed files that are associated with the same backup target are included in the first external file.
10. The non-transitory machine-readable medium of claim 9 , wherein the backup target comprises a directory name or a virtual machine (VM) name.
11. The non-transitory machine-readable medium of claim 8 , wherein a file hash for each file is calculated based on a combination of a backup server identifier, a backup identifier, a file full path, and a time of last modification associated with the file.
12. The non-transitory machine-readable medium of claim 8 , wherein the one or more file changes introduced in the new backup comprise one or more files newly added in the new backup, one or more files that have been deleted in the new backup, or a combination thereof.
13. The non-transitory machine-readable medium of claim 12 , wherein updating the file index in the search engine to reflect the one or more file changes comprises performing at least one of adding one entry to the file index for each of the one or more files newly added in the new backup or associating a respective entry for each of the one or more files that have been deleted in the new backup with a deleted status.
14. The non-transitory machine-readable medium of claim 8 , wherein the file hashes in the first and second external files are sorted based on their values before determining the one or more file changes.
15. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform data storage operations, the operations including:
generating a first external file, the first external file comprising file hashes for files already indexed in a file index in a search engine of a data backup storage system that are not associated with a deleted status;
generating a second external file, the second external file comprising file hashes for files in a new backup;
determining one or more file changes introduced in the new backup based on a comparison between the file hashes in the first external file and the file hashes in the second external file; and
updating the file index in the search engine to reflect the one or more file changes introduced in the new backup.
16. The data processing system of claim 15 , wherein the new backup is associated with a backup target indicative of a source of the new backup, and only file hashes for already-indexed files that are associated with the same backup target are included in the first external file.
17. The data processing system of claim 16 , wherein the backup target comprises a directory name or a virtual machine (VM) name.
18. The data processing system of claim 15 , wherein a file hash for each file is calculated based on a combination of a backup server identifier, a backup identifier, a file full path, and a time of last modification associated with the file.
19. The data processing system of claim 15 , wherein the one or more file changes introduced in the new backup comprise one or more files newly added in the new backup, one or more files that have been deleted in the new backup, or a combination thereof.
20. The data processing system of claim 19 , wherein updating the file index in the search engine to reflect the one or more file changes comprises performing at least one of adding one entry to the file index for each of the one or more files newly added in the new backup or associating a respective entry for each of the one or more files that have been deleted in the new backup with a deleted status.
21. The data processing system of claim 15 , wherein the file hashes in the first and second external files are sorted based on their values before determining the one or more file changes.