Systems and methods for data retention while migrating objects and object metadata stored in object storage environments migrated across cloud ecosystems
Systems and methods for novel uses and/or improvements to data migration. In particular, systems and methods for data migration of metadata stored in object storage from one container to another, especially in instances when the metadata is destined for ingestion by an artificial intelligence application. The systems and methods ensure that all metadata (e.g., metadata stored in object storage) is preserved during data migration, including metadata such as content type, object lock mode, object lock retain until date, and/or custom metadata.
1. A system for retention of data while migrating metadata, stored in object storage of objects being migrated, across container ecosystems comprising computer storage to preserve the data for artificial intelligence applications, the system comprising:
one or more processors; and
one or more non-transitory, computer-readable mediums comprising instructions that when executed by the one or more processors causes operations comprising:
receiving, at a user interface, a first user input requesting that a first object, at a source location, be migrated to a target location;
determining first object-stored metadata for the first object;
generating an array of the first object-stored metadata;
generating a first hash value of the first object-stored metadata;
determining a first source identifier, wherein the first source identifier corresponds to the first object at the source location;
generating a first supplemental data structure comprising the array, the first hash value, and the first source identifier;
generating a third object-stored metadata for the first object, wherein the third object-stored metadata is not included in the first object-stored metadata;
including the third object-stored metadata in the first supplemental data structure;
storing the first supplemental data structure at a staging location;
executing a first function that causes the first object to be migrated from the source location to the target location and changes the first source identifier in the first supplemental data structure to a second source identifier, wherein the second source identifier corresponds to the first object at the target location;
in response to executing the first function, generating a second object at the target location, wherein the second object corresponds to the first object, and wherein the second object comprises the third object-stored metadata;
populating second object-stored metadata for the second object based on the first object-stored metadata;
validating the second object-stored metadata using the first hash value; and
generating for display, on the user interface, a data migration confirmation based on validating the second object-stored metadata using the first hash value, wherein the data migration confirmation indicates that the first object-stored metadata was successfully migrated to the target location.
2. A method for data retention while migrating metadata stored in object storage of objects being migrated across container ecosystems comprising computer storage, the method comprising:
receiving a first object, at a source location, for migration to a target location, wherein the source location comprises a first computing device, and wherein the target location comprises a second computing device;
determining first object-stored metadata for the first object;
generating an array of the first object-stored metadata;
determining a first source identifier, wherein the first source identifier corresponds to the first object at the source location;
generating a first supplemental data structure comprising the array and the first source identifier;
generating a third object-stored metadata for the first object, wherein the third object-stored metadata is not included in the first object-stored metadata;
including the third object-stored metadata in the first supplemental data structure;
storing the first supplemental data structure at a staging location;
executing a first function that causes the first object to be migrated from the source location to the target location and changes the first source identifier in the first supplemental data structure to a second source identifier, wherein the second source identifier corresponds to the first object at the target location; and
in response to executing the first function, generating a second object at the target location, wherein the second object corresponds to the first object, and wherein the second object comprises the third object-stored metadata.
3. The method of claim 2 , further comprising:
generating a first hash value of the first object-stored metadata;
including the first hash value in the first supplemental data structure; and
validating, using the first hash value, that the first object-stored metadata has been successfully migrated to the target location.
4. The method of claim 2 , further comprising:
receiving a data migration confirmation that the first object and the first object-stored metadata have been successfully migrated to the target location; and
deleting the first supplemental data structure from the staging location.
5. The method of claim 2 , further comprising:
in response to executing the first function, generating a second object at the target location, wherein the second object corresponds to the first object;
populating second object-stored metadata for the second object based on the first object-stored metadata;
validating the second object-stored metadata using a first hash value that generated based on the first object-stored metadata; and
generating for display, on a user interface, a data migration confirmation based on validating the second object-stored metadata using the first hash value, wherein the data migration confirmation indicates that the first object-stored metadata was successfully migrated to the target location.
6. The method of claim 2 , further comprising:
comparing fourth object-stored metadata to the third object-stored metadata, wherein the fourth object-stored metadata is generated at the target location in response to migrating the first object;
determining an inconsistency between the fourth object-stored metadata and the third object-stored metadata; and
modifying the fourth object-stored metadata based on the inconsistency.
7. The method of claim 2 , further comprising:
generating a first name of the first object-stored metadata;
including the first name in the first supplemental data structure;
in response to executing the first function, generating a second object at the target location, wherein the second object corresponds to the first object, and wherein the second object comprises a second name;
determining that the first supplemental data structure corresponds to the second object based on comparing the first name to the second name; and
in response to determining that the first supplemental data structure corresponds to the second object, populating second object-stored metadata for the second object based on the first supplemental data structure.
8. The method of claim 2 , further comprising:
generating a first content type of the first object-stored metadata;
including the first content type in the first supplemental data structure;
in response to executing the first function, generating a second object at the target location, wherein the second object corresponds to the first object, and wherein the second object comprises a second content type;
determining that the first supplemental data structure corresponds to the second object based on comparing the first content type to the second content type; and
in response to determining that the first supplemental data structure corresponds to the second object, populating second object-stored metadata for the second object based on the first supplemental data structure.
9. The method of claim 2 , wherein generating the first supplemental data structure further comprises:
determining a compatibility requirement of the target location; and
formatting the first supplemental data structure based on the compatibility requirement.
10. The method of claim 2 , wherein generating the array of the first object-stored metadata further comprises:
inputting the first object-stored metadata into an artificial intelligence model, wherein the artificial intelligence model is trained on object-stored metadata samples to generate arrays of data; and
receiving an output from the artificial intelligence model, wherein the array is based on the output.
11. The method of claim 2 , wherein the first object-stored metadata comprises a third object that is linked to the first object.
12. The method of claim 2 , wherein executing the first function further comprises:
determining a first application programming interface (“API”) for the target location;
generating a first API request, using the first API, wherein the first API request communicates the first object from the source location to the target location; and
generating a second API request, using the first API, wherein the second API request communicates the first object-stored metadata from the staging location to the target location.
13. The method of claim 2 , further comprising:
receiving a notification that the first function is executed; and
in response to receiving the notification, executing a second function that causes the first object-stored metadata to be migrated from the staging location to the target location based on the second source identifier.
14. The method of claim 2 , further comprising:
receiving, at a user interface, a first user input requesting that the first object be migrated;
receiving, at the user interface, a second user input identifying the source location; and
receiving, at the user interface, a third user input identifying the target location.
15. The method of claim 2 , further comprising:
generating a first entry in a retention log of the staging location, wherein the first entry corresponds to the first object; and
generating a second entry in the retention log of the staging location, wherein the second entry corresponds to the first supplemental data structure.
16. One or more non-transitory, computer-readable mediums, comprising instructions that, when executed by one or more processors, cause operations comprising:
receiving a first object, at a source location, for migration to a target location;
determining first object-stored metadata for the first object;
generating an array of the first object-stored metadata;
determining a first source identifier, wherein the first source identifier corresponds to the first object at the source location;
generating a first supplemental data structure comprising the array and the first source identifier;
generating a third object-stored metadata for the first object, wherein the third object-stored metadata is not included in the first object-stored metadata;
including the third object-stored metadata in the first supplemental data structure;
storing the first supplemental data structure at a staging location;
executing a first function that causes the first object to be migrated from the source location to the target location and changes the first source identifier in the first supplemental data structure to a second source identifier, wherein the second source identifier corresponds to the first object at the target location; and
in response to executing the first function, generating a second object at the target location, wherein the second object corresponds to the first object, and wherein the second object comprises the third object-stored metadata.
17. The one or more non-transitory, computer-readable mediums of claim 16 , wherein the operations further comprise:
generating a first hash value of the first object-stored metadata;
including the first hash value in the first supplemental data structure; and
validating, using the first hash value, that the first object-stored metadata has been successfully migrated to the target location.
18. The one or more non-transitory, computer-readable mediums of claim 16 , wherein the operations further comprise:
receiving a data migration confirmation that the first object and the first object-stored metadata have been successfully migrated to the target location; and
deleting the first supplemental data structure from the staging location.
19. The one or more non-transitory, computer-readable mediums of claim 16 , wherein the operations further comprise:
in response to executing the first function, generating a second object at the target location, wherein the second object corresponds to the first object;
populating second object-stored metadata for the second object based on the first object-stored metadata;
validating the second object-stored metadata using a first hash value that generated based on the first object-stored metadata; and
generating for display, on a user interface, a data migration confirmation based on validating the second object-stored metadata using the first hash value, wherein the data migration confirmation indicates that the first object-stored metadata was successfully migrated to the target location.