CONVERGING OF DATA MANAGEMENT AND DATA ANALYSIS
Various embodiments of the present disclosure provide a solution for converging a data management system and a data analysis system at the level of storage. In some embodiments, the present disclosure provides a computer-implemented method. The method includes obtaining, a data management system, a first file in a first format. The method also includes, in response to determining that the first format is different from a predetermined second format supported by a data analysis system, converting the first file into a second file in the second format. The method further includes storing the first and second files to a data storage system. The data storage system is accessible to the data management system and the data analysis system.
1 . A computer-implemented method, comprising:
obtaining, by a data management system, a first file in a first format;
in response to determining that the first format is different from a predetermined second format supported by a data analysis system, converting thee first file into a second file in the second format; and
storing the first and second files to a data storage system accessible to the data management system and the data analysis system.
2 . The method according to claim 1 , wherein converting the first file into the second file in the second format comprises:
determining, based on a predefined security policy, whether data included in the first file is accessible to the data analysis system; and
in response to determining that the data is accessible to the data analysis system, converting the first file into the second file.
3 . The method according to claim 1 , further comprising:
generating metadata for the first and second files; and
storing to the data storage system the metadata in association with the first and second files.
4 . The method according to claim 1 , further comprising:
in response to determining that the first format is identical to the second format, storing the first file to the data storage system.
5 . The method according to claim 1 , further comprising:
in response to a request for deleting the first file stored in the data storage system, deleting the first file from the data storage system; and
in response to determining that the first file has been converted into the second file, deleting the second file from the data storage system.
6 . The method according to claim 5 , further comprising:
in response to an update of the first file stored in the data storage system, generating the request for deleting the stored first file.
7 . The method according to claim 6 , further comprising:
in response to determining that the first format is different from the predetermined second format, converting the updated first file into a third file in the second format; and
storing the updated first file and the third file to the data storage system.
8 . The method according to claim 3 , wherein the metadata includes a version identifier for data in the first file, the method further comprising:
in response to obtaining a fourth file in the first format as a different version of the first file, updating the version identifier.
9 . The method according to claim 1 . further comprising:
in response to obtaining a fourth file in the first format as a different version of the first file, converting the fourth file into a fifth file in the second format; and
storing the fourth and fifth files to the data storage system.
10 . The method according to any of claims 1 , further comprising:
merging at least one of the first and second files with at least one further file to obtain a merged file; and
storing the merged file into the data storage system.
11 . The method according to claim 10 , further comprising:
generating an associated index file for a respective file in the merged file, the index file including an identifier of the merged file, an identifier of the respective file, and an offset of the respective file in the merged file; and
storing the index file into the data storage system.
12 . A device, comprising:
at least one processing unit; and
at least one memory coupled to the at least one processing unit and storing instructions thereon, the instructions, when executed by the at least one processing, unit, cause the device to perform actions including:
obtaining a first file in a first format;
in response to determining that the first format is different from a predetermined second format supported by a data analysis system, converting the first file into a second file in the second format; and
storing the first and second files to a data storage system accessible to the device and the data analysis system.
13 . The device according to claim 12 , wherein converting the first file into the second file in the second format comprises:
determining, based on a predefined security policy, whether data included in the first file is accessible to the data analysis system; and
in response to determining that the data is accessible to the data analysis system, converting the first file into the second file.
14 . The device according to claim 12 , wherein the actions further include:
generating metadata for the first and second files; and
storing to the data storage system the metadata in association with the first and second files.
15 . The device according to claim 12 , wherein the actions further include:
in response to determining that the first format is identical to the second format, storing the first file to the data storage system.
16 . The device according to claim 12 , wherein the actions further include:
in response to a request for deleting the first file stored in the data storage system, deleting the first file from the data storage system; and
in response to determining that the first file has been converted into the second file, deleting the second file from the data storage system.
17 . The device according to claim 16 , wherein the actions further include:
in response to an update of the first file stored in the data storage system, generating the request for deleting the stored first file.
18 . The device according to claim 16 , wherein the actions further include:
in response to determining that the first format is different from the predetermined second format, converting the updated first file into a third file in the second format; and
storing the updated first file and the third file to the data storage system.
19 . The device according to claim 14 , wherein the metadata includes a version identifier for data in the first file, the actions further including:
in response to obtaining a fourth file in the first format as a different version of the first file, updating the version identifier.
20 . The device according to claim 12 , wherein the actions further include:
in response to obtaining a fourth file in the first format as a different version of the first file, converting the fourth file into a fifth file in the second format; and
storing the fourth and fifth files to the data storage system.
21 - 24 . (canceled)