Optimizing governed data transfer in a multi-cloud environment using lineage data

1 . A computer-implemented method, comprising:

receiving, by a processor set of a mediator server, a data transfer request to transfer a dataset stored on a source cloud to a destination cloud, wherein the mediator server is a part of a cloud-based, infrastructure-level platform configured for orchestrating secure data governance across companies and other platforms;

determining, by the processor set, a target view of the data transfer request based on one or more policies;

determining, by the processor set, using lineage metadata created by the mediator server for one or more previous data transfer requests for the dataset stored on the source cloud, a first portion of the target view exists in one or more copies of a dataset stored on the destination cloud;

determining, using the lineage metadata, a second portion of the target view does not exist in the one or more copies of the dataset stored on the destination cloud;

extracting, by the processor set, data corresponding to the first portion from the one or more copies of the dataset stored on the destination cloud;

transferring data corresponding to the second portion from the dataset stored on the source cloud;

creating, by the processor set, the target view using the extracted data and the transferred data; and

serving, by the processor set, the data transfer request using the created target view.

2 . The computer-implemented method of claim 1 , wherein the one or more copies of the dataset stored on the destination cloud are a result of one or more previous data transfer requests for the dataset stored on the source cloud.

3 . The computer-implemented method of claim 1 , wherein the one or more copies of the dataset stored on the destination cloud are a different dataset than the dataset stored on the source cloud.

4 . The computer-implemented method of claim 3 , further comprising identifying the one or more copies of the dataset stored on the destination cloud using data lineage information of the source cloud.

5 . The computer-implemented method of claim 1 , wherein the extracting the data comprises constructing a column in the target view using a transformation relationship.

6 . The computer-implemented method of claim 1 , further comprising adding rows to or deleting rows from the created target view based on changes being made to the dataset stored on the source cloud.

7 . A computer program product comprising one or more computer readable storage media having program instructions collectively stored on the one or more computer readable storage media, the program instructions executable to:

receive, by a processor set of a mediator server, a data transfer request to transfer a dataset stored on a source cloud to a destination cloud;

determine a target view of the data transfer request based on one or more policies;

determine, by the processor set, using lineage metadata created by the mediator server, for one or more previous data transfer requests for the dataset stored on the source cloud, that at least a first portion of the target view is constructible from data residing on the destination cloud as a result of the one or more previous data transfer requests;

determine, using the lineage metadata, a second portion of the target view does not exist in one or more copies of a dataset stored on the destination cloud;

transfer data corresponding to the second portion from the dataset stored on the source cloud;

create the target view using the transferred data and the data residing on the destination cloud corresponding to the first portion, thereby reducing an amount of data that is transferred from the source cloud to fulfill the data transfer request; and

serve the data transfer request using the created target view.

8 . The computer program product of claim 7 , wherein the one or more copies of the dataset stored on the destination cloud are the result of one or more previous data transfer requests for the dataset stored on the source cloud.

9 . The computer program product of claim 7 , wherein the one or more copies of the dataset stored on the destination cloud are a different dataset than the dataset stored on the source cloud.

10 . The computer program product of claim 9 , wherein the program instructions are executable to identify the one or more copies of the dataset stored on the destination cloud using data lineage information of the source cloud, and wherein the data lineage information defines input datasets on the source cloud, output datasets on the source cloud, and transformations used to generate respective ones of the output datasets from respective ones of the input datasets.

11 . The computer program product of claim 7 , wherein the constructing the first portion of the target view comprises constructing a column in the target view using a transformation relationship.

12 . The computer program product of claim 7 , wherein the program instructions are executable to add rows to or delete rows from the created target view based on changes being made to the dataset stored on the source cloud.

13 . A system comprising:

a processor set, one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable to:

receive a data transfer request to transfer a dataset stored on a source cloud to a destination cloud, wherein the dataset stored on the source cloud comprises plural columns of data;

determine a target view of the data transfer request based on one or more policies, wherein the target view comprises one or more of the plural columns;

determine, using lineage metadata, a first portion of the target view exists in one or more copies of a dataset stored on the destination cloud;

determine, using the lineage metadata, a second portion of the target view does not exist in the one or more copies of a dataset stored on the destination cloud;

extract data corresponding to the first portion from the one or more copies of the dataset stored on the destination cloud;

transfer data corresponding to the second portion from the dataset stored on the source cloud;

create the target view using the extracted data and the transferred data; and

serve the data transfer request using the created target view.

14 . The system of claim 13 , wherein the one or more copies of the dataset stored on the destination cloud are a result of one or more previous data transfer requests for the dataset stored on the source cloud.

15 . The system of claim 13 , wherein the one or more copies of the dataset stored on the destination cloud are a different dataset than the dataset stored on the source cloud, and the program instructions are executable to identify the one or more copies of the dataset stored on the destination cloud using data lineage information of the source cloud.

16 . The system of claim 13 , wherein the first portion of the target view comprises a first subset of the one or more of the plural columns, and wherein the second portion of the target view comprises a second subset of the one or more of the plural columns.

17 . The system of claim 13 , wherein the extracting the data comprises constructing a column in the target view using a transformation relationship.

18 . The system of claim 13 , wherein the program instructions are executable to add rows to or delete rows from the created target view based on changes being made to the dataset stored on the source cloud.