Direct access of data lake files
In an example embodiment, a solution is provided that allows users to interact directly with underlying object data lake storages in hyperscalers when accessing user data. This feature is known as direct access. Use of this solution increases performance of the user systems and the cloud systems, without impacting the functionality nor increasing the complexity. This is true even for Spark users, where the driver abstracts the communication path between the Spark application and HDL files. This results in an opt-in solution that allows users to reduce their cost by increasing complexity.
1 . A system comprising:
at least one hardware processor;
a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
receiving, at a first virtual private cloud, from a user, a request to access a file stored in a data lake object storage of a hyperscaler;
generating a request, to authenticate the user, to a data lake file application on a second virtual private cloud, separate and distinct from the first virtual private cloud, the data lake object storage being separate and distinct from both the first virtual private cloud and the second virtual private cloud;
receiving a response from the data lake file application on the second virtual private cloud, the response indicating that access rights to the file have been granted to the user; and
in response to the receiving the response, directly accessing the data lake storage object from the first virtual private cloud to access the file.
2 . The system of claim 1 , wherein the directly accessing uses an Application Program Interface provided in a data lake files client library on the first virtual private cloud.
3 . The system of claim 2 , wherein the data lake file application determines that the user should be granted access rights to the file based on trusted authorities listed in a cluster file container on the second virtual private cloud.
4 . The system of claim 1 , wherein the request to access the file contains a specialized header where the user requests direct access.
5 . The system of claim 1 , wherein the response comprises hyperscaler-specific data.
6 . The system of claim 5 , wherein the hyperscaler-specific data comprises a pre-signed uniform resource locator (URL).
7 . The system of claim 5 , wherein the hyperscaler-specific data comprises temporary credentials.
8 . A method comprising:
receiving, at a first virtual private cloud, from a user, a request to access a file stored in a data lake object storage of a hyperscaler;
generating a request, to authenticate the user, to a data lake file application on a second virtual private cloud, separate and distinct from the first virtual private cloud, the data lake object storage being separate and distinct from both the first virtual private cloud and the second virtual private cloud;
receiving a response from the data lake file application on the second virtual private cloud, the response indicating that access rights to the file have been granted to the user; and
in response to the receiving the response, directly accessing the data lake storage object from the first virtual private cloud to access the file.
9 . The method of claim 8 , wherein the directly accessing the data lake storage object uses an Application Program Interface provided in a data lake files client library on the first virtual private cloud.
10 . The method of claim 9 , wherein the data lake file application determines that the user should be granted access rights to the file based on trusted authorities listed in a cluster file container on the second virtual private cloud.
11 . The method of claim 8 , wherein the request to access the file contains a specialized header where the user requests direct access.
12 . The method of claim 8 , wherein the response comprises hyperscaler-specific data.
13 . The method of claim 12 , wherein the hyperscaler-specific data comprises a pre-signed uniform resource locator (URL).
14 . The method of claim 12 , wherein the hyperscaler-specific data comprises temporary credentials.
15 . A non-transitory machine-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving, at a first virtual private cloud, from a user, a request to access a file stored in a data lake object storage of a hyperscaler;
generating a request, to authenticate the user, to a data lake file application on a second virtual private cloud, separate and distinct from the first virtual private cloud, the data lake object storage being separate and distinct from both the first virtual private cloud and the second virtual private cloud;
receiving a response from the data lake file application on the second virtual private cloud, the response indicating that access rights to the file have been granted to the user; and
in response to the receiving the response, directly accessing the data lake storage object from the first virtual private cloud to access the file.
16 . The non-transitory machine-readable medium of claim 15 , wherein the directly accessing the data lake storage object uses an Application Program Interface provided in a data lake files client library on the first virtual private cloud.
17 . The non-transitory machine-readable medium of claim 16 , wherein the data lake file application determines that the user should be granted access rights to the file based on trusted authorities listed in a cluster file container on the second virtual private cloud.
18 . The non-transitory machine-readable medium of claim 17 , wherein the request to access the file contains a specialized header where the user requests direct access.
19 . The non-transitory machine-readable medium of claim 15 , wherein the response comprises hyperscaler-specific data.
20 . The non-transitory machine-readable medium of claim 19 , wherein the hyperscaler-specific data comprises a pre-signed uniform resource locator (URL).