Policy enforcement for dataset access in distributed computing environment

1 . A method for enforcing data access in a distributed execution framework, the method comprising:

receiving a data access request from a data processing application executing in the distributed execution framework, the data access request associated with a particular user;

directing the data access request to a security data application executing in the distributed execution framework;

modifying, by the security data application, the data access request;

executing, by the security data application, the modified data access request to obtain data that is responsive to the modified data access request;

generating, by the security data application, executor processes to store the obtained data;

storing the obtained data in the generated executor processes; and

providing the obtained data and a location of the generated executor processes storing the obtained data to the data processing application.

2 . The method of claim 1 , wherein modifying the data access request comprises:

retrieving, by the security data application, at least one data policy or rule for the particular user that specifies the user’s access to the requested data; and

modifying, by the security data application, the data access request based on the at least one data access policy or rule for the particular user.

3 . The method of claim 2 , wherein retrieving the at least one data policy or rule for the particular user comprises requesting data provisioning rules and policies for the particular user from a policy engine.

4 . The method of claim 3 , wherein the policy engine is embedded in the distributed execution framework.

5 . The method of claim 3 , wherein the policy engine is external to the distributed execution framework.

6 . The method of claim 1 , further comprising:

blocking direct data access requests associated with data processing application users.

7 . The method of claim 1 , wherein a data access method of the data processing application calls a data access method of the security data application to make the data access request.

8 . The method of claim 7 , wherein the data access method of the data processing application calls the data access method of the security data application using a direct application programming interface method.

9 . The method of claim 7 , wherein the data access method of the security data application overrides the data access method of the data processing application so that the data access method of the data processing application implicitly calls the data access method of the security data application.

10 . The method of claim 1 , wherein executing, by the security data application, the modified data access request to obtain data that is responsive to the modified data access request comprises:

sending the modified request to a distributed database or file system; and

obtaining data from the distributed database or file system that match criteria of the modified data access request.

11 . The method of claim 1 , further comprising:

in response to a request from the data processing application, transferring obtained data from the generated executor processes to executor processes of the data processing application.

12 . The method of claim 1 , further comprising:

receiving a second data access request from a second data processing application executing in the distributed execution framework, the second data access request associated with a second particular user;

directing the second data access request to the security data application executing in the distributed execution framework to obtain data access.

13 . The one or more non-transitory computer-readable storage mediums of claim 1 , wherein modifying the data access request comprises:

retrieving, by the security data application, at least one data policy or rule for the particular user that specifies the user’s access to the requested data; and

modifying, by the security data application, the data access request based on the at least one data access policy or rule for the particular user.

14 . A system comprising:

one or more computers; and

one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving a data access request from a data processing application executing in a distributed execution framework, the data access request associated with a particular user;

directing the data access request to a security data application executing in the distributed execution framework;

modifying, by the security data application, the data access request;

executing, by the security data application, the modified data access request to obtain data that is responsive to the modified data access request;

generating, by the security data application, security executor processes to store the obtained data;

storing the obtained data in the generated security executor processes; and

providing the obtained data and a location of the generated security executor processes storing the obtained data to the data processing application.

15 . The system of claim 14 wherein modifying the data access request comprises:

retrieving, by the security data application, at least one data policy or rule for the particular user that specifies the user’s access to the requested data; and

modifying, by the security data application, the data access request based on the at least one data access policy or rule for the particular user.

16 . The system of claim 15 , wherein retrieving the at least one data policy or rule for the particular user comprises requesting data provisioning rules and policies for the particular user from a policy engine.

17 . The system of claim 14 , wherein executing, by the security data application, the modified data access request to obtain data that is responsive to the modified data access request comprises:

sending the modified request to a distributed database or file system; and

obtaining data from the distributed database or file system that match criteria of the modified data access request.

18 . The system of claim 14 , wherein the operations further comprise:

in response to a request from the data processing application, transferring obtained data from the generated security executor processes to executor processes of the data processing application.

19 . One or more non-transitory computer-readable storage mediums comprising instructions stored thereon that are executable by a processing device and upon such execution cause the processing device to perform operations comprising:

receiving a data access request from a data processing application executing in a distributed execution framework, the data access request associated with a particular user;

directing the data access request to a security data application executing in the distributed execution framework;

modifying, by the security data application, the data access request;

executing, by the security data application, the modified data access request to obtain data that is responsive to the modified data access request;

storing the obtained data in a security executor; and

providing the obtained data and a location of the security executor storing the obtained data to the data processing application.

20 . The one or more non-transitory computer-readable storage mediums of claim 19 , wherein the operations further comprise:

in response to a request from the data processing application, transferring obtained data from the security executor to executor processes of the data processing application.