Securing data lakes via object store monitoring

1. A method comprising:

periodically executing database operations, each database operation to access data stored in an object store;

in response to each database operation, generating an access log set and storing the access log set in an access log storage;

periodically aggregating, by a processing device, access log data from a plurality of access log sets currently stored in the access log storage to generate aggregated log data; and

monitoring the aggregated log data over time to identify one or more security incidents of the object store, wherein

the access log data from each of the plurality of access log sets corresponds to one or more access events, and includes a set of characteristics of each of the one or more access events, the set of characteristics include:

a source from which the access event originated, a user, a role, or a group who initiated the access e vent, prefixes accessed by the access event, partitions accessed by the access event, sub-partitions accessed by the access event, a time of the access event, and a number of bytes transferred during the access event, and

the access log data from each of the plurality of access log sets is aggregated based on a set of characteristics of each of the one or more access events from each of the plurality of access log sets.

2. The method of claim 1 , wherein the aggregated log data represents a single event and indicates for the single event: a source from which the single event originated, a user, role, or group who initiated the single event, prefixes accessed by the single event, partitions accessed by the single event, sub-partitions accessed by the single event, a time of the single event, and a number of bytes transferred during the single event.

3. The method of claim 2 , wherein monitoring the aggregated log data over time comprises:

identifying security incidents based on one or more of a source from which the single event originated, a user, role, or group who initiated the single event, prefixes accessed by the single event, partitions accessed by the single event, sub-partitions accessed by the single event, a time of the single event, and a number of bytes transferred during the single event.

4. The method of claim 1 , wherein the access log data from a plurality of access log sets currently stored in the access log storage is aggregated at regular intervals.

5. The method of claim 1 , wherein the object store is a data lake comprising multiple data repositories.

6. A system comprising:

a memory; and

a processing device, operatively coupled to the memory, the processing device to:

periodically execute database operations, each database operation to access data stored in an object store;

in response to each database operation, generate an access log set and storing the access log set in an access log storage;

periodically aggregate access log data from a plurality of access log sets currently stored in the access log storage to generate aggregated log data; and

monitor the aggregated log data over time to identify one or more security incidents of the object store, wherein

the access log data from each of the plurality of access log sets corresponds to one or more access events, and includes a set of characteristics of each of the one or more access events, the set of characteristics include:

a source from which the access event originated, a user a role, or a group who initiated the access event, prefixes accessed by the access event, partitions accessed by the access event, sub-partitions accessed by the access event, a time of the access event, and a number of bytes transferred during the access event, and

the access log data from each of the plurality of access log sets is aggregated based on a set of characteristics of each of the one or more access events from each of the plurality of access log sets.

7. The system of claim 6 , wherein the aggregated log data represents a single event and indicates for the single event: a source from which the single event originated, a user, role, or group who initiated the single event, prefixes accessed by the single event, partitions accessed by the single event, sub-partitions accessed by the single event, a time of the single event, and a number of bytes transferred during the single event.

8. The system of claim 7 , wherein to monitor the aggregated log data over time, the processing device is to:

identify security incidents based on one or more of a source from which the single event originated, a user, role, or group who initiated the single event, prefixes accessed by the single event, partitions accessed by the single event, sub-partitions accessed by the single event, a time of the single event, and a number of bytes transferred during the single event.

9. The system of claim 6 , wherein the access log data from a plurality of access log sets currently stored in the access log storage is aggregated at regular intervals.

10. The system of claim 6 , wherein the object store is a data lake comprising multiple data repositories.

11. A non-transitory computer-readable medium having instructions stored thereon which, when executed by a processing device, cause the processing device to:

periodically execute database operations, each database operation to access data stored in an object store;

in response to each database operation, generate an access log set and storing the access log set in an access log storage;

periodically aggregate, by the processing device, access log data from a plurality of access log sets currently stored in the access log storage to generate aggregated log data; and

monitor the aggregated log data over time to identify one or more security incidents of the object store, wherein

the access log data from each of the plurality of access log sets corresponds to one or more access events, and includes a set of characteristics of each of the one or more access events, the set of characteristics include:

a source from which the access event originated, a user, a role, or a group who initiated the access event, prefixes accessed by the access event, partitions accessed by the access event, sub-partitions accessed by the access event, a time of the access event, and a number of bytes transferred during the access event, and

the access log data from each of the plurality of access log sets is aggregated based on a set of characteristics of each of the one or more access events from each of the plurality of access log sets.

12. The non-transitory computer-readable medium of claim 11 , wherein the aggregated log data represents a single event and indicates for the single event: a source from which the single event originated, a user, role, or group who initiated the single event, prefixes accessed by the single event, partitions accessed by the single event, sub-partitions accessed by the single event, a time of the single event, and a number of bytes transferred during the single event.

13. The non-transitory computer-readable medium of claim 12 , wherein to monitor the aggregated log data over time, the processing device is to:

identify security incidents based on one or more of a source from which the single event originated, a user, role, or group who initiated the single event, prefixes accessed by the single event, partitions accessed by the single event, sub-partitions accessed by the single event, a time of the single event, and a number of bytes transferred during the single event.

14. The non-transitory computer-readable medium of claim 11 , wherein the access log data from a plurality of access log sets currently stored in the access log storage is aggregated at regular intervals.