Methods for big data usage monitoring, entitlements and exception analysis
The invention relates to methods for implementing a unified framework for big data usage analytics and security profiling across heterogeneous platforms and multiple clusters. An embodiment of the present invention may be applied in various applications and use cases. This may include auditing for Big Data lakes; capacity assessment and charge back functionality for Big Data infrastructure and clusters and resource usage analytics.
1. A method that manages data from a plurality of disparate big data sources, the method comprising the steps of:
receiving, via a communication network, the data from a plurality of independent big data platforms, wherein the data is managed in a unified framework implemented across heterogeneous platforms and multiple clusters;
identifying, via an access limits file system, one or more entitlements associated with the data;
monitoring, via a computer processor, data usage comprising authentication failures, authorization failures, personally identifiable information (PII) queries and Data Manipulation Language (DML) queries wherein the data usage is parsed by a selected filter according to one of platform, cluster, application, service, environment, action type and action name; and
graphically displaying, via a user interactive interface, each of: authentication failures, authorization failures, PII queries and DML queries based on the selected filter.
2. The method of claim 1 , wherein the user interactive interface comprises an exception investigation status interface.
3. The method of claim 2 , wherein the exception investigation status interface comprises data access, login failure, privilege access and access failure.
4. The method of claim 1 , wherein the user interactive interface comprises a user entitlements and permissions interface.
5. The method of claim 4 , wherein the user entitlements and permissions interface comprises a node arrangement view.
6. The method of claim 1 , wherein the user interactive interface comprises an user search and audit tool interface.
7. The method of claim 6 , wherein the user search and audit tool interface enables a search function based on user identifier, date, platform, cluster, application, service, environment, database, table, action type and action name.
8. The method of claim 1 , wherein for each of: authentication failures, authorization failures, PII queries and DML queries, an option to view action by user is provided.
9. The method of claim 8 , wherein the view action by user comprises user identifier and record level detail.
10. The method of claim 1 , wherein for each of: authentication failures, authorization failures, PII queries and DML queries, an option to view details is provided, the view details comprising a heatmap by action type and platform, leaderboard by user name and weekly trend by application.
11. A system that manages data from a plurality of disparate big data sources, the system comprising:
an interface that communicates with a plurality of independent big data platforms;
an access limits file system that manages user and data entitlements;
an user interactive interface that manages and presents monitored data usage; and
a hardware computer processor, coupled to the interface and the access limits file system, the computer processor is further configured to:
receive, via a communication network, the data from the plurality of independent big data platforms, wherein the data is managed in a unified framework implemented across heterogeneous platforms and multiple clusters;
identify one or more entitlements associated with the data;
monitor the data usage comprising authentication failures, authorization failures, personally identifiable information (PII) queries and Data Manipulation Language (DML) queries wherein the data usage is parsed by a selected filter according to one of platform, cluster, application, service, environment, action type and action name; and
graphically display, via the user interactive interface, each of: authentication failures, authorization failures, PII queries and DML queries based on the selected filter.
12. The system of claim 11 , wherein the user interactive interface comprises an exception investigation status interface.
13. The system of claim 12 , wherein the exception investigation status interface comprises data access, login failure, privilege access and access failure.
14. The system of claim 11 , wherein the user interactive interface comprises a user entitlements and permissions interface.
15. The system of claim 14 , wherein the user entitlements and permissions interface comprises a node arrangement view.
16. The system of claim 11 , wherein the user interactive interface comprises an user search and audit tool interface.
17. The system of claim 16 , wherein the user search and audit tool interface enables a search function based on user identifier, date, platform, cluster, application, service, environment, database, table, action type and action name.
18. The system of claim 11 , wherein for each of: authentication failures, authorization failures, PII queries and DML queries, an option to view action by user is provided.
19. The system of claim 18 , wherein the view action by user comprises user identifier and record level detail.
20. The system of claim 11 , wherein for each of: authentication failures, authorization failures, PII queries and DML queries, an option to view details is provided, the view details comprising a heatmap by action type and platform, leaderboard by user name and weekly trend by application.