Cloud access security broker systems and methods via a distributed worker pool

1. A Cloud Access Security Broker (CASB) system comprising:

a controller;

a message broker connected to the controller;

an authentication provider configured to perform authentication of one or more tenants;

a plurality of workers connected to the message broker and connected to one or more cloud providers having a plurality of files contained therein for the one or more tenants, and

a plurality of queues configured to store files from the plurality of files for analysis by workers of the plurality of workers based on assignments from the message broker,

wherein the controller is configured to crawl through the plurality of files for the one or more tenants, based on policy and configuration for the one or more tenants and push a file from the plurality of files into one of the plurality of queues responsive to the file being one of a unique file entry or a new event, and wherein a first crawl includes,

fetching Information Technology (IT) admin change logs for file-related activities for a tenant in batches, processing the batch for unique file entries in the batch, pushing the file info into a queue, repeating until the entire IT log is crawled, and storing the log's stream-position for a next crawl, wherein IT admin change logs include statistics and transactions, and

responsive to an IT admin change log not being accessible, crawling through the entire list of entities for a tenant, storing the list of entities for each tenant, storing a last change for each entity, updating tenant info, and repeating for all tenants,

wherein the first crawl is synchronized with Data Leakage Prevention (DLP) scan frequency and subsequent crawls performed periodically crawl incrementally.

2. The CASB system of claim 1 , wherein one of the plurality of workers are further configured to cause an action of one of the files in the plurality of queues responsive to the provided assignments.

3. The CASB system of claim 2 , wherein the action includes any of allowing a file, deleting a file, quarantining a file, and providing a notification.

4. The CASB system of claim 1 , further comprising

a Data Leakage Prevention (DLP) engine configured to scan the plurality of files based on the policy and the configuration, and to provide an action based on the scan.

5. The CASB system of claim 1 , further comprising

a sandbox configured to execute a file of the plurality of files, and provide an action based on the execution and based on the policy and the configuration.

6. The CASB system of claim 1 , wherein the plurality of workers include a plurality of types of workers, each being configured to perform a specific task in the CASB system.

7. The CASB system of claim 1 , further comprising

a connection between the controller and a cloud-based security system, wherein the cloud-based security system is configured to analyze files of the plurality of files and provide an action.

8. The CASB system of claim 1 , wherein the controller includes a regulator that monitors the performance of all the workers and performs control based thereon.

9. A non-transitory computer-readable storage medium having computer-readable code stored thereon for programming a processor to perform steps of:

in a Cloud Access Security Broker (CASB) system having a message broker connected to a controller, an authentication provider configured to perform authentication of one or more tenants, and a plurality of workers connected to the message broker and connected to one or more cloud providers having a plurality of files contained therein for the one or more tenants, obtaining policy and configuration for the one or more tenants provided via the controller;

providing assignments from the message broker to the plurality of workers;

crawling through the plurality of files for the one or more tenants, based on the policy and the configuration, to find files being one of a unique file entry or a new event, wherein a first crawl includes,

fetching Information Technology (IT) admin change logs for file-related activities for a tenant in batches, processing the batch for unique file entries in the batch, pushing the file info into a queue, repeating until the entire IT log is crawled, and storing the log's stream-position for a next crawl, wherein IT admin change logs include statistics and transactions, and

responsive to an IT admin change log not being accessible, crawling through the entire list of entities for a tenant, storing the list of entities for each tenant, storing a last change for each entity, updating tenant info, and repeating for all tenants,

wherein the first crawl is synchronized with Data Leakage Prevention (DLP) scan frequency and subsequent crawls performed periodically crawl incrementally; and

pushing the found files into one of a plurality of queues.

10. The non-transitory computer-readable storage medium of claim 9 , wherein one of the plurality of workers is further configured to cause an action of one of the files in the plurality of queues responsive to the provided assignments.

11. The non-transitory computer-readable storage medium of claim 10 , wherein the action includes any of allowing a file, deleting a file, quarantining a file, and providing a notification.

12. The non-transitory computer-readable storage medium of claim 9 , further comprising the steps of

scanning the plurality of files based on the policy and the configuration for Data Leakage Prevention (DLP); and

providing an action based on the scanning.

13. The non-transitory computer-readable storage medium of claim 9 , further comprising the steps of

executing a file of the plurality of files in a sandbox; and

providing an action based on the execution and based on the policy and the configuration.

14. The non-transitory computer-readable storage medium of claim 9 , wherein the plurality of workers include a plurality of types of workers, each being configured to perform a specific task in the CASB system.

15. A method comprising:

in a Cloud Access Security Broker (CASB) system having a message broker connected to a controller, an authentication provider configured to perform authentication of one or more tenants, and a plurality of workers connected to the message broker and connected to one or more cloud providers having a plurality of files contained therein for the one or more tenants, obtaining policy and configuration for the one or more tenants provided via the controller;

providing assignments from the message broker to the plurality of workers;

crawling through the plurality of files for the one or more tenants, based on the policy and the configuration, to find files being one of a unique file entry or a new event, wherein a first crawl includes,

fetching Information Technology (IT) admin change logs for file-related activities for a tenant in batches, processing the batch for unique file entries in the batch, pushing the file info into a queue, repeating until the entire IT log is crawled, and storing the log's stream-position for a next crawl, wherein IT admin change logs include statistics and transactions, and

responsive to an IT admin change log not being accessible crawling through the entire list of entities for a tenant, storing the list of entities for each tenant, storing a last change for each entity, updating tenant info, and repeating for all tenants,

wherein the first crawl synchronized with Data Leakage Prevention (DLP) scan frequency and subsequent crawls performed periodically crawl incrementally; and

pushing the found files into one of a plurality of queues.

16. The method of claim 15 , further comprising

scanning the plurality of files based on the policy and the configuration for Data Leakage Prevention (DLP); and

providing an action based on the scanning.