Enterprise-scalable scanning using grid-based architecture with remote agents
View Patent ↗An enterprise coordinator coupled to one or more site coordinators provides configuration and scheduling of tasks across a plurality of sites, and accumulates results. Each of the site coordinators optionally manages one or more respective agents to perform agent-local ones of the tasks, and optionally manages one or more respective grid workers to perform, in a distributed fashion, site-local ones of the tasks. Each of the site coordinators optionally apportions a state file into respective work lists that are assigned to the respective grid workers, and concatenates at least some portions of received results to produce a new state file. Each of the grid workers performs operations in accordance with the assigned work list and returns results including an updated version of the work list. In some usage scenarios, a bootstrapping technique is used to install an agent program on unprovisioned ones of the agents and/or the grid workers.
1. A method comprising:
controlling each of a first group of controlled computers of a site to perform respective local scans of respective file subsets, each respective file subset comprising files that are of respective local volumes implemented via respective local disks associated with the respective controlled computer;
controlling a second group of controlled computers of the site to collectively perform a distributed scan of a file subgroup, the file subgroup comprising files that are of global volumes implemented by one or more storage servers of the site;
managing a state file that specifies the file subgroup;
aggregating results of the local scans and the distributed scan;
providing the aggregated results to a scheduling computer; and
wherein the acts of controlling, the act of managing, and the act of aggregating results are via a controlling computer of the site that is enabled to communicate with the first and the second groups of controlled computers via a local area network of the site, and further enabled to communicate with the scheduling computer via a wide area network.
2. The method of claim 1 , wherein scanning a plurality of files of a plurality of storage volumes comprises the acts of controlling and the act of managing, and the local and the global volumes comprise the plurality of storage volumes.
3. The method of claim 1 , wherein the first group of controlled computers comprises one or more agents.
4. The method of claim 1 , wherein the second group of controlled computers comprises one or more grid workers.
5. The method of claim 1 , wherein the controlling computer is a site coordinator that is specific to the site.
6. The method of claim 1 , wherein at least one of the storage servers comprises a storage attached network or network attached storage.
7. The method of claim 1 , wherein the scheduling computer is an enterprise coordinator.
8. A system comprising:
a processing element enabled, via a local area network, to communicate with a first group of controlled computers of a site, each of the first group of controlled computers having respective local disks providing respective local volumes, the processing element being further enabled, via the local area network, to communicate with a second group of controlled computers of the site, each of the second group of controlled computers having access to one or more global volumes provided by one or more storage servers of the site, the processing element being further enabled, via a wide area network, to communicate with a scheduling computer; and
a storage medium coupled to the processing element, the storage medium enabled to store instructions that when executed by the processing element cause the processing element to perform functions comprising
controlling each of the first group of controlled computers to perform respective local scans of respective file subsets, each respective file subset comprising files of the respective local volumes of the respective controlled computer,
controlling the second group of controlled computers to collectively perform a distributed scan of a file subgroup, the file subgroup comprising files of the global volumes,
managing a state file that specifies the file subgroup,
aggregating results of the local scans and the distributed scan, and
providing the aggregated results to the scheduling computer.
9. The system of claim 8 , wherein a site coordinator that is specific to the site comprises the processing element.
10. The system of claim 8 , wherein the functions further comprise providing a respective directory path to each of the second group of controlled computers.
11. The system of claim 8 , wherein the functions further comprise receiving a list of directories from at least one of the second group of controlled computers, and based at least in part on the list, providing a directory path to one or more of the second group of controlled computers.
12. The system of claim 8 , wherein at least one of the first group of controlled computers is a temporary agent.
13. The system of claim 8 , wherein at least one of the first group of controlled computers is a permanent agent.
14. A non-transitory computer readable medium having a set of instructions stored therein which when executed by a processing element of a controlling computer of a site causes the processing element to perform functions comprising:
controlling, via a local area network of the site, each of a first group of controlled computers of the site to perform respective local scans of respective file subsets, each respective file subset comprising files that are of respective local volumes implemented via respective local disks associated with the respective controlled computer;
controlling, via the local area network, a second group of controlled computers of the site to collectively perform a distributed scan of a file subgroup, the file subgroup comprising files that are of global volumes implemented by one or more storage servers of the site; managing a state file that specifies the file subgroup;
aggregating results of the local scans and the distributed scan, the results comprising identification of the files of the local and the global volumes that are new, modified, or deleted; and
providing, via a wide area network, the aggregated results to a scheduling computer.
15. The non-transitory computer readable medium of claim 14 , wherein the first group of controlled computers comprises one or more agents, at least one of the agents is a temporary agent and at least one of the agents is a permanent agent, the second group of controlled computers comprises one or more grid workers, the controlling computer is all or any portion of a site coordinator that is specific to the site, at least one of the storage servers is either a storage attached network or comprises network attached storage, and the scheduling computer is all or any portion of an enterprise coordinator.
16. The non-transitory computer readable medium of claim 14 , wherein the functions further comprise providing a directory path to at least one of the second group of controlled computers.
17. The non-transitory computer readable medium of claim 14 , wherein the functions further comprise receiving respective lists of directories from each of a first subgroup of the second group of controlled computers, and based at least in part on the lists, providing a respective directory path to each of a second subgroup of the second group of controlled computers.
18. The non-transitory computer readable medium of claim 14 , wherein the functions further comprise managing a bootstrap communication with one of the first group of controlled computers.
19. The non-transitory computer readable medium of claim 14 , wherein the functions further comprise communicating a same directory path to two or more of the second group of controlled computers, the same directory path to be analyzed at least partially in parallel by the two or more of the second group of controlled computers as part of the distributed scan, and further communicating unique respective ranges to each of the two or more of the second group of controlled computers, the unique respective ranges being with respect to the same directory path.
20. The non-transitory computer readable medium of claim 14 , wherein the functions further comprise providing one or more hashes of one or more corresponding directories to one or more of the second group of controlled computers.