System and methods for subdividing an unknown tree for execution of operations by multiple compute engines
An electronic system for subdividing an unknown directory tree for execution of operations on directories and files therein has a plurality of computer engines. Each computer engine has a processor. A memory is coupled to the processor, the memory storing program instructions. The program instructions when executed by a processor of a first of the computer engines, causes the processor of the first computer engine to: enumerate a file system comprising directories and files into ordered lists of tasks; subdivide the lists of tasks into a number of subtasks, each of the subtasks being smaller in size than each of the tasks; and deliver the subtasks to a plurality of different computer engines.
1 . An electronic system for executing operations on directories and files in a file system comprising:
a plurality of computer engines, wherein each computer engine comprises:
a processor;
a memory coupled to the processor, the memory storing program instructions;
wherein the program instructions when executed by a first processor of a first computer engine of the plurality of computer engines, causes the first processor of the plurality of computer engines to:
store a corresponding sorted list of directories;
traverse over the corresponding sorted list of directories, wherein known directories of the corresponding sorted list of directories become markers between unknown sublists of directories and files, wherein each corresponding unknown sublist of directory and files in the corresponding unknown sublist of directory are mutually exclusive;
execute a designated operation on each known directory and files in each known directory;
divide the corresponding sorted list of directories into least two parts, wherein at least one of the two part contains a portion of the unknown sublists of directories and files;
assigning the portion of the unknown sublist a second computer engine from the plurality of computer engines; and
communicate to the second computer engine when no more directories in the sorted list of directories remain;
wherein the program instructions when executed by the processor of the second computer engine from the plurality of computer engines causes the processor of the second computer engine to:
receive the portion of the unknown sublist assigned from the first computer engine;
execute a designated operation on the portion of the unknown sublist assigned from the first computer engine;
communicate to the first computer engine a status of the designated operation on the portion of the unknown sublist assigned; and
communicate to the first computer engine when the designated operation on the portion of the unknown sublist assigned is finished.
2 . The electronic system of claim 1 , wherein the memory storing program instructions that when executed by the processors of the first and second compute engines, causes each processor the first and second computer engines to analyze metadata of the directories and files forming a respective subtask being analyzed by a corresponding one of the first and second computer engines.
3 . The electronic system of claim 1 , wherein the memory storing program instructions that when executed by the processors of the first and second computer engines, causes each processor of the first and second engines to analyze in-file metadata or full-file content of the files forming a respective subtask being analyzed by a corresponding one of the first and second computer engines.
4 . The electronic system of claim 3 , wherein the analyze metadata of the directories and files forming the respective subtask is stored in a database.
5 . The electronic system of claim 1 , wherein the memory storing program instructions that when executed by the processors of the first and second compute engines, causes the processors of the first and second computer engines to copy corresponding directories and files to a different file system.
6 . The electronic system of claim 1 , wherein the memory storing program instructions that when executed by the processors of the first and second computer engines, causes the processors of the first and second computer engines to move corresponding directories and files to a different file system.
7 . The electronic system of claim 1 , wherein the memory storing program instructions that when executed by the processors of the first and second computer engines, causes the processors of the first and second computer engines to perform any file management operations to corresponding directories and files.
8 . The electronic system of claim 1 , wherein the memory storing program instructions that when executed by the processors of the first and second computer engines, causes the processors of the first and second computer engines to move and link corresponding directories and files.
9 . The electronic system of claim 1 , wherein the program instructions when executed by the processor of the second computer engine from the plurality of computer engines causes the processor of the second computer engine to request to the first computer engine additional unknown sublists when the designated operation on the portion of the unknown sublist assigned is finished.