Parallel tasking application framework
View Patent ↗Particular embodiments generally relate to a parallel tasking framework. Network devices may use discovery messages to determine which services are being offered by different devices. The services provided using the framework allow parallel processing of tasks using modules found in devices. Distribution of modules from one of the devices to other devices may be provided to process a project, which may be any analysis that needs to be performed on data. A project may be made up of tasks that need to be performed where each task may process at least part of the data for the project. The parallel tasking framework allows a device to use processing resources from other devices when other devices are not using the full availability of their processing resources.
1. A method for processing data, the method comprising:
receiving one or more discovery messages including services found on one or more networked devices, the services provided in a framework allowing parallel processing of tasks using modules compatible with the framework;
determining a project in which data should be processed;
determining one of the networked devices that has processing resources to perform a service for the project, the service determined to be able to perform a task in processing at least a part of the data for the project;
distributing a module from the framework for the service to the one of the networked devices; and
distributing data for the task to the one of the networked devices, the one of the networked devices processing the data using the module for the project.
2. The method of claim 1 , further comprising:
storing the services found on the one or more networked devices in a registry; and
analyzing the services in the registry to determine which of the one or more networked devices is capable of performing the task for the projects using a service.
3. The method of claim 1 , further comprising:
determining a service found on a network device in the one or more networked devices that can be transferred to the first device; and
sending a request of the service to the network device.
4. The method of claim 1 , further comprising:
detecting a new device on the network;
determining if the new device has processing resources; and
distributing a second task to the new device if the new device has available processing resources.
5. The method of claim 1 , wherein the one or more networked devices include the framework, which allows distribution of modules among the one or more networked devices.
6. The method of claim 1 , further comprising:
polling the one or more network devices dynamically to determine available processing resources; and
analyzing the project to determine the one of the networked devices to perform the task based on the dynamically determined available processing resources.
7. The method of claim 6 , wherein the one of the networked devices is selected based on capabilities included in the one of the networked devices.
8. The method of claim 1 , further comprising:
determining if the data includes segmentation restrictions; and
if the data includes segmentation restrictions, segmenting the data into a data unit based on the segmentation restrictions and the one of the networked devices such that the segmentation restrictions are not violated.
9. The method of claim 8 , further comprising if the data does not include segmentation restrictions, segmenting the data based on the capabilities of the one of the networked devices.
10. The method of claim 1 , wherein the modules comprises modules that are compatible with multiple computing architectures, wherein a module compatible with the one of the networked devices is sent to the one of the networked devices.
11. An apparatus comprising:
one or more processors; and
logic encoded in one or more computer-readable media for execution by the one or more processors and when executed operable to:
receive one or more discovery messages including services found on one or more networked devices, the services provided in a framework allowing parallel processing of tasks using modules compatible with the framework;
determine a project in which data should be processed;
determine one of the networked devices that has processing resources to perform a service for the project, the service determined to be able to perform a task in processing at least a part of the data for the project;
distribute a module from the framework for the service to the one of the networked devices; and
distribute data for the task to the one of the networked devices, the one of the networked devices processing the data using the module for the project.
12. The apparatus of claim 11 , wherein the logic when executed is further operable to:
store the services found on the one or more networked devices in a registry; and
analyze the services in the registry to determine which of the one or more networked devices is capable of performing the task for the projects using a service.
13. The apparatus of claim 11 , wherein the logic when executed is further operable to:
determine a service found on a network device in the one or more networked devices that can be transferred to the first device; and
send a request of the service to the network device.
14. The apparatus of claim 11 , wherein the logic when executed is further operable to:
detect a new device on the network;
determine if the new device has processing resources; and
distribute a second task to the new device if the new device has available processing resources.
15. The apparatus of claim 11 , wherein the one or more networked devices include the framework, which allows distribution of modules among the one or more networked devices.
16. The apparatus of claim 11 , wherein the logic when executed is further operable to:
poll the one or more network devices dynamically to determine available processing resources; and
analyze the project to determine the one of the networked devices to perform the task based on the dynamically determined available processing resources.
17. The apparatus of claim 16 , wherein the one of the networked devices is selected based on capabilities included in the one of the networked devices.
18. The apparatus of claim 11 , wherein the logic when executed is further operable to:
determine if the data includes segmentation restrictions; and
if the data includes segmentation restrictions, segment the data into a data unit based on the segmentation restrictions and the one of the networked devices such that the segmentation restrictions are not violated.
19. The apparatus of claim 18 , wherein the logic when executed is further operable to if the data does not include segmentation restrictions, segment the data based on the capabilities of the one of the networked devices.
20. A system for processing data, the method comprising:
a plurality of target computing devices, wherein each of the target computing devices include different processing resources; and
a master computing device configured to:
receive one or more discovery messages including services found on the plurality of target computing devices, the services provided in a framework allowing parallel processing of tasks using modules compatible with the framework;
determine a project in which data should be processed;
determine processing resources available in the plurality of target computing devices to perform one or more services for the project, the one or more services determined to be able to perform one or more tasks in processing at least a part of the data for the project;
distribute a module in the one or more modules from the framework for a service to each of the plurality of target computing devices; and
distribute data for a task to each of the plurality of target computing devices, wherein each of the plurality of target computing devices processes the data distributed using the distributed module for the project.