Method and system for high-throughput distributed computing of computational jobs

1. A method for distributed computing of computational jobs composed of job data, comprising:

providing a data storage system comprising one or more databases,

setting up a grid of user devices from user entities, the user devices being remote from the storage system and at least periodically in signal communication with the storage system, each user device having a predetermined computing capacity,

periodically querying each user device about a respective device status, the device status comprising computing capacity and availability parameters, wherein querying comprises requesting each user device to generate and send to the storage system a status answer, the status answer including the device status of the user device,

defining a grid status as a function of status answers that were successfully sent by the user devices and received in the storage system,

providing customer entities with an informatic customer platform,

receiving job specifications, uploaded in the storage system through the customer platform by the customer entities, each job specification including parameters characterizing a respective computational job,

selecting for each computational job a respective job partitioning scheme as a function of the grid status and of the job specification,

receiving input streams of data, wherein each input stream is received for a respective computational job after receipt of the job specification of the computational job, includes the job data of the computational job, and is uploaded through the customer platform by a customer entity,

splitting the job data included in each input stream in input chunks of data, according to splitting parameters of the partitioning scheme selected for the computational job,

generating one or more tasks for each input chunk, each task including the respective input chunk and executable instructions to execute computations on the respective input chunk,

wherein the job partitioning scheme selected for each computational job comprises one or more replication parameters, and generating the tasks comprises generating plural replicated tasks for at least some input chunks, such that the tasks are more than the input chunks according to the one or more replication parameters,

assigning and sending each task to one or more respective user devices, that are selected based on distribution parameters of the partitioning scheme selected for the respective computational job, wherein replicated tasks generated for a common input chunk are assigned to distinct user devices,

receiving in the storage system output chunks of data from the user devices, each output chunk being obtained by a user device by executing a task,

verifying that at least one output chunk has timely arrived for each input chunk,

outputting to the customer entities job results through the customer platform, each job result including data obtained by assembling the output chunks related to the corresponding computational job.

2. The method of claim 1 , wherein:

for each computational job, receiving the input stream of the computational job takes an upload time interval,

the steps of splitting the job data, generating the tasks and sending the tasks to the user devices begin during the upload time interval,

each input chunk is temporarily stored in the storage system, from a respective save instant, in which the input chunk is saved in the storage system, to a respective delete instant, in which the input chunk is deleted from the storage system,

the save instant of each input chunk occurs during the upload time interval of the respective computational job,

for at least one input chunk, the delete instant occurs during the upload time interval of the respective computational job.

3. The method of claim 1 , wherein:

the distribution parameters of the partitioning scheme include one or more target ranges for respective computing capacity and availability parameters or combinations thereof, the target ranges being determined as a function of the job specifications,

the partitioning scheme provides, for the purpose of assigning tasks of a computational job to user devices, to discard incompatible user devices, having device status with computing capacity and availability parameters that are incompatible with the one or more target ranges, and to at least pre-elect the remaining compatible user devices, having device status with computing capacity and availability parameters that are compatible with the one or more target ranges.

4. The method of claim 1 , wherein:

selecting the job partitioning scheme comprises grouping the user devices in clusters, each cluster comprising user devices that are selected based on values of their computing capacity and availability parameters,

assigning the tasks to the user devices comprises:

pre-assigning cluster task groups to distinct clusters, of the cluster task groups being arranged as cluster queues,

for each cluster, feeding all the tasks of the cluster queue of the computational job_into the individual device queue of each user device of the cluster, preferably wherein each individual device queue include task labels, indicating assignment statuses of each task as being already assigned or not yet assigned,

selecting by an available user device of the cluster a specific task, from the individual device queue of the user device, that has not been so far assigned to another user device, and confirming assignment of the selected specific task to said available user device.

5. The method of claim 4 , wherein:

feeding the tasks of the cluster queues into the device queues comprises shuffling the tasks of each cluster queue with different orders, preferably random orders, for the different user devices of the cluster, and feeding the tasks in the individual device queues in the respective different orders,

selecting the specific task by the available user device comprises selecting the first task in the device queue of the user device that has not been assigned so far to another user device.

6. The method of claim 4 , wherein:

selecting the job partitioning scheme comprises assigning to the clusters, as a splitting parameter, respective different cluster chunk sizes, and as a distribution parameter, respective cluster chunk numbers, based on the job specifications, preferably wherein higher cluster chunk sizes are assigned to clusters having higher values of computing capacity and availability parameters, and lower cluster chunk sizes are assigned to clusters having lower values of computing capacity and availability parameters,

splitting the job data generates for each cluster, and feeds into the cluster queue of the cluster, and

inputting chunks in a number not exceeding a cluster chunk number of the cluster, preferably the number equals to the cluster chunk number of the cluster, and with a size not exceeding a cluster chunk size of the cluster, preferably the size equals to the cluster chunk size of the cluster.

7. The method of claim 1 , wherein, after verifying timely arrival of the output chunks, in case no output chunk has been received for a specific input chunk within a predetermined maximum time, the method comprises re-assigning the task related to the specific input chunk to another user device or to a software validator service running on the storage system, the validator service being configured in this case to obtain an output chunk by executing the task.

8. The method of claim 1 , wherein:

defining the grid status comprises processing the computing capacity and availability parameters of each device status to obtain for each user device one or more device scores,

defining the grid status comprises ordering the user devices in one or more grid indexes based on respective device scores, and

said distribution parameters of the job partitioning scheme are set to assign the tasks to the user devices based on the device positions in said one or more grid indexes.

9. The method of claim 1 , wherein the computing capacity and availability parameters of each device status comprise one or more of:

power on/off, availability time window, current activity of the user device, remaining battery, core temperature, user device location and/or accelerometer data,

values of installed, available and/or usage threshold for RAM, CPU, storage and/or bandwidth,

network type, speed, latency and/or IP address,

performance on a recent test tasks or on a recent task generated from an input chunk.

10. Electronic system for distributed computing of computational jobs by the method of claim 1 , comprising:

a customer data processing system including a memory and a processor configured to execute a customer-side software including said customer platform, with algorithms for inputting said job specifications, inputting said input streams of data, and outputting said job results,

said data storage system including a memory and a processor configured to execute a software internal service system, with algorithms for querying the user devices about the device status, receiving said status answers and defining said grid status, selecting said job partitioning schemes, splitting said job data, generating said tasks, assigning and sending the tasks to the user devices, receiving said output chunks from the user devices, verifying timely arrival of the output chunks, and assembling the output chunks in said job results,

said user device, each user device including a memory and a processor configured to execute a user-side software, with algorithms for assessing the computing capacity and availability parameters of a user device, generating and sending said status answers, receiving the tasks assigned to the user device, executing the tasks to generate said output chunks, and sending the output chunks to the storage system.