Geo-distributed computation and analytics based on cost of transporting and computational cost

1. A Multi-site Integrated Session-oriented Transaction (MIST) computing system, the system comprising:

one or more field nodes;

a central processing facility comprising one or more cloud nodes, the central processing facility geographically remote from each of the one or more field nodes;

a wide area network comprising one or more virtual links, each virtual link of the one or more virtual links connecting a corresponding field node of the one or more field nodes to at least one of the cloud nodes;

a MIST controller comprising a MIST compiler and a MIST mapper, executing on a computing platform,

wherein the MIST compiler is configured to process a distributed compute session request for a distributed dataset to generate a compiled distributed compute session request comprising one or more functions and one or more data dependencies, wherein the distributed dataset comprises one or more geographically-distributed sets of data, the sets of data including at least one remote set of data that is proximate to one of the field nodes, and

wherein the MIST mapper is configured to:

process the compiled distributed compute session request to determine, for each of the at least one remote sets of data, whether to process the remote set of data with a proximate field node of the field nodes that is proximate to the remote set of data or with a cloud node of one of the cloud nodes, based on a comparison of a first total cost comprising a cost of transporting the remote set of data from the proximate field node to the cloud node via a corresponding virtual link of the plurality of virtual links and a computational cost for processing the remote set of data at the cloud node with a second total cost comprising a computation cost for processing the remote set of data at the proximate field node to generate output data and a cost of transporting the output data from the proximate field node to the cloud node via the corresponding virtual link;

in response to determining to process the remote set of data with the cloud node, output a data retrieval request to the proximate field node to cause the proximate field node to send the remote set of data to the cloud node for processing at the cloud node; and

in response to determining to process the remote set of data with the proximate field node, output a computation request to the proximate field node to cause the proximate field node to process the remote set of data.

2. The system of claim 1 , comprising:

a MIST cluster comprising the one or more cloud nodes and the one or more field nodes interconnected with the one or more virtual links.

3. The system of claim 2 , wherein each of the one or more virtual links comprises a secure virtual tunnel.

4. The system of claim 1 , comprising:

a cluster manager configured to track one or more of resources of the one or more cloud nodes and the one or more field nodes, data proximity for sets of data to the field nodes, and resource costs,

wherein the MIST mapper is configured to determine whether to process the remote set of data with one of the field nodes that is proximate to the remote set of data or with the cloud node of one of the cloud nodes further based on one of more of the resources of the one or more cloud nodes and the one or more field nodes, the data proximity for sets of data to the field nodes, or the resource costs.

5. The system of claim 1 , wherein, to determine whether to process the remote set of data with one of the field nodes that is proximate to the remote set of data or with the cloud node, the MIST mapper is configured to determine a first proximity of a data source for the computation request relative to the field node and determine a second proximity of the data source for the computation request relative to the cloud node.

6. The system of claim 5 , wherein the MIST mapper is configured to select the field node from the plurality of field nodes in response to determining that the first proximity is less than a proximity of each other node of the plurality of field nodes.

7. The system of claim 1 , wherein, to determine whether to process the remote set of data with one of the field nodes that is proximate to the remote set of data or with the cloud node, the MIST mapper is configured to apply machine learning.

8. The system of claim 1 , wherein the compiled distributed compute session request is a first compiled distributed compute session request, and wherein the MIST mapper is configured to:

process a second compiled distributed compute session request to determine, for each of one or more geographically-distributed sets of data for the second compiled distributed compute session request, whether to process the remote set of data for the second compiled distributed compute session request with one of the field nodes that is proximate to the remote set of data or with the cloud node, based on a cost of transporting the remote set of data from the proximate field node to the cloud node via one of the plurality of virtual links;

in response to determining to process the remote set of data for the second compiled distributed compute session request with the cloud node, output a data retrieval request for the second compiled distributed compute session request to the proximate field node to cause the proximate field node to send the remote set of data for the second compiled distributed compute session request to the cloud node for processing at the cloud node; and

in response to determining to process the remote set of data for the second compiled distributed compute session request with the field node, output a computation request for the second compiled distributed compute session request to the proximate field node to cause the proximate field node to process the remote set of data.

9. The system of claim 1 , wherein the computation request comprises a request to process streaming network data for network and information security and intrusion detection.

10. The system of claim 9 ,

wherein the MIST compiler determines to process the computation request at the field node; and

wherein, to process the streaming network data, the field node is configured to generate behavioral analysis information and send information for the streaming network data to the cloud node.

11. The system of claim 10 , wherein the cloud node determines malware is detected in the MIST computing system using the behavioral analysis information.

12. The system of claim 1 , wherein, to determine whether to process the remote set of data with one of the field nodes that is proximate to the remote set of data or with the cloud node, the MIST mapper is configured to calculate a cost function based on the cost of transporting data for the computation request on the plurality of virtual links to the cloud node.

13. The system of claim 12 , wherein, to calculate the cost function, the MIST mapper is configured to calculate the cost function further based on one or more of a computational graph for a session for the computation request, data at the cloud node, data at one or more of the plurality of field nodes, a bandwidth at the plurality of virtual links, or a latency at the plurality of virtual links.

14. A method for Multi-site Integrated Session-oriented Transaction (MIST) computing comprising:

processing, by processing circuitry, a distributed compute session request for a distributed dataset to generate a compiled distributed compute session request comprising one or more functions and one or more data dependencies, wherein the distributed dataset comprises one or more geographically-distributed sets of data, the sets of data including at least one remote set of data that is proximate to one of one or more field nodes, wherein a wide area network comprises one or more virtual links, each virtual link of the one or more virtual links connecting a corresponding field node of the one or more field nodes to at least a cloud node of one of one or more cloud nodes of a central processing facility, the central processing facility being geographically remote from each of the one or more field nodes;

processing, by the processing circuitry, the compiled distributed compute session request to determine, for each of the at least one remote sets of data, whether to process the remote set of data with a proximate field node of the field nodes that is proximate to the remote set of data or with a cloud node of one of the cloud nodes, based on a comparison of a first total cost comprising a cost of transporting the remote set of data from the proximate field node to the cloud node via a corresponding virtual link of the plurality of virtual links and a computational cost for processing the remote set of data at the cloud node with a second total cost comprising a computation cost for processing the remote set of data at the proximate field node to generate output data and a cost of transporting the output data from the proximate field node to the cloud node via the corresponding virtual link;

in response to determining to process the remote set of data with the cloud node, outputting, by the processing circuitry, a data retrieval request to the proximate field node to cause the proximate field node to send the remote set of data to the cloud node for processing at the cloud node; and

in response to determining to process the remote set of data with the proximate field node, outputting, by the processing circuitry, a computation request to the proximate field node to cause the proximate field node to process the remote set of data.

15. The method of claim 14 , wherein a MIST cluster comprises the one or more cloud nodes and the one or more field nodes interconnected with the one or more virtual links.

16. The method of claim 15 , wherein each of the one or more virtual links comprises a secure virtual tunnel.

17. The method of claim 14 , further comprising:

tracking one or more of resources of the one or more cloud nodes and the one or more field nodes, data proximity for sets of data to the field nodes, and resource costs,

wherein determining whether to process the remote set of data with one of the field nodes that is proximate to the remote set of data or with the cloud node of one of the cloud nodes further is based on one of more of the resources of the one or more cloud nodes and the one or more field nodes, the data proximity for sets of data to the field nodes, or the resource costs.

18. The method of claim 14 , wherein determining whether to process the remote set of data with one of the field nodes that is proximate to the remote set of data or with the cloud node comprises determining a first proximity of a data source for the computation request relative to the field node and determine a second proximity of the data source for the computation request relative to the cloud node.

19. The method of claim 18 , further comprising:

selecting the field node from the plurality of field nodes in response to determining that the first proximity is less than a proximity of each other node of the plurality of field nodes.

20. A non-transitory computer-readable storage device comprising instructions that, when executed, cause one or more processors to perform operations including:

processing a distributed compute session request for a distributed dataset to generate a compiled distributed compute session request comprising one or more functions and one or more data dependencies, wherein the distributed dataset comprises one or more geographically-distributed sets of data, the sets of data including at least one remote set of data that is proximate to one of one or more field nodes, wherein a wide area network comprises one or more virtual links, each virtual link of the one or more virtual links connecting a corresponding field node of the one or more field nodes to at least a cloud node of one of one or more cloud nodes of a central processing facility, the central processing facility being geographically remote from each of the one or more field nodes;

processing the compiled distributed compute session request to determine, for each of the at least one remote sets of data, whether to process the remote set of data with a proximate field node of the field nodes that is proximate to the remote set of data or with a cloud node of one of the cloud nodes, based on a comparison of a first total cost comprising a cost of transporting the remote set of data from the proximate field node to the cloud node via a corresponding virtual link of the plurality of virtual links and a computational cost for processing the remote set of data at the cloud node with a second total cost comprising a computation cost for processing the remote set of data at the proximate field node to generate output data and a cost of transporting the output data from the proximate field node to the cloud node via the corresponding virtual link;

in response to determining to process the remote set of data with the cloud node, outputting a data retrieval request to the proximate field node to cause the proximate field node to send the remote set of data to the cloud node for processing at the cloud node; and

in response to determining to process the remote set of data with the proximate field node, outputting a computation request to the proximate field node to cause the proximate field node to process the remote set of data.