Data processing

1. A method implemented in a network apparatus used in a network, the method comprising:

collecting information about network topology from a network controller;

collecting information about data movement;

deciding routing in the network according to the information about network topology and the information about data movement; and

providing information about the routing to the network controller,

wherein the network controller enforces the routing in the network,

wherein the information about data movement comprises shuffle phase data movement, and

wherein the information about data movement is collected during the shuffle phase data movement by:

modifying a reduce task code,

modifying a JobTracker code, or

running an agent continuously scanning a log file.

2. The method as in claim 1 , wherein the network comprises a software defined network or an OpenFlow network.

3. The method as in claim 1 , wherein the information about network topology is inferred by:

sending Link Layer Discovery Protocol (LLDP) packets from a switch; and

examining rules installed in the switch.

4. The method as in claim 1 , wherein the information about data movement comprises Hadoop Distributed File System (HDFS) related data movement.

5. The method as in claim 1 , wherein the routing in the network is decided by:

estimating network demand;

detecting a network hotspot; and

scheduling a network flow.

6. The method as in claim 1 , wherein the information about the routing comprises:

a pair of endpoints; and

a list of one or more switches that traffic between the pair of endpoints traverses.

7. The method as in claim 6 , wherein the network controller creates a set of network rules to be installed in each switch.

8. The method as in claim 7 , wherein each network rule includes:

a match part used to match said each rule against packets between the pair of endpoints; and

an action part used to forward traffic to a next switch on a path.

9. A non-transitory computer readable medium storing a program causing a computer to execute a process including:

collecting information about network topology from a network controller;

collecting information about data movement;

deciding routing in the network according to the information about network topology and the information about data movement; and

providing information about the routing to the network controller,

wherein the network controller enforces the routing in the network,

wherein the information about data movement comprises shuffle phase data movement, and

wherein the information about data movement is collected during the shuffle phase data movement by:

modifying a reduce task code,

modifying a JobTracker code, or

running an agent continuously scanning a log file.

10. The non-transitory computer readable medium as in claim 9 , wherein the network comprises a software defined network or an OpenFlow network.

11. The non-transitory computer readable medium as in claim 9 , wherein the information about network topology is inferred by:

sending Link Layer Discovery Protocol (LLDP) packets from a switch; and

examining rules installed in the switch.

12. The non-transitory computer readable medium as in claim 9 , wherein the information about data movement comprises Hadoop Distributed File System (HDFS) related data movement.

13. The non-transitory computer readable medium as in claim 9 , wherein the routing in the network is decided by:

estimating network demand;

detecting a network hotspot; and

scheduling a network flow.

14. The non-transitory computer readable medium as in claim 9 , wherein the information about the routing comprises:

a pair of endpoints; and

a list of one or more switches that traffic between the pair of endpoints traverses.

15. A network comprising:

a network apparatus; and

a network controller,

wherein the network apparatus collects information about network topology from the network apparatus, collects information about data movement, decides routing in the network according to the information about network topology and the information about data movement, and provides information about the routing to the network controller,

wherein the network controller provides the information about network topology, creates a set of network rules to be installed in each switch according to information about the routing, and enforces the routing in the network,

wherein the information about data movement comprises shuffle phase data movement, and

wherein the information about data movement is collected during the shuffle phase data movement by:

modifying a reduce task code,

modifying a JobTracker code, or

running an agent continuously scanning a log file.