Debugging a map reduce application on a cluster
View Patent ↗A method, apparatus, system, article of manufacture, and data structure provide the ability to debug a map-reduce application on a cluster. A cluster of two or more computers is defined by installing a map-reduce framework (that includes an integrated development environment [IDE]) onto each computer. The cluster is formatted by identifying and establishing communication between each computer so that the cluster functions as a unit. Data is placed into the cluster. A function to be executed by the framework on the cluster is obtained, debugged, and executed directly on the cluster using the IDE and the data in the cluster.
1. A computer-implemented method for debugging a map-reduce application on a cluster comprising:
(A) defining a cluster of two or more computers by:
(1) installing a map-reduce framework onto each of the two or more computers, wherein the map-reduce framework comprises an integrated development environment (IDE); and
(2) formatting the cluster by:
(i) identifying the two or more computers in the cluster;
(ii) establishing communication between the two or more computers; and
(iii) enabling the cluster of two or more computers to function as a unit;
(B) placing data into the cluster;
(C) obtaining a function to be executed by the map-reduce framework on the cluster; and
(D) debugging the function directly on the cluster using the IDE, wherein the function being debugged is directly executed on the cluster using the data in the cluster, wherein the debugging further comprises:
(1) dynamically accepting code for the function that is input into a debugger of the IDE;
(2) dynamically determining, as the code is dynamically typed into the debugger, if the code specifies a heap allocation without reusing memory; and
(3) if the code specifies a heap allocation without reusing memory, dynamically highlighting the code in the debugger.
2. The computer-implemented method of claim 1 , wherein the formatting identifies all of the computers in the cluster as well as the computer that the formatting command is being issued from.
3. The computer-implemented method of claim 1 , wherein the debugging is initialized on any of the two or more computers without regard to functionality available on the two or more computers.
4. The computer-implemented method of claim 1 , further comprising providing, as part of the IDE, a skip to reduce function configured to skip a map portion of the map-reduce function during the debugging.
5. The computer-implemented method of claim 1 , further comprising utilizing a scroll widget that is configured to specify a location in the data to begin the debugging.
6. The computer-implemented method of claim 1 , wherein the function is obtained by dynamically developing the function as part of the IDE using a just-in-time compiler that compiles code of the function at execution time.
7. The computer-implemented method of claim 1 further comprising accepting an alternative string data type for the data in the cluster, wherein the alternative string data type reuses its own memory stack within the IDE.
8. An apparatus for debugging a map-reduce application on a computer cluster comprising:
(A) a computer cluster comprising two or more computers;
(B) a map-reduce framework installed onto each of the two or more computers, wherein:
(1) the map-reduce framework defines the cluster;
(2) the map-reduce framework comprises an integrated development environment (IDE); and
(3) the cluster is formatted by:
(i) identifying the two or more computers in the cluster;
(ii) establishing communication between the two or more computers; and
(iii) enabling the cluster of two or more computers to function as a unit;
(C) data that is placed into the cluster; and
(D) a function configured to be executed by the map-reduce framework on the cluster; and
(E) a debugger that is part of the IDE, wherein:
(1) the function is debugged directly on the cluster using the debugger; and
(2) the function being debugged is directly executed on the cluster using the data in the cluster;
(3) the debugger is configured to dynamically accept code for the function that is input into a debugger of the IDE;
(4) the debugger is configured to dynamically determine, as the code is dynamically typed into the debugger, if the code specifies a heap allocation without reusing memory; and
(5) if the code specifies a heap allocation without reusing memory, the debugger is configured to dynamically highlight the code in the debugger.
9. The apparatus of claim 8 , wherein the cluster is formatted by identifying all of the computers in the cluster as well as the computer that the formatting command is being issued from.
10. The apparatus of claim 8 , wherein the debugger is initialized on any of the two or more computers without regard to functionality available on the two or more computers.
11. The apparatus of claim 8 , wherein the debugger comprises a skip to reduce function configured to skip a map portion of the map-reduce function during the debugging.
12. The apparatus of claim 8 , wherein the debugger comprises a scroll widget that is configured to specify a location in the data to begin the debugging.
13. The apparatus of claim 8 , wherein the function is obtained by dynamically developing the function as part of the IDE using a just-in-time compiler that compiles code of the function at execution time.
14. The apparatus of claim 8 , wherein the debugger is further configured to accept an alternative string data type for the data in the cluster, wherein the alternative string data type reuses its own memory stack within the IDE.
15. A system for debugging a map-reduce application on a cluster comprising:
(A) two or more computers communicatively coupled to each other to define a cluster that functions as a unit;
(B) a map-reduce framework installed on each of the two or more computers, wherein the map-reduce framework comprises an integrated development environment (IDE);
(C) a function configured to be executed by the map-reduce framework on data in the cluster;
(D) a debugger installed as part of the IDE, wherein the debugger is configured to:
(1) debug the function directly on the cluster;
(2) dynamically accept code for the function that is input into a debugger of the IDE;
(3) dynamically determine, as the code is dynamically typed into the debugger, if the code specifies a heap allocation without reusing memory; and
(4) if the code specifies a heap allocation without reusing memory, dynamically highlight the code in the debugger.
16. The system of claim 15 , wherein a debug operation is initialized on the debugger on any of the two or more computers without regard to functionality available on the two or more computers.
17. The system of claim 15 , wherein the debugger further comprises, a skip to reduce function configured to skip a map portion of the map-reduce function during a debugging operation.
18. The system of claim 15 , wherein the debugger further comprises a scroll widget that is configured to specify a location in the data to begin the debugging.
19. The system of claim 15 , wherein the function is obtained by dynamically developing the function as part of the IDE using a just-in-time compiler that compiles code of the function at execution time.
20. The system of claim 15 , wherein the debugger is further configured to accept an alternative string data type for the data in the cluster, wherein the alternative string data type reuses its own memory stack within the IDE.