Distributed steam processing

1. A system comprising:

one or more processors;

one or more non-transitory computer-readable media containing instructions that, in response to being executed by the one or more processors, cause the system to perform one or more operations comprising:

receive, from a master node in a cluster of nodes, a command for a first slave node to process a file, the first slave node operating within a first operating system, the cluster of nodes configured to utilize a second operating system different from the first operating system, the cluster of nodes including the first slave node that operates within the first operating system and a plurality of second slave nodes that utilize the second operating system;

in response to the command, initiate a specialized class within the first operating system to facilitate access to the cluster, the specialized class configured to, prior to attempting to access the file for processing as instructed by the master node, determine:

whether the first slave node is operating within the first operating system different from the second operating system of the cluster; and

whether the file is accessible via a local file system of the first operating system;

based on determining both (i) the first slave node is operating within the first operating system, and (ii) the file is inaccessible via the local file system of the first operating system, send a request, by the specialized class, to a client-side runner module for the file, the client-side runner module configured to request the file from a shared storage server via a server-side runner module, the server-side runner module configured to communicate with the shared storage server on behalf of the first slave node using the second operating system;

in response to receiving the file from the server-side runner module, store the file locally in the local file system of the first operating system within which the first slave node is operating; and

process the file as commanded by the master node by first accessing the file directly from the local file system.

2. The system of claim 1 , wherein the second operating system natively supports Hadoop.

3. The system of claim 2 , wherein

the cluster of nodes includes the master node that utilizes the second operating system and is configured to distribute data among the first slave node and the plurality of second slave nodes; and

the operations further comprise:

receive a server key from the master node, and

store the server key at the first slave node such that the first slave node accesses the cluster of nodes without a passphrase being manually entered.

4. The system of claim 1 , wherein the operations further comprise:

send a message to add the first slave node to the cluster of nodes, the message being sent to the server-side runner module, and

update the shared storage server to indicate the first slave node as being added to the cluster of nodes.

5. The system of claim 1 , wherein the operations further comprise:

send a message to remove the first slave node from the cluster of nodes, the message being sent to the server-side runner module, and

update the shared storage server to indicate the first slave node as being removed from the cluster of nodes.

6. The system of claim 1 , wherein:

the first operating system executes an emulating program that emulates an environment of the second operating system; and

the first slave node is configured to make an operating system call that is natively supported by the second operating system based on the emulating program executed by the first operating system.

7. The system of claim 1 , wherein the cluster of nodes processes log files of a content provider stored in the cluster of nodes, and the file is at least part of the log files.

8. A method comprising:

receiving, from a master node in a cluster of nodes, a command for a first slave node to process a file, the first slave node operating within a first operating system, the cluster of nodes configured to utilize a second operating system different from the first operating system, the cluster of nodes including the first slave node;

in response to the command, initiating a specialized class within the first operating system to facilitate access to the cluster, the specialized class configured to, prior to attempting to access the file for processing as instructed by the master node, determine:

whether the first slave node is operating within the first operating system that is different from the second operating system of the cluster; and

whether the file is accessible via a local file system of the first operating system;

based on determining both (i) the first slave node is operating within the first operating system, and (ii) the file is inaccessible via the local file system of the first operating system, sending a request, by the specialized class, to a client-side runner module for the file, the client-side runner module configured to request the file from a shared storage server via a server-side runner module, the server-side runner module configured to communicate with the shared storage server using the second operating system and on behalf of the first slave node; and

in response to receiving the file from the server-side runner module, storing the file locally in the local file system of the first operating system within which the first slave node is operating such that the first slave node can directly access the file using the first operating system.

9. The method of claim 8 , wherein the second operating system natively supports Hadoop.

10. The method of claim 9 , wherein:

the cluster of nodes includes the master node that utilizes the second operating system and is configured to distribute data among the first slave node and a plurality of second slave nodes, the plurality of second slave nodes utilizing the second operating system; and

the method further comprises:

receiving a server key from the master node, and

storing the server key at the first slave node such that the first slave node accesses the cluster of nodes without a passphrase being manually entered.

11. The method of claim 8 , further comprising:

sending a message to add the first slave node to the cluster of nodes, the message being sent to the server-side runner module, and

updating the shared storage server to indicate the first slave node as being added to the cluster of nodes.

12. The method of claim 8 , the method further comprising:

sending a message to remove the first slave node from the cluster of nodes, the message being sent to the server-side runner module, and

updating the shared storage server to indicate the first slave node as being removed from the cluster of nodes.

13. The method of claim 8 , wherein the cluster of nodes processes log files of a content provider stored in the cluster of nodes, and the file is at least part of the log files.

14. A non-transitory computer-readable medium containing instructions that, in response to being executed by one or more processors, cause a machine to perform operations, the operations comprising:

receiving, from a master node in a cluster of nodes, a command for a first slave node to process a file, the first slave node operating within a first operating system, the cluster of nodes configured to utilize a second operating system different from the first operating system, the cluster of nodes including the first slave node and a plurality of second slave nodes that utilize the second operating system;

in response to the command, initiating a specialized class within the first operating system to facilitate access to the cluster, the specialized class configured to, prior to attempting to access the file for processing as instructed by the master node, determine:

whether the first slave node is operating within the first operating system that is different from the second operating system of the cluster; and

whether the file is accessible via a local file system of the first operating system;

based on determining both (i) the first slave node is operating within the first operating system, and (ii) the file is inaccessible via the local file system of the first operating system, sending a request, by the specialized class, to a client-side runner module for the file, the client-side runner module configured to request the file from a shared storage server via a server-side runner module, the server-side runner module configured to communicate with the shared storage server using the second operating system; and

in response to receiving the file from the server-side runner module, store the file locally in the first slave node.

15. The computer-readable medium of claim 14 , wherein the second operating system natively supports Hadoop.

16. The computer-readable medium of claim 15 , wherein:

the cluster of nodes includes the master node that utilizes the second operating system, and is configured to distribute data among the first slave node and the plurality of second slave nodes; and

the operations further comprise:

receive a server key from the master node, and

store the server key at the first slave node such that the first slave node accesses the cluster of nodes without a passphrase being manually entered.

17. The computer-readable medium of claim 14 , wherein the operations further comprise:

send a message to add the first slave node to the cluster of nodes, the message being sent to the server-side runner module, and

update the shared storage server to indicate the first slave node as being added to the cluster of nodes.

18. The computer-readable medium of claim 14 , wherein the operations further comprise:

send a message to remove the first slave node from the cluster of nodes, the message being sent to the server-side runner module, and

update the shared storage server to indicate the first slave node as being removed from the cluster of nodes.

19. The computer-readable medium of claim 14 , wherein:

the first operating system executes an emulating program that emulates an environment of the second operating system; and

the first slave node is configured to make an operating system call that is natively supported by the second operating system based on the emulating program executed by the first operating system.

20. The computer-readable medium of claim 14 , wherein the cluster of nodes processes log files of a content provider stored in the cluster of nodes, and the file is at least part of the log files.