System and method for identifying external connections in a streaming application
A method, computer program product, and computing device for monitoring network activity associated with streaming a data load through a stream application including a plurality of stream operators deployed on a plurality of computing devices. One or more stream operators with one or more external connections may be identified from the plurality of stream operators. The identified one or more stream operators may be deployed based upon, at least in part, the one or more external connections.
1. A computer-implemented method using a stream computing management application, the method comprising:
identifying one or more stream operators with one or more external connections from a plurality of stream operators associated with a streaming data load, the identified one or more stream operators having operator code not directly controlled by a stream computing management application; and
deploying the identified one or more stream operators based upon, at least in part, the one or more external connections when compiling a stream application, deploying the identified one or more stream operators including wrapping at least one of the identified one or more stream operators with exception handling, the exception handling allowing the identified one or more stream operators to be unfused from the plurality of stream operators associated with the streaming data load without a risk of crashing in case of an exception.
2. The computer-implemented method of claim 1 , further comprising:
monitoring, at a computing device, network activity associated with the streaming data load through the stream application including the plurality of stream operators deployed on a plurality of computing devices.
3. The computer-implemented method of claim 2 , wherein monitoring the network activity associated with the streaming data load includes:
profiling at least one stream operator of the plurality of stream operators with expected connection information during development before compilation of the stream application; and
associating the expected connection information with the at least one stream operator.
4. The computer-implemented method of claim 2 , wherein identifying the one or more stream operators includes:
identifying one or more internal connections between the plurality of stream operators from the monitored network activity associated with the streaming data load; and
filtering the one or more stream operators with external connections from the plurality of stream operators based upon, at least in part, identifying the one or more internal connections between the plurality of stream operators.
5. The computer-implemented method of claim 1 , wherein deploying the identified one or more stream operators with the one or more external connections includes:
deploying at least one of the identified one or more stream operators on one or more computing devices designated for external connections.
6. The computer-implemented method of claim 1 , wherein deploying the identified one or more stream operators with the one or more external connections includes:
fusing at least one of the identified one or more stream operators to one or more other stream operators of the plurality of stream operators, the fusing performed based upon an external connection identification process.
7. The computer-implemented method of claim 1 , wherein deploying the identified one or more stream operators with the one or more external connections includes:
deploying at least one of the identified one or more stream operators in a consistent region, the consistent region guaranteed to process each tuple of a data stream through the consistent region at least one time.
8. The computer-implemented method of claim 1 , further comprising:
displaying external connection information associated with the one or more external connections from the plurality of stream operators in a user interface.
9. The computer-implemented method of claim 8 , wherein deploying the identified one or more stream operators includes deploying the identified one or more stream operators via the user interface, the user interface configured to allow a user to configure computing devices for external connections.
10. A computer program product comprising a non-transitory computer readable storage medium having a plurality of instructions stored thereon, which, when executed by a processor, cause the processor to perform operations comprising:
identifying one or more stream operators with one or more external connections from a plurality of stream operators associated with a streamed data load, the identified one or more stream operators having operator code not directly controlled by a stream computing management application; and
deploying the identified one or more stream operators based upon, at least in part, the one or more external connections when compiling a stream application, deploying the identified one or more stream operators including wrapping at least one of the identified one or more stream operators with exception handling, the exception handling allowing the identified one or more stream operators to be unfused from the plurality of stream operators associated with the streaming data load without a risk of crashing in case of an exception.
11. The computer program product of claim 10 , further comprising:
monitoring, at a computing device, network activity associated with the streaming data load through the stream application including the plurality of stream operators deployed on a plurality of computing devices.
12. The computer program product of claim 11 , wherein monitoring the network activity associated with the streaming data load includes:
profiling at least one stream operator of the plurality of stream operators with expected connection information during development before compilation of the stream application; and
associating the expected connection information with the at least one stream operator.
13. The computer program product of claim 11 , wherein identifying the one or more stream operators includes:
identifying one or more internal connections between the plurality of stream operators from the monitored network activity associated with the streaming data load; and
filtering the one or more stream operators with external connections from the plurality of stream operators based upon, at least in part, identifying the one or more internal connections between the plurality of stream operators.
14. The computer program product of claim 10 , wherein deploying the identified one or more stream operators with the one or more external connections includes:
deploying at least one of the identified one or more stream operators on one or more computing devices designated for external connections.
15. The computer program product of claim 10 , wherein deploying the identified one or more stream operators with the one or more external connections includes:
fusing at least one of the identified one or more stream operators to one or more other stream operators of the plurality of stream operators, the fusing performed based upon an external connection identification process.
16. The computer program product of claim 10 , wherein deploying the identified one or more stream operators with the one or more external connections includes:
deploying at least one of the identified one or more stream operators in a consistent region, the consistent region guaranteed to process each tuple of a data stream through the consistent region at least one time.
17. The computer program product of claim 10 , further comprising:
displaying external connection information associated with the one or more external connections from the plurality of stream operators in a user interface.
18. A computing system including one or more processors and one or more memories configured to perform operations comprising:
identifying one or more stream operators with one or more external connections from a plurality of stream operators associated with a streaming data load, the identified one or more stream operators having operator code not directly controlled by a stream computing management application; and
deploying the identified one or more stream operators based upon, at least in part, the one or more external connections when compiling a stream application, deploying the identified one or more stream operators including wrapping at least one of the identified one or more stream operators with exception handling, the exception handling allowing the identified one or more stream operators to be unfused from the plurality of stream operators associated with the streaming data load without risk of crashing in case of an exception.