Platform for multi-stream sampling and visualization
A stream tool is disclosed that allows a user to seamlessly connect with the different data streams, regardless of the streams' transmission platforms or communication protocols, in order to visually see a representation of the type of data that the data streams are transmitting. A user may specify a particular data stream and provide corresponding connection details. A collection of abstracted software functions enable interaction with the different stream platforms and protocols. Using these abstracted functions, a stream-processing service accesses a requested data stream and samples its data events for either sample timeframe or up to a threshold number of data events. The sampled data events are parsed and visually presented to the user in an easy-to-understand format. The user may then inspect the data stream's data for use in developing robust applications that may integrate and use such data.
1. A method for generating visual representations of data associated with a plurality of data streams that transmit data using different stream platforms, the method comprising:
providing a unified stream platform capable of interacting with different stream platforms using a common set of abstracted functions;
receiving a request to access a first data stream, the request comprising connection information for a first stream producer supplying the first data stream using a first stream platform or protocol;
accessing the first data stream from the first stream producer using the common set of abstracted functions;
sampling a subset of data events of the accessed first data stream for a threshold sample timeframe or a threshold sample count; and
generating a visual representation comprising a user interface (UI) showing both the sampled subset of data events of the accessed first data stream for presentation to a user and the backend instructions performed to sample the sampled subset of data events, wherein the backend instructions indicate one or more conditions specifying duration of sampling of the first data stream.
2. The method of claim 1 , further comprising identifying a message type associated with the subset of data events.
3. The method of claim 2 , wherein the message type comprises at least one of a JSON, TXT, AVRO, protobuf, or parquet message.
4. The method of claim 2 , further comprising:
parsing the sampled subset of data events of the accessed first data stream to identify data and data types in at least one of the subset of data events; and
presenting the data and the data types in the visual representation of the sampled subset of data events of the accessed first data stream.
5. The method of claim 1 , further comprising presenting, on a client device of the user, the visual representation of the one or more sampled data events.
6. The method of claim 1 , wherein said sampling of the subset of data events of the accessed first data stream is triggered to occur upon receipt of a first data event of the first data stream.
7. The method of claim 6 , wherein the common set of abstracted functions are mapped to the different stream providers through translating at least some of the common set of abstracted functions to a plurality of functions of the different stream providers.
8. The method of claim 1 , wherein the sample timeframe is about ten seconds.
9. The method of claim 8 , further comprising:
determining the subset of data events were not received during the sample timeframe; and
sampling for the subset of data events for a subsequent sample timeframe incident to the subset of data events not being received during the sample timeframe.
10. The method of claim 1 , wherein the connection information comprises a host name and a port identifier associated with the first stream provider.
11. The method of claim 1 , further comprising:
accessing a second stream producer that uses a different stream platform than the first stream producer using the common set of abstracted functions of the unified stream platform;
sampling a second set of data events of a second data stream from the second stream producer; and
generating a second visual representation of the second sampled subset of data events of the sampled second set of data events of the second data stream for presentation to the user.
12. The method of claim 1 , wherein the first stream comprises a stream of at least one of clickstreams, log data, telemetry data for devices, sensor data for devices, web data, social-media data, gesture data, customer data, gaming data, video data, audio data, image data, blockchain data, or financial transactions.
13. The method of claim 1 , wherein the first stream platform comprises at least one of Apache Kafka, Apache Pulsar, Azure® Event Hubs, Rabbit MQ, RabbitMQ, or Amazon® Kinesis.
14. A system for generating visual representations of data associated with a plurality of data streams that transmit data using different stream platforms, the system comprising:
memory embodied with instructions for accessing a plurality of different stream providers using a unified stream platform capable of interacting with different stream platforms using a common set of abstracted functions; and
one or more processors programmed to:
access a first data stream from a first stream producer using the common set of abstracted functions,
sample at least one data event of the accessed first data stream, and
parse the at least one data event to identify data types and corresponding data therein, and
generate a visual representation comprising a user interface (UI) of the at least one data event showing the data types and corresponding data identified through said parsing and the backend instructions performed to sample the sampled subset of data events, wherein the backend instructions indicate one or more conditions specifying duration of sampling of the first data stream.
15. The system of claim 14 , wherein the one or more processors are programmed to sample the at least one data event of the accessed first data stream for only a particular sample timeframe.
16. The system of claim 14 , wherein the one or more processors are programmed to sample the at least one data event of the accessed first data stream until a preset sample count of data events has been received.
17. The system of claim 14 , wherein the one or more processors are programmed to begin sampling the one or more data events upon receipt of an initial data event.
18. The system of claim 14 , wherein said parsing comprises identifying a type of message of the sampled at least one data event.
19. One or more computer storage memory embodied with computer-executable components configured for generating visual representations of data associated with a plurality of data streams that transmit data using different stream platforms, said components comprising:
an abstraction component executable to provide a unified stream platform capable of interacting with the different stream platforms using a common set of abstracted functions;
a sampling component to sample subsets of data events from the plurality of data streams that transmit data using different stream platforms;
a parser component executable to identify different types of data in the subsets of data events; and
a visual plug-in component executable to generate visual representations comprising a user interface (UI) of the plurality of data streams of the sampled subsets of data events using the identified different types of data and the backend instructions performed to sample the sampled subset of data events, wherein the backend instructions indicate one or more conditions specifying duration of sampling of the first data stream.
20. The computer storage media of claim 17 , wherein the sampling component is executable to only sample data for one or more sample timeframes or one or more sample counts.