Platform for multi-stream sampling and visualization
A stream tool is disclosed that allows a user to seamlessly connect with the different data streams, regardless of the streams' transmission platforms or communication protocols, in order to visually see a representation of the type of data that the data streams are transmitting. A user may specify a particular data stream and provide corresponding connection details. A collection of abstracted software functions enable interaction with the different stream platforms and protocols. Using these abstracted functions, a stream-processing service accesses a requested data stream and samples its data events for either sample timeframe or up to a threshold number of data events. The sampled data events are parsed and visually presented to the user in an easy-to-understand format. The user may then inspect the data stream's data for use in developing robust applications that may integrate and use such data.
1 . A method comprising:
receiving a request for access to a first data stream, the request comprising connection information for a first stream producer supplying the first data stream using a first stream platform;
receiving a sample of the first data stream;
parsing the sample of the first data stream to identify a data type and a corresponding data value in the sample, wherein parsing the sample of the first data stream comprises identifying a message type associated with the sample by analyzing a schema of a message of the sample; and
displaying, on a user interface (UI), a stream preview UI portion showing visual representations of the identified data type and the corresponding data value in a visual format that differs from the sample of the first data stream.
2 . The method of claim 1 , wherein the connection information comprises at least one of a host name, a host identifier, a port identifier, or a uniform resource location (URL).
3 . The method of claim 1 , wherein the visual representations are generated by at least one of a cloud computing environment or a client device that includes the UI.
4 . The method of claim 1 , wherein parsing the sample of the first data stream comprises searching the message of the sample for at least one of the schema or a metadata.
5 . The method of claim 1 , wherein parsing the sample of the first data stream comprises
selecting at least one of the schema, a metadata, or a search function based on the identified message type.
6 . The method of claim 1 , further comprising storing a list of different parser rules for different parser types.
7 . The method of claim 1 , wherein the message type comprises at least one of a JSON, TXT, A VRO, protobuf, or parquet message.
8 . The method of claim 1 , wherein the visual format of the visual representations displayed on the stream preview UI portion are in an easier-to-read format as compared to the sample of the first data stream.
9 . The method of claim 1 , wherein displaying the stream preview UI portion showing the visual representations of the identified data type and the corresponding data value comprises showing character strings representing the identified data type and the corresponding data value.
10 . The method of claim 1 , further comprising displaying, on the UI, a backend UI portion showing computer-executable code of backend instructions performed to sample the first data stream, wherein the computer-executable code of the backend instructions displays the at least one of a threshold sample timeframe or a threshold sample count of the sample.
11 . The method of claim 1 , wherein the identified data type comprises a location.
12 . A system comprising:
memory embodied with instructions that cause a processor to:
receive a request for access to a first data stream, the request comprising connection information for a first stream producer supplying the first data stream using a first stream platform;
receive a sample of the first data stream;
parse the sample of the first data stream to identify a data type and a corresponding data value in the sample, wherein parsing the sample of the first data stream comprises identifying a message type associated with the sample by analyzing a schema of a message of the sample; and
display, on a user interface (UI), a stream preview UI portion showing visual representations of the identified data type and the corresponding data value in a visual format that differs from the sample of the first data stream.
13 . The system of claim 12 , wherein parsing the sample of the first data stream comprises searching the message of the sample for at least one of the schema or a metadata.
14 . The system of claim 12 , wherein parsing the sample of the first data stream comprises
selecting at least one of the schema, a metadata, or a search function based on the identified message type.
15 . The system of claim 12 , wherein the visual representations are generated by at least one of a cloud computing environment or a client device that includes the UI.
16 . The system of claim 12 , wherein the visual format of the visual representations displayed on the stream preview UI portion are in an easier-to-read format as compared to the sample of the first data stream.
17 . A method comprising:
providing a unified stream platform capable of interacting with different stream platforms using common abstracted functions that sample data streams at least one of upon receipt of a first data event, upon occurrence of a system error, or according to a sample timeframe;
receiving a request for access to a first data stream, the request comprising connection information for a first stream producer;
accessing the first data stream from the first stream producer using the common abstracted functions;
parsing a sample of the first data stream to identify a data type and a corresponding data value in the sample, wherein parsing the sample of the first data stream comprises identifying a message type associated with the sample by analyzing a schema of a message of the sample; and
displaying, on a user interface (UI), a stream preview UI portion showing visual representations of the identified data type and the corresponding data value in a visual format that differs from the sample of the first data stream.
18 . The method of claim 1 , wherein parsing the sample of the first data stream comprises selecting at least one of the schema or a metadata to search for, based on the identified message type.
19 . The system of claim 12 , wherein parsing the sample of the first data stream comprises selecting at least one of the schema or a metadata to search for based on the identified message type.
20 . The method of claim 17 , wherein parsing the sample of the first data stream comprises selecting at least one of the schema or a metadata to search for based on the identified message type.