Machine learned inference of protocols from banner data
Systems and methods are disclosed to infer, using a machine learned model, a service protocol of a server based on the banner data produced by the server. In embodiments, the machine learned model is implemented by a network scanner configured to receive banner data from open ports on servers. A received banner is parsed into a set of features, such as the counts or presence of particular characters or strings in the banner. In embodiments, certain types of banner content such as network addresses, hostnames, dates, and times, are replaced with special characters. The machine learned model is applied to the features to infer a most likely protocol of the server port that produced the banner. Advantageously, the model can be trained to perform the inference task with high accuracy and without using human-specified rules, which can be brittle for unconventional banner data and carry undesired biases.
1. A system, comprising:
one or more hardware processors with associated memory that implement a protocol inference system, configured to:
obtain banner data generated by a server, wherein the server is configured to provide one or more services to clients via one or more respective service protocols, and the banner data is generated as an initial communication to a client;
parse the banner data to obtain a set of features of the banner data;
apply a machine learned model to the set of features, wherein the machine learned model is trained using a machine learning technique to infer, based at least in part on the set of features, a service protocol supported by the server; and
provide output of the machine learned model, wherein the output indicates the service protocol inferred.
2. The system of claim 1 , wherein protocol inference system is implemented as part of a network scanning system configured to send an initial request to the server as the client, and the banner data is generated in response to the initial request.
3. The system of claim 2 , wherein the network scanning system is configured to:
determine, based at least in part on the output of the machine learned model, that a port on the server that outputted the banner data is configured to support the service protocol; and
cause the client to send, to the port, one or more subsequent service requests to the port according to the service protocol.
4. The system of claim 1 , wherein the protocol inference system is implemented as part of a network scanning system configured to monitor network traffic to or from servers in a network, and the generation of the banner data is not caused by the network scanning system.
5. The system of claim 4 , wherein the network scanning system is configured to:
determine, based at least in part on the output of the machine learned model, that a port on the server that outputted the banner data is configured to support the service protocol; and
store metadata that associates observed network traffic to or from the port with the service protocol.
6. The system of claim 1 , wherein the protocol inference system is configured to provide the output of the machine learned model is provided via a graphical user interface (GUI) to cause the GUI to indicate the service protocol as a machine-inferred protocol of the server.
7. The system of claim 1 , wherein the protocol inference system is configured to:
determine that a port on the server that produced the banner data has a port number that is different from a standard port number associated with the service protocol; and
generate an alert indicating that the service protocol is detected on a non-standard port number.
8. A method comprising:
performing, by one or more hardware processors with associated memory that implement a protocol inference system:
obtaining banner data generated by a server, wherein the server is configured to provide one or more services to clients via one or more respective service protocols, and the banner data is generated as an initial communication to a client;
parsing the banner data to obtain a set of features of the banner data;
applying a machine learned model to the set of features, wherein the machine learned model is trained using a machine learning technique to infer, based at least in part on the set of features, a service protocol supported by the server; and
providing output of the machine learned model, wherein the output indicates the service protocol inferred.
9. The method of claim 8 , wherein the inferring of the service protocol comprises selecting one of a plurality of different protocols, including two or more of:
Post Office Protocol version 3 (POP3),
File Transfer Protocol (FTP),
Telnet protocol,
Secure Shell protocol (SSH),
Internet Message Access Protocol (IMAP),
Network News Transfer Protocol (NNTP),
Simple Mail Transfer Protocol (SMTP),
Hypertext Transfer Protocol (HTTP), and
Hypertext Transfer Protocol Secure (HTTPS).
10. The method of claim 9 , wherein the providing of the output of the machine learned model comprises providing an output vector indicating respectively probabilities of individual ones the plurality of different protocols.
11. The method of claim 8 , further comprising performing, by the protocol inference system:
determining, using the machine learned model and for another set of features obtained from another banner data, that no service protocol can be inferred from the other banner data with a sufficient level of confidence; and
providing an output of the machine learned model indicating an indeterminate result for the other banner data.
12. The method of claim 8 , wherein:
the parsing of the banner data comprises parsing a text sequence into individual text characters; and
the obtaining of the set of features comprises determining one or more of:
a count of a particular character in the text sequence,
an indication of presence or absence of a particular character in the text sequence,
a count of alphabetical characters in the text sequence, and
a count of numerical characters in the text sequence.
13. The method of claim 12 , wherein the obtaining of the set of features comprises determining one or more subsequences of characters in text sequence that matches a name associated with a particular service protocol.
14. The method of claim 12 , wherein:
the parsing of the banner data comprises determining and replacing one or more types of content in the banner data with one or more respective special characters or tokens, the one or more types of content indicating one or more of:
a network address,
a hostname,
a date or time, and
a time zone; and
the obtaining of the set of features comprises determining a count or an indication of presence or absence of individual ones of the one or more special characters or tokens in the banner data.
15. One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors implement at least a portion of a network scanning system, wherein the program instructions when executed on or across the one or more processors cause the network scanning system to:
obtain banner data generated by a server, wherein the server is configured to provide one or more services to clients via one or more respective service protocols, and the banner data is generated as an initial communication to a client;
parse the banner data to obtain a set of features of the banner data;
apply a machine learned model to the set of features, wherein the machine learned model is trained using a machine learning technique to infer, based at least in part on the set of features, a service protocol supported by the server; and
provide output of the machine learned model, wherein the output indicates the service protocol inferred.
16. The one or more non-transitory computer-accessible storage media of claim 15 , wherein the program instructions when executed on or across the one or more processors cause the network scanning system to:
send a connection request to a Transmission Control Protocol (TCP) port on the server; and
determine, based at least in part on the output of the machine learned model, that the UDP port is configured to support the inferred service protocol.
17. The one or more non-transitory computer-accessible storage media of claim 15 , wherein the program instructions when executed on or across the one or more processors cause the network scanning system to:
send a packet to a User Datagram Protocol (UDP) port on the server; and
determine, based at least in part on the output of the machine learned model, that the UDP port is configured to support the inferred service protocol.
18. The one or more non-transitory computer-accessible storage media of claim 15 , wherein the program instructions when executed on or across the one or more processors cause the network scanning system to implement a random forest model as the machine learned model.
19. The one or more non-transitory computer-accessible storage media of claim 15 , wherein the program instructions when executed on or across the one or more processors cause the network scanning system to:
determine that a particular service protocol inferred from a particular banner data is incorrect; and
store the particular banner data in a banner repository, wherein banner data in the banner repository are used to compile training datasets for further training of the machine learned model.
20. The one or more non-transitory computer-accessible storage media of claim 19 , wherein to parse the banner data to obtain the set of features, the program instructions when executed on or across the one or more processors cause the network scanning system to:
replace one or more types of content in the banner data with one or more respective special tokens, the one or more types of content indicating one or more of:
a network address,
a hostname,
a date or time, and
a time zone; and
determine, as one or more features, a count or a presence or absence of individual ones of the one or more special tokens in the banner data.