System and method for generating data sets for learning to identify user actions
Embodiments for generating appropriate data sets for learning to identify user actions. A user uses one or more applications over a suitable period of time. As the user uses the applications, a monitoring device, acting as a “man-in-the-middle,” intermediates the exchange of encrypted communication between the applications and the servers that serve the applications. The monitoring device obtains, for each action performed by the user, two corresponding (bidirectional) flows of communication: an encrypted flow, and an unencrypted flow. Since the unencrypted flow indicates the type of action that was performed by the user, the correspondence between the encrypted flow and the unencrypted flow may be used to automatically label the encrypted flow, without decrypting the encrypted flow. Features of the encrypted communication may then be stored in association with the label to automatically generate appropriately-sized learning set for each application of interest.
1. A system, for use with a monitoring device intermediating an exchange, between an application and a server, of encrypted communication that was generated in response to one or more actions performed by a user of the application, the system comprising:
a network interface; and
at least one processor, configured:
to receive from the monitoring device, via the network interface, the encrypted communication, and corresponding unencrypted communication that was derived, by the monitoring device, from the encrypted communication,
to ascertain a correspondence between the unencrypted communication and the encrypted communication based on respective types of the one or more actions, and
to extract features from encrypted communication and labels from the unencrypted communication having a correspondence and storing the features of the encrypted communication in association with the labels.
2. The system according to claim 1 ,
wherein the encrypted communication includes a plurality of encrypted blocks, each of the encrypted blocks including at least a first group of packets sent from the application to the server, and a second group of packets sent from the server to the application in response to the first group of packets,
wherein the unencrypted communication includes a plurality of unencrypted blocks, each of the unencrypted blocks including at least a first group of packets sent from the application to the server, and a second group of packets sent from the server to the application in response to the first group of packets, and
wherein the processor is configured to ascertain the respective types of the actions by:
ascertaining a correspondence between the encrypted blocks and the unencrypted blocks, and
based on the correspondence, ascertaining, for each encrypted block of the encrypted blocks, the type of the action in response to which the encrypted block was generated.
3. The system according to claim 2 , wherein the processor is configured to apply the labels by storing features of each encrypted block of the encrypted blocks in association with a respective one of the labels that indicates the type of the action in response to which the encrypted block was generated.
4. The system according to claim 2 , wherein the processor is further configured to separate the encrypted communication into the encrypted blocks, based on respective durations of time between successive pairs of the encrypted blocks being greater than a threshold.
5. The system according to claim 2 , wherein the processor is configured to ascertain that any particular one of the encrypted blocks corresponds to a particular one of the unencrypted blocks, based on a duration of time between the particular one of the encrypted blocks and the particular one of the unencrypted blocks being less than a threshold.
6. The system according to claim 2 , wherein the processor is configured to ascertain the correspondence between the encrypted blocks and the unencrypted blocks by comparing respective durations of the encrypted blocks to respective durations of the unencrypted blocks.
7. The system according to claim 2 , wherein the processor is configured to ascertain the correspondence between the encrypted blocks and the unencrypted blocks by comparing respective amounts of data in the encrypted blocks to respective amounts of data in the unencrypted blocks.
8. The system according to claim 1 , wherein the processor is further configured to learn, using the stored features and labels, to ascertain, for subsequent encrypted communication exchanged between the application and the server, respective types of actions in response to which the subsequent encrypted communication was generated.
9. The system according to claim 1 , wherein the processor is further configured:
to form a plurality of clusters, each of the clusters including (i) at least one representative block of encrypted communication received from the monitoring device, and (ii) other encrypted communication, and
to apply, to all of the encrypted communication in each cluster of the clusters, a label, of the labels, applied to the representative block for the cluster.
10. The system according to claim 9 , wherein the processor is configured to form the plurality of clusters by clustering blocks of the other encrypted communication based on respective communication patterns exhibited by the blocks.
11. A method for intermediating an exchange between an application and a server using a monitoring device, the exchange including encrypted communication that was generated in response to one or more actions performed by a user of the application, the method comprising:
receiving, from the monitoring device, the encrypted communication, and corresponding unencrypted communication that was derived, by the monitoring device, from the encrypted communication;
using a processor to automatically ascertain a correspondence between the unencrypted communication and the encrypted communication based on respective types of the one or more actions, and
to extract features from encrypted communication and labels from the unencrypted communication having a correspondence and storing the features of the encrypted communication in association with the labels.
12. The method according to claim 11 ,
wherein the encrypted communication includes a plurality of encrypted blocks, each of the encrypted blocks including at least a first group of packets sent from the application to the server, and a second group of packets sent from the server to the application in response to the first group of packets,
wherein the unencrypted communication includes a plurality of unencrypted blocks, each of the unencrypted blocks including at least a first group of packets sent from the application to the server, and a second group of packets sent from the server to the application in response to the first group of packets, and
wherein ascertaining the respective types of the actions comprises:
ascertaining a correspondence between the encrypted blocks and the unencrypted blocks, and
based on the correspondence, ascertaining, for each encrypted block of the encrypted blocks, the type of the action in response to which the encrypted block was generated.
13. The method according to claim 12 , wherein applying the labels comprises applying the labels by storing features of each encrypted block of the encrypted blocks in association with a respective one of the labels that indicates the type of the action in response to which the encrypted block was generated.
14. The method according to claim 12 , further comprising separating the encrypted communication into the encrypted blocks, based on respective durations of time between successive pairs of the encrypted blocks being greater than a threshold.
15. The method according to claim 14 , wherein ascertaining the correspondence between the encrypted blocks and the unencrypted blocks comprises ascertaining that any particular one of the encrypted blocks corresponds to a particular one of the unencrypted blocks, based on a duration of time between the particular one of the encrypted blocks and the particular one of the unencrypted blocks being less than a threshold.
16. The method according to claim 14 , wherein ascertaining the correspondence between the encrypted blocks and the unencrypted blocks comprises ascertaining the correspondence by comparing respective durations of the encrypted blocks to respective durations of the unencrypted blocks.
17. The method according to claim 14 , wherein ascertaining the correspondence between the encrypted blocks and the unencrypted blocks comprises ascertaining the correspondence by comparing respective amounts of data in the encrypted blocks to respective amounts of data in the unencrypted blocks.
18. The method according to claim 11 , further comprising, using the stored features and labels, automatically learning to ascertain, for subsequent encrypted communication exchanged between the application and the server, respective types of actions in response to which the subsequent encrypted communication was generated.
19. The method according to claim 11 , further comprising:
automatically forming a plurality of clusters, each of the clusters including (i) at least one representative block of encrypted communication received from the monitoring device, and (ii) other encrypted communication, and
applying, to all of the encrypted communication in each cluster of the clusters, a label, of the labels, applied to the representative block for the cluster.
20. The method according to claim 19 , wherein automatically forming the plurality of clusters comprises automatically forming the plurality of clusters by clustering blocks of the other encrypted communication based on communication patterns exhibited by the blocks.