Systems and methods for identifying a source of a user interface from a fingerprint of the user interface
Systems and methods are provided herein for determining aspects of a user interface, and determining therefrom a service or device that corresponds to the user interface. In doing so, a user's activity can be profiled, and aggregate usage data over a population of particular devices and OTT applications can be tracked. Moreover, losses and gains of popularity of services or devices may be monitored.
1. A method for resolving an unknown user interface based on a relationship to known user interfaces, the method comprising:
detecting a display of an unknown user interface;
in response to detecting the display of the unknown user interface, determining, by fingerprinting the unknown user interface, a first characteristic of the unknown user interface and a second characteristic of the unknown user interface;
comparing each of the first characteristic and the second characteristic to characteristics indicated by entries of a database, the entries each indicating characteristics of a respective known user interface of a plurality of known user interfaces;
determining, from the comparing, that the first characteristic partially matches characteristics of a first respective known user interface of the plurality of known user interfaces, and that the second characteristic partially matches characteristics of a second respective known user interface of the plurality of known user interfaces;
determining whether the first respective known user interface and the second respective known user interface correspond to a same application; and
in response to determining that the first respective known user interface and the second respective known user interface correspond to the same application, determining that the unknown user interface is a user interface for the same application.
2. The method of claim 1 , wherein fingerprinting the unknown user interface comprises:
identifying elements of an image that include the unknown user interface;
identifying elements of the image that do not include the unknown user interface; and
isolating the elements of the image that include the unknown user interface from the elements of the image that do not include the unknown user interface.
3. The method of claim 1 , wherein determining whether the first respective known user interface and the second respective known user interface correspond to the same application comprises:
accessing a knowledge graph that describes relationships between the entries;
identifying, from the described relationships, a relationship between the first respective known user interface and the second respective known user interface, wherein the relationship comprises a service common to the first respective known user interface and the second respective known user interface; and
based on identifying the relationship, determining that the first respective known user interface and the second respective known user interface correspond to the same application.
4. The method of claim 3 , wherein the knowledge graph describes the described relationships by indicating discrete components of user interfaces that substantially match discrete components of other user interfaces.
5. The method of claim 1 , wherein determining that the first characteristic partially matches characteristics of the first respective known user interface, and that the second characteristic partially matches characteristics of the second respective known user interface, comprises:
identifying a first portion of the characteristics of the first respective known user interface;
identifying a second portion of the characteristics of the second respective known user interface;
determining that the first portion matches the first characteristic; and
determining that the second portion matches the second characteristic.
6. The method of claim 1 , further comprising generating an entry in the database for the unknown user interface, as a third respective known user interface of the plurality of known user interfaces corresponding to the same application.
7. The method of claim 6 , further comprising:
determining, by fingerprinting a second unknown user interface, a third characteristic of the second unknown user interface and a fourth characteristic of the second unknown user interface;
comparing each of the third characteristic and the fourth characteristic to characteristics indicated by the entries of the database; and
determining that the second unknown user interface is the third respective known user interface based on the comparing.
8. The method of claim 1 , wherein comparing each of the first characteristic and the second characteristic to the characteristics indicated by the entries of the database, further comprises:
determining whether the characteristics of the unknown user interface fully match the characteristics of the first respective known user interface; and
in response to determining that the characteristics of the unknown user interface fully match the characteristics of the first respective known user interface, determining that the unknown user interface corresponds to the first respective known user interface.
9. The method of claim 1 , wherein comparing each of the first characteristic and the second characteristic to the characteristics indicated by the entries of the database, further comprises:
determining that the first characteristic of the unknown user interface represents a first discrete component and the second characteristic of the unknown user interface represents a second discrete component; and
determining whether the characteristics of an entry of the entries include the first discrete component or the second discrete component.
10. The method of claim 1 , wherein fingerprinting the unknown user interface comprises using at least one of object recognition, image recognition, or characteristic recognition to determine the first characteristic of the unknown user interface and the second characteristic of the unknown user interface.
11. A system for resolving an unknown user interface based on a relationship to known user interfaces, the system comprising:
control circuitry configured to:
detect a display of an unknown user interface;
in response to detecting the display of the unknown user interface, determine, by fingerprinting the unknown user interface, a first characteristic of the unknown user interface and a second characteristic of the unknown user interface;
compare each of the first characteristic and the second characteristic to characteristics indicated by entries of a database, the entries each indicating characteristics of a respective known user interface of a plurality of known user interfaces;
determine, from the comparing, that the first characteristic partially matches characteristics of a first respective known user interface of the plurality of known user interfaces, and that the second characteristic partially matches characteristics of a second respective known user interface of the plurality of known user interfaces;
determine whether the first respective known user interface and the second respective known user interface correspond to a same application; and
in response to determining that the first respective known user interface and the second respective known user interface correspond to the same application, determine that the unknown user interface is a user interface for the same application.
12. The system of claim 11 , wherein the control circuitry is further configured, when fingerprinting the unknown user interface, to:
identify elements of an image that include the unknown user interface;
identify elements of the image that do not include the unknown user interface; and
isolate the elements of the image that include the unknown user interface from the elements of the image that do not include the unknown user interface.
13. The system of claim 11 , wherein the control circuitry is further configured, when determining whether the first respective known user interface and the second respective known user interface correspond to the same application, to:
access a knowledge graph that describes relationships between the entries;
identify, from the described relationships, a relationship between the first respective known user interface and the second respective known user interface, wherein the relationship comprises a service common to the first respective known user interface and the second respective known user interface; and
based on identifying the relationship, determine that the first respective known user interface and the second respective known user interface correspond to the same application.
14. The system of claim 13 , wherein the knowledge graph describes the described relationships by indicating discrete components of user interfaces that substantially match discrete components of other user interfaces.
15. The system of claim 11 , wherein the control circuitry is further configured, when determining that the first characteristic partially matches characteristics of the first respective known user interface, and that the second characteristic partially matches characteristics of the second respective known user interface, to:
identify a first portion of the characteristics of the first respective known user interface;
identify a second portion of the characteristics of the second respective known user interface;
determine that the first portion matches the first characteristic; and
determine that the second portion matches the second characteristic.
16. The system of claim 11 , wherein the control circuitry is further configured to generate an entry in the database for the unknown user interface, as a third respective known user interface of the plurality of known user interfaces corresponding to the same application.
17. The system of claim 16 , wherein the control circuitry is further configured to:
determine, by fingerprinting a second unknown user interface, a third characteristic of the second unknown user interface and a fourth characteristic of the second unknown user interface;
compare each of the third characteristic and the fourth characteristic to characteristics indicated by the entries of the database; and
determine that the second unknown user interface is the third respective known user interface based on the comparing.
18. The system of claim 11 , wherein the control circuitry is further configured, when comparing each of the first characteristic and the second characteristic to the characteristics indicated by the entries of the database, to:
determine whether the characteristics of the unknown user interface fully match the characteristics of the first respective known user interface; and
in response to determining that the characteristics of the unknown user interface fully match the characteristics of the first respective known user interface, determine that the unknown user interface corresponds to the first respective known user interface.
19. The system of claim 11 , wherein the control circuitry is further configured, when comparing each of the first characteristic and the second characteristic to the characteristics indicated by the entries of the database, to:
determine that the first characteristic of the unknown user interface represents a first discrete component and the second characteristic of the unknown user interface represents a second discrete component; and
determine whether the characteristics of an entry of the entries include the first discrete component or the second discrete component.
20. The system of claim 11 , wherein the control circuitry is further configured, when fingerprinting the unknown user interface, to use at least one of object recognition, image recognition, or characteristic recognition to determine the first characteristic of the unknown user interface and the second characteristic of the unknown user interface.