Detection and ranking of entities from mobile onscreen content

1. A method comprising:

receiving an image captured from a mobile device display for a first mobile application;

determining a window that includes a chronological set of images, the images each representing a respective screen captured from a display of a mobile device and having an associated timestamp;

identifying entities appearing in images in a first portion of the window using text for images in a remaining portion of the window as context to disambiguate ambiguous entity references; and

providing the identified entities to the mobile device for customizing a second mobile application executing on the mobile device.

2. The method of claim 1 , wherein the first portion is a middle portion and the remaining portion includes a front portion of images having a timestamp prior to timestamps for images in the middle portion and a back portion of images having a timestamp after the timestamps for the images in the middle portion.

3. The method of claim 1 , wherein selecting the set of images includes:

determining that a next chronological image is associated with a mobile application that differs from a mobile application associated with a last image in the set; and

closing the set in response to the determining.

4. The method of claim 3 , wherein the first portion is a last portion and the remaining portion includes images having a timestamp prior to timestamps for images in the last portion.

5. The method of claim 1 , wherein selecting the set of images includes:

determining that a next chronological image is associated with a third mobile application that differs from the first mobile application, which is associated with a last image in the set;

determining that the first mobile application launched the third mobile application; and

including the next chronological image in the set.

6. The method of claim 1 , wherein the entities are entities in a data graph storing entities connected by labeled edges that represent relationships.

7. The method of claim 1 , further comprising:

calculating a rank for a first entity of the entities identified in the first portion based on an amount of time the first entity appears in the images.

8. The method of claim 7 , wherein calculating the rank includes:

determining that a location of the first entity remains constant; and

downgrading the rank in response to the determining.

9. The method of claim 1 , wherein the method further comprises:

determining a quantity of mobile applications associated with images having a first entity; and

boosting a rank of the first entity when the first entity is associated with more than one mobile application.

10. The method of claim 1 , wherein the window represents a fixed time period.

11. The method of claim 1 , wherein the window represents a quantity of images.

12. The method of claim 1 , wherein disambiguating ambiguous entity mentions includes:

determining a first candidate entity and a second candidate entity that each correspond to a same recognized item for the image;

determine that the first candidate entity shares a category with the first mobile application; and

select the first candidate entity over the second candidate entity based on sharing the category.

13. The method of claim 12 , wherein the second candidate entity has a higher prior probability than the first candidate entity.

14. The method of claim 1 , wherein each of the images has a mobile application associated with the image and the method further includes:

using the associated mobile applications to determine the window.

15. The method of claim 1 , wherein the first mobile application and the second mobile application are the same mobile application.

16. A mobile device comprising:

at least one processor; and

memory storing:

a plurality of images of screens captured on the mobile device, each of the plurality of images having an associated timestamp, and

instructions that, when executed by the at least one processor, cause the mobile device to:

receive an image captured from a mobile device display for a first mobile application,

determine a window that includes a chronological set of the plurality of images,

identify entities appearing in images in a first portion of the window using text for images in a remaining portion of the window as context to disambiguate ambiguous entity references, and

using the identified entities for customizing mobile applications executing on the mobile device.

17. The mobile device of claim 16 , wherein the first portion is a middle portion and the remaining portion includes a front portion of images having a timestamp prior to timestamps for images in the middle portion and a back portion of images having a timestamp after the timestamps for the images in the middle portion.

18. The mobile device of claim 16 , wherein each of the plurality of images has an associated mobile application and determining the set of images in the window includes:

using the associated mobile applications to determine the window.

19. The mobile device of claim 16 , wherein each of the plurality of images has an associated mobile application and determining the set of images in the window includes:

determining that a next chronological image is associated with a mobile application that differs from a mobile application associated with a last image in the set; and

closing the set in response to the determining.

20. The mobile device of claim 19 , wherein the first portion is a last portion in the set and the remaining portion includes images having a timestamp prior to timestamps for images in the last portion.

21. The mobile device of claim 16 , wherein each of the plurality of images has an associated mobile application and determining the set of images in the window includes:

determining that a next chronological image is associated with a third mobile application that differs from the first mobile application, which is associated with a last image in the set;

determining that the first mobile application launched the third mobile application; and

including the next chronological image in the set.

22. The mobile device of claim 16 , wherein the instructions further include instructions that, when executed by the at least one processor, cause the mobile device to:

determine, from the entities in the first portion, at least one entity that is an outlier; and

store, in a user profile in the memory, the entities identified in the first portion except for the at least one entity.

23. The mobile device of claim 16 , wherein the instructions further include instructions that, when executed by the at least one processor, cause the mobile device to:

calculate a rank for a first entity of the entities identified in the first portion based on an amount of time the first entity appears in the images.

24. The mobile device of claim 23 , wherein calculating the rank includes:

determining that a location of the first entity remains constant across images; and

downgrading the rank in response to the determining.

25. A computer system comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the system to:

receive, from a mobile device, an image of a screen generated by a first mobile application executing on the mobile device,

determine a window that includes a chronological set of images, the images each representing a respective screen captured from a screen generated by a mobile application executing on the mobile device, and each of the images and having an associated timestamp,

identify entities appearing in images in a first portion of the window using text recognized in images in a remaining portion of the window as context to disambiguate ambiguous entity references, and

provide the identified entities to the mobile device for customizing mobile applications executing on the device.

26. The system of claim 25 , wherein the memory further stores instructions that, when executed by the at least one processor, causes the system to:

determine that a first candidate entity and a second candidate entity each correspond to a same recognized item for a first image in the first portion of the window;

determine that the first candidate entity shares a category with a mobile application associated with the first image; and

select the first candidate entity over the second candidate entity based on sharing the category.

27. The system of claim 26 , wherein the second candidate entity has a higher prior probability than the first candidate entity.

28. The system of claim 25 , wherein the memory further stores instructions that, when executed by the at least one processor, causes the system to:

determine that a next chronological image is associated with a mobile application that differs from a mobile application associated with a last image in the set; and

close the set in response.

29. The system of claim 25 , wherein the memory further stores instructions that, when executed by the at least one processor, causes the system to:

determine that a next chronological image is associated with a second mobile application that differs from the first mobile application, which is associated with a last image in the set;

determine that the first mobile application launched the second mobile application; and

include the next chronological image in the set.

30. The system of claim 25 , wherein the instructions further include instructions that, when executed by the at least one processor, cause the system to:

calculate a rank for a first entity of the entities identified in the first portion based on an amount of time the first entity appears in the images; and

store the first entity with the rank in the memory.

31. The system of claim 30 , wherein calculating the rank includes:

determining that a location of the first entity remains constant across images; and

downgrading the rank in response to the determining.