IP Library Granted Patent US 11,995,299
Granted Patent B2
US 11,995,299 · App. 18/188,369 · Granted May 28, 2024

Restoring full online documents from scanned paper fragments

Inventors: Alexander Pashintsev (Cupertino, CA); Eugene Livshitz (San Mateo, CA); Vitaly Glazkov (Moscow, RU); P. J. Tanzillo (Austin, TX)
Assignee: Bending Spoons S.p.A.
G06F3/04842G06F3/017G06F40/166H04N23/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,995,299
App. No.
18/188,369
Granted
May 28, 2024
Kind
B2
Abstract

Searching for documents includes retrieving objects from a physical media image using a camera from a smartphone, a user selecting a subset of the objects, forming a search query based on the subset of objects, and applying the search query to a search engine to search for the documents. Retrieving objects from a media image may include waiting for a view of the camera to stabilize. Waiting for the view of the camera to stabilize may include detecting changing content of a video flow provided to the camera and/or using motion sensors of the camera to detect movement. Retrieving objects may include the smartphone identifying possible subsets of objects in the media image. The user selecting a subset of the objects may include the smartphone presenting at least some of the possible subsets to the user and the user selecting one of the possible subsets.

Claims (63)

1. A method comprising:

at a computing device including one or more processors:

receiving, by a camera of a device, a video stream of media content;

in accordance with a determination that motion of the device is below a predetermined threshold, identifying, within one or more frames of the video stream, one or more preferred objects within the media content, wherein the one or more preferred objects are a subset of identified objects within the video stream based on at least a scene layout;

detecting a selection of a preferred object of the one or more preferred objects;

performing a search in a collection of digital documents based on content of the preferred object; and

in accordance with a determination that one or more search results include at least a portion of the content of the preferred object, displaying the one or more search results.

2. The method of claim 1 , comprising:

while displaying the one or more search results:

detecting input to select a respective search result of the one or more search results; and

storing the preferred object with information that corresponds to the respective search result.

3. The method of claim 2 , wherein the one or more search results include source documents that include at least a portion of the content of the preferred object.

4. The method of claim 1 , wherein the preferred object is a subset of preferred objects and identifying, within the one or more frames of the video stream, the one or more preferred objects within the media content further includes:

simultaneously displaying, by a display of the device, a plurality of subsets of preferred objects, wherein:

each subset includes at least one preferred object of the one or more preferred objects, and

each subset is distinct.

5. The method of claim 4 , wherein:

displaying the plurality of subsets of preferred objects includes displaying a plurality of pictograms;

a respective pictogram of the plurality of pictograms includes a representation of a subset of preferred objects; and

at least one preferred object of the subset of preferred objects is emphasized.

6. The method of claim 4 , further comprising;

while displaying the subset of preferred objects, determining whether the camera of the device is moved to target a portion of the subset of preferred objects or another preferred object of one or more preferred objects; and

identifying, within one or more frames of the video stream, another subset of preferred objects within the media content, the other subset of preferred objects based on the targeted portion of the subset of the plurality of subsets of objects or the other preferred object of one or more preferred objects.

7. The method of claim 1 , wherein the one or more preferred objects are further based on one or more of user preferences and heuristics.

8. The method of claim 1 , wherein performing the search based on the content of the preferred object includes:

determining, for the preferred object, one or more fragments, wherein each fragment of the one or more fragments is a subquery; and

performing the search based on one or more subqueries.

9. The method of claim 8 , wherein determining, for the preferred object, one or more fragments includes:

identifying text within the preferred object; and

removing one or more stopwords from the text within the preferred object, wherein text between the removed stopwords is a fragment.

10. The method of claim 8 , further comprising:

combining subqueries of the one or more fragments generate a search query; and

performing the search based on the search query.

11. The method of claim 1 , wherein the one or more preferred objects include at least one of a column, a header, a text block, a drawing, a chart, a building, or a person.

12. The method of claim 1 , wherein identifying the one or more preferred objects within the media content is delayed for a period of time that corresponds to a camera stabilization operation.

13. The method of claim 12 , wherein performing the camera stabilization operation includes at least one of: detecting changing content of video captured by the camera and using one or more motion sensors of the camera to detect movement.

14. The method of claim 1 , wherein detecting the selection of the preferred object includes detecting, by the device, a user input gesture that corresponds to the preferred object during presentation thereof.

15. The method of claim 1 , wherein performing the search produces a plurality of search results that are prioritized based on at least one of: publication time, author credentials, or distribution chain.

16. The method of claim 1 , wherein the one or more search results include a full online version of one or more source documents restored using the selected preferred object.

17. A non-transitory computer readable medium storing one or more programs, the one or more programs comprising instructions, which when executed by a device with a camera, cause the device to:

receive, by a camera of a device, a video stream of media content;

in accordance with a determination that motion of the device is below a predetermined threshold, identify, within one or more frames of the video stream, one or more preferred objects within the media content, wherein the one or more preferred objects are a subset of identified objects within the video stream based on at least a scene layout;

detect a selection of a preferred object of the one or more preferred objects;

perform a search in a collection of digital documents based on content of the preferred object; and

in accordance with a determination that one or more search results include at least a portion of the content of the preferred object, display the one or more search results.

18. The non-transitory computer readable medium of claim 17 , wherein the instructions, when executed by the device with the camera, further cause the device to:

while displaying the one or more search results:

detect input to select a respective search result of the one or more search results; and

store the preferred object with information that corresponds to the respective search result.

19. The non-transitory computer readable medium of claim 17 , wherein the preferred object is a subset of preferred objects and identifying, within the one or more frames of the video stream, the one or more preferred objects within the media content further includes:

simultaneously displaying, by a display of the device, a plurality of subsets of preferred objects, wherein:

each subset includes at least one preferred object of the one or more preferred objects, and

each subset is distinct.

20. A device comprising:

a display;

a camera;

one or more processors; and

memory storing one or more instructions that, when executed by the one or more processors, cause the device to:

receive, by a camera of a device, a video stream of media content;

in accordance with a determination that motion of the device is below a predetermined threshold, identify, within one or more frames of the video stream, one or more preferred objects within the media content, wherein the one or more preferred objects are a subset of identified objects within the video stream based on at least a scene layout;

detect a selection of a preferred object of the one or more preferred objects;

perform a search in a collection of digital documents based on content of the preferred object; and

in accordance with a determination that one or more search results include at least a portion of the content of the preferred object, display the one or more search results.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 12, 2024
From: EVERNOTE CORPORATION
To: BENDING SPOONS S.P.A.
Reel/Frame 066288/0195 →
Continuity (5)
Continuation 17714075 · Apr 5, 2022
Continuation 16988530 · Aug 7, 2020
Continuation 15241510 · Aug 19, 2016
Provisional Application 62209046 · Aug 24, 2015
Related Publication 20230229292A1 · Jul 20, 2023