IP Library › Granted Patent US 10,636,074
Granted Patent B1
US 10,636,074 · App. 14/858,972 · Granted Apr 28, 2020

Determining and executing application functionality based on text analysis

Inventors: Peter Van Tuyl Bentley (Seattle, WA); Angela Lam (Seattle, WA); Michael Douglas McQueen (San Francisco, CA); Robert Steven Murdock (Fairfax, CA)
Assignee: Amazon Technologies, Inc.
G06Q30/0633G06F3/041G06F3/04883G06K9/00422
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,636,074
App. No.
14/858,972
Granted
Apr 28, 2020
Kind
B1
Abstract

Systems, methods, and computer-readable media are disclosed for detecting handwritten text on a writing surface, determining one or more layout and/or content attributes of the handwritten text, determining application functionality that corresponds to the layout and/or content attribute(s), executing the application functionality to obtain an execution result, and presenting an indication of the execution result on the writing surface in association with at least a portion of the handwritten text.

Claims (81)

1. A method, comprising:

receiving a first indication of handwritten text detected on a writing surface associated with a user device;

determining that the handwritten text includes two or more indicators comprising at least one of a content type indicator associated with a type of content included in the handwritten text;

determining, based at least in part on a first indicator and a second indicator of the two or more indicators, a first application functionality, wherein determining the first application functionality comprises:

determining, based at least in part on the first indicator, two or more application functionalities; and

determining, based at least in part on the second indicator, the first application functionality of the two or more application functionalities; and

executing the first application functionality.

2. The method of claim 1 , wherein determining that the handwritten text includes the two or more indicators further comprises determining that the handwritten text includes a content layout indicator associated with a layout of the handwritten text, the content layout indicator being a graphical indicator, the method further comprising:

determining that the handwritten text includes a first text line;

determining that the graphical indicator is associated with the first text line;

determining that the first text line is a title corresponding to a listing of content elements; and

determining that one or more additional text lines represent the listing of content elements.

3. The method of claim 2 , wherein determining that the handwritten text includes the two or more indicators further comprises determining that the first text line includes the content type indicator, the content type indicator being a keyword, wherein determining the first application functionality to execute comprises determining the first application functionality based at least in part on an association between the first application functionality and the keyword, and wherein executing the first application functionality comprises:

determining that the listing of content elements is a listing of items;

determining that a first additional text line of the one or more additional text lines represents a first item in the listing of items;

determining a first product that corresponds to the first item; and

sending a second indication of an execution result, wherein the second indication of the execution result comprises a third indication of the first product.

4. The method of claim 3 , wherein executing the first application functionality further comprises:

determining historical data associated with a user profile, the historical data indicating that the first product was selected for acquisition during a prior application session; and

automatically initiating an order for the first product,

wherein the third indication of the first product comprises a fourth indication that the order for the first product was initiated.

5. The method of claim 3 , further comprising:

sending a third indication of the listing of items to a mobile device;

receiving, from a mobile application executing on the mobile device, a fourth indication of a user selection associated with a second item in the listing of items, receiving the fourth indication of the user selection comprising receiving a fifth indication that a second product that corresponds to the second item has been acquired; and

causing presentation of a sixth indication that the second product has been acquired on the writing surface in association with a second additional text line of the one or more additional text lines that represents the second item.

6. The method of claim 2 , wherein determining that the handwritten text includes the two or more indicators further comprises determining that the first text line includes the content type indicator, the content type indicator being a keyword, wherein determining the first application functionality to execute comprises determining the first application functionality based at least in part on an association between the first application functionality and the keyword, and wherein executing the first application functionality comprises:

determining that the listing of content elements is a listing of tasks;

determining that a first additional text line of the one or more additional text lines represents a first task in the listing of tasks;

determining a first executable application that corresponds to the first task; and

performing the first task using the first executable application; and

sending a second indication of an execution result, wherein the second indication of the execution result comprises a third indication that the first task has been completed.

7. The method of claim 1 , wherein determining that the handwritten text includes the two or more indicators comprises determining that the handwritten text includes an application function indicator associated with an application function to be performed, the application function indicator being a search function indicator, the method further comprising:

determining that the handwritten text includes a first text line; and

determining that the search function indicator is associated with the first text line,

wherein executing the first application functionality comprises:

generating a search query based at least in part on one or more words included in the first text line; and

executing the search query to obtain an execution result, the execution result being a set of search results.

8. The method of claim 1 , further comprising causing an indication that the first application functionality is being executed to be displayed on the writing surface in association with the at least a portion of the handwritten text.

9. The method of claim 1 , further comprising receiving an execution result associated with executing the first application functionality.

10. The method of claim 1 , wherein determining that the handwritten text includes the two or more indicators comprises determining that the handwritten text includes one or more text lines extending in a direction, wherein the one or more text lines comprise a first text line extending in the direction and a second text line below the first text line and extending in the direction, wherein determining that the handwritten text includes one or more indicators further comprises determining that the second text line extends substantially in the direction.

11. An electronic device, comprising:

at least one memory storing computer-executable instructions; and

at least one processor operatively coupled to the at least one memory, wherein the at least one processor is configured to execute the computer-executable instructions to:

receive image data associated with a writing surface, the image data including a representation of handwritten text;

determine that the handwritten text includes two or more indicators comprising at least one of a content type indicator indicating a type of content included in the handwritten text;

determine, based at least in part on a first indicator and a second indicator of the two or more indicators, a first application functionality, wherein determining the first application functionality comprises:

determining, based at least in part on the first indicator, two or more application functionalities; and

determining, based at least in part on the second indicator, the first application functionality of the two or more application functionalities; and

execute the first application functionality.

12. The electronic device of claim 11 , further comprising an image capture device configured to generate the image data, wherein the image data is representative of an image of the writing surface, and wherein the writing surface is a non-electronic surface that includes the handwritten text.

13. The electronic device of claim 12 , wherein the image is a first image, wherein the electronic device is communicatively coupled to an image projector, and wherein the at least one processor is configured to

cause presentation of an execution result on the writing surface by executing the computer-executable instructions to send a signal to the image projector that causes the image projector to project a second image on the writing surface, the second image including an indication of the execution result.

14. The electronic device of claim 12 , further comprising receiving an execution result associated with executing the first application functionality.

15. The electronic device of claim 11 , wherein the at least one processor is configured to determine that the handwritten text includes the two or more indicators by executing the computer-executable instructions to:

send the image data to a remote system comprising one or more servers, wherein the remote system is configured to detect the handwritten text in the image data and perform natural language processing on the handwritten text to determine the two or more indicators; and

receive, from the remote system, an identification of the two or more indicators; and

wherein the at least one processor is configured to execute the first application functionality by interacting with the remote system.

16. The electronic device of claim 11 , wherein the writing surface is an electronic display of the electronic device, and wherein the image data comprises input data detected at the electronic display, the input data being indicative of the handwritten text generated as a result of contact between an input device and the electronic display.

17. An electronic device, comprising:

at least one memory storing computer-executable instructions; and

at least one processor operatively coupled to the at least one memory, wherein the at least one processor is configured to execute the computer-executable instructions to:

receive a first indication of handwritten text detected on a writing surface associated with a user device;

determine that the handwritten text includes two or more indicators comprising at least one of a content type indicator associated with a type of content included in the handwritten text; and

determine, based at least in part on a first indicator and a second indicator of the two or more indicators, a first application functionality to execute, wherein determining the first application functionality comprises:

determining, based at least in part on the first indicator, two or more application functionalities; and

determining, based at least in part on the second indicator, the first application functionality of the two or more application functionalities; and

execute the first application functionality.

18. The electronic device of claim 17 , wherein to determine that the handwritten text includes the two or more indicators comprises to determine that the handwritten text includes a content layout indicator associated with a layout of the handwritten text, the content layout indicator being a graphical indicator, wherein the at least one processor is further configured to execute the computer-executable instructions to:

determine that the handwritten text includes a first text line;

determine that the graphical indicator is associated with the first text line;

determine that the first text line is a title corresponding to a listing of content elements; and

determine that one or more additional text lines represent the listing of content elements.

19. The electronic device of claim 18 , wherein determine that the handwritten text includes the two or more indicators further comprises determining that the first text line includes the content type indicator, the content type indicator being a keyword, wherein determine the first application functionality to execute comprises determining the first application functionality based at least in part on an association between the first application functionality and the keyword, and wherein to execute the first application functionality comprises to:

determine that the listing of content elements is a listing of items;

determine that a first additional text line of the one or more additional text lines represents a first item in the listing of items;

determine a first product that corresponds to the first item; and

send a second indication of an execution result, wherein the second indication of the execution result comprises a third indication of the first product.

20. The electronic device of claim 19 , wherein to execute the first application functionality further comprises to:

determine historical data associated with a user profile, the historical data indicating that the first product was selected for acquisition during a prior application session; and

automatically initiate an order for the first product,

wherein the third indication of the first product comprises a fourth indication that the order for the first product was initiated.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 3, 2016
From: BENTLEY, PETER VAN TUYL; LAM, ANGELA; MCQUEEN, MICHAEL DOUGLAS; MURDOCK, ROBERT STEVEN
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 038444/0066 →
Cited By (3)
US 12,299,383 US 12,314,668 US 12,346,658