IP Library › Granted Patent US 10,671,844
Granted Patent B2
US 10,671,844 · App. 15/721,710 · Granted Jun 2, 2020

Handwritten text recognition

Inventors: Ryan S. Dixon (Mountain View, CA); Adrien Delaye (Sunnyvale, CA); Daniel E. Gobera Rubalcava (San Jose, CA); Dominic L. Howell (San Jose, CA); Youssouf K. Chherawala (Sunnyvale, CA); Martin Bresler (Nová Ves nad Nisou, CZ)
Assignee: Apple Inc.
G06K9/00422G06F3/017G06F3/03545G06F3/0412G06F3/04883G06K9/00409G06K9/222G06K9/42G06K9/6217G06K9/6267
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,671,844
App. No.
15/721,710
Granted
Jun 2, 2020
Kind
B2
Abstract

The subject technology provides for receiving a new input stroke. The subject technology determines whether the new input stroke is associated with an existing line group based on a writing direction estimate of the existing line group. The subject technology merges the new input stroke with the existing line group in response to determining that the new input stroke is associated with the existing line group. The subject technology determines a local orientation of the existing line group including the new input stroke based on an estimate of a direction of writing and a scale of each stroke. The subject technology normalizes the existing line group including the new input stroke using the determined location orientation.

Claims (72)

1. A method comprising:

receiving a new input stroke;

determining whether the new input stroke is associated with an existing line group based on a writing direction estimate of the existing line group, wherein the existing line group includes a plurality of strokes representing a line of text;

merging the new input stroke with the existing line group in response to determining that the new input stroke is associated with the existing line group;

determining a local orientation of the existing line group including the new input stroke based on an estimate of a direction of writing and a scale of each stroke, wherein the local orientation of the existing line group including the new input stroke is based at least in part on an average of vectors corresponding to a respective center of the new input stroke and centers of strokes around the new input stroke; and

normalizing the existing line group including the new input stroke using the determined local orientation.

2. The method of claim 1 , wherein determining the local orientation of the existing line group is further based on an estimate of an ordering of the strokes of the existing line group, the ordering of the strokes being based on the writing direction estimate.

3. The method of claim 1 , further comprising:

performing handwritten character recognition on the normalized existing line group to provide a set of recognized characters.

4. The method of claim 3 , further comprising:

storing a word formed by the set of recognized characters in an index for searching.

5. The method of claim 4 , further comprising:

using the word formed by the set of recognized characters as a title of a document corresponding to handwritten content including the set of recognized characters.

6. The method of claim 5 , further comprising:

receiving a search keyword;

performing a search using the index to locate the document corresponding to handwritten content; and

providing, as a search result, the document corresponding to handwritten content.

7. The method of claim 1 , wherein determining whether the new input stroke is associated with the existing line group based on the writing direction estimate of the existing line group further comprises:

determining a last stroke of the existing line group;

determining an oriented distance from the last stroke to the new input stroke; and

determining that the oriented distance meets a threshold value.

8. The method of claim 7 , further comprising:

determining a last stroke of a second existing line group;

determining the oriented distance from the last stroke of the second existing line group to the new input stroke; and

determining whether the oriented distance from the last stroke of the second existing line group meets the threshold value.

9. The method of claim 8 , further comprising:

determining, using timestamp information, a more recent stroke between the last stroke of the existing line group and the last stroke of the second existing line group; and

merging the new input stroke with a corresponding line group associated with the more recent stroke.

10. The method of claim 1 , further comprising:

determining that no existing line group exists; and

creating a new line group including the new input stroke.

11. The method of claim 1 , wherein normalizing the existing line group including the new input stroke using the determined local orientation further comprises:

rotating each stroke in the existing line group; and

moving each stroke, except for a first stroke, in the existing line group.

12. The method of claim 1 , further comprising:

determining that non-textual information is included in a second line group; and

disregarding the non-textual information for handwritten character recognition.

13. A system comprising:

a processor;

a memory device containing instructions, which when executed by the processor cause the processor to:

receive a new input stroke;

determine whether the new input stroke is associated with an existing line group based on a writing direction estimate of the existing line group, wherein the existing line group includes a plurality of strokes representing a line of text;

merge the new input stroke with the existing line group in response to determining that the new input stroke is associated with the existing line group;

determine a local orientation of the existing line group including the new input stroke based on an estimate of a direction of writing and a scale of each stroke, wherein the local orientation of the existing line group including the new input stroke is based at least in part on an average of vectors corresponding to a respective center of the new input stroke to centers of strokes around the new input stroke; and

normalize the existing line group including the new input stroke using the determined local orientation.

14. The system of claim 13 , wherein to determine whether the new input stroke is associated with the existing line group further comprises:

determining a last stroke of the existing line group;

determining an oriented distance from the last stroke to the new input stroke; and

determining that the oriented distance meets a threshold value.

15. The system of claim 14 , wherein the memory device contains further instructions, which when executed by the processor further cause the processor to:

determine a last stroke of a second existing line group;

determine the oriented distance from the last stroke of the second existing line group to the new input stroke; and

determine whether the oriented distance from the last stroke of the second existing line group meets the threshold value.

16. The system of claim 15 , wherein the memory device contains further instructions, which when executed by the processor further cause the processor to:

determine, using timestamp information, a more recent stroke between the last stroke of the existing line group and the last stroke of the second existing line group; and

merge the new input stroke with a corresponding line group associated with the more recent stroke.

17. The system of claim 13 , wherein the memory device contains further instructions, which when executed by the processor further cause the processor to:

determine that no existing line group exists; and

create a new line group including the new input stroke.

18. The system of claim 13 , wherein to normalize the existing line group including the new input stroke using the determined local orientation further comprises:

rotating each stroke in the existing line group; and

moving each stroke, except for a first stroke, in the existing line group.

19. The system of claim 13 , wherein the memory device contains further instructions, which when executed by the processor further cause the processor to:

determine that non-textual information is included in a second line group; and

disregard the non-textual information for handwritten character recognition.

20. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:

receiving a new input stroke;

determining whether the new input stroke is associated with an existing line group based on a writing direction estimate of the existing line group, wherein the existing line group includes a plurality of strokes representing a line of text;

merging the new input stroke with the existing line group in response to determining that the new input stroke is associated with the existing line group;

determining a local orientation of the existing line group including the new input stroke based on an estimate of a direction of writing and a scale of each stroke, wherein the local orientation of the existing line group including the new input stroke is based at least in part on an average of vectors corresponding to a respective center of the new input stroke to centers of strokes around the new input stroke; and

normalizing the existing line group including the new input stroke using the determined local orientation.

21. The method of claim 1 , wherein the centers of strokes are within a neighborhood around the new input stroke, the neighborhood is based on a window specified by a particular size, and the local orientation is based on a directional vector of each stroke in the existing line group, the directional vector having a norm of a particular value.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 2, 2017
From: DIXON, RYAN S.; DELAYE, ADRIEN; RUBALCAVA, DANIEL E. GOBERA; HOWELL, DOMINIC L.; CHHERAWALA, YOUSSOUF K.; BRESLER, MARTIN
To: APPLE INC.
Reel/Frame 043759/0856 →
Continuity (2)
Provisional Application 62514772 · Jun 2, 2017
Related Publication 20180349692A1 · Dec 6, 2018