Stroke attribute matrices
Methods, systems, and computer program products are provided for stroke attribute matrices. User input strokes may be converted into attributes encoded in one or more stroke attribute matrices (SAMs), such as bitmaps, for image or other multidimensional analysis. One or more convolutional neural networks (CNNs) may recognize letters, symbols, shapes and gestures in SAMs. A selector may select output classifications from among multiple CNNs. A sequence analyzer may select a sequence of selected CNN outputs. Stroke information may comprise, for example, velocity (e.g. direction and speed), tilt, pressure, line width, pen up/down events, hover height, etc. Stroke information may be stored, for example, in bitmap color channels (e.g. to facilitate human review). For example, an x, y velocity vector and x, y tilt may be encoded, respectively, as RGBA components of pixel data. Stroke crossings may be encoded, for example, by combining attribute values at pixels where strokes intersect.
1. A method, comprising:
receiving, by a computing device, stroke data based on strokes input by a user;
generating at least one stroke attribute from the stroke data;
encoding the at least one stroke attribute into a bitmap, wherein the encoded at least one stroke attribute indicates relative positions of points in the strokes; and
providing the bitmap as input to at least one machine-learning (ML) classifier to determine what the strokes represent.
2. The method of claim 1 , wherein said encoding of the at least one stroke attribute into a bitmap comprises encoding a plurality of stroke attributes as a plurality of different colors in the bitmap.
3. The method of claim 2 , wherein said encoding of the plurality of stroke attributes as the plurality of different colors in the bitmap comprises encoding an angle as a first color and encoding a magnitude as a second color or encoding a first direction and speed as the first color and encoding a second direction and speed as the second color.
4. The method of claim 1 , wherein the at least one stroke attribute comprises a direction attribute indicating stroke directions at the points in the strokes.
5. The method of claim 1 , wherein the at least one ML classifier comprises a plurality of ML classifiers, the method further comprising:
selecting a classification from a plurality of classifications generated by the plurality of ML classifiers, the selected classification being among a sequence of classifications for a sequence of bitmaps encoding the stroke data; and
determining what the strokes represent based on the sequence of selected classifications.
6. The method of claim 1 , wherein the strokes comprise first and second stroke segments that intersect, wherein said generating the at least one stroke attribute or said encoding the at least one stroke attribute into a bitmap comprises combining a value of the at least one stroke attribute for the first stroke segment with a value of the at least one stroke attribute for the second stroke segment.
7. The method of claim 6 , further comprising:
determining what the strokes represent based at least in part on the intersection.
8. The method of claim 1 , wherein said generating the at least one stroke attribute or said encoding the at least one stroke attribute into a bitmap comprises encoding a temporal discrepancy indicating at least one of (i) when stroking begins or ends or (ii) when hovering begins or ends.
9. A system, comprising:
one or more processors; and
one or more memory devices that store program code configured to be executed by the one or more processors, the program code comprising:
a stroke recognizer configured to:
receive stroke data based on strokes input by a user;
generate stroke attributes from the stroke data;
encode the stroke attributes into bitmaps, wherein the encoded stroke attributes indicate relative positions of points in the strokes; and
provide the bitmaps as input to machine-learning (ML) classifiers to determine what the strokes represent.
10. The system of claim 9 , wherein said stroke recognizer is configured to encode the stroke attributes as different colors in the bitmaps.
11. The system of claim 10 , wherein said stroke recognizer is configured to (i) encode a first attribute as a first color and encode a second attribute as a second color or (ii) encode a first vector attribute as the first color and encode a second vector attribute as the second color.
12. The system of claim 9 , wherein the stroke attributes comprise a direction attribute indicating stroke directions at the points in the strokes.
13. The system of claim 9 , said stroke recognizer further configured to:
generate a set of classifications for each of the bitmaps;
select a classification from the set of classifications for each of the bitmaps, creating a sequence of classifications; and
determine what the strokes represent based on the sequence of classifications.
14. The system of claim 9 , wherein said stroke recognizer is further configured to:
combine attribute values for strokes at a point of intersection.
15. A computer-readable storage medium having program instructions recorded thereon that, when executed by a processing circuit, perform a method comprising:
receiving stroke data based on strokes input by a user;
generating stroke attributes from the stroke data;
encoding the stroke attributes into bitmaps, wherein the encoded stroke attributes indicate relative positions of points in the strokes and wherein the stroke attributes comprise a direction attribute indicating stroke directions at the points in the strokes; and
providing the bitmaps as input to machine-learning (ML) classifiers to determine what the strokes represent.
16. The computer-readable storage medium of claim 15 , wherein encoding the stroke attributes into bitmaps comprises:
encoding the stroke attributes as different colors in the bitmaps.
17. The computer-readable storage medium of claim 15 , the method further comprising:
generating a set of classifications for each bitmap in a sequence of bitmaps encoded from stroke attributes based on the strokes;
selecting a sequence of classifications from the set of classifications for each bitmap in the set of bitmaps; and
determining what the strokes represent based on the sequence of selected classifications.
18. The computer-readable storage medium of claim 15 , the method further comprising:
combining attribute values for strokes at a point of intersection.
19. The system of claim 9 , wherein the stroke data comprise first and second stroke segments that intersect, wherein said generate stroke attributes or said encode stroke attributes into bitmaps comprises combining a value of a respective stroke attribute for the first stroke segment with a value of a respective stroke attribute for the second stroke segment.
20. The computer-readable storage medium of claim 16 , wherein said encoding of the plurality of stroke attributes as different colors in the bitmap comprises:
encoding an angle as a first color and encoding a magnitude as a second color; or
encoding a first direction and speed as the first color and encoding a second direction and speed as the second color.