Object detection and segmentation for inking applications
An ink parsing system receives ink strokes at an inking device input and render the received ink strokes into an image in a pixel space. Writing strokes are detected in the image and labeled. Pixels corresponding to the labeled writing strokes are removed from the image. Drawing strokes in the image having the removed pixels are detected using and labeled. Writing objects and drawing objects corresponding, respectively, to the labeled writing strokes and the labeled drawing strokes are output. A digital ink parsing pipeline is thereby provided having accurate ink stroke detection and segmentation.
1. An ink parsing system comprising:
a memory associated with a computing device, the memory including a writing detector component and a drawing detector component; and
at least one processor that executes an ink parsing engine that uses the writing detector component and the drawing detector component to:
receive ink strokes at an inking device input;
render the received ink strokes into an image;
detect one or more writing strokes in the image using the writing detector component and label the writing strokes;
remove one or more pixels corresponding to the labeled writing strokes from the image;
detect one or more drawing strokes in the image, after detecting the one or more writing strokes and removing the one or more pixels corresponding to the labeled writing strokes using the drawing detector component, and label the drawing strokes; and
output one or more writing objects and one or more drawing objects corresponding, respectively, to the labeled writing strokes and the labeled drawing strokes.
2. The ink parsing system of claim 1 , wherein the at least one processor executes the ink parsing engine to perform a semantic segmentation with the writing detector component, the semantic segmentation using pixel-wise classification to detect the writing strokes.
3. The ink parsing system of claim 2 , wherein the at least one processor executes the ink parsing engine to use predicted writing masks to label the writing strokes.
4. The ink parsing system of claim 2 , wherein the at least one processor executes the ink parsing engine to use a neural network having a plurality of layers, each with a fixed number of kernels, to perform the semantic segmentation.
5. The ink parsing system of claim 1 , wherein the at least one processor executes the ink parsing engine to perform drawing stroke detection with the drawing detector component using a convolutional network.
6. The ink parsing system of claim 5 , wherein the at least one processor executes the ink parsing engine to decode detected drawing strokes using a plurality of bounding boxes with corresponding predicted labels, and further uses a decision tree as a binary classifier to detect the drawing strokes.
7. The ink parsing system of claim 1 , wherein (i) the ink parsing engine has a parsing pipeline configured to perform detection using the writing detector component before performing detection using the drawing detector component, (ii) the writing detector is configured to remove the one or more pixels to generate a modified inking input, (iii) the drawing detector component is configured to perform detection of the one or more drawing strokes on the modified inking input that, comprises a non-writing image, and (iv) the modified inking input, the one or more writing objects, and the one or more drawing objects are output to a next stage for processing.
8. A computerized method for digital ink parsing, the computerized method comprising:
receiving ink strokes at an inking device input;
rendering the received ink strokes into an image;
detecting one or more writing strokes in the image and label the writing strokes;
removing one or more pixels corresponding to the labeled writing strokes from the image;
detecting one or more drawing strokes in the image, after detecting the one or more writing strokes and removing the one or more pixels corresponding to the labeled writing strokes, and label the drawing strokes; and
outputting one or more writing objects and one or more drawing objects corresponding, respectively, to the labeled writing strokes and the labeled drawing strokes.
9. The computerized method of claim 8 , further comprising performing a semantic segmentation using pixel-wise classification to detect the writing strokes.
10. The computerized method of claim 9 , further comprising using predicted writing masks to label the writing strokes.
11. The computerized method of claim 9 , further comprising using a neural network having a plurality of layers, each with a fixed number of kernels, to perform the semantic segmentation.
12. The computerized method of claim 8 , further comprising performing drawing stroke detection using a convolutional network.
13. The computerized method of claim 12 , further comprising decoding detected drawing strokes using a plurality of bounding boxes with corresponding predicted labels, and using a decision tree as a binary classifier to detect the drawing strokes.
14. The computerized method of claim 8 , further comprising using a parsing pipeline that is configured to detect writing strokes before detecting drawing strokes.
15. One or more computer storage media having computer-executable instructions for digital ink parsing that, upon execution by at least one processor, cause the at least one processor to at least:
receive ink strokes at an inking device input;
render the received ink strokes into an image;
detect one or more writing strokes in the image and label the writing strokes;
remove one or more pixels corresponding to the labeled writing strokes from the image;
detect one or more drawing strokes in the image, after detecting the one or more writing strokes and removing the one or more pixels corresponding to the labeled writing strokes, and label the drawing strokes; and
output one or more writing objects and one or more drawing objects corresponding, respectively, to the labeled writing strokes and the labeled drawing strokes.
16. The one or more computer storage media of claim 15 having further computer-executable instructions that, upon execution by the at least one processor, cause the at least processor to at least perform a semantic segmentation using pixel-wise classification to detect the writing strokes.
17. The one or more computer storage media of claim 16 having further computer-executable instructions that, upon execution by the at least one processor, cause the at least processor to at least use predicted writing masks to label the writing strokes.
18. The one or more computer storage media of claim 16 having further computer-executable instructions that, upon execution by the at least one processor, cause the at least processor to at least use a neural network having a plurality of layers, each with a fixed number of kernels, to perform the semantic segmentation.
19. The one or more computer storage media of claim 15 having further computer-executable instructions that, upon execution by the at least one processor, cause the at least processor to at least perform drawing stroke detection using a convolutional network, decode detected drawing strokes using a plurality of bounding boxes with corresponding predicted labels, and use a decision tree as a binary classifier to detect the drawing strokes.
20. The one or more computer storage media of claim 15 having further computer-executable instructions that, upon execution by the at least processor, cause the at least processor to at least use a parsing pipeline that is configured to detect writing strokes before detecting drawing strokes.