System for stylus input latency compensation
A stylus provides input via a touchscreen comprising a touch sensor and a display. Latency between placement of a stylus tip and corresponding presentation of visual indicia on the display is reduced or eliminated by determining a predicted path of the stylus tip during a stroke. Visual indicia is presented on the display, based on the predicted path. Inputs from the touch sensor may include hover events associated with detection of the tip while not in contact with the touchscreen and touch events associated with presence of the tip on the touchscreen. A machine learning network may be trained to determine the predicted path. A portion of the network may be trained to accept dynamic-length sequences of events and generate fixed length sequences, reducing subsequent network complexity. A hand may be detected and used to determine the predicted path. The end of a stroke may be predicted, reducing overshoot.
1 . A system comprising:
a touch sensor;
a display device;
one or more memories, storing first computer-executable instructions; and
one or more hardware processors to execute the first computer-executable instructions to:
acquire, using the touch sensor, first event data comprising a first set of events associated with a first set of times;
determine first display data based on the first set of events;
present first visual indicia on the display device based on the first display data;
determine, using one or more trained machine learning networks to process the first event data, coordinate data, wherein the coordinate data is indicative of a predicted set of coordinates associated with a second set of times that are after the first set of times;
determine second display data based on the coordinate data; and
present second visual indicia using the display device based on the second display data.
2 . The system of claim 1 , wherein the one or more trained machine learning networks comprises a first trained machine learning network and a second trained machine learning network; and
the one or more hardware processors to further execute the first computer-executable instructions to:
determine, using the first event data and the first trained machine learning network, first data comprising a fixed sequence length of embedding vector data; and
determine, using the first data and the second trained machine learning network, second data, wherein the coordinate data is based on the second data.
3 . The system of claim 1 , the one or more hardware processors to further execute the first computer-executable instructions to:
determine, using the first event data and a first classifier, hand data indicative of a hand associated with input to the touch sensor; and
determine, using the one or more trained machine learning networks, the coordinate data based at least in part on the hand data.
4 . The system of claim 1 , the one or more hardware processors to further execute the first computer-executable instructions to:
determine mode data indicative of input associated with one or more of text or drawings; and
determine, using the one or more trained machine learning networks, the coordinate data based at least in part on the mode data.
5 . The system of claim 1 , the one or more hardware processors to further execute the first computer-executable instructions to:
determine, using the one or more trained machine learning networks and the first event data, first data indicative of a predicted set of events associated with the second set of times; and
wherein the coordinate data is determined using a regression algorithm and the first data.
6 . The system of claim 1 , the one or more hardware processors to further execute the first computer-executable instructions to:
determine, using the one or more trained machine learning networks and the first event data, first data indicative of the predicted set of coordinates associated with the second set of times; and
determine, based on the first data, confidence data that is indicative of a confidence that respective ones of the predicted set of coordinates are correct predictions of a future touch event;
determine, based on the first data, stroke data that is indicative of a likelihood that respective ones of the predicted set of coordinates are part of a stroke; and
wherein the coordinate data is determined using the first data and the stroke data.
7 . The system of claim 1 , the one or more hardware processors to further execute the first computer-executable instructions to:
determine input data based on the coordinate data; and
providing the input data to one or more applications.
8 . The system of claim 1 , the one or more hardware processors to further execute the first computer-executable instructions to:
determine a second set of events associated with the second set of times using the one or more trained machine learning networks to process the first event data, wherein the second set of events represent a predicted set of events; and
determine the coordinate data based on the second set of events.
9 . A computer-implemented method comprising:
acquiring, using a touch sensor, first event data comprising a first set of events associated with a first set of times;
determining first display data based on the first set of events;
presenting first visual indicia on a display device based on the first display data;
determining, using one or more trained machine learning networks to process the first event data, coordinate data, wherein the coordinate data is indicative of a predicted set of coordinates associated with a second set of times that are after the first set of times;
determining second display data based on the coordinate data; and
presenting second visual indicia on the display device based on the second display data.
10 . The method of claim 9 , wherein the first set of events are indicative of:
touch events associated with contact between a stylus tip and the touch sensor; and
hover events associated with the stylus tip proximate to, but not in contact with, the touch sensor.
11 . The method of claim 9 , wherein the one or more trained machine learning networks comprises a first trained machine learning network and a second trained machine learning network; and
the method further comprising:
determining, using the first event data and the first trained machine learning network, first data comprising a fixed sequence length of embedding vector data; and
determining, using the first data and the second trained machine learning network, second data, wherein the coordinate data is based on the second data.
12 . The method of claim 9 , comprising:
determining, using the first event data and a first classifier, hand data indicative of a hand associated with input to the touch sensor; and
determining, using the one or more trained machine learning networks, the coordinate data based at least in part on the hand data.
13 . The method of claim 9 , further comprising:
determining mode data indicative of input associated with one or more of text or drawings; and
determining, using the one or more trained machine learning networks, the coordinate data based at least in part on the mode data.
14 . The method of claim 9 , further comprising:
determining, using the one or more trained machine learning networks and the first event data, first data indicative of a predicted set of events associated with the second set of times; and
wherein the coordinate data is determined using a regression algorithm and the first data.
15 . The method of claim 9 , further comprising:
determining, using the one or more trained machine learning networks and the first event data, first data indicative of the predicted set of coordinates associated with the second set of times; and
determining, based on the first data, confidence data that is indicative of a confidence that respective ones of the predicted set of coordinates are correct predictions of a future touch event; and
determining, based on the first data, stroke data that is indicative of a likelihood that respective ones of the predicted set of coordinates are part of a stroke; and
wherein the coordinate data is determined using the first data and the stroke data.
16 . The method of claim 9 , further comprising:
determining input data based on the coordinate data; and
providing the input data to one or more applications.
17 . The method of claim 9 , the determining the coordinate data comprising:
determining a second set of events associated with the second set of times using the one or more trained machine learning networks to process the first event data, wherein the second set of events represent a predicted set of events; and
determining the coordinate data based on the second set of events.
18 . The method of claim 17 , wherein the second set of events is indicative of:
the predicted set of coordinates with respect to one or more of the touch sensor or the display device; and one or more of:
an orientation of a stylus with respect to the one or more of the touch sensor or the display device,
a pressure on the stylus, or
a tilt of the stylus.
19 . A system comprising:
a touch sensor;
one or more memories, storing first computer-executable instructions; and
one or more hardware processors to execute the first computer-executable instructions to:
acquire, using the touch sensor, first event data comprising a first set of events associated with a first set of times;
determine, using one or more trained machine learning networks to process the first event data, a predicted set of events indicative of a predicted path with respect to the touch sensor, wherein the predicted set of events occur after the first set of times;
determine input data based at least in part on the predicted set of events; and
provide the input data to one or more applications.
20 . The system of claim 19 , wherein the one or more trained machine learning networks comprises a first trained machine learning network and a second trained machine learning network; and
the one or more hardware processors to further execute the first computer-executable instructions to:
determine, using the first event data and the first trained machine learning network, first data comprising a fixed sequence length of embedding vector data; and
determine, using the first data and the second trained machine learning network, second data; and
wherein the predicted set of events is based on the second data.