System and method for automatic language detection for handwritten text
Methods for automatic language detection for handwritten text are performed by systems and devices. Such automatic language detection is performed prior to sending representations of the handwritten text to a language recognition engine. Handwritten inputs including one or more writing strokes are received from an input interface, and are associated with coordinates of the inputs and times that the inputs are made. The handwritten inputs are grouped into words based on the coordinates and times. Writing strokes are normalized, and then the words are individually transformed to generate language vectors, such as through a recurrent neural network. The language vectors are used to determine language probabilities for the handwritten inputs. Based on the language probabilities, the handwritten inputs are provided to a specific language recognition engine to determine the language thereof prior to translation or transcription.
1 . A language detection system comprising:
a processing system comprising one or more processors; and
a memory configured to store program code to be executed by the one or more processors, the program code including:
an input analyzer configured to:
receive handwritten inputs from an input interface, the handwritten inputs including one or more writing strokes;
associate the handwritten inputs with sets of coordinate-time pairs, wherein said sets of coordinate-time pairs represent at least one or more times when the strokes were made; and
group the handwritten inputs into words based at least on the sets of coordinate-time pairs;
a detection manager configured to:
normalize the one or more writing strokes that correspond to the words to generate normalized words;
transform the normalized words individually to generate language vectors for the normalized words; and
determine at least one language probability based at least on the language vectors; and
an engine selector configured to:
select a specific language recognition engine from a plurality of language recognition engines based at least on the at least one language probability; and
provide the handwritten inputs to the selected specific language recognition engine to recognize the handwritten inputs.
2 . The language detection system of claim 1 , wherein the detection manager is configured to analyze the normalized words individually to generate the language vectors for the normalized words using a recurrent neural network (RNN) where the language vectors are outputs of the RNN.
3 . The language detection system of claim 2 , wherein the RNN is a bi-directional RNN that generates first vectors from a forward portion of the bi-directional RNN and second vectors from a backward portion of the bi-directional RNN; and
wherein the detection manager is configured to concatenate the first vectors and the second vectors to generate the language vectors.
4 . The language detection system of claim 1 , wherein each of the writing strokes includes one or more of the sets of coordinate-time pairs; and
wherein the input analyzer is configured to:
compare spatial and temporal attributes of each of the one or more writing strokes with others of the one or more writing strokes based at least on the one or more sets of coordinate time pairs; and
group the handwritten inputs into words based at least on the spatial and temporal attributes.
5 . The language detection system of claim 4 , wherein the spatial and temporal attributes include at least one of:
a direction of a successive word with respect to a previous word;
amounts of time between the one or more writing strokes;
an amount of time taken to complete a writing stroke; or
a proximity of a writing stroke to others of the one or more writing strokes.
6 . The language detection system of claim 1 , wherein the handwritten inputs comprise handwriting strokes captured in approximately real-time, and wherein the input interface comprises a user interface that is one or more of a touchscreen, a touchpad, or a camera; or
wherein the handwritten inputs comprise a stored representation of handwriting, and wherein the input interface comprises an access interface to a storage where the representation is stored.
7 . The language detection system of claim 1 , wherein the input analyzer is configured to:
detect symbols in the handwritten inputs; and
provide the detected symbols to the detection manager to determine the at least one language probability.
8 . The language detection system of claim 1 , further comprising:
the specific language recognition engine that is configured to:
generate a language translation or a transcription of the handwritten inputs; and
a user interface configured to:
provide the language translation or the transcription of the handwritten inputs generated by the specific language recognition engine to a display.
9 . A computer-implemented method for language detection, the method comprising:
receiving handwritten inputs from an input interface, the handwritten inputs including one or more writing strokes;
grouping the handwritten inputs into words based at least on sets of coordinate-time pairs corresponding to the handwritten inputs, wherein said sets of coordinate-time pairs represent at least one or more times when the strokes were made;
transforming the words individually to generate language vectors for the words;
determining at least one language probability based at least on the language vectors;
based at least on the at least one language probability, selecting a specific language recognition engine from a plurality of language recognition engines corresponding to different languages; and
providing the handwritten inputs to the selected specific language recognition engine to recognize the handwritten inputs.
10 . The computer-implemented method of claim 9 , further comprising:
associating the handwritten inputs with the sets of coordinate-time pairs; and
normalizing the one or more writing strokes that correspond to writing strokes of the words to generate normalized words.
11 . The computer-implemented method of claim 9 , further comprising at least one of:
analyzing the normalized words individually to generate the language vectors for the normalized words using a recurrent neural network (RNN) where the language vectors are outputs of the RNN; or
analyzing the normalized words individually to generate the language vectors for the normalized words using a bi-directional recurrent neural network (RNN) that generates first vectors from a forward portion of the bi-directional RNN and second vectors from a backward portion of the bi-directional RNN, where the first vectors and the second vectors are concatenated to generate the language vectors that are outputs of the RNN.
12 . The computer-implemented method of claim 9 , wherein each of the writing strokes includes one or more of the sets of coordinate-time pairs, the method further comprising:
comparing spatial and temporal attributes of each of the one or more writing strokes with others of the one or more writing strokes based at least on the one or more sets of coordinate time pairs; and
grouping the handwritten inputs into words based at least on the spatial and temporal attributes.
13 . The computer-implemented method of claim 12 , wherein the spatial and temporal attributes include at least one of:
a direction of a successive word with respect to a previous word;
amounts of time between the one or more writing strokes;
an amount of time taken to complete a writing stroke; or
a proximity of a writing stroke to others of the one or more writing strokes.
14 . The computer-implemented method of claim 9 , wherein the handwritten inputs comprise handwriting strokes captured in approximately real-time, and wherein the input interface comprises a user interface that is one or more of a touchscreen, a touchpad, or a camera; or
wherein the handwritten inputs comprise a stored representation of handwriting, and wherein the input interface comprises an access interface to a storage where the representation is stored.
15 . The computer-implemented method of claim 9 , further comprising:
detecting symbols in the handwritten inputs; and
providing the detected symbols to a detection manager to determine the at least one language probability.
16 . The computer-implemented method of claim 9 , further comprising:
generating a language translation or a transcription of the handwritten inputs; and
providing the language translation or the transcription of the handwritten inputs generated by the specific language recognition engine to a user interface of a display.
17 . The computer-implemented method of claim 9 , wherein said determining at least one language probability based at least on the language vectors is also based at least on one or more user profile attributes.
18 . A computer-readable storage medium having program instructions recorded thereon that, when executed by a processing device, perform a method for language detection, the method comprising:
receiving handwritten inputs that include one or more writing strokes;
grouping the handwritten inputs into words based at least on sets of coordinate-time pairs corresponding to the handwritten inputs, wherein said sets of coordinate-time pairs represent at least one or more times when the strokes were made;
transforming the words individually to generate language vectors for the words;
determining at least one language probability based at least on the language vectors;
selecting a specific language recognition engine from a plurality of language recognition engines based at least on the at least one language probability; and
providing the handwritten inputs to a specific language recognition engine to recognize the handwritten inputs.
19 . The computer-readable storage medium of claim 18 , wherein the method further comprises:
comparing spatial and temporal attributes of each of the one or more writing strokes with others of the one or more writing strokes based at least on the one or more sets of coordinate time pairs;
grouping the handwritten inputs into words based at least on the spatial and temporal attributes; and
analyzing the normalized words individually to generate the language vectors for the normalized words using a bi-directional recurrent neural network (RNN) that generates first vectors from a forward portion of the bi-directional RNN and second vectors from a backward portion of the bi-directional RNN, where the first vectors and the second vectors are concatenated to generate the language vectors that are outputs of the RNN.
20 . The computer-readable storage medium of claim 18 , wherein the handwritten inputs comprise handwriting strokes captured in approximately real-time, and wherein the input interface comprises a user interface that is one or more of a touchscreen, a touchpad, or a camera; or
wherein the handwritten inputs comprise a stored representation of handwriting, and wherein the input interface comprises an access interface to a storage where the representation is stored.