Bidirectional probabilistic natural language rewriting and selection
A speech recognition and natural language understanding system performs insertion, deletion, and replacement edits of tokens at positions with low probabilities according to both a forward and a backward statistical language model (SLM) to produce rewritten token sequences. Multiple rewrites can be produced with scores depending on the probabilities of tokens according to the SLMs. The rewritten token sequences can be parsed according to natural language grammars to produce further weighted scores. Token sequences can be rewritten iteratively using a graph-based search algorithm to find the best rewrite. Mappings of input token sequences to rewritten token sequences can be stored in a cache, and searching for a best rewrite can be bypassed by using cached rewrites when present. Analysis of various initial token sequences that produce the same new rewritten token sequence can be useful to improve natural language grammars.
1. A computer-implemented method of rewriting an input token sequence of a user query when providing query results to the user, the method comprising:
receiving, over a computer network from an application executing on a remote client device, a user query from a system user, the query comprising the input token sequence;
determining forward probabilities, according to a forward statistical language model, for a plurality of tokens in the input token sequence;
determining backward probabilities, according to a backward statistical language model, for a plurality of tokens in the input token sequence;
inserting a new token at a location after a first token having a low backward probability and before an adjacent second token having a low forward probability to create a new rewritten token sequence;
processing the new rewritten token sequence to produce a result; and
providing a response, indicating the result of the processing, to the system user.
2. A computer-implemented method of rewriting an input token sequence, the method comprising:
determining forward probabilities, according to a forward statistical language model, for a plurality of tokens in the input token sequence;
determining backward probabilities, according to a backward statistical language model, for a plurality of tokens in the input token sequence;
computing a probability score for each of the plurality of tokens based on a lowest one of the forward probabilities and a lowest one of the backward probabilities; and
replacing, with a new token, a token having a lowest probability score among the computed probability scores for the plurality of tokens to create a new rewritten token sequence.
3. A computer-implemented method of rewriting an input token sequence, the method comprising:
determining forward probabilities, according to a forward statistical language model (SLM), for a plurality of tokens in the input token sequence;
determining backward probabilities, according to a backward statistical language model, for a plurality of tokens in the input token sequence; and
replacing a suspicious token having a forward probability below a first threshold and a backward probability below a second threshold with a new token to create a new rewritten token sequence.
4. The method of claim 3 , further comprising:
substituting a tag for at least one token in the input token sequence prior to determining probabilities.
5. The method of claim 3 , further comprising:
choosing, as the new token, one that is both in a list of highest probability tokens according to the forward SLM and a list of highest probability tokens according to the backward SLM.
6. The method of claim 5 , further comprising:
performing a syntactic analysis of the input token sequence according to syntax rules; and
restricting the choosing to only tokens that are syntactically legal in context of neighboring tokens according to the syntax rules.
7. The method of claim 5 , further comprising:
computing, for the new rewritten token sequence, a rewrite score that depends at least on the probability of the new token in the forward SLM and the probability of the new token in the backward SLM.
8. The method of claim 7 , further comprising:
scaling the rewrite score based on the probability of the new token in a diverse corpus SLM that was built from expressions related to a wide variety of topics.
9. The method of claim 5 , further comprising:
replacing the suspicious token with an alternative new token to create an alternative rewritten token sequence;
computing, for the alternative rewritten token sequence, an alternative score as a combination of both the probability of the alternative new token in the forward SLM and the probability of the alternative new token in the backward SLM; and
choosing whichever of the new rewritten token sequence and the alternative rewritten token sequence has a higher score.
10. The method of claim 5 , further comprising:
maintaining a token buffer of hypothesized tokens from recent continuous speech,
wherein the input token sequence is a sequence of tokens in the token buffer.
11. The method of claim 3 , further comprising:
storing a history cache of tokens present in recent token sequences;
for replacing the suspicious token, choosing the new token from each of a list of forward most probable tokens and a list of backward most probable tokens; and
increasing the probability score of at least one token that is present in the history cache.
12. The method of claim 3 , further comprising:
parsing the new rewritten token sequence according to a grammar using a natural language parser to produce a parse score.
13. The method of claim 12 , further comprising:
replacing an alternative suspicious token different from the suspicious token to create an alternative rewritten token sequence;
parsing the alternative rewritten token sequence according to the grammar using the natural language parser to produce an alternative parse score; and
choosing whichever of the new rewritten token sequence and the alternative rewritten token sequence has a higher parse score.
14. The method of claim 12 , further comprising:
replacing the suspicious token with an alternative new token to create an alternative rewritten token sequence;
parsing the alternative rewritten token sequence according to the grammar using the natural language parser to produce an alternative parse score; and
choosing whichever of the new rewritten token sequence and the alternative rewritten token sequence has a higher parse score.
15. The method of claim 12 , further comprising:
using a tree-based algorithm to iteratively perform rewrites and compute scores for each rewrite to produce a set of rewrites from which to choose one with a best score.
16. The method of claim 3 , further comprising:
storing the input token sequence in a cache;
storing the new rewritten token sequence in the cache in association with the input token sequence; and
searching the cache for the input token sequence.
17. The method of claim 16 , further comprising:
analyzing the cache to identify, for the new rewritten token sequence, a most frequent input token sequence that was rewritten to the new rewritten token sequence.
18. The method of claim 17 , further comprising:
creating a grammar rule to cover the most frequent input token sequence that was rewritten to the new rewritten token sequence.
19. The method of claim 17 , further comprising:
adapting a grammar rule to cover the most frequent input token sequence that was rewritten to the new rewritten token sequence.