Mixture of n-gram language models
View Patent ↗Methods, systems, and apparatus, including computer programs encoded on computer storage media, for creating a static language model from a mixture of n-gram language models. One of the methods includes receiving a set of development sentences W, receiving a set of language models G M , determining a set of n-gram language model weights λ M based on the development sentences W and the set of language models G M , determining a set of sentence cluster weights γ C , each of the sentence cluster weights corresponding to a cluster in a set of sentence clusters, each cluster in the set of sentence clusters associated with at least one sentence from the set of development sentences W, and generating a language model from the set of language models G M , the set of n-gram language model weights λ M , the set of sentence clusters, and the set of sentence cluster weights γ C .
1. A method comprising:
receiving, by a training system, a set of development sentences W;
receiving a set of two or more language models G M , each language model in the set of two or more language models G M for use by an automatic speech recognition system when generating text from speech;
determining a set of n-gram language model weights λ M using the development sentences W and the set of two or more language models G M ;
determining a set of sentence cluster weights γ C , each of the sentence cluster weights corresponding to a cluster in a set of sentence clusters, each cluster in the set of sentence clusters associated with at least one sentence from the set of development sentences W;
generating a language model from the set of two or more language models G M , the set of n-gram language model weights λ M , the set of sentence clusters, and the set of sentence cluster weights γ C ; and
providing, by the training system, the language model for use by a particular automatic speech recognition system to determine n-grams included in an utterance encoded in an audio signal.
2. The method of claim 1 , further comprising:
determining, for each sentence in the set of development sentences W, a probability p of the sentence using the set of n-gram language model weights λ M , and the set of sentence cluster weights γ C .
3. The method of claim 2 , wherein the probability p of the sentence w from the set of development sentences W satisfies
p
(
w
)
=
∏
i
=
1
w
∑
m
=
1
M
α
m
,
h
i
p
m
(
w
i
❘
h
i
)
where
α
m
,
h
i
=
∑
c
=
1
C
p
(
c
❘
h
i
)
λ
c
,
m
,
p
(
c
❘
h
i
)
=
p
(
h
i
❘
c
)
γ
c
∑
c
=
1
C
p
(
h
i
❘
c
)
γ
c
,
p
(
h
i
❘
c
)
=
∏
j
=
1
i
∑
m
=
1
M
λ
c
,
m
p
m
(
w
j
❘
h
j
)
,
λ is the n-gram language model weight, γ is the sentence cluster weight, C is the quantity of clusters in the set of sentence clusters, M is the quantity of language models in the set of two or more language models G M , |w| is the quantity of n-grams in the sentence w, and p m (w i |h i ) is the probability that n-gram w i follows history h i as estimated by a language model G m from the set of two or more language models G M .
4. The method of claim 2 , further comprising:
maximizing a combination of the probabilities by repeating the determination of at least one of the weights in the set of n-gram language model weights λ M , the determination of at least one of the weights in the set of sentence cluster weights γ C , and the probabilities p w of the sentences in the set of development sentences W.
5. The method of claim 4 , wherein maximizing the combination of the probabilities comprises maximizing a product of the probabilities.
6. The method of claim 4 , wherein:
each of the n-gram language model weights λ m from n-gram language model weights λ M satisfies
λ
c
,
m
′
=
1
c
∑
w
∈
c
∑
i
=
1
w
λ
c
,
m
p
m
(
w
i
❘
h
i
)
∑
j
=
1
M
λ
c
,
j
p
j
(
w
j
❘
h
j
)
,
and
each of the sentence cluster weights γ c from the set of sentence cluster weights γ C satisfies
γ
c
=
c
W
,
where c is one of the clusters in the set of sentence clusters, |c| is the quantity of sentences in cluster c, m is a language model from the set of two or more language models G M , M is the quantity of language models in the set of two or more language models G M , |W| is the quantity of sentences in set of development sentences W, |w| is the quantity of n-grams in the sentence w from the set of development sentences W, and p m (w i |h i ) is the probability that n-gram w i follows history h i as estimated by a language model G m from the set of two or more language models G M .
7. The method of claim 4 , wherein:
each of the n-gram language model weights λ m from n-gram language model weights λ M satisfies
λ
c
,
m
′
=
1
N
c
∑
w
∈
W
p
(
c
❘
w
)
∑
i
=
1
w
λ
c
,
m
p
m
(
w
i
❘
h
i
)
∑
j
=
1
M
λ
c
,
j
p
j
(
w
j
❘
h
j
)
,
and
each of the sentence cluster weights γ c from the set of sentence cluster weights γ C satisfies
γ
c
′
=
1
W
∑
w
∈
W
p
(
c
❘
w
)
,
where
N
c
=
∑
w
∈
W
p
(
c
❘
w
)
w
,
p
(
c
❘
w
)
=
γ
c
p
c
(
w
)
∑
i
=
1
C
γ
i
p
i
(
w
)
,
p
c
(
w
)
=
∏
i
=
1
w
∑
m
=
1
M
λ
c
,
m
p
m
(
w
i
❘
h
i
)
,
c is one of the clusters in the set of sentence clusters, C is the quantity of clusters in the set of sentence clusters, m is a language model from the set of two or more language models G M , M is the quantity of language models in the set of two or more language models G M , |w| is the quantity of n-grams in the sentence w from the set of development sentences W, and p m (w i |h i ) is the probability that n-gram w i , follows history h i as estimated by a language model G m from the set of two or more language models G M .
8. The method of claim 1 , further comprising training, prior to receiving the set of two or more language models G M , each of the language models in the set of two or more language models G M , each of the language models in the set of two or more language models G M trained from a different corpus.
9. The method of claim 1 , wherein:
receiving the set of two or more language models G M comprises receiving the set of two or more language models G M , each language model in the set of two or more language models G M for use by an automatic speech recognition when generating text from speech for a particular task; and
providing the language model for use by the particular automatic speech recognition system to determine n-grams included in an utterance comprises providing the language model for use determining n-grams included in an utterance independent from the task associated with the utterance.
10. The method of claim 1 , wherein providing the language model for use by the particular automatic speech recognition system to determine n-grams included in an utterance comprising providing the language model to a user device.
11. The method of claim 1 , wherein providing the language model for use by the particular automatic speech recognition system to determine n-grams included in an utterance comprises providing the language model for use transcribing the n-grams included in the utterance.
12. A non-transitory computer storage medium encoded with instructions that, when executed by a device, cause the device to perform operations comprising:
receiving, by the device, a set of development sentences W;
receiving a set of two or more language models G M , each language model in the set of two or more language models G M for use by an automatic speech recognition system when generating text from speech;
determining a set of n-gram language model weights λ M using the development sentences W and the set of two or more language models G M ;
determining a set of sentence cluster weights γ C , each of the sentence cluster weights corresponding to a cluster in a set of sentence clusters, each cluster in the set of sentence clusters associated with at least one sentence from the set of development sentences W;
generating a language model from the set of two or more language models G M , the set of n-gram language model weights λ M , the set of sentence clusters, and the set of sentence cluster weights γ C ; and
providing, by the device, the language model for use by an automatic speech recognition system to determine n-grams included in an utterance encoded in an audio signal.
13. The computer storage medium of claim 12 , the operations further comprising:
determining, for each sentence in the set of development sentences W, a probability p of the sentence using the set of n-gram language model weights λ M , and the set of sentence cluster weights γ C .
14. The computer storage medium of claim 13 , wherein the probability p of the sentence w from the set of development sentences W satisfies
p
(
w
)
=
∏
i
=
1
w
∑
m
=
1
M
α
m
,
h
i
p
m
(
w
i
❘
h
i
)
where
α
m
,
h
i
=
∑
c
=
1
C
p
(
c
❘
h
i
)
λ
c
,
m
,
p
(
c
❘
h
i
)
=
p
(
h
i
❘
c
)
γ
c
∑
c
=
1
C
p
(
h
i
❘
c
)
γ
c
,
p
(
h
i
❘
c
)
=
∏
j
=
1
i
∑
m
=
1
M
λ
c
,
m
p
m
(
w
j
❘
h
j
)
,
λ is the n-gram language model weight, γ is the sentence cluster weight, C is the quantity of clusters in the set of sentence clusters, M is the quantity of language models in the set of two or more language models G M , |w| is the quantity of n-grams in the sentence w, and p m(w i |h i ) is the probability that n-gram w i follows history h i as estimated by a language model G m from the set of two or more language models G M .
15. The computer storage medium of claim 13 , the operations further comprising:
maximizing a combination of the probabilities by repeating the determination of at least one of the weights in the set of n-gram language model weights λ M , the determination of at least one of the weights in the set of sentence cluster weights γ C , and the probabilities p w of the sentences in the set of development sentences W.
16. The computer storage medium of claim 15 , wherein:
each of the n-gram language model weights λ m from n-gram language model weights λ M satisfies
λ
c
,
m
′
=
1
c
∑
w
∈
c
∑
i
=
1
w
λ
c
,
m
p
m
(
w
i
❘
h
i
)
∑
j
=
1
M
λ
c
,
j
p
j
(
w
j
❘
h
j
)
,
and
each of the sentence cluster weights γ c from the set of sentence cluster weights γ C satisfies
γ
c
=
c
W
,
where c is one of the clusters in the set of sentence clusters, |c| is the quantity of sentences in cluster c, m is a language model from the set of two or more language models G M , M is the quantity of language models in the set of two or more language models G M , |W| is the quantity of sentences in set of development sentences W, |w| is the quantity of n-grams in the sentence w from the set of development sentences W, and p m (w i |h i ) is the probability that n-gram w i follows history h i as estimated by a language model G m from the set of two or more language models G M .
17. The computer storage medium of claim 15 , wherein:
each of the n-gram language model weights λ m from n-gram language model weights λ M satisfies
λ
c
,
m
′
=
1
N
c
∑
w
∈
W
p
(
c
❘
w
)
∑
i
=
1
w
λ
c
,
m
p
m
(
w
i
❘
h
i
)
∑
j
=
1
M
λ
c
,
j
p
j
(
w
j
❘
h
j
)
,
and
each of the sentence cluster weights γ c from the set of sentence cluster weights λ C satisfies
γ
c
′
=
1
W
∑
w
∈
W
p
(
c
❘
w
)
,
where
N
c
=
∑
w
∈
W
p
(
c
❘
w
)
w
,
p
(
c
❘
w
)
=
γ
c
p
c
(
w
)
∑
i
=
1
C
γ
i
p
i
(
w
)
,
p
c
(
w
)
=
∏
i
=
1
w
∑
m
=
1
M
λ
c
,
m
p
m
(
w
i
❘
h
i
)
,
c is one of the clusters in the set of sentence clusters, C is the quantity of clusters in the set of sentence clusters, m is a language model from the set of two or more language models G M , M is the quantity of language models in the set of two or more language models G M , |w| is the quantity of n-grams in the sentence w from the set of development sentences W, and p m (w i |h i ) is the probability that n-gram w i follows history h i as estimated by a language model G m from the set of two or more language models G M .
18. A system comprising:
a device and one or more storage devices storing instructions that are operable, when executed by the device, to cause the device to perform operations comprising:
receiving, by the device, a set of development sentences W;
receiving a set of two or more language models G M , each language model in the set of two or more language models G M for use by an automatic speech recognition system when generating text from speech;
determining a set of n-gram language model weights λ M using the development sentences W and the set of two or more language models G M ;
determining a set of sentence cluster weights γ C , each of the sentence cluster weights corresponding to a cluster in a set of sentence clusters, each cluster in the set of sentence clusters associated with at least one sentence from the set of development sentences W;
generating a language model from the set of two or more language models G M , the set of n-gram language model weights λ M , the set of sentence clusters, and the set of sentence cluster weights γ C ; and
providing, by the device, the language model for use by an automatic speech recognition system to determine n-grams included in an utterance encoded in an audio signal.
19. The system of claim 18 , the operations further comprising:
determining, for each sentence in the set of development sentences W, a probability p of the sentence using the set of n-gram language model weights λ M , and the set of sentence cluster weights γ C .
20. The system of claim 19 , the operations further comprising:
maximizing a combination of the probabilities by repeating the determination of at least one of the weights in the set of n-gram language model weights λ M , the determination of at least one of the weights in the set of sentence cluster weights γ C , and the probabilities p w of the sentences in the set of development sentences W.