Training speech processing models using pseudo tokens
A speech processing model may be trained using pseudo tokens. Training a speech processing model with pseudo tokens may allow for training with a smaller amount of labeled training data and accordingly lower costs. A set of pseudo tokens may be determined by computing feature vectors from unlabeled training data, clustering the feature vectors, and performing token compression using the clustered feature vectors. A first speech processing model may be trained using unlabeled training data by determining sequences of pseudo tokens corresponding to the unlabeled training data. A second speech processing model may be initialized using the first speech processing model and then trained using labeled training data. The second speech processing model may then be deployed to a speech processing application.
1 . A computer-implemented method, comprising:
obtaining a first training corpus of first training samples, wherein the first training samples are unlabeled;
computing feature vectors from the first training samples;
determining a set of pseudo tokens from the feature vectors by:
clustering at least a portion of the feature vectors into a plurality of clusters,
assigning a cluster-token to each cluster of the plurality of clusters,
determining a plurality of cluster-token sequences using the plurality of clusters, and
determining the set of pseudo tokens by performing token compression on the plurality of cluster-token sequences;
determining a plurality of pseudo-token sequences using the feature vectors and the set of pseudo tokens;
training a first neural network using the first training samples and the plurality of pseudo-token sequences;
obtaining a second training corpus of second training samples, wherein the second training corpus is labeled;
training a second neural network using the second training corpus, wherein at least a portion of the second neural network is initialized using the first neural network; and
deploying the second neural network to a production system.
2 . The computer-implemented method of claim 1 , wherein clustering the feature vectors comprises using k-means clustering.
3 . The computer-implemented method of claim 1 , comprising performing a deduplication operation on the plurality of cluster-token sequences.
4 . The computer-implemented method of claim 1 , wherein performing token compression comprises performing token compression using byte-pair encoding.
5 . The computer-implemented method of claim 1 , wherein performing token compression comprises performing token compression using subword regularization.
6 . The computer-implemented method of claim 1 , wherein the first neural network comprises a first encoder and a first decoder.
7 . The computer-implemented method of claim 1 , wherein the first neural network comprises a transducer neural network.
8 . The computer-implemented method of claim 7 , wherein the transducer neural network comprises a joint network.
9 . A system, comprising at least one server computer comprising at least one processor and at least one memory, the at least one server computer configured to:
obtain a first training corpus of first training samples, wherein the first training samples are unlabeled;
compute feature vectors from the first training samples;
determine a set of pseudo tokens from the feature vectors by:
clustering at least a portion of the feature vectors into a plurality of clusters,
assigning a cluster-token to each cluster of the plurality of clusters,
determining a plurality of cluster-token sequences using the plurality of clusters, and
determining the set of pseudo tokens by performing token compression on the plurality of cluster-token sequences;
determine a plurality of pseudo-token sequences using the feature vectors and the set of pseudo tokens;
train a first neural network using the first training samples and the plurality of pseudo-token sequences;
obtain a second training corpus of second training samples, wherein the second training corpus is labeled;
train a second neural network using the second training corpus, wherein at least a portion of the second neural network is initialized using the first neural network; and
deploy the second neural network to a production system.
10 . The system of claim 9 , wherein the first neural network comprises a first encoder and a first decoder.
11 . The system of claim 10 , wherein:
a second encoder of the second neural network is initialized using the first encoder of the first neural network; and
a second decoder of the second neural network is initialized using the first decoder of the first neural network.
12 . The system of claim 10 , wherein the first encoder comprises a transformer encoder and the first decoder comprises a transformer decoder.
13 . The system of claim 10 , wherein the first encoder comprises a recurrent neural network.
14 . The system of claim 10 , wherein the first encoder comprises a convolutional neural network.
15 . The system of claim 9 , wherein the at least one server computer is configured to compute the feature vectors by performing average pooling.
16 . One or more non-transitory, computer-readable media comprising computer-executable instructions that, when executed, cause at least one processor to perform actions comprising:
obtaining a first training corpus of first training samples, wherein the first training samples are unlabeled;
computing feature vectors from the first training samples;
determining a set of pseudo tokens from the feature vectors by:
clustering at least a portion of the feature vectors into a plurality of clusters,
assigning a cluster-token to each cluster of the plurality of clusters,
determining a plurality of cluster-token sequences using the plurality of clusters, and
determining the set of pseudo tokens by performing token compression on the plurality of cluster-token sequences;
determining a plurality of pseudo-token sequences using the feature vectors and the set of pseudo tokens;
training a first neural network using the first training samples and the plurality of pseudo-token sequences;
obtaining a second training corpus of second training samples, wherein the second training corpus is labeled;
training a second neural network using the second training corpus, wherein at least a portion of the second neural network is initialized using the first neural network; and
deploying the second neural network to a production system.
17 . The one or more non-transitory, computer-readable media of claim 16 , wherein the first neural network comprises a first encoder and a first decoder.
18 . The one or more non-transitory, computer-readable media of claim 16 , wherein the second training corpus is labeled with text corresponding to the second training samples and the production system performs speech recognition.
19 . The one or more non-transitory, computer-readable media of claim 16 , wherein training the first neural network comprises optimizing a connectionist temporal classification objective function.
20 . The one or more non-transitory, computer-readable media of claim 16 , wherein a portion of the second neural network is initialized with random values.