Attentive scoring function for speaker identification

1 . A computer-implemented method for speaker identification executed on data processing hardware that causes the data processing hardware to perform operations comprising:

receiving audio data corresponding to an utterance captured by a user device;

processing, using a speaker identification model, the audio data to generate an evaluation attentive d-vector (ad-vector) representing voice characteristics of the utterance, the evaluation ad-vector comprising a first routing vector, the first routing vector comprising environmental or channel information associated with the audio data the evaluation ad-vector was generated from;

generating, using a self-attention mechanism, at least one multi-condition attention score that indicates a likelihood that the evaluation ad-vector matches a respective reference ad-vector associated with a respective user, the reference ad-vector comprising a second routing vector, the second routing vector comprising environmental or channel information associated with at least one reference utterance associated with the reference ad-vector; and

identifying the speaker of the utterance as the respective user associated with the respective reference ad-vector based on the multi-condition attention score,

wherein the first routing vector and the second routing vector are configured to identify matching conditions between the utterance associated with the evaluation ad-vector and the at least one reference utterance associated with the reference ad-vector to allow the self-attention mechanism to condition the reference ad-vector to the evaluation ad-vector when generating the at least one multi-condition attention score.

2 . The computer-implemented method of claim 1 , wherein identifying the speaker of the utterance comprises:

determining whether the multi-condition attention score satisfies a threshold score; and

when the multi-condition attention score satisfies the threshold score, determining that the speaker of the utterance includes the respective user associated with the respective reference ad-vector.

3 . The computer-implemented method of claim 1 , wherein:

generating the at least one multi-condition attention score comprises using the self-attention mechanisms to generate multiple multi-condition attention scores each indicating a respective likelihood that the evaluation ad-vector matches a respective one of multiple reference ad-vectors, each reference ad-vector is associated with a respective one of one or more enrolled users of the user device; and

identifying the speaker of the utterance comprises identifying the speaker of the utterance as the respective enrolled user of the user device that is associated with the respective reference ad-vector corresponding to the greatest multi-condition attention score.

4 . The computer-implemented method of claim 3 , wherein:

the utterance captured by the user device comprises a query specifying an action to perform;

each of the one or more different enrolled users of the user device has permissions for accessing a different respective set of personal resources; and

performance of the action specified by the query requires access to the respective set of personal resources associated with the respective enrolled user identified as the speaker of the utterance.

5 . The computer-implemented method of claim 1 , wherein processing the audio data to generate the reference ad-vector comprises:

generating, as output from the speaker identification model comprising a neural network, a non-attentive d-vector representing voice characteristics of the utterance; and

applying a set of linear and non-linear transformations to transform the non-attentive d-vector into the reference ad-vector.

6 . The computer-implemented method of claim 1 , wherein the reference ad-vector is generated by the speaker identification model in response to receiving one or more previous utterances spoken by the respective user.

7 . The computer-implemented method of claim 1 , wherein:

the evaluation ad-vector further comprises n e style classes each comprising a respective value vector concatenated with a corresponding first routing vector; and

the reference ad-vector further comprises n r style classes each comprising a respective value vector concatenated with a corresponding second routing vector.

8 . The computer-implemented method of claim 7 , wherein:

the speaker identification model comprises a neural network having an input layer, a plurality of hidden layers, and an output layer comprising multiple sets of output nodes, each set of output nodes in the n sets of output nodes of the output layer designated to learn to generate speaker-related information specific to a respective one of the n e style classes; and

processing the audio data to generate the evaluation ad-vector comprises using the neural network to process the audio data to generate each of the n e style classes for the evaluation ad-vector as output from the respective set of output nodes of the output layer that is designated to learn to generate the speaker-related information specific to the respective style class.

9 . The computer-implemented method of claim 7 , wherein each value vector comprises a same first dimensionality and each routing vector comprises a same second dimensionality that is less than the first dimensionality of each value vector.

10 . The computer-implemented method of claim 7 , wherein the reference ad-vector is generated by:

receiving, as input to the speaker identification model, m enrollment utterances spoken by the respective user;

for each enrollment utterance of the m enrollment utterances, generating, as output from the speaker identification model, a respective enrollment ad-vector having n e style classes; and

combining a superset of the n e style classes of the enrollment ad-vectors generated as output from the speaker identification model for the m enrollment utterances into the reference ad-vector.

11 . The computer-implemented method of claim 10 , wherein, when generating the at least one multi-condition attention score that indicates the likelihood that the evaluation ad-vector matches the respective reference ad-vector associated with the respective user, the self-attention mechanism automatically aligns the style classes among the evaluation ad-vector and the multiple reference ad-vectors.

12 . The computer-implemented method of claim 7 , wherein at least one style class of the n r and n e style classes is dependent on a fixed term or phrase.

13 . A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

receiving audio data corresponding to an utterance captured by a user device;

processing, using a speaker identification model, the audio data to generate an evaluation attentive d-vector (ad-vector) representing voice characteristics of the utterance, the evaluation ad-vector comprising a first routing vector, the first routing vector comprising environmental or channel information associated with the audio data the evaluation ad-vector was generated from;

generating, using a self-attention mechanism, at least one multi-condition attention score that indicates a likelihood that the evaluation ad-vector matches a respective reference ad-vector associated with a respective user, the reference ad-vector comprising a second routing vector, the second routing vector comprising environmental or channel information associated with at least one reference utterance associated with the reference ad-vector; and

identifying the speaker of the utterance as the respective user associated with the respective reference ad-vector based on the multi-condition attention score,

wherein the first routing vector and the second routing vector are configured to identify matching conditions between the utterance associated with the evaluation ad-vector and the at least one reference utterance associated with the reference ad-vector to allow the self-attention mechanism to condition the reference ad-vector to the evaluation ad-vector when generating the at least one multi-condition attention score.

14 . The system of claim 13 , wherein identifying the speaker of the utterance comprises:

determining whether the multi-condition attention score satisfies a threshold score; and

when the multi-condition attention score satisfies the threshold score, determining that the speaker of the utterance includes the respective user associated with the respective reference ad-vector.

15 . The system of claim 13 , wherein:

generating the at least one multi-condition attention score comprises using the self-attention mechanisms to generate multiple multi-condition attention scores each indicating a respective likelihood that the evaluation ad-vector matches a respective one of multiple reference ad-vectors, each reference ad-vector is associated with a respective one of one or more enrolled users of the user device; and

identifying the speaker of the utterance comprises identifying the speaker of the utterance as the respective enrolled user of the user device that is associated with the respective reference ad-vector corresponding to the greatest multi-condition attention score.

16 . The system of claim 15 , wherein:

the utterance captured by the user device comprises a query specifying an action to perform;

each of the one or more different enrolled users of the user device has permissions for accessing a different respective set of personal resources; and

performance of the action specified by the query requires access to the respective set of personal resources associated with the respective enrolled user identified as the speaker of the utterance.

17 . The system of claim 13 , wherein processing the audio data to generate the reference ad-vector comprises:

generating, as output from the speaker identification model comprising a neural network, a non-attentive d-vector representing voice characteristics of the utterance; and

applying a set of linear and non-linear transformations to transform the non-attentive d-vector into the reference ad-vector.

18 . The system of claim 13 , wherein the reference ad-vector is generated by the speaker identification model in response to receiving one or more previous utterances spoken by the respective user.

19 . The system of claim 13 , wherein:

the evaluation ad-vector further comprises n e style classes each comprising a respective value vector concatenated with a corresponding first routing vector; and

the reference ad-vector further comprises n r style classes each comprising a respective value vector concatenated with a corresponding second routing vector.

20 . The system of claim 19 , wherein:

the speaker identification model comprises a neural network having an input layer, a plurality of hidden layers, and an output layer comprising multiple sets of output nodes, each set of output nodes in the n sets of output nodes of the output layer designated to learn to generate speaker-related information specific to a respective one of the n e style classes; and

processing the audio data to generate the evaluation ad-vector comprises using the neural network to process the audio data to generate each of the n e style classes for the evaluation ad-vector as output from the respective set of output nodes of the output layer that is designated to learn to generate the speaker-related information specific to the respective style class.

21 . The system of claim 19 , wherein each value vector comprises a same first dimensionality and each routing vector comprises a same second dimensionality that is less than the first dimensionality of each value vector.

22 . The system of claim 19 , wherein the reference ad-vector is generated by:

receiving, as input to the speaker identification model, m enrollment utterances spoken by the respective user;

for each enrollment utterance of the m enrollment utterances, generating, as output from the speaker identification model, a respective enrollment ad-vector having n e style classes; and

combining a superset of the n e style classes of the enrollment ad-vectors generated as output from the speaker identification model for the m enrollment utterances into the reference ad-vector.

23 . The system of claim 22 , wherein, when generating the at least one multi-condition attention score that indicates the likelihood that the evaluation ad-vector matches the respective reference ad-vector associated with the respective user, the self-attention mechanism automatically aligns the style classes among the evaluation ad-vector and the multiple reference ad-vectors.

24 . The system of claim 19 , wherein at least one style class of the n r and n e style classes is dependent on a fixed term or phrase.