Cross-attention between sparse external features and contextual word embeddings to improve text classification

1. A method comprising:

obtaining a dense representation of external features, said dense representation of external features being obtained by multiplying a sparse representation of said external features with a first dense matrix;

obtaining, by a natural language understanding (NLU) application, a dense representation of text, said dense representation of text being obtained by multiplying a sparse representation of said text with a second dense matrix;

obtaining a mask that associates said external features to tokens of said text, the mask being a sparse matrix that represents which external feature is associated with which token of said text;

performing, by a cross-attention process that utilizes said mask, an information fusion of said dense representation of said external features and said tokens of said text;

based on said information fusion, generating, by the NLU application, a joint representation of said external features and said tokens of said text; and

performing, by the NLU application, named entity recognition using the joint representation.

2. The method of claim 1 , also comprising:

performing a classification operation that processes said joint representation to produce annotated text.

3. The method of claim 2 , wherein said classification operation utilizes a self-attention layer to produce a contextual joint representation of said joint representation.

4. The method of claim 3 , wherein said classification operation:

normalizes said contextual joint representation to generate a normalized contextual joint representation,

employs a classification layer that transforms said normalized contextual joint representation into a discrete probability density distribution, and

utilizes said discrete probability density distribution to produce said annotated text.

5. The method of claim 1 , also comprising:

calculating cross-attention scores for each pair of said tokens and said external features; and

setting, using the mask, a cross-attention score to zero between a token-external feature pair not represented in said text.

6. The method of claim 1 , also comprising comparing token units of text to different categories of external features to generate said mask.

7. The method of claim 1 , also comprising:

prior to performing said information fusion, employing a normalization function on said dense representation of said external features to generate normalized data,

wherein said cross-attention process performs said information fusion on said normalized data.

8. The method of claim 1 , also comprising:

performing a sum and normalize operation of said joint representation and said tokens of text to generate a normalized joint representation.

9. A system comprising:

a processor; and

a memory that contains instructions that are readable by said processor to cause said processor to perform operations of:

obtaining a dense representation of external features, said dense representation of external features being obtained by multiplying a sparse representation of said external features with a first dense matrix;

obtaining, by a natural language understanding (NLU) application, a dense representation of text, said dense representation of text being obtained by multiplying a sparse representation of said text with a second dense matrix;

obtaining a mask that associates said external features to tokens of said text, the mask being a sparse matrix that represents which external feature is associated with which token of said text;

performing, by a cross-attention process that utilizes said mask, an information fusion of said dense representation of said external features and said tokens of said text;

based on said information fusion, generating, by the NLU application, a joint representation of said external features and said tokens of said text; and

performing, by the NLU application, named entity recognition using the joint representation.

10. The system of claim 9 , wherein said operations also include:

performing a classification operation that processes said joint representation to produce annotated text.

11. The system of claim 10 , wherein said classification operation utilizes a self-attention layer to produce a contextual joint representation of said joint representation.

12. The system of claim 11 , wherein said classification operation:

normalizes said contextual joint representation to generate a normalized contextual joint representation,

employs a classification layer that transforms said normalized contextual joint representation into a discrete probability density distribution, and

utilizes said discrete probability density distribution to produce said annotated text.

13. The system of claim 9 , wherein said operations also include employing an external features embedding generator to transform external features into said dense representation of external features.

14. The system of claim 9 , wherein said operations also include comparing token units of text to different categories of external features to generate said mask.

15. The system of claim 9 , wherein said operations also include:

prior to performing said information fusion, employing a normalization function on said dense representation of said external features to generate normalized data,

wherein said cross-attention process performs said information fusion on said normalized data.

16. The system of claim 9 , wherein said operations also include:

receiving, by the NLU application, an utterance from a user; and

transforming the utterance into said text.

17. A storage device that is non-transitory, comprising instructions that are readable by a processor to cause said processor to perform operations of:

obtaining a dense representation of external features, said dense representation of external features being obtained by multiplying a sparse representation of said external features with a first dense matrix;

obtaining, by a natural language understanding (NLU) application, a dense representation of text, said dense representation of text being obtained by multiplying a sparse representation of said text with a second dense matrix;

obtaining a mask that associates said external features to tokens of said text, the mask being a sparse matrix that represents which external feature is associated with which token of said text;

performing, by a cross-attention process that utilizes said mask, an information fusion of said dense representation of said external features and said tokens of said text;

based on said information fusion, generating, by the NLU application, a joint representation of said external features and said tokens of said text; and

performing, by the NLU application, named entity recognition using the joint representation.

18. The storage device of claim 17 , wherein said operations also include:

performing a classification operation that processes said joint representation to produce annotated text.

19. The storage device of claim 18 , wherein said classification operation utilizes a self-attention layer to produce a contextual joint representation of said joint representation.

20. The storage device of claim 19 , wherein said classification operation:

normalizes said contextual joint representation to generate a normalized contextual joint representation,

employs a classification layer that transforms said normalized contextual joint representation into a discrete probability density distribution, and

utilizes said discrete probability density distribution to produce said annotated text.