Domain adaptation of AI NLP encoders with knowledge distillation

1. A computer-implemented method comprising:

accessing natural language processing (NLP) general encoder data, wherein the NLP general encoder data is generated from a general-domain dataset that is not domain specific;

accessing a domain specific dataset;

generating filtered encoder data using a subset of the NLP general encoder data;

training the filtered encoder data using the domain specific dataset to generate distilled encoder data; and

generating tuning values for the distilled encoder data to configure task outputs associated with the domain specific dataset.

2. The computer-implemented method of claim 1 , wherein the domain specific dataset is unlabeled data.

3. The computer-implemented method of claim 1 , wherein the distilled encoder data is configured to identify categories associated with labels of the domain specific dataset.

4. The computer-implemented method of claim 1 , wherein the domain specific dataset includes unlabeled data.

5. The computer-implemented method of claim 1 , wherein the distilled encoder data is configured to identify real-time data in a two-way communication associated with an actionable request.

6. The computer-implemented method of claim 1 , wherein training the filtered encoder data using the domain specific dataset includes performing dynamic masking with randomly selected words in the domain specific dataset in sufficient quantity to leverage gradient accumulation.

7. The computer-implemented method of claim 1 , wherein training the filtered encoder data using the domain specific dataset includes training the filtered encoder data using a masked language model.

8. The computer-implemented method of claim 1 , wherein the domain specific dataset is smaller than a threshold size, and wherein the threshold size is smaller than the general-domain dataset.

9. The computer-implemented method of claim 1 , wherein the general-domain dataset is distinct from the domain specific dataset.

10. The computer-implemented method of claim 1 further comprising:

accessing the general-domain dataset used to generate the NLP general encoder data; and

generating the domain specific dataset as a subset of the general-domain dataset.

11. The computer-implemented method of claim 1 further comprising:

accessing the distilled encoder data;

accessing a real-time stream of domain specific data; and

processing the real-time stream of domain specific data using the distilled encoder data to generate task output data.

12. A device comprising:

a memory storing instructions; and

one or more processors coupled to the memory, the one or more processors configured by the instructions to perform operations comprising:

accessing natural language processing (NLP) general encoder data, wherein the NLP general encoder data is generated from a general-domain dataset that is not domain specific;

accessing a domain specific dataset;

generating filtered encoder data using a subset of the NLP general encoder data;

training the filtered encoder data using the domain specific dataset to generate distilled encoder data; and

generating tuning values for the distilled encoder data to configure task outputs associated with the domain specific dataset.

13. The device of claim 12 , wherein the domain specific dataset is unlabeled data.

14. The device of claim 12 , wherein the distilled encoder data is configured to identify categories associated with labels of the domain specific dataset.

15. The device of claim 12 , wherein the domain specific dataset includes unlabeled data.

16. The device of claim 12 , wherein the distilled encoder data is configured to identify real-time data in a two-way communication associated with an actionable request.

17. The device of claim 12 , wherein training the filtered encoder data using the domain specific dataset includes performing dynamic masking with randomly selected words in the domain specific dataset in sufficient quantity to leverage gradient accumulation.

18. The device of claim 12 , wherein training the filtered encoder data using the domain specific dataset includes training the filtered encoder data using a masked language model.

19. The device of claim 12 , wherein the domain specific dataset is smaller than a threshold size, and wherein the threshold size is smaller than the general-domain dataset.

20. The device of claim 12 , wherein the general-domain dataset is distinct from the domain specific dataset.

21. The device of claim 12 , wherein the one or more processors are configured for operations further comprising:

accessing the general-domain dataset used to generate the NLP general encoder data; and

generating the domain specific dataset as a subset of the general-domain dataset.

22. The device of claim 12 , wherein the one or more processors are configured for operations further comprising:

accessing the distilled encoder data;

accessing a real-time stream of domain specific data; and

processing the real-time stream of domain specific data using the distilled encoder data to generate task output data.

23. A non-transitory computer readable storage medium comprising instructions that, when executed by one or more processors of a device, cause the device to perform operations comprising:

accessing natural language processing (NLP) general encoder data, wherein the NLP general encoder data is generated from a general-domain dataset that is not domain specific;

accessing a domain specific dataset;

generating filtered encoder data using a subset of the NLP general encoder data;

training the filtered encoder data using the domain specific dataset to generate distilled encoder data; and

generating tuning values for the distilled encoder data to configure task outputs associated with the domain specific dataset.

24. The non-transitory computer readable storage medium of claim 23 , wherein the domain specific dataset is unlabeled data.

25. The non-transitory computer readable storage medium of claim 23 , wherein the distilled encoder data is configured to identify categories associated with labels of the domain specific dataset.

26. The non-transitory computer readable storage medium of claim 23 , wherein the domain specific dataset includes unlabeled data.

27. The non-transitory computer readable storage medium of claim 23 , wherein the distilled encoder data is configured to identify real-time data in a two-way communication associated with an actionable request.

28. The non-transitory computer readable storage medium of claim 23 , wherein training the filtered encoder data using the domain specific dataset includes performing dynamic masking with randomly selected words in the domain specific dataset in sufficient quantity to leverage gradient accumulation.

29. The non-transitory computer readable storage medium of claim 23 , training the filtered encoder data using the domain specific dataset includes training the filtered encoder data using a masked language model.

30. The non-transitory computer readable storage medium of claim 23 , wherein the domain specific dataset is smaller than a threshold size, and wherein the threshold size is smaller than the general-domain dataset.

31. The non-transitory computer readable storage medium of claim 23 , wherein the general-domain dataset is distinct from the domain specific dataset.

32. The non-transitory computer readable storage medium of claim 23 , wherein the instructions further configure the device for operations comprising:

accessing the general-domain dataset used to generate the NLP general encoder data; and

generating the domain specific dataset as a subset of the general-domain dataset.

33. The non-transitory computer readable storage medium of claim 23 , wherein the instructions further configure the device for operations comprising:

accessing the distilled encoder data;

accessing a real-time stream of domain specific data; and

processing the real-time stream of domain specific data using the distilled encoder data to generate task output data.

34. A computer-implemented method comprising:

accessing encoder data, wherein the encoder data is generated from a general-domain dataset that is not domain specific;

accessing a domain specific dataset;

generating filtered encoder data using a subset of the encoder data;

training the filtered encoder data using the domain specific dataset to generate distilled encoder data; and

generating tuning values for the distilled encoder data to configure task outputs associated with the domain specific dataset.

35. The computer-implemented method of claim 34 , wherein the domain specific dataset is unlabeled data.

36. The computer-implemented method of claim 34 , wherein the distilled encoder data is configured to identify categories associated with labels of the domain specific dataset.

37. The computer-implemented method of claim 34 , wherein the domain specific dataset includes unlabeled data.

38. The computer-implemented method of claim 34 , wherein the distilled encoder data is configured to identify real-time data in a two-way communication associated with an actionable request.

39. The computer-implemented method of claim 34 , wherein training the filtered encoder data using the domain specific dataset includes performing dynamic masking with randomly selected words in the domain specific dataset in sufficient quantity to leverage gradient accumulation.

40. The computer-implemented method of claim 34 , wherein training the filtered encoder data using the domain specific dataset includes training the filtered encoder data using a masked language model.

41. The computer-implemented method of claim 34 , wherein the domain specific dataset is smaller than a threshold size, and wherein the threshold size is smaller than the general-domain dataset.

42. The computer-implemented method of claim 34 , wherein the general-domain dataset is distinct from the domain specific dataset.

43. The computer-implemented method of claim 34 further comprising:

accessing the general-domain dataset used to generate the encoder data; and

generating the domain specific dataset as a subset of the general-domain dataset.

44. The computer-implemented method of claim 34 further comprising:

accessing the distilled encoder data;

accessing a real-time stream of domain specific data; and

processing the real-time stream of domain specific data using the distilled encoder data to generate task output data.

45. A device, comprising:

a memory; and

one or more processors coupled to the memory and configured to perform operations comprising:

accessing encoder data, wherein the encoder data is generated from a general-domain dataset that is not domain specific;

accessing a domain specific dataset;

generating filtered encoder data using a subset of the encoder data;

training the filtered encoder data using the domain specific dataset to generate distilled encoder data; and

generating tuning values for the distilled encoder data to configure task outputs associated with the domain specific dataset.

46. The device of claim 45 , wherein the domain specific dataset is unlabeled data.

47. The device of claim 45 , wherein the distilled encoder data is configured to identify categories associated with labels of the domain specific dataset.

48. The device of claim 45 , wherein the domain specific dataset includes unlabeled data.

49. The device of claim 45 , wherein the distilled encoder data is configured to identify real-time data in a two-way communication associated with an actionable request.

50. The device of claim 45 , wherein training the filtered encoder data using the domain specific dataset includes performing dynamic masking with randomly selected words in the domain specific dataset in sufficient quantity to leverage gradient accumulation.

51. The device of claim 45 , wherein training the filtered encoder data using the domain specific dataset includes training the filtered encoder data using a masked language model.

52. The device of claim 45 , wherein the domain specific dataset is smaller than a threshold size, and wherein the threshold size is smaller than the general-domain dataset.

53. The device of claim 45 , wherein the general-domain dataset is distinct from the domain specific dataset.

54. The device of claim 45 , wherein the one or more processors are further configured for operations comprising:

accessing the general-domain dataset used to generate the encoder data; and

generating the domain specific dataset as a subset of the general-domain dataset.

55. The device of claim 45 , wherein the one or more processors are further configured for operations comprising:

accessing the distilled encoder data;

accessing a real-time stream of domain specific data; and

processing the real-time stream of domain specific data using the distilled encoder data to generate task output data.

56. A non-transitory computer readable storage medium comprising instructions that, when executed by one or more processors of a device, cause the device to perform operations comprising:

accessing encoder data, wherein the encoder data is generated from a general-domain dataset that is not domain specific;

accessing a domain specific dataset;

generating filtered encoder data using a subset of the encoder data;

training the filtered encoder data using the domain specific dataset to generate distilled encoder data; and

generating tuning values for the distilled encoder data to configure task outputs associated with the domain specific dataset.

57. The non-transitory computer readable storage medium of claim 56 , wherein the domain specific dataset is unlabeled data.

58. The non-transitory computer readable storage medium of claim 56 , wherein the distilled encoder data is configured to identify categories associated with labels of the domain specific dataset.

59. The non-transitory computer readable storage medium of claim 56 , wherein the domain specific dataset includes unlabeled data.

60. The non-transitory computer readable storage medium of claim 56 , wherein the distilled encoder data is configured to identify real-time data in a two-way communication associated with an actionable request.

61. The non-transitory computer readable storage medium of claim 56 , wherein training the filtered encoder data using the domain specific dataset includes performing dynamic masking with randomly selected words in the domain specific dataset in sufficient quantity to leverage gradient accumulation.

62. The non-transitory computer readable storage medium of claim 56 , wherein training the filtered encoder data using the domain specific dataset includes training the filtered encoder data using a masked language model.

63. The non-transitory computer readable storage medium of claim 56 , wherein the domain specific dataset is smaller than a threshold size, and wherein the threshold size is smaller than the general-domain dataset.

64. The non-transitory computer readable storage medium of claim 56 , wherein the general-domain dataset is distinct from the domain specific dataset.

65. The non-transitory computer readable storage medium of claim 56 , wherein the instructions further configure the device for operations comprising:

accessing the general-domain dataset used to generate the encoder data; and

generating the domain specific dataset as a subset of the general-domain dataset.

66. The non-transitory computer readable storage medium of claim 56 , further comprising:

accessing the distilled encoder data;

accessing a real-time stream of domain specific data; and

processing the real-time stream of domain specific data using the distilled encoder data to generate task output data.