Representation learning using multi-task deep neural networks

1. A system comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, configure the one or more processors to perform operations comprising:

receiving linguistic items, each comprising a query or a document and being associated with one of multiple disparate tasks, at least one of the linguistic items being associated with a first one of the tasks and at least one of the linguistic items being associated with a second one of the tasks;

mapping an initial input representation of the linguistic items into lower-dimensional task-specific output representations using a multi-task deep neural network comprising a plurality of operational layers and transitions between the operational layers,

each operational layer corresponding to a respective representation of the linguistic items and each transition between two adjacent layers mapping a higher-dimensional representation of the linguistic items at one of the adjacent layers to a lower-dimensional representation of the linguistic items at the other one of the adjacent layers,

at least one of the operational layers corresponding to a representation of the linguistic items that is shared among the multiple disparate tasks and results from a shared transition to that layer, and

at least a last one of the operational layers corresponding to the task-specific output representations, which are unshared among the multiple disparate tasks and result from respective unshared, task-specific transitions to the last layer; and

using the lower-dimensional task-specific output representations of the linguistic items to perform the respective tasks associated therewith.

2. The system of claim 1 , wherein at least two of the operational layers each correspond to a representation of the linguistic items that is shared among the multiple disparate tasks.

3. The system of claim 1 , wherein the first and second disparate tasks include query classifying and web searching.

4. The system of claim 1 , wherein a first one of the operational layers, corresponding to the initial input representation of the linguistic items, represents each linguistic item as a bag of words.

5. The system of claim 1 , wherein the at least one operational layer corresponding to a shared representation comprises an operational layer representing the linguistic items as letter n-grams, and wherein the transition to that layer comprises hashing words of the linguistic items into the n-grams.

6. The system of claim 5 , wherein the at least one operational layer corresponding to a shared representation further comprises an operational layer corresponding to semantic representations of the n-grams.

7. The system of claim 1 , wherein the at least one operational layer comprises an operational layer corresponding to a semantic representation of the linguistic items.

8. A computing device comprising:

an input port to receive objects representing linguistic information, each object being associated with one of multiple disparate tasks, at least one of the objects being associated with a first one of the tasks and at least one of the objects being associated with a second one of the tasks; and

a processor and memory storing instructions that, when executed by the processor, configure the processor to:

represent each of the objects as an n-dimensional representation at a shared operational layer of a multi-task deep neural network comprising a plurality of operational layers, the n-dimensional representation being shared between or among the first and second disparate tasks;

transform the n-dimensional representations into respective task-specific m-dimensional representations at an unshared operational layer of the multi-task deep neural network, wherein m is less than n, and wherein the n-dimensional representation of the at least one object associated with the first task is specific to the first task and the n-dimensional representation of the at least one object associated with the second task is specific to the second task; and

use the task-specific m-dimensional representations to perform the respective disparate tasks.

9. The computing device of claim 8 , wherein the processor is configured to:

transform the n-dimensional representations into the task-specific m-dimensional representations based, at least in part, on machine learning.

10. The computing device of claim 8 , wherein the n-dimensional representations comprise bag-of-words or one or more one-hot vectors.

11. The computing device of claim 8 , wherein the first and second disparate tasks include query classifying and web searching.

12. The computing device of claim 8 , wherein the linguistic information comprises a query or a document.

13. A method comprising:

receiving a first linguistic object and a second linguistic object, the first and second linguistic objects being associated with respective first and second disparate tasks;

transforming the first linguistic object to a first task-specific representation using a first plurality of operations comprising at least one operation shared among the first and second disparate tasks and at least one unshared operation specific to the first task;

transforming the second linguistic object to a second task-specific representation using a second plurality of operations comprising the at least one operation shared among the first and second disparate tasks and at least one unshared operation specific to the second task; and

using the first and second semantic representations to perform the first and second disparate tasks, respectively.

14. The method of claim 13 ,

wherein the at least one operation shared among the first and second disparate tasks generates first and second semantic representations of the first and second linguistic objects, respectively;

wherein the at least one unshared operation specific to the first tasks transforms the first semantic representation to the first task-specific representation; and

wherein the at least one unshared operation specific to the second task transforms the second semantic representation to the second task-specific representation.

15. The method of claim 13 , wherein the first linguistic object and the second linguistic object are each one-hot vectors based, at least in part, on a language vocabulary, and wherein the first task-specific representation and the second task-specific representation each comprise a vector having dimensionality less than about 150.

16. The method of claim 14 , wherein transforming the first and second linguistic objects to the respective first and second semantic representations comprises generating n-grams from bags-of-words.

17. The method of claim 13 , wherein the first and second pluralities of operations are based, at least in part, on multi-task deep neural networks.