Performing computing tasks using decoupled models for different data types
A technique executes tasks using a data store of machine-trained models. The data store specifically includes a subset of encoder-type machine-trained models for converting input data items having different input data types into respective embeddings in a vector space, and a subset of decoder-type machine-trained models for converting embeddings in the same vector space into data items having respective different output data types. When executing a particular task that involves one or more data types, the technique selects one or more machine-trained models that match those data types. In some implementations, the technique provides a clipboard store for storing embeddings produced by the encoder-type machine-trained models and consumable by the decoder-type machine-trained models. The technique includes provisions for ensuring that any decoder-type machine-model is capable of processing embeddings produced by different versions of the encoder-type machine-trained models.
1 . A computer-implemented method comprising:
receiving an input data item having a particular input data type;
executing at least part of a task that includes an instruction of pasting the input data item into a target item having a particular output data type that is different than the input data type, comprising:
executing a particular encoder-type machine-trained model to convert the input data item having the particular input data type to a particular embedding,
wherein the particular encoder-type machine-trained model is one of a subset of encoder-type machine-trained models that map input data items having different input data types to respective embeddings in a single shared vector space;
storing the particular embedding in a clipboard store;
executing a particular decoder-type machine-trained model that is associated with the particular output data type, to convert the particular embedding to an output data item of the particular output data type of the target item,
wherein the particular decoder-type machine-trained model is one of a subset of decoder-type machine-trained models that map the embeddings in the same single shared vector space to respective output data items having different output data types; and
pasting the output data item into the target item.
2 . The method of claim 1 , wherein the method is executed, at least in part, by a control system of a computing system.
3 . The method of claim 2 , wherein the control system also manages versions of the subset of encoder-type machine-trained models and the subset of decoder-type machine-trained models.
4 . The method of claim 1 , wherein the embeddings output by the encoder-type machine-trained models and input to the decoder-type machine-trained models are distributed-representation vectors mapped to the single shared vector space, and wherein a distance between any two vectors in the vector space reflects an extent of similarity between the two vectors.
5 . The method of claim 4 , wherein the encoder-type machine-trained models and the decoder-type machine-trained models have been trained to produce or consume embeddings in the single shared vector space.
6 . The method of claim 1 , wherein the clipboard store also includes a predecessor embedding produced by an earlier version of the particular encoder-type machine-trained model for the input data item, prior to generating the particular embedding.
7 . The method of claim 6 , wherein the particular embedding includes a base part that matches information in the predecessor embedding, and another part that includes information that is not present in the predecessor embedding.
8 . The method of claim 1 ,
wherein the method further includes generating output information for presentation in a user interface presentation that represents contents of the clipboard store, and
wherein the output information includes an image that conveys semantic contents of the particular embedding, for presentation in the user interface presentation together with a representation of the particular embedding.
9 . The method of claim 1 , wherein a given decoder-type machine-trained model in the subset of decoder-type machine-trained models operates on two or more input data items, the two or more input data items including a particular embedding that expresses semantic content of a particular input data item.
10 . The method of claim 9 , wherein another of the two or more input data items is an image mask that identifies a portion of the particular input data item.
11 . The method of claim 1 , further comprising:
generating a supplemental item in addition to the particular embedding; and
storing the supplemental item in the clipboard store along with the particular embedding.
12 . The method of claim 11 , wherein the supplemental item is an instance of randomly-generated noise information.
13 . The method of claim 11 , wherein the particular decoder-type machine-trained produces the output data item based on the particular embedding and the supplemental item.
14 . The method of claim 1 , wherein at least one of the decoder-type machine-trained models of the subset of decoder-type machine-trained models receives two or more input data items from the clipboard store, and maps the two more input data items into a single output data item.
15 . The method of claim 1 , further comprising storing the input data item in the clipboard store prior to producing the particular embedding.
16 . The method of claim 1 , wherein at least one entry in the clipboard store provides an identifier for an embedding and an indication of a source data item from which the embedding originated.
17 . A computing system, comprising:
a processing system for executing machine-readable instructions; and
a storage device for storing the machine-readable instructions,
the storage device also storing a set of machine-trained models, the set of machine-trained models including a subset of encoder-type machine-trained models that map input data items having different input data types to respective embeddings in a single shared vector space, and a subset of decoder-type machine-trained models that map the embeddings in the single shared vector space to respective output data items having different output data types, wherein any of the subset of encoder-type machine-trained models is capable of producing an embedding that is consumable by any of the subset of subset of decoder-type machine-trained models,
the processing system executing the machine-readable instructions for:
receiving an input data item having a particular input data type;
executing at least part of a task that includes an instruction of pasting the input data item into a target item having a particular output data type that is different than the input data type, comprising:
executing a particular encoder-type machine-trained model of the subset of decoder-type machine-trained models to convert the input data item having the particular input data type to a particular embedding;
storing the particular embedding in a clipboard store;
executing a particular decoder-type machine-trained model, of the subset of decoder-type machine-trained models, that is associated with the particular output data type, to convert the particular embedding to an output data item of the particular output data type of the target item; and
pasting the output data item into the target item.
18 . The computing system of claim 17 , wherein the embeddings output by the encoder-type machine-trained models and input to the decoder-type machine-trained models are distributed-representation vectors mapped to the single shared vector space, and wherein a distance between any two vectors in the vector space reflects an extent of similarity between the two vectors.
19 . A non-transitory computer-readable storage medium for storing computer-readable instructions, wherein a processing system executing the computer-readable instructions performs operations comprising:
receiving an input data item having a particular input data type;
executing at least part of a task that includes an instruction of pasting the input data item into a target item having a particular output data type that is different than the input data type, comprising:
executing a particular encoder-type machine-trained model to convert the input data item having the particular input data type to a particular embedding,
wherein the particular encoder-type machine-trained model is one of a subset of encoder-type machine-trained models stored in the computer-readable storage medium that map input data items having different input data types to respective embeddings in a single shared vector space;
storing the particular embedding in a clipboard store;
executing a particular decoder-type machine-trained model that is associated with the particular output data type, to convert the particular embedding to an output data item of the particular output data type of the target item,
wherein the particular decoder-type machine-trained model is one of a subset of decoder-type machine-trained models stored in the computer-readable storage medium that map the embeddings in the same single shared vector space to respective output data items having different output data types; and
pasting the output data item into the target item.
20 . The non-transitory computer-readable storage medium of claim 19 ,
wherein the clipboard store also includes a predecessor embedding produced by an earlier version of the particular encoder-type machine-trained model for the input data item, prior to generating the particular embedding, and
wherein the particular embedding includes a base part that matches information in the predecessor embedding, and another part that includes information that is not present in the predecessor embedding.