Multi-modal hierarchical tokenization deep neural network
A system is disclosed for encoding a data string of a first modality into a hierarchical tokenized representation for processing by a text-based deep neural network (DNN) trained on a second modality. The data string comprises multiple units, each having one or more attributes. Each attribute is represented in the tokenized string as a sequence of hierarchical tokens, with a first hierarchical token encoding one or more most significant bits and a subsequent hierarchical token encoding one or more less significant bits. The DNN processes the data string bidirectionally, across the sequence of units and within the token hierarchy, to select tokens that capture attribute information. The selected hierarchical tokens output by the DNN from a representation of the original data string that preserves attribute detail while enabling cross-modal processing using models trained on text.
1 . A multi-modal artificial intelligence (AI) system comprising:
an encoder configured to encode a data string into a tokenized string, the data string corresponding to a first modality, wherein the data string comprises a plurality of units, wherein each unit is encoded in the tokenized string as a plurality of hierarchical tokens representing one or more attributes of the unit, wherein a first hierarchical token in a hierarchy represents one or more most significant bits (MSBs) of an attribute of the unit and a second hierarchical token represents one or more less significant bits (LSBs) of the attribute; and
a computing system comprising memory and one or more processors, the memory storing executable instructions, wherein the instructions, when executed, cause the one or more processors to:
instantiate a text-based deep neural network (DNN), wherein text used to train the text-based DNN corresponds to a second modality;
receive the data string;
apply the text-based DNN to the data string to select the hierarchical tokens in the tokenized string, wherein selecting the hierarchical tokens is performed according to both a first direction of the data string and a second direction of the hierarchy of the hierarchical tokens from the MSBs to the LSBs; and
output, by the text-based DNN, selected hierarchical tokens as a representation of the data string, wherein the selected hierarchical tokens contain information of the attributes of the plurality of units in the data string.
2 . The system of claim 1 , wherein the text-based DNN is trained using the text in the first modality as training data and then trained using a plurality of data strings in the second modality.
3 . The system of claim 1 , wherein the text-based DNN is trained using a plurality of data strings in the second modality, and wherein training of the text-based DNN comprises:
determining, in forward propagation, a plurality of tokenized strings, the plurality of tokenized strings representing different configurations corresponding to the data string;
determining first differences in attribute values among the plurality of tokenized strings, wherein the first differences in the attribute values are determined through the forward propagation of the text-based DNN;
comparing the first differences with second differences obtained from ground truth; and
adjusting parameters of the text-based DNN based on comparing the first differences with the second differences.
4 . The system of claim 3 , wherein the first differences in the attribute values are determined at least partially through aggregating values obtained from a plurality of attention blocks in the text-based DNN.
5 . The system of claim 1 , wherein the attribute of the unit with the MSBs and the LSBs is a coordinate of the unit.
6 . The system of claim 5 , wherein encoding the data string into the tokenized string comprises:
tokenizing the data string to produce a tokenized structural expression;
tokenizing, for each unit in the data string, coordinates for the unit to produce tokenized coordinates for the unit, wherein the tokenized coordinates are in the hierarchy of hierarchical tokens; and
combining the tokenized structural expression and the tokenized coordinates for each unit to produce the tokenized string.
7 . The system of claim 6 , wherein tokenizing, for each unit in the data string, the coordinates for the unit comprises:
pixelating a rendered sphere to produce a set of pixels each corresponding to a location on a surface of the rendered sphere;
tokenizing, for a first unit in the data string, coordinates at a center of the sphere; and
tokenizing, for each additional units in the data string, coordinates corresponding to a pixel selected from the set of pixels based on a location of the additional unit relative to the center of the sphere.
8 . The system of claim 7 , wherein the hierarchy of the hierarchical tokens is relative to the center of the sphere.
9 . The system of claim 6 , wherein the coordinates are in a Cartesian coordinate system, an xyz coordinate system, an octree coordinate system, a polar coordinate system, a cylindrical coordinate system, or a barycentric coordinate system.
10 . The system of claim 1 , wherein selecting the hierarchical tokens is through inference of the text-based DNN based on aggregating activation outputs from attention blocks in the text-based DNN.
11 . A computer-implemented method, comprising:
encoding a data string into a tokenized string, the data string corresponding to a first modality, wherein the data string comprises a plurality of units, wherein each unit is encoded in the tokenized string as a plurality of hierarchical tokens representing one or more attributes of the unit, wherein a first hierarchical token in a hierarchy represents one or more most significant bits (MSBs) of an attribute of the unit and a second hierarchical token represents one or more less significant bits (LSBs) of the attribute
instantiating a text-based deep neural network (DNN), wherein text used to train the text-based DNN corresponds to a second modality;
applying the text-based DNN to the data string to select the hierarchical tokens in the tokenized string, wherein selecting the hierarchical tokens is performed according to both a first direction of the data string and a second direction of the hierarchy of the hierarchical tokens from the MSBs to the LSBs; and
outputting, by the text-based DNN, selected hierarchical tokens as a representation of the data string, wherein the selected hierarchical tokens contain information of the attributes of the plurality of units in the data string.
12 . The computer-implemented method of claim 11 , wherein the text-based DNN is trained using the text in the first modality as training data and then trained using a plurality of data strings in the second modality.
13 . The computer-implemented method of claim 11 , wherein the text-based DNN is trained using a plurality of data strings in the second modality, and wherein training of the text-based DNN comprises:
determining, in forward propagation, a plurality of tokenized strings, the plurality of tokenized strings representing different configurations corresponding to the data string;
determining first differences in attribute values among the plurality of tokenized strings, wherein the first differences in the attribute values are determined through the forward propagation of the text-based DNN;
comparing the first differences with second differences obtained from ground truth; and
adjusting parameters of the text-based DNN based on comparing the first differences with the second differences.
14 . The computer-implemented method of claim 13 , wherein the first differences in the attribute values are determined at least partially through aggregating values obtained from a plurality of attention blocks in the text-based DNN.
15 . The computer-implemented method of claim 11 , wherein the attribute of the unit with the MSBs and the LSBs is a coordinate of the unit.
16 . The computer-implemented method of claim 15 , wherein encoding the data string into the tokenized string comprises:
tokenizing the data string to produce a tokenized structural expression;
tokenizing, for each unit in the data string, coordinates for the unit to produce tokenized coordinates for the unit, wherein the tokenized coordinates are in the hierarchy of hierarchical tokens; and
combining the tokenized structural expression and the tokenized coordinates for each unit to produce the tokenized string.
17 . The computer-implemented method of claim 16 , wherein tokenizing, for each unit in the data string, the coordinates for the unit comprises:
pixelating a rendered sphere to produce a set of pixels each corresponding to a location on a surface of the rendered sphere;
tokenizing, for a first unit in the data string, coordinates at a center of the sphere; and
tokenizing, for each additional units in the data string, coordinates corresponding to a pixel selected from the set of pixels based on a location of the additional unit relative to the center of the sphere.
18 . The computer-implemented method of claim 17 , wherein the hierarchy of the hierarchical tokens is relative to the center of the sphere.
19 . The computer-implemented method of claim 16 , wherein the coordinates are in a Cartesian coordinate computer-implemented method, an xyz coordinate computer-implemented method, an octree coordinate computer-implemented method, a polar coordinate computer-implemented method, a cylindrical coordinate computer-implemented method, or a barycentric coordinate computer-implemented method.
20 . A non-transitory computer-readable medium configured to store code comprising instructions, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform steps comprising:
encoding a data string into a tokenized string, the data string corresponding to a first modality, wherein the data string comprises a plurality of units, wherein each unit is encoded in the tokenized string as a plurality of hierarchical tokens representing one or more attributes of the unit, wherein a first hierarchical token in a hierarchy represents one or more most significant bits (MSBs) of an attribute of the unit and a second hierarchical token represents one or more less significant bits (LSBs) of the attribute
instantiating a text-based deep neural network (DNN), wherein text used to train the text-based DNN corresponds to a second modality;
applying the text-based DNN to the data string to select the hierarchical tokens in the tokenized string, wherein selecting the hierarchical tokens is performed according to both a first direction of the data string and a second direction of the hierarchy of the hierarchical tokens from the MSBs to the LSBs; and
outputting, by the text-based DNN, selected hierarchical tokens as a representation of the data string, wherein the selected hierarchical tokens contain information of the attributes of the plurality of units in the data string.