Magnitude invariant multimodal agent for efficient image-text interface automation
A system for magnitude-invariant image-text agentic interface automation is disclosed. A bit vectorization logic is configured to convert image patches in a plurality of image patches into magnitude-invariant bit vectors, and generate a plurality of lines of magnitude-invariant bit vectors. A tokenization logic is configured to translate the input text sequence into a sequence of input text tokens, and to translate the successive lines of magnitude-invariant bit vectors interleaved with a newline character into a sequence of input magnitude-invariant bit vector tokens. A linear projection logic is configured to linearly project a single token stream of the sequence of input text tokens and the sequence of input magnitude-invariant bit vector tokens into a decoder-only Transformer logic, wherein the linear projection of the single token stream bypasses any embedding lookup.
1 . A system for magnitude-invariant image-text agentic interface automation, comprising:
memory storing an input image and an input text sequence;
patch extraction logic configured to extract image patches from the input image on a line-by-line basis, and generate a plurality of lines of image patches for the input image;
bit vectorization logic configured to convert image patches in the plurality of lines of image patches into magnitude-invariant bit vectors, and generate a plurality of lines of magnitude-invariant bit vectors;
newline insertion logic configured to interleave a newline character between successive lines of magnitude-invariant bit vectors in the plurality of lines of magnitude-invariant bit vectors, wherein the newline character specifies an end of a line in the input image;
tokenization logic configured to translate the input text sequence into a sequence of input text tokens, and to translate the successive lines of magnitude-invariant bit vectors interleaved with the newline character into a sequence of input magnitude-invariant bit vector tokens;
linear projection logic configured to linearly project a single token stream of the sequence of input text tokens and the sequence of input magnitude-invariant bit vector tokens into a decoder-only Transformer logic, wherein the linear projection of the single token stream bypasses any embedding lookup; and
the decoder-only Transformer logic configured to process the linearly projected, embedding lookup-bypassed single token stream to generate a sequence of output tokens that are responsive to the input image and the input text sequence.
2 . The system of claim 1 , wherein the bit vectorization logic is further configured to apply a RGB555 format compression to convert the image patches in the plurality of lines of image patches into the magnitude-invariant bit vectors, and generate the plurality of lines of magnitude-invariant bit vectors.
3 . The system of claim 2 , wherein the RGB555 format compression produces three 5-bit values, one for each of subpixel channels R (red), G (green), and B (blue).
4 . The system of claim 3 , wherein the three 5-bit values take either a 1 value or a −1 value.
5 . The system of claim 4 , wherein the three 5-bit values are magnitude-invariant to scale modification functions of the decoder-only Transformer logic.
6 . The system of claim 5 , wherein a layer normalization (LayerNorm) function is one of the scaling functions of the decoder-only Transformer logic.
7 . The system of claim 1 , wherein the bit vectorization logic is further configured to apply a RGB888 format compression to convert the image patches in the plurality of lines of image patches into the magnitude-invariant bit vectors, and generate the plurality of lines of magnitude-invariant bit vectors.
8 . The system of claim 7 , wherein the RGB888 format compression produces three 8-bit values, one for each of subpixel channels R (red), G (green), and B (blue).
9 . The system of claim 8 , wherein the three 8-bit values take either a 1 value or a −1 value.
10 . The system of claim 9 , wherein the three 8-bit values are magnitude-invariant to scale modification functions of the decoder-only Transformer logic.
11 . The system of claim 10 , wherein a layer normalization (LayerNorm) function is one of the scaling functions of the decoder-only Transformer logic.
12 . The system of claim 1 , wherein the bit vectorization logic is further configured to apply a RGB565 format compression to convert the image patches in the plurality of lines of image patches into the magnitude-invariant bit vectors, and generate the plurality of lines of magnitude-invariant bit vectors.
13 . The system of claim 12 , wherein the RGB565 format compression produces 5-bit values for R (red) and B (blue) subpixel channels and 6-bit values for G (green) subpixel channel.
14 . The system of claim 13 , wherein the 5-bit and the 6-bit values take either a 1 value or a −1 value.
15 . The system of claim 14 , wherein the 5-bit and the 6-bit values are magnitude-invariant to scale modification functions of the decoder-only Transformer logic.
16 . The system of claim 15 , wherein a layer normalization (LayerNorm) function is one of the scaling functions of the decoder-only Transformer logic.
17 . A system for magnitude-invariant image-text agentic interface automation, comprising:
memory storing an input image;
patch extraction logic configured to extract image patches from the input image on a line-by-line basis, and generate a plurality of lines of image patches for the input image;
bit vectorization logic configured to convert image patches in the plurality of lines of image patches into magnitude-invariant bit vectors, and generate a plurality of lines of magnitude-invariant bit vectors;
newline insertion logic configured to interleave a newline character between successive lines of magnitude-invariant bit vectors in the plurality of lines of magnitude-invariant bit vectors, wherein the newline character specifies an end of a line in the input image;
tokenization logic configured to translate the successive lines of magnitude-invariant bit vectors interleaved with the newline character into a sequence of input magnitude-invariant bit vector tokens;
linear projection logic configured to linearly project the sequence of input magnitude-invariant bit vector tokens into a decoder-only Transformer logic, wherein the linear projection of the sequence of input magnitude-invariant bit vector tokens bypasses any embedding lookup; and
the decoder-only Transformer logic configured to process the linearly projected, embedding lookup-bypassed sequence of input magnitude-invariant bit vector tokens to generate a sequence of output tokens that are responsive to the input image.
18 . A system for magnitude-invariant image-text agentic interface automation, comprising:
memory storing an input image;
patch extraction logic configured to extract image patches from the input image on a line-by-line basis, and generate a plurality of lines of image patches for the input image;
bit vectorization logic configured to convert image patches in the plurality of lines of image patches into magnitude-invariant bit vectors, and generate a plurality of lines of magnitude-invariant bit vectors;
tokenization logic configured to translate lines of the plurality of lines of magnitude-invariant bit vectors into a sequence of input magnitude-invariant bit vector tokens;
linear projection logic configured to linearly project the sequence of input magnitude-invariant bit vector tokens into a decoder-only Transformer logic; and
the decoder-only Transformer logic configured to process the linearly projected sequence of input magnitude-invariant bit vector tokens to generate a sequence of output tokens that are responsive to the input image.