Multimodal agent for efficient image-text interface automation
A system for image-text agentic interface automation is disclosed. A multimodal agent is configured to process arbitrary-length text sequences and arbitrary-resolution images. A newline insertion logic is configured to interleave a newline character between successive lines of image patches in a plurality of lines of image patches, wherein the newline character specifies an end of a line in an input image. A tokenization logic is configured to translate the input text sequence into a sequence of input text tokens, and to translate the successive lines of image patches interleaved with the newline character into a sequence of input image tokens. A linear projection logic is configured to linearly project a single token stream of the sequence of input text tokens and the sequence of input image tokens into a decoder-only Transformer logic, wherein the linear projection of the single token stream bypasses any embedding lookup.
1. A system for image-text agentic interface automation, comprising:
a multimodal agent configured to process arbitrary-length text sequences and arbitrary-resolution images:
memory storing an input image and an input text sequence;
patch extraction logic configured to extract image patches from the input image on a line-by-line basis, and generate a plurality of lines of image patches for the input image;
newline insertion logic configured to interleave a newline character between successive lines of image patches in the plurality of lines of image patches, wherein the newline character specifies an end of a line in the input image;
tokenization logic configured to translate the input text sequence into a sequence of input text tokens, and to translate the successive lines of image patches interleaved with the newline character into a sequence of input image tokens;
linear projection logic configured to linearly project a single token stream of the sequence of input text tokens and the sequence of input image tokens into a decoder-only Transformer logic, wherein the linear projection of the single token stream bypasses any embedding lookup; and
the decoder-only Transformer logic configured to process the linearly projected, embedding lookup-bypassed single token stream to generate a sequence of output tokens that are responsive to the input image and the input text sequence.
2. The system of claim 1 , wherein the line in the input image is a row of image patches.
3. The system of claim 1 , wherein the line in the input image is a column of image patches.
4. The system of claim 1 , wherein the successive lines of image patches are arranged in a raster-scan order.
5. The system of claim 1 , wherein the decoder-only Transformer logic is further configured without any image-specific position embeddings.
6. The system of claim 5 , wherein the decoder-only Transformer logic is further configured to be trained on images of arbitrary size at training time, thereby obviating separate high and low-resolution training stages.
7. The system of claim 1 , wherein the decoder-only Transformer logic is further configured without a pooling logic.
8. The system of claim 1 , wherein the decoder-only Transformer logic is further configured without a causal attention logic.
9. The system of claim 1 , wherein the decoder-only Transformer logic is further configured to decouple input embeddings from output embeddings.
10. The system of claim 1 , wherein the decoder-only Transformer logic is further configured to use a squared rectified linear unit (ReLU) activation function.
11. The system of claim 1 , wherein the decoder-only Transformer logic is further configured to use a rotary positional embedding (RoPE).
12. The system of claim 1 , wherein the decoder-only Transformer logic is further configured to add a layer normalization (LayerNorm) function to Query (Q) and Key (K) embeddings before the Q and K embeddings enter attention calculations.
13. A system for image-text agentic interface automation, comprising:
a multimodal agent configured to process arbitrary-resolution images:
memory storing an input image;
patch extraction logic configured to extract image patches from the input image on a line-by-line basis, and generate a plurality of lines of image patches for the input image;
newline insertion logic configured to interleave a newline character between successive lines of image patches in the plurality of lines of image patches, wherein the newline character specifies an end of a line in the input image;
tokenization logic configured to translate the successive lines of image patches interleaved with the newline character into a sequence of input image tokens;
linear projection logic configured to linearly project the sequence of input image tokens into a decoder-only Transformer logic, wherein the linear projection of the sequence of input image tokens bypasses any embedding lookup; and
the decoder-only Transformer logic configured to process the linearly projected, embedding lookup-bypassed sequence of input image tokens to generate a sequence of output tokens that are responsive to the input image.
14. The system of claim 13 , wherein the line in the input image is a row of image patches.
15. The system of claim 13 , wherein the line in the input image is a column of image patches.
16. The system of claim 13 , wherein the decoder-only Transformer logic is further configured without any image-specific position embeddings.
17. The system of claim 16 , wherein the decoder-only Transformer logic is further configured to be trained on images of arbitrary size at training time, thereby obviating separate high and low-resolution training stages.
18. A computer-implemented method for image-text agentic interface automation, including:
storing an input image;
extracting image patches from the input image on a line-by-line basis, and generating a plurality of lines of image patches for the input image;
interleaving a newline character between successive lines of image patches in the plurality of lines of image patches, wherein the newline character specifies an end of a line in the input image;
translating the successive lines of image patches interleaved with the newline character into a sequence of input image tokens;
linearly projecting the sequence of input image tokens into a decoder-only Transformer logic, wherein the linear projection of the sequence of input image tokens bypasses any embedding lookup; and
processing the linearly projected, embedding lookup-bypassed sequence of input image tokens through the decoder-only Transformer logic to generate a sequence of output tokens that are responsive to the input image.