Hybrid model for vision systems
Methods and systems for generating a feature map from an image is disclosed. The vision system includes a vision model or processing the image to generate the feature map according a neural network. The vision model includes a first convolutional block for downsampling a set of image data to obtain a first stage convoluted data; a second convolutional block for downsampling the first stage convoluted data to obtain a second stage convoluted data, wherein one or both of the first convolutional block and the second convolutional block is a mobile convolution block (MBConv) that includes: a first Gaussian Error Linear Unit (GELU) layer, a depth-wise convolution (DWConv) layer having, and a resizing convolution layer; and a transformer block (TFB) generating the feature map from the second stage convoluted data.
1 . A vision system for generating a feature map from an image, the vision system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:
a vision model that processes the image to generate the feature map implemented on a neural network, wherein the vision model comprises:
a first convolutional block that downsamples a set of image data to obtain a first stage convoluted data;
a second convolutional block that downsamples the first stage convoluted data to obtain a second stage convoluted data, wherein
one or both of the first convolutional block and the second convolutional block is a mobile convolution (MBConv) block that includes: a Gaussian Error Linear Unit (GELU) convolutional layer, a GELU depth-wise convolutional (DWConv) layer, and a resizing convolutional layer, wherein the GELU convolutional layer precedes the GELU DWConv layer in the MBConv block; and
a transformer block (TFB) generating the feature map from the second stage convoluted data.
2 . The vision system of claim 1 , wherein the GELU convolutional layer, having a first kernel size and a first channel size, expands the first channel size to a second channel size.
3 . The vision system of claim 2 , wherein the resizing convolutional layer returns from the second channel size to the first channel size.
4 . The vision system of claim 1 , wherein the GELU DWConv layer, having a second kernel size, captures spatial interaction.
5 . The vision system of claim 1 , wherein the vision model further comprises a stem convolutional block having two identical kernel sized convolutional layers, the stem convolutional block processing the image to obtain stemmed image data that are provided as the set of image data to the first convolutional block.
6 . The vision system of claim 1 , wherein a number of blocks and a number of channels progressively increase from the first convolutional block to the second convolutional block and to the TFB.
7 . The vision system of claim 1 , wherein the TFB comprises a self-attention (SA) residual block and a feed-forward network (FFN) residual block.
8 . The vision system of claim 7 , wherein a first layer in the SA residual block includes a layer normalization (LN) layer.
9 . The vision system of claim 7 , wherein a first layer in the FFN residual block is a layer normalization (LN) layer.
10 . The vision system of claim 7 , wherein an output of a layer normalization (LN) layer of the FFN is provided to both a first linear layer and a second linear layer, and an output of the second linear layer is processed by a second GELU layer.
11 . The vision system of claim 10 , wherein an output of the first linear layer and an output of the second GELU layer are combined to provide an input of a subsequent linear layer.
12 . The vision system of claim 1 , wherein the neural network is trained with a Contrastive Language-Image Pretraining (CLIP) framework.
13 . The vision system of claim 1 , wherein the neural network is trained with Locked-Text Tuning that includes:
initializing a text encoder with a pre-trained model;
freezing the text encoder that is initialized with the pre-trained model; and
training the neural network to obtain the vision model, wherein the training includes training with an image dataset to determine weights of nodes in the neural network until a loss function is less or equal to a predetermined value.
14 . The vision system of claim 1 , wherein the vision model is generated by fitting a data set, having a plurality of image-text pairs, to determine weights of nodes such that a contrastive loss is below a predetermined threshold.
15 . The vision system of claim 1 , wherein the vision model is validated by validating the vision model until a contrastive loss according to a loss function is reduced to a predetermined threshold.
16 . A method for generating a vision model, comprising:
a first convolutional block that downsamples a set of image data to obtain a first stage convoluted data;
a second convolutional block that downsamples the first stage convoluted data to obtain a second stage convoluted data, wherein
one or both of the first convolutional block and the second convolutional block is a mobile convolution (MBConv) block that includes: a Gaussian Error Linear Unit (GELU) convolutional layer, a GELU depth-wise convolutional (DWConv) layer, and a resizing convolutional layer, wherein the GELU convolutional layer precedes the GELU DWConv layer in the MBConv block; and
a transformer block (TFB) generating a feature map from the second stage convoluted data,
wherein the method comprising:
benchmarking a plurality of vision models in a test-bed of models, wherein the test-bed benchmarks the plurality of vision models according to a short schedule to fast benchmark the vision models under a contrastive language-image pretraining (CLIP), and a long schedule in determining performance of the plurality of vision models, wherein
the benchmarking includes analyzing the plurality of vision models on at least classification capability, retrieval capability, open-vocabulary detection capability, or large multi-modal model performance in determining data scalability, model scalability, and feature resolution; and
generating the vision model using the short schedule and the long schedule.
17 . The method of claim 16 , wherein the benchmarking includes analyzing the plurality of vision models on classification capacity and retrieval capacity, and one or both of the open-vocabulary detection capability or the large multi-modal model performance.
18 . The method of claim 16 , wherein the method further comprises training the vision model by:
initializing a text encoder with a pre-trained model;
freezing the text encoder that is initialized with the pre-trained model; and
training a randomly initialized image encoder, using an image dataset having a plurality of image-text pairs, for obtaining a neural network.
19 . A method of validating a vision model comprising:
a first convolutional block that downsamples a set of image data to obtain a first stage convoluted data;
a second convolutional block that downsamples the first stage convoluted data to obtain a second stage convoluted data, wherein
one or both of the first convolutional block and the second convolutional block is a mobile convolution (MBConv) block that includes: a Gaussian Error Linear Unit (GELU) convolutional layer, a GELU depth-wise convolutional (DWConv) layer, and a resizing convolutional layer, wherein the GELU convolutional layer precedes the GELU DWConv layer in the MBConv block; and
a transformer block (TFB) generating a feature map from the second stage convoluted data,
wherein the method comprising:
processing an image dataset using the vision model; and
validating the vision model using classification capability, retrieval capability, open-vocabulary detection capability, and large multi-modal model performance in determining data scalability, model scalability, and feature resolution.
20 . The method of claim 19 , wherein the validating further comprises validating the vision model using zero-shot accuracy on the classification capability and the retrieval capability.