Systems and methods for continuous adaptation of semantic image segmentation model

1 . A semantic image segmentation (SIS) system, comprising:

a neural network module trained to generate semantic image segmentation maps based on input images, the semantic image segmentation maps grouping pixels of the input images under respective class labels, respectively;

a minimum entropy module configured to, at a first time, determine first minimum entropies of pixels, respectively, in the semantic image segmentation maps generated for a received image and N images received before the received image,

where N is an integer greater than or equal to 1; and

an adaptation module configured to selectively adjust parameters of the neural network module based on optimization of a loss function that minimizes the first minimum entropies.

2 . The SIS system of claim 1 wherein the adaptation module is configured to adjust batch norm parameters of the neural network module based on the optimization of a loss function that minimizes the first minimum entropies.

3 . The SIS system of claim 2 wherein the batch norm parameters include β and γ of each layer of the neural network module.

4 . The SIS system of claim 1 wherein the neural network module includes a ResNet-50 convolutional neural network.

5 . The SIS system of claim 1 wherein the neural network module includes a visual network having the Transformer architecture.

6 . The SIS system of claim 1 further comprising a buffer module configured to store the N images received before the received image.

7 . The SIS system of claim 6 wherein the received image and the N images received before the received image are captured consecutively in time.

8 . The SIS system of claim 7 wherein the received image and the N images received before the received image are captured non-consecutively in time.

9 . The SIS system of claim 1 wherein:

the minimum entropy module is configured to, at a second time after the first time, determine second minimum entropies of the pixels, respectively, in the semantic image segmentation maps generated for a second received image and N images received before the second received image; and

the adaptation module is configured to selectively adjust the parameters of the neural network module based on the second minimum entropies.

10 . The SIS system of claim 9 wherein the received image, the second received image, and the N images form a continuous video stream.

11 . The SIS system of claim 1 wherein the neural network module is further configured to, after the adjustment of the parameters, determine a semantic image segmentation map based on the received image.

12 . A robot comprising:

a camera;

the SIS system of claim 1 , wherein the received image is captured using the camera; and

a control module configured to actuate an actuator of the robot based on one of the semantic image segmentation maps from the neural network module.

13 . The system of claim 1 wherein the neural network module is configured to receive the received image from a camera.

14 . The system of claim 1 wherein the neural network module is configured to receive the received image from a video stored in memory.

15 . A semantic image segmentation (SIS) method, comprising:

by a neural network module, generating semantic image segmentation maps based on input images, the semantic image segmentation maps grouping pixels of the input images under respective class labels, respectively;

at a first time, determining first minimum entropies of pixels, respectively, in the semantic image segmentation maps generated for a received image and N images received before the received image,

where N is an integer greater than or equal to 1; and

selectively adjusting parameters of the neural network module based on optimization of a loss function that minimizes the first minimum entropies.

16 . The SIS method of claim 15 wherein the selectively adjusting includes adjusting batch norm parameters of the neural network module based on the optimization of a loss function that minimizes the first minimum entropies.

17 . The SIS method of claim 16 wherein the batch norm parameters include β and γ of each layer of the neural network module.

18 . The SIS method of claim 15 wherein the neural network module includes one of:

a ResNet-50 convolutional neural network; and

a visual network having the Transformer architecture.

19 . The SIS method of claim 15 further comprising storing the N images received before the received image in a buffer module.

20 . The SIS method of claim 15 wherein one of:

the received image and the N images received before the received image are captured consecutively in time; and

the received image and the N images received before the received image are captured non-consecutively in time.

21 . A semantic image segmentation (SIS) method, comprising:

by a neural network module, generating semantic image segmentation maps based on input images that group pixels of the input images under respective class labels, respectively;

at a first time, determining first minimum entropies of pixels, respectively, in the semantic image segmentation maps generated for a received image and N images received before the received image,

where N is an integer greater than or equal to 1; and

selectively adjusting the respective class labels of the semantic image segmentation map of the received image based on a function that minimizes the first minimum entropies of the pixels.