Method for categorizing a scene comprising a sub-scene with machine learning

1 . A computer-implemented method for categorizing visual content in a scene, the method comprising: receiving, by a computing device, a plurality of data points corresponding to the scene; detecting, from the plurality of data points, at least one subset indicative of a sub-scene presented by a display device that is part of the scene; disregarding, in a process of categorizing, the data points corresponding to the sub-scene so that the sub-scene is not taken into account in the categorizing; and generating, by the computing device executing a multi-task model trained to jointly recognize at least two of scene type, action, and pose, based on remaining data points, a categorization of at least one of: the scene, a subject in the scene, a pose in the scene, or an action in the scene, together with an optional confidence value.

2 . The method of claim 1 , wherein the display device is selected from a television, computer screen, billboard, poster, printed fabric, or other display.

3 . The method of claim 1 , wherein categorizing is performed in near real-time.

4 . The method of claim 1 , wherein detecting the sub-scene comprises identifying human keypoints or object features using Mask R-CNN or Part Affinity Fields, and using the resulting features to localize a display region.

5 . The method of claim 1 , wherein the categorization pipeline comprises a convolutional neural network initialized from an ImageNet-pretrained backbone selected from VGG, AlexNet, ResNet, GoogLeNet, Inception, or Xception and fine-tuned with newly added layers.

6 . The method of claim 1 , further comprising storing a label associated with the generated categorization for inclusion in training data used for future determinations.

7 . The method of claim 1 , wherein excluding comprises generating a segmentation mask that spatially delimits pixels of the display device and prevents their use by a scene classifier.

8 . The method of claim 1 , wherein detecting the sub-scene includes evaluating a size feature of a candidate display region relative to an average display device size of at least 350 square inches.

9 . A method of training a machine learning model to perform sub-scene-aware categorization, the method comprising:

obtaining a training corpus of images or image sequences that include scenes containing display devices;

for each training sample, labeling a subset of data points that correspond to a sub-scene presented on a display device;

training, using the labeled data, a model configured to down-weight or ignore features originating from the labeled subset during inference, while classifying at least one of scene, subject, pose, or action; and

optimizing the model parameters using an optimization method selected from Adam or RMSProp.

10 . The method of claim 9 , wherein the loss function comprises Softmax loss or Hinge loss.

11 . The method of claim 9 , wherein the model is initialized from weights trained on ImageNet.

12 . The method of claim 9 , wherein the training corpus includes indoor and outdoor scenes.

13 . The method of claim 9 , further comprising deploying the trained model in a monitoring system to count a number of people in a public square while disregarding images of people on a wide screen.

14 . A vehicle control system comprising an image capturing device and a computing device, the computing device configured to:

receive data points corresponding to a scene from the image capturing device;

detect at least one subset of the data points indicative of a sub-scene presented by a display device;

categorize the scene while disregarding the sub-scene; and

based on the categorization, adjust at least one physical parameter of the vehicle selected from speed, direction, pose, position, or orientation.

15 . The system of claim 14 , wherein the display device comprises a billboard selected from a traditional poster, a digital billboard, or a screen configured to display static images, image series, or video.

16 . The system of claim 14 , wherein detecting the sub-scene employs a Faster-RCNN detector to localize display regions.

17 . The system of claim 14 , wherein the computing device generates driving commands with an end-to-end latency less than 20 milliseconds.

18 . The system of claim 14 , wherein the scene includes at least one view from inside the vehicle looking out of the vehicle.

19 . The system of claim 14 , wherein the computing device (i) identifies candidate display devices whose apparent area, after distance correction, corresponds to at least 350 square inches, (ii) applies a pixel mask to exclude detections within a polygon covering each such device, and (iii) counts a number of people present outside the masks for driver-assistance reporting.