Fast object detection in video via scale separation
Techniques and apparatuses enabling high accuracy video object detection using reduced system resource requirements (e.g., reduced computational load, shallower neural network designs, etc.) are described. For example, a search domain of an object detection scheme (e.g., a target object class, a target object size, a target object rotation angle, etc.) may be separated into subdomains (e.g., such as subdomains of object classes, subdomains of object sizes, subdomains object rotation angles, etc.). Specialized, subdomain-level object detection/segmentation tasks may then be separated across sequential video frames. As such, different subdomain-level processing techniques (e.g., via specialized neural networks) may be implemented across different frames of a video sequence. Moreover, redundancy information of consecutive video frames may be leveraged, such that specialized object detection tasks combined with visual object tracking across consecutive frames may enable more efficient (e.g., more accurate, less computationally intensive, etc.) full domain object detection and object segmentation schemes.
1. A method for image processing, comprising:
dividing a video into a plurality of frames including a first frame and a second frame;
selecting the first frame for processing with a first neural network and the second frame for processing with a second neural network based on an ordering of the plurality of frames;
generating first object information for a first object in the first frame by performing a first computer vision task using the first neural network;
generating second object information for a second object in the second frame by performing a second computer vision task using the second neural network; and
generating label data for the video including a first label based on the first object information and a second label based on the second object information.
2. The method of claim 1 , further comprising:
alternately applying the first neural network to a first subset of the plurality of frames to obtain first label data corresponding to the first computer vision task and applying the second neural network to a second subset of the plurality of frames to obtain second label data corresponding to the second computer vision task, wherein the first subset and the second subset alternate according to a temporal ordering of the plurality of frames.
3. The method of claim 1 , further comprising:
reducing a resolution of the second frame to obtain a low-resolution version of the second frame, wherein the second neural network takes the low-resolution version as input and the first neural network takes a high-resolution version of the first frame as input.
4. The method of claim 1 , further comprising:
detecting a third object in a third frame of the video using a third neural network trained to detect objects by performing a third computer vision task different from the first computer vision task and second computer vision task.
5. The method of claim 1 , wherein:
the first computer vision task generates object information for objects that are small in size relative to objects which have object information generated by the second computer vision task.
6. The method of claim 1 , wherein:
the first computer vision task and the second computer vision task overlap,
the first frame is a high-resolution frame,
the first computer vision task is performed based on the first frame to detect objects that are small in size,
the second frame is a low-resolution frame, and
the second computer vision task is performed based on the second frame to detect objects that are large in size.
7. The method of claim 1 , wherein:
the second computer vision task is different from the first computer vision task.
8. The method of claim 1 , further comprising:
identifying available system resources for generating the label data; and
selecting a frequency for applying the first neural network and the second neural network based on the available system resources.
9. A method for training a machine learning model, the method comprising:
identifying first training data corresponding to a first computer vision task;
training a first neural network to perform the first computer vision task using the first training data;
identifying second training data corresponding to a second computer vision task;
training a second neural network to perform the second computer vision task using the second training data; and
alternately applying the first neural network and the second neural network to different frames of a video to generate label data corresponding to the first computer vision task and the second computer vision task, respectively.
10. The method of claim 9 , wherein:
the second training data has a lower resolution than the first training data.
11. The method of claim 9 , wherein:
the first neural network and the second neural network are trained independently.
12. The method of claim 9 , further comprising:
identifying third training data corresponding to a third computer vision task;
training a third neural network to perform the third computer vision task using the third training data; and
alternately applying the third neural network to the frames of the video to generate the label data, wherein the label data includes data corresponding to the third computer vision task.
13. The method of claim 9 , wherein:
the first training data includes first label data for objects that are small in size relative to objects for which label data is generated by the second computer vision task.
14. The method of claim 9 , wherein:
the first training data and the second training data overlap,
the first computer vision task is performed based on a first frame which is a high-resolution frame to detect objects that are small in size, and
the second computer vision task is performed based on a second frame which is a low-resolution frame to detect objects that are large in size.
15. A system for image processing, comprising:
a frame selection component to divide a video into a plurality of frames including a first frame and a second frame based on a temporal ordering of the plurality of frames;
a first neural network configured to generate first object information for a first object in the first frame by performing a first computer vision task;
a second neural network configured to generate second object information for a second object in the second frame by performing a second computer vision task; and
a data synthesis component configured to generate label data for the video including a first label based on the first object information and a second label based on the second object information.
16. The system of claim 15 , further comprising:
a third neural network configured to generate third object information for a third object in a third frame, wherein the third neural network is trained to perform a third computer vision task different from the first computer vision task and second computer vision task.
17. The system of claim 15 , further comprising:
a resolution reduction component configured to reduce a resolution of the second frame to obtain a low-resolution version of the second frame, wherein the second neural network takes the low-resolution version as input and the first neural network takes a high-resolution version of the first frame as input.
18. The system of claim 15 , further comprising:
a resource optimization component configured to identify available system resources for generating the label data and to select a frequency for applying the first neural network and the second neural network based on the available system resources.
19. The system of claim 15 , wherein:
the second neural network has fewer parameters than the first neural network,
the first frame is a high-resolution frame,
the first computer vision task is performed based on the first frame to detect objects that are small in size,
the second frame is a low-resolution frame, and
the second computer vision task is performed based on the second frame to detect objects that are large in size.
20. The system of claim 15 , further comprising:
a training component configured to train the first neural network and the second neural network.