Systems and methods for reducing power consumption of executing learning models in vehicle systems
A device may receive video data that includes a plurality of video frames, and may utilize a scheduling policy to divide the plurality of video frames into a first set of video frames and a second set of video frames. The device may process the first set of video frames, with a first convolutional neural network (CNN) model that includes one or more saliency gates, to generate first predictions and saliency maps, and may generate a trained first CNN model based on the first predictions and the saliency maps. The device may process the second set of video frames and the saliency maps, with a second CNN model that includes a saliency propagation module, to generate second predictions, and may generate a trained second CNN model based on the second predictions. The device may perform actions based on the trained first CNN model and the trained second CNN model.
1 . A method, comprising:
receiving, by a device, video data that includes a plurality of video frames;
utilizing, by the device, a scheduling policy to divide the plurality of video frames into a first set of video frames and a second set of video frames;
processing, by the device, the first set of video frames, with a first convolutional neural network (CNN) model that includes one or more saliency gates, to generate first predictions and saliency maps;
generating, by the device, a trained first CNN model based on the first predictions and the saliency maps;
processing, by the device, the second set of video frames and the saliency maps, with a second CNN model that includes a saliency propagation module, to generate second predictions;
generating, by the device, a trained second CNN model based on the second predictions,
wherein the trained second CNN model is configured to be utilized more than the trained first CNN model without losing accuracy of predictions; and
performing, by the device, one or more actions based on the trained first CNN model and the trained second CNN model.
2 . The method of claim 1 , wherein utilizing the scheduling policy to divide the plurality of video frames into the first set of video frames and the second set of video frames comprises:
selecting a first quantity of the plurality of video frames as the first set of video frames; and
selecting a second quantity of the plurality of video frames as the second set of video frames,
wherein the second quantity is greater than the first quantity.
3 . The method of claim 1 , wherein a first parameter size of the first CNN model is greater than a second parameter size of the second CNN model.
4 . The method of claim 3 , wherein a first input resolution of the first CNN model is greater than a second input resolution of the second CNN model.
5 . The method of claim 1 , wherein each of the saliency maps identifies salient image regions in a video frame of the first set of video frames.
6 . The method of claim 1 , wherein the one or more saliency gates calculate the saliency maps.
7 . The method of claim 1 , wherein each of the one or more saliency gates is provided after a convolutional block of the first CNN model.
8 . A device, comprising:
one or more processors configured to:
receive video data that includes a plurality of video frames;
select a first quantity of the plurality of video frames as a first set of video frames;
select a second quantity of the plurality of video frames as a second set of video frames,
wherein the second quantity is greater than the first quantity;
process the first set of video frames, with a first convolutional neural network (CNN) model that includes one or more saliency gates, to generate first predictions and saliency maps;
generate a trained first CNN model based on the first predictions and the saliency maps;
process the second set of video frames and the saliency maps, with a second CNN model that includes a saliency propagation module, to generate second predictions;
generate a trained second CNN model based on the second predictions,
wherein the trained second CNN model is configured to be utilized more than the trained first CNN model without losing accuracy of predictions; and
perform one or more actions based on the trained first CNN model and the trained second CNN model.
9 . The device of claim 8 , wherein each of the one or more saliency gates calculates one of the saliency maps based on a hidden representation calculated by a convolutional block of the first CNN model and a last latent representation calculated by the first CNN model.
10 . The device of claim 8 , wherein the saliency propagation module injects spatial priors of the saliency maps into the second CNN model and corrects spatial misalignment due to elapsed time.
11 . The device of claim 8 , wherein the one or more processors, to perform the one or more actions, are configured to:
modify the first quantity of the plurality of video frames or the second quantity of the plurality of video frames based on the trained first CNN model and the trained second CNN model.
12 . The device of claim 8 , wherein the one or more processors, to perform the one or more actions, are configured to:
modify a quantity of the one or more saliency gates based on the trained first CNN model and the trained second CNN model.
13 . The device of claim 8 , wherein the one or more processors, to perform the one or more actions, are configured to one or more of:
process real time video data with the trained first CNN model and the trained second CNN model to generate classifications for the real time video data; or
process real time temporal-based data with the trained first CNN model and the trained second CNN model.
14 . The device of claim 8 , wherein the one or more processors, to perform the one or more actions, are configured to:
implement the trained first CNN model and the trained second CNN model at a traffic location or in a vehicle.
15 . A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:
one or more instructions that, when executed by one or more processors of a device, cause the device to:
receive video data that includes a plurality of video frames;
utilize a scheduling policy to divide the plurality of video frames into a first set of video frames and a second set of video frames;
process the first set of video frames, with a first convolutional neural network (CNN) model that includes one or more saliency gates, to generate first predictions and saliency maps;
generate a trained first CNN model based on the first predictions and the saliency maps;
process the second set of video frames and the saliency maps, with a second CNN model that includes a saliency propagation module, to generate second predictions,
wherein a first parameter size of the first CNN model is greater than a second parameter size of the second CNN model, and
wherein a first input resolution of the first CNN model is greater than a second input resolution of the second CNN model;
generate a trained second CNN model based on the second predictions,
wherein the trained second CNN model is configured to be utilized more than the trained first CNN model without losing accuracy of predictions; and
perform one or more actions based on the trained first CNN model and the trained second CNN model.
16 . The non-transitory computer-readable medium of claim 15 , wherein the one or more instructions, that cause the device to utilize the scheduling policy to divide the plurality of video frames into the first set of video frames and the second set of video frames, cause the device to:
select a first quantity of the plurality of video frames as the first set of video frames; and
select a second quantity of the plurality of video frames as the second set of video frames, wherein the second quantity is greater than the first quantity.
17 . The non-transitory computer-readable medium of claim 15 , wherein each of the saliency maps identifies salient image regions in a video frame of the first set of video frames, and
wherein the one or more saliency gates calculate the saliency maps.
18 . The non-transitory computer-readable medium of claim 15 , wherein each of the one or more saliency gates is provided after a convolutional block of the first CNN model.
19 . The non-transitory computer-readable medium of claim 15 , wherein each of the one or more saliency gates calculates one of the saliency maps based on a hidden representation calculated by a convolutional block of the first CNN model and a last latent representation calculate by the first CNN model.
20 . The non-transitory computer-readable medium of claim 15 , wherein the saliency propagation module injects spatial priors of the saliency maps into the second CNN model and corrects spatial misalignment due to elapsed time.