JOINT TRAINING OF NETWORK ARCHITECTURE SEARCH AND MULTI-TASK DENSE PREDICTION MODELS FOR EDGE DEPLOYMENT
Implementations are described herein for performing joint optimization of multi-task learning of dense predictions (MT-DP) and hardware-aware neural architecture search (NAS). In various implementations, a set of tasks to be performed using a resource-constrained edge computing system may be determined. Based on a base multi-task dense-prediction (MT-DP) architecture template, the set of tasks, and a plurality of hardware-based constraints of a target edge computing system, a network architecture search (NAS) may be used to sample candidate MT-DP architecture(s) from a search space of neural network architecture components. Each sampled candidate MT-DP architecture may include a distinct assembly of sampled neural network architecture components applied to the base MT-DP architecture template. Image data may be processed using the candidate MT-DP architecture(s) to determine performance metrics. These performance metrics may be used to jointly train the MT-DP architecture(s) and/or the NAS.
1 . A method implemented using one or more processors and comprising:
obtaining a set of tasks to be performed using a resource-constrained edge computing system;
based on a base multi-task dense-prediction (MT-DP) architecture template, the set of tasks, and a plurality of hardware-based constraints of the edge computing system, and using a network architecture search (NAS), sampling one or more candidate MT-DP architectures from a search space of neural network architecture components, wherein each sampled candidate MT-DP architecture comprises a distinct assembly of sampled neural network architecture components applied to the base MT-DP architecture template; and
processing image data using the one or more candidate MT-DP architectures to determine one or more performance metrics for each of the one or more candidate MT-DP architectures.
2 . The method of claim 1 , further comprising training the NAS based on the one or more performance metrics for each of the one or more candidate MT-DP architectures.
3 . The method of claim 1 , further comprising selecting and deploying, on the edge computing system, one or more of the candidate MT-DP architectures based on one or more of the performance metrics.
4 . The method of claim 1 , further comprising partially training the one or more candidate MT-DP architectures to a degree short of convergence, wherein the one or more performance metrics are determined from the partially-trained candidate MT-DP architectures.
5 . The method of claim 4 , wherein at least one of the tasks comprises pixel-wise depth estimation, and the partially training is performed using both mean absolute error (MAE) and mean relative error (MRE).
6 . The method of claim 1 , wherein each of the neural network architecture components in the search space comprises a neural network layer having one or more layer parameters.
7 . The method of claim 6 , wherein the one or more layer parameters include a layer type selected from inverted bottleneck (IBN) and fused-MN.
8 . The method of claim 6 , wherein the one or more layer parameters include a kernel size.
9 . The method of claim 6 , wherein the one or more layer parameters include an output channel multiplier or stride.
10 . The method of claim 6 , wherein the one or more layer parameters include an expansion ratio.
11 . A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions, cause the one or more processors to:
obtain a set of tasks to be performed using a resource-constrained edge computing system;
based on a base multi-task dense-prediction (MT-DP) architecture template, the set of tasks, and a plurality of hardware-based constraints of the edge computing system, and using a network architecture search (NAS), sample one or more candidate MT-DP architectures from a search space of neural network architecture components, wherein each sampled candidate MT-DP architecture comprises a distinct assembly of sampled neural network architecture components applied to the base MT-DP architecture template; and
process image data using the one or more candidate MT-DP architectures to determine one or more performance metrics for each of the one or more candidate MT-DP architectures.
12 . The system of claim 11 , further comprising instructions to train the NAS based on the one or more performance metrics for each of the one or more candidate MT-DP architectures.
13 . The system of claim 11 , further comprising instructions to select and deploy, on the edge computing system, one or more of the candidate MT-DP architectures based on one or more of the performance metrics.
14 . The system of claim 11 , further comprising instructions to partially train the one or more candidate MT-DP architectures to a degree short of convergence, wherein the one or more performance metrics are determined from the partially-trained candidate MT-DP architectures.
15 . The system of claim 4 , wherein at least one of the tasks comprises pixel-wise depth estimation, and the one or more candidate MT-DP architectures are partially trained using both mean absolute error (MAE) and mean relative error (MRE).
16 . The system of claim 11 , wherein each of the neural network architecture components in the search space comprises a neural network layer having one or more layer parameters.
17 . The system of claim 16 , wherein the one or more layer parameters include a layer type selected from inverted bottleneck (IBN) and fused-MN.
18 . The system of claim 16 , wherein the one or more layer parameters include a kernel size or an output channel multiplier.
19 . A method implemented using one or more processors and comprising:
obtaining a plurality of images capturing crops growing in an agricultural plot;
processing the plurality of images using one or more candidate multi-task dense-prediction (MT-DP) machine learning models to perform a plurality of agricultural prediction tasks, including one or more agricultural prediction tasks that generate pixel-level predictions for the plurality of images, wherein each of the one or more MT-DP machine learning models was assembled using neural network layers sampled from a search space of neural network layers having different parameters using a network architecture search (NAS); and
operating one or more agricultural vehicles in the agricultural plot based on the pixel-level predictions for the plurality of images.
20 . The method of claim 19 , further comprising jointly training the NAS and one or more of the candidate MT-DP machine learning models.