Granted Patent B2

US 12,657,465 · App. 17/503,150 · Granted Jun 16, 2026

Inventors: Ye Yu (Redmond, WA); Gaurav Mittal (Redmond, WA); Mei Chen (Bellevue, WA); Yu Gong (Burnaby, CA)

Assignee: Microsoft Technology Licensing, LLC.

G06N3/082G06F18/214G06F18/2178

Abstract

Claims (86)

1 . A system comprising:

a processor; and

a computer-readable medium storing instructions that are operative upon execution by the processor to:

receive a computer vision task;

for a neural network (NN) having a plurality of modules, determine a task objective for at least a final module of the plurality of modules;

for the NN, determine a balancing objective using at least an output of the final module and an output of a first intermediate module of the plurality of modules, wherein determining the balancing objective comprises:

for each intermediate module output for a plurality of intermediate modules of the plurality of modules and the output of the final module:

determining mutual information between the intermediate module output and the output of the final module;

extracting self-information from the intermediate module output; and

combining the mutual information and the self-information;

determine an overall objective, wherein determining the overall objective comprises combining the task objective with the balancing objective;

adjust weights of the NN to minimize the overall objective;

determine whether a compressed NN meets an accuracy requirement for the computer vision task;

based on determining the compressed NN meets the accuracy requirement, compress the NN by deleting the final module;

continue deleting an additional module from the plurality of modules while the compressed NN meets the accuracy requirement; and

deploy the compressed NN to perform the computer vision task with the compressed NN.

2 . The system of claim 1 , wherein determining the overall objective comprises:

negating a positive value of the balancing objective, such that maximizing the positive value of the balancing objective minimizes the balancing objective.

3 . The system of claim 2 , wherein determining the balancing objective further comprises:

for each intermediate module output for the plurality of intermediate modules of the plurality of modules and the output of the final module, determining adaptive mutual information, wherein determining the adaptive mutual information comprises combining:

the mutual information between the intermediate module output and the output of the final module

with the self-information for the intermediate module output, and

wherein the negated positive value of the balancing objective comprises the adaptive mutual information.

4 . The system of claim 3 , wherein combining the mutual information between the intermediate module output and the output of the final module with the self-information for the intermediate module output comprises multiplying the mutual information between the intermediate module output and the output of the final module by the self-information for the intermediate module output.

5 . The system of claim 1 , wherein the instructions are further operative to:

during deployment of the NN, perform fine-tuning of the NN with user feedback.

6 . The system of claim 1 , wherein determining the task objective further comprises:

using at least an output of a second intermediate module of the plurality of modules.

7 . The system of claim 1 , wherein the instructions are further operative to:

receive a second computer vision task, the second computer vision task with a lesser accuracy requirement; and

compress the NN to a second compressed NN by deleting two or more modules from the NN, wherein the second compressed NN meets the lesser accuracy requirement, and wherein the second compressed NN has fewer modules than the compressed NN.

8 . A computerized method comprising:

receiving a computer vision task;

for a neural network (NN) having a plurality of modules, determining a task objective for at least a final module of the plurality of modules;

for the NN, determining a balancing objective using at least an output of the final module and an output of a first intermediate module of the plurality of modules, wherein determining the balancing objective comprises:

for each intermediate module output for a plurality of intermediate modules of the plurality of modules and the output of the final module:

determining mutual information between the intermediate module output and the output of the final module;

extracting self-information from the intermediate module output; and

combining the mutual information and the self-information;

determining an overall objective, wherein determining the overall objective comprises combining the task objective with the balancing objective;

adjusting weights of the NN to minimize the overall objective;

determining whether a compressed NN meets an accuracy requirement the computer vision task;

based on determining the compressed NN meets the accuracy requirement, compressing the NN by deleting the final module;

continuing deleting an additional module from the plurality of modules while the compressed NN meets the accuracy requirement; and

deploying the compressed NN to perform the computer vision task with the compressed NN.

9 . The computerized method of claim 8 , wherein determining the overall objective comprises:

negating a positive value of the balancing objective, such that maximizing the positive value of the balancing objective minimizes the balancing objective.

10 . The computerized method of claim 9 , wherein determining the balancing objective further comprises:

for each intermediate module output for the plurality of intermediate modules of the plurality of modules and the output of the final module, determining adaptive mutual information, wherein determining the adaptive mutual information comprises combining

the mutual information between the intermediate module output and the output of the final module

with the self-information for the intermediate module output,

and wherein the negated positive value of the balancing objective comprises the adaptive mutual information.

11 . The computerized method of claim 10 , wherein combining the mutual information between the intermediate module output and the output of the final module with the self-information for the intermediate module output comprises multiplying the mutual information between the intermediate module output and the output of the final module by the self-information for the intermediate module output.

12 . The computerized method of claim 8 , wherein combining the task objective with the balancing objective comprises summing the task objective and the balancing objective.

13 . The computerized method of claim 8 , wherein determining the task objective further comprises:

using at least an output of a second intermediate module of the plurality of modules.

14 . The computerized method of claim 8 , further comprising:

receiving a second computer vision task, the second computer vision task with a lesser accuracy requirement; and

compressing the NN to a second compressed NN by deleting two or more modules from the NN, wherein the second compressed NN meets the lesser accuracy requirement, and wherein the second compressed NN has fewer modules than the compressed NN.

15 . One or more computer storage devices having computer-executable instructions stored thereon, which, on execution by a computer, cause the computer to perform operations comprising:

receive a computer vision task;

for a deployed neural network (NN) having a plurality of modules, determining a task objective for at least a final module of the plurality of modules;

for the NN, determining a balancing objective using at least an output of the final module and an output of a first intermediate module of the plurality of modules, wherein determining the balancing objective comprises:

for each intermediate module output for a plurality of intermediate modules of the plurality of modules and the output of the final module:

determining mutual information between the intermediate module output and the output of the final module;

extracting self-information from the intermediate module output; and

combining the mutual information and the self-information;

determining an overall objective, wherein determining the overall objective comprises combining the task objective with the balancing objective;

receiving user feedback on an output result of the NN;

based on at least the user feedback, perform fine-tuning of the NN, wherein the fine-tuning comprises adjusting weights of the NN to minimize the overall objective;

determine whether a compressed NN meets an accuracy requirement for accuracy for the computer vision task;

based on determining the compressed NN meets the accuracy requirement, compress the NN by deleting the final module;

continue deleting an additional module from the plurality of modules while the compressed NN meets the accuracy requirement; and

deploy the compressed NN to perform the computer vision task with the compressed NN.

16 . The one or more computer storage devices of claim 15 , wherein determining the overall objective comprises:

negating a positive value of the balancing objective, such that maximizing the positive value of the balancing objective minimizes the balancing objective.

17 . The one or more computer storage devices of claim 16 , wherein determining the balancing objective further comprises:

for each intermediate module output for a plurality of intermediate modules of the plurality of modules and the output of the final module, determining adaptive mutual information, wherein determining the adaptive mutual information comprises combining

the mutual information between the intermediate module output and the output of the final module

with the self-information for the intermediate module output, and

wherein the negated positive value of the balancing objective comprises the adaptive mutual information.

18 . The one or more computer storage devices of claim 17 , wherein combining the mutual information between the intermediate module output and the output of the final module with the self-information for the intermediate module output comprises multiplying the mutual information between the intermediate module output and the output of the final module by the self-information for the intermediate module output.

19 . The one or more computer storage devices of claim 15 , wherein combining the task objective with the balancing objective comprises summing the task objective and the balancing objective, and wherein determining the task objective further comprises using at least an output of a second intermediate module of the plurality of modules.

20 . The one or more computer storage devices of claim 15 , wherein the operations further comprise:

receiving a second computer vision task, the second computer vision task with a lesser accuracy requirement; and

compressing the NN to a second compressed NN by deleting two or more modules from the NN, wherein the second compressed NN meets the lesser accuracy requirement, and wherein the second compressed NN has fewer modules than the compressed NN.

Assignments (1)

Continuity (1)

References Cited (47)