IP Library › Granted Patent US 10,891,514
Granted Patent B2
US 10,891,514 · App. 16/222,905 · Granted Jan 12, 2021

Image classification pipeline

Inventors: Youjun Liu (Palo Alto, CA); Ji Li (Sunnyvale, CA); Amit Srivastava (San Jose, CA)
Assignee: Microsoft Technology Licensing, LLC
G06K9/6227G06K9/00986G06N3/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,891,514
App. No.
16/222,905
Granted
Jan 12, 2021
Kind
B2
Abstract

The present disclosure relates to processing operations configured for an image recognition pipeline that is used to tailor real-time management of image recognition processing for technical scenarios across a plurality of different applications/services. Image recognition processing is optimized at run-time to ensure that latency requirements are met so that image recognition processing results are returned in a timely manner that aids task execution in an application-specific instances. An image recognition pipeline may manage a plurality of image recognition models that comprise a combination of image analysis service (IAS) models and deep learning models. A scheduler of the image recognition pipeline optimizes image recognition processing by selecting at least: a subset of the image recognition models for image recognition processing and a device configuration for execution of the subset of image recognition models, in order to return image recognition results within a threshold time period that satisfies application-specific execution.

Claims (59)

1. A method comprising:

detecting, through an image recognition pipeline service interfacing with a productivity application or service that is configured to provide access to content, a usage of image content through the productivity application or service;

in response to detecting access to the image content through the productivity application or service, loading configuration data for the image recognition pipeline service;

generating an application-specific configuration for image recognition processing of the image content, wherein the generating of the application-specific configuration comprises execution of processing operations that:

select, from a plurality of available image recognition models provided through the image recognition pipeline service, a subset of the available image recognition models based on application-specific parameters identified in the configuration data for the productivity application or service, and

select a device configuration for execution of image recognition processing of the image content based on analysis of the application-specific parameters identified in the configuration data for the productivity application or service;

propagating the image content for image recognition processing based on a selection of the subset of the available image recognition models and a selection of the device configuration;

receiving image recognition results for the image content; and

generating a suggestion for the usage of the image content based on the image recognition results.

2. The method of claim 1 , wherein the subset of the available image recognition models is selected based on a comparative analysis of a timing requirement, identified by the productivity application or service, for requiring return of the image recognition results and a latency estimation for execution of image recognition processing by individual image recognition models of the plurality of available image recognition models.

3. The method of claim 2 , wherein the available image recognition models comprise a plurality of image analysis service (IAS) models configured to execute image recognition processing in a time period that satisfies a latency threshold and a plurality of deep learning models configured to execute image recognition processing in a time period that exceeds the latency threshold, and wherein the subset of the available image recognition models comprises a combination of one or more IAS models and one or more deep learning models.

4. The method of claim 3 , further comprising:

detecting a processing time for retrieval of the image recognition results for the subset of available image recognition models; and

in response to determining that the image recognition results are retrieved within a threshold time period that corresponds with the timing requirement identified by the productivity application or service, propagating the image content to a second subset of available image recognition models for image recognition processing.

5. The method of claim 3 , further comprising; receiving image recognition results from the second subsequent of available image recognition models, and wherein the suggestion for the usage of the image content is generated based on the image recognition results received from the subset of available image recognition models and the image recognition results from the second subset of available image recognition models.

6. The method of claim 2 , further comprising:

selecting, from the plurality of available image recognition models, a second subset of available image recognition models for image recognition processing; and

propagating, through the image recognition pipeline service, the image content to the second subset of available image recognition models, wherein the image recognition results from processing by the second subset of available image recognition models is received after a threshold time period that corresponds with the timing requirement identified by the productivity application or service.

7. The method of claim 1 , further comprising: propagating the suggestion for usage of the image content to productivity application or service, and wherein the productivity application or service is configured to present, through a user interface, the suggestion for the usage of the image content.

8. The method of claim 1 , wherein the configuration data comprises:

a first configuration file providing data that comprises data for managing the plurality of available image recognition models; and a second configuration file providing data that comprises application-specific parameters associated with the productivity application or service, and wherein a scheduling component of the image recognition pipeline service utilizes the first configuration file and the second configuration file to select the subset of available image recognition models and select the device configuration.

9. A system comprising:

at least one processor; and

a memory, operatively connected with the at least one processor, storing computer-executable instructions that, when executed by the at least one processor, causes the at least one processor to execute a method that comprises:

detecting, through an image recognition pipeline service interfacing with a productivity application or service that is configured to provide access to content, a usage of image content through the productivity application or service;

in response to detecting access to the image content through the productivity application or service, loading configuration data for the image recognition pipeline service;

generating an application-specific configuration for image recognition processing of the image content, wherein the generating of the application-specific configuration comprises execution of processing operations that:

select, from a plurality of available image recognition models provided through the image recognition pipeline service, a subset of the available image recognition models based on application-specific parameters identified in the configuration data for the productivity application or service, and

select a device configuration for execution of image recognition processing of the image content based on analysis of the application-specific parameters identified in the configuration data for the productivity application or service;

propagating the image content for image recognition processing based on a selection of the subset of the available image recognition models and a selection of the device configuration;

receiving image recognition results for the image content; and

generating a suggestion for the usage of the image content based on the image recognition results.

10. The system of claim 9 , wherein the subset of the available image recognition models is selected based on a comparative analysis of a timing requirement, identified by the productivity application or service, for requiring return of the image recognition results and a latency estimation for execution of image recognition processing by individual image recognition models of the plurality of available image recognition models.

11. The system of claim 10 , wherein the available image recognition models comprise a plurality of image analysis service (IAS) models configured to execute image recognition processing in a time period that satisfies a latency threshold and a plurality of deep learning models configured to execute image recognition processing in a time period that exceeds the latency threshold, and wherein the subset of the available image recognition models comprises a combination of one or more IAS models and one or more deep learning models.

12. The system of claim 11 , wherein the method, executed by the at least one processor, further comprises:

detecting a processing time for retrieval of the image recognition results for the subset of available image recognition models; and

in response to determining that the image recognition results are retrieved within a threshold time period that corresponds with the timing requirement identified by the productivity application or service, propagating the image content to a second subset of available image recognition models for image recognition processing.

13. The system of claim 12 , wherein the method, executed by the at least one processor, further comprises: receiving image recognition results from the second subsequent of available image recognition models, and wherein the suggestion for the usage of the image content is generated based on the image recognition results received from the subset of available image recognition models and the image recognition results from the second subset of available image recognition models.

14. The system of claim 10 , wherein the method, executed by the at least one processor, further comprises:

selecting, from the plurality of available image recognition models, a second subset of available image recognition models for image recognition processing; and

propagating, through the image recognition pipeline service, the image content to the second subset of available image recognition models, wherein the image recognition results from processing by the second subset of available image recognition models is received after a threshold time period that corresponds with the timing requirement identified by the productivity application or service.

15. The system of claim 9 , wherein the method, executed by the at least one processor, further comprises: propagating the suggestion for usage of the image content to productivity application or service, and wherein the productivity application or service is configured to present, through a user interface, the suggestion for the usage of the image content.

16. The system of claim 9 , wherein the configuration data comprises:

a first configuration file providing data that comprises data for managing the plurality of available image recognition models; and a second configuration file providing data that comprises application-specific parameters associated with the productivity application or service, and wherein a scheduling component of the image recognition pipeline service utilizes the first configuration file and the second configuration file to select the subset of available image recognition models and select the device configuration.

17. A computer-readable storage media storing computer-executable instructions that, when executed by at least one processor, causes the at least one processor to execute a method comprising:

detecting, through an image recognition pipeline service interfacing with a productivity application or service that is configured to provide access to content, a usage of image content through the productivity application or service;

in response to detecting access to the image content through the productivity application or service, loading configuration data for the image recognition pipeline service;

generating an application-specific configuration for image recognition processing of the image content, wherein the generating of the application-specific configuration comprises execution of processing operations that:

select, from a plurality of available image recognition models provided through the image recognition pipeline service, a subset of the available image recognition models based on application-specific parameters identified in the configuration data for the productivity application or service, and

select a device configuration for execution of image recognition processing of the image content based on analysis of the application-specific parameters identified in the configuration data for the productivity application or service;

propagating the image content for image recognition processing based on a selection of the subset of the available image recognition models and a selection of the device configuration;

receiving image recognition results for the image content; and

generating a suggestion for the usage of the image content based on the image recognition results.

18. The computer-readable storage media of claim 17 , wherein the subset of the available image recognition models is selected based on a comparative analysis of a timing requirement, identified by the productivity application or service, for requiring return of the image recognition results and a latency estimation for execution of image recognition processing by individual image recognition models of the plurality of available image recognition models.

19. The computer-readable storage media of claim 17 , wherein the executed method further comprising:

detecting a processing time for retrieval of the image recognition results for the subset of available image recognition models; and

in response to determining that the image recognition results are retrieved within a threshold time period that corresponds with the timing requirement identified by the productivity application or service, propagating the image content to a second subset of available image recognition models for image recognition processing; and

receiving image recognition results from the second subsequent of image recognition models, and wherein the suggestion for the usage of the image content is generated based on the image recognition results received from the subset of image recognition models and the image recognition results from the second subset of image recognition models.

20. The computer-readable storage media of claim 17 , wherein the configuration data comprises: a first configuration file providing data that comprises data for managing the plurality of available image recognition models; and a second configuration file providing data that comprises application-specific parameters associated with the productivity application or service, and wherein a scheduling component of the image recognition pipeline service utilizes the first configuration file and the second configuration file to select the subset of available image recognition models and select the device configuration.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 17, 2018
From: LIU, YOUJUN; LI, JI; SRIVASTAVA, AMIT
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 047800/0241 →
Continuity (1)
Related Publication 20200193218A1 · Jun 18, 2020
Cited By (1)
US 12,749,296