IP Library Granted Patent US 9,286,657
Granted Patent B1
US 9,286,657 · App. 14/750,809 · Granted Mar 15, 2016

Efficient image processing using dynamically sized tiles

Inventor: David Hammond Thorup, Jr. (Bothell, WA)
Assignee: Mylio, LLC
G06T5/00G06K9/46G06T7/0051H04N19/44G06K2009/4666
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,286,657
App. No.
14/750,809
Granted
Mar 15, 2016
Kind
B1
Abstract

Systems and methods for processing an image through an ordered series of image filters (i.e., an image filter chain) to produce an output image. The systems and methods of the present disclosure function to partition or break-up an image into dynamically sized tiles which are processed efficiently (e.g., using minimal memory), quickly and on-demand. The systems and methods allow tile sizes to be calculated dynamically based on what each image filter in an image filter chain needs. By processing each tile on-demand, significant memory and time savings are achieved because only the parts of an image required for display are processed. Further, such functionality allows for decoding of only a portion of the image, which requires less memory and operates much faster than if the entire image was decoded.

Claims (118)

1. A method of operation for an image processor system to process a source image, the method comprising:

receiving, by at least one processor of the image processor system, image data from the source image;

receiving, by the at least one processor, an image filter chain specification which specifies an ordered series of one or more image filters between an image filter chain input and an image filter chain output;

determining, by the at least one processor, an output image size by forward iterating through each of the one or more image filters in the image filter chain specification from the image filter chain input to the image filter chain output, the output image size being a size of an output image output from the image filter chain specification;

determining, by the at least one processor, geometries for a plurality of output tiles based at least in part on the determined output image size; and

generating, by the at least one processor, a plurality of input tiles which each correspond to one of the output tiles by reverse iterating each of the output tiles through each of the one or more image filters in the image filter chain specification.

2. The method of claim 1 wherein determining an output image size comprises:

providing, by the at least one processor, an input image size to at least a subset of the image filters in the image filter chain specification; and

receiving, by the at least one processor, an output image size from the at least a subset of the image filters in the image filter chain specification.

3. The method of claim 1 wherein the image filter chain specification specifies at least a first image filter and a second image filter, and determining an output image size comprises:

providing, by the at least one processor, an input image size to first image filter in the image filter chain specification;

receiving, by the at least one processor, a first output image size from the first image filter in the image filter chain specification;

providing, by the at least one processor, the first output image size to the second image filter in the image filter chain specification; and

receiving, by the at least one processor, a second output image size from the second image filter in the image filter chain specification.

4. The method of claim 1 wherein generating a plurality of input tiles comprises:

for each output tile,

providing, by the at least one processor, position information and dimension information as input to each of the image filters in the image filter chain specification;

receiving, from each of the image filters in the image filter chain specification, source rectangle information indicative of position information and dimension information for a source rectangle required to generate an image having the provided position information and dimension information; and

receiving, from each of the image filters in the image filter chain specification, region of interest information indicative of a region of interest within the source rectangle.

5. The method of claim 1 wherein receiving image data from the source image comprises:

determining, by the at least one processor, a source image region of interest based at least in part on the input tiles; and

receiving, by the at least one processor, image data for the determined source image region of interest.

6. The method of claim 5 , further comprising:

decoding, by the at least one processor, the determined source image region of interest of the source image to provide the image data.

7. The method of claim 1 , further comprising:

populating, by the at least one processor, each of the input tiles with the image data from the source image;

processing, by the at least one processor, each of the populated input tiles through each of the one or more image filters in the image filter chain specification; and

generating, by the at least one processor, the output image by populating each of the output tiles with respective processed image data from each of the corresponding input tiles.

8. The method of claim 7 wherein processing each of the populated input tiles comprises:

generating, by the at least one processor, an input working tile and an output working tile;

storing, by the at least one processor, the input working tile and the output working tile in at least one nontransitory processor-readable medium;

designating, by the at least one processor, the input working tile as a first input tile for a first one of the image filters;

designating, by the at least one processor, the output working tile as a first output tile for the first one of the image filters;

processing the first input tile of the first one of the image filters through the first one of the image filters;

designating, by the at least one processor, the output working tile as a second input tile for a second one of the image filters; and

designating, by the at least one processor, the input working tile as a second output tile for the second one of the image filters.

9. The method of claim 8 , further comprising:

prior to the generating of the input working tile and the output working tile,

determining, by the at least one processor, a maximum width of the input tiles and the output tiles;

determining, by the at least one processor, a maximum height of the input tiles and the output tiles; and

determining, by the at least one processor, a size for the input working tile and the output working tile based at least in part on the determined maximum width and maximum height,

wherein generating the input working tile and the output working tile comprises generating the input working tile and the output working tile based at least in part on the determined size for the input working tile and the output working tile.

10. The method of claim 1 , further comprising:

storing, by the at least one processor, the image data in a nontransitory processor-readable medium operatively coupled with a graphics processing unit (GPU).

11. The method of claim 1 , further comprising:

storing, by the at least one processor, a scaled version of the source image in a nontransitory processor-readable medium,

wherein receiving image data comprises receiving the scaled version of the source image from the nontransitory processor-readable medium.

12. The method of claim 1 , further comprising:

determining, by the at least one processor, whether at least one component of the image processor system is capable of reading from each of a plurality of bit depths; and

determining, by the at least one processor, whether the at least one component of the image processor system is capable of writing to each of a plurality of bit depths.

13. The method of claim 12 wherein receiving image data from the source image comprises receiving image data having a bit depth based at least in part on the determination of whether at least one component of the image processor system is capable of reading from each of a plurality of bit depths.

14. The method of claim 12 , further comprising:

determining, by the at least one processor, which bit depth of the plurality of bit depths is the highest precision bit depth which the at least one component of the image processor system is capable of writing to and read from; and

setting the determined bit depth as a bit depth for each of the image filters in the image filter chain specification.

15. The method of claim 12 , further comprising:

determining, by the at least one processor, a bit depth for an output component associated with the image processor system; and

setting a bit depth of the output tiles based at least in part on the determined bit depth for the output component.

16. The method of claim 15 wherein determining a bit depth for an output component comprises determining a bit depth for at least one of a display, a printer, or a file.

17. An image processor system to process a source image, the image processor system comprising:

at least one nontransitory processor-readable medium that stores processor-executable instructions; and

at least one processor communicably coupled to the at least one nontransitory processor-readable medium, the at least one processor:

receives image data from the source image;

receives an image filter chain specification which specifies an ordered series of one or more image filters between an image filter chain input and an image filter chain output;

determines an output image size by forward iterating through each of the one or more image filters in the image filter chain specification from the image filter chain input to the image filter chain output, the output image size being a size of an output image output from the image filter chain specification;

determines geometries for a plurality of output tiles based at least in part on the determined output image size; and

generates a plurality of input tiles which each correspond to one of the output tiles by reverse iterating each of the output tiles through each of the one or more image filters in the image filter chain specification.

18. The image processor system of claim 17 wherein the at least one processor:

provides an input image size to at least a subset of the image filters in the image filter chain specification; and

receives an output image size from the at least a subset of the image filters in the image filter chain specification.

19. The image processor system of claim 17 wherein the image filter chain specification specifies at least a first image filter and a second image filter, and the at least one processor:

provides an input image size to first image filter in the image filter chain specification;

receives a first output image size from the first image filter in the image filter chain specification;

provides the first output image size to the second image filter in the image filter chain specification; and

receives a second output image size from the second image filter in the image filter chain specification.

20. The image processor system of claim 17 wherein the at least one processor:

for each output tile,

provides position information and dimension information as input to each of the image filters in the image filter chain specification;

receives, from each of the image filters in the image filter chain specification, source rectangle information indicative of position information and dimension information for a source rectangle required to generate an image having the provided position information and dimension information; and

receives, from each of the image filters in the image filter chain specification, region of interest information indicative of a region of interest within the source rectangle.

21. The image processor system of claim 17 wherein the at least one processor:

determines a source image region of interest based at least in part on the input tiles; and

receives image data for the determined source image region of interest.

22. The image processor system of claim 21 wherein the at least one processor:

decodes the determined source image region of interest of the source image to provide the image data.

23. The image processor system of claim 17 wherein the at least one processor:

populates each of the input tiles with the image data from the source image;

processes each of the populated input tiles through each of the one or more image filters in the image filter chain specification; and

generates the output image by populating each of the output tiles with respective processed image data from each of the corresponding input tiles.

24. The image processor system of claim 23 wherein the at least one processor:

generates an input working tile and an output working tile;

stores the input working tile and the output working tile in at the least one nontransitory processor-readable medium;

designates the input working tile as a first input tile for a first one of the image filters;

designates the output working tile as a first output tile for the first one of the image filters;

processes the first input tile of the first one of the image filters through the first one of the image filters;

designates the output working tile as a second input tile for a second one of the image filters; and

designates the input working tile as a second output tile for the second one of the image filters.

25. The image processor system of claim 24 wherein, before the at least one processor generates the input working tile and the output working tile, the at least one processor:

determines a maximum width of the input tiles and the output tiles;

determines a maximum height of the input tiles and the output tiles; and

determines a size for the input working tile and the output working tile based at least in part on the determined maximum width and maximum height,

wherein the at least one processor generates the input working tile and the output working tile based at least in part on the determined size for the input working tile and the output working tile.

26. The image processor system of claim 17 wherein the at least one processor comprises at least one graphics processing unit (GPU), and the at least one nontransitory processor-readable medium comprises at least one nontransitory processor-readable medium operatively coupled to the at least one GPU, and wherein the at least one processor:

stores the image data in the at least one nontransitory processor-readable medium operatively coupled to the at least one GPU.

27. The image processor system of claim 17 wherein the at least one processor:

stores a scaled version of the source image in the at least one nontransitory processor-readable medium; and

receives the scaled version of the source image from the at least one nontransitory processor-readable medium.

28. The image processor system of claim 17 wherein the at least one processor:

determines whether at least one component of the image processor system is capable of reading from each of a plurality of bit depths; and

determines whether the at least one component of the image processor system is capable of writing to each of a plurality of bit depths.

29. A method of operation for an image processor system to process a source image, the method comprising:

providing, by the at least one processor, an image filter chain specification which specifies an ordered series of one or more image filters;

determining, by the at least one processor, a size of an output image output from the image filter chain specification;

determining, by the at least one processor, geometries for a plurality of output tiles based at least in part on the determined size of the output image; and

generating, by the at least one processor, a plurality of input tiles which each correspond to one of the output tiles by reverse iterating each of the output tiles through each of the one or more image filters in the image filter chain specification.

30. The method of claim 29 , further comprising:

populating, by the at least one processor, each of the input tiles with the image data;

processing, by the at least one processor, each of the populated input tiles through each of the one or more image filters in the image filter chain specification; and

generating, by the at least one processor, the output image by populating each of the output tiles with respective processed image data from each of the corresponding input tiles.

Assignments (3)
SECURITY INTEREST Recorded Dec 3, 2024
From: MYLIO, LLC
To: GATES FRONTIER, LLC
Reel/Frame 069468/0135 →
SECURITY INTEREST Recorded Jul 17, 2018
From: MYLIO, LLC
To: GATES VENTURES, LLC, AS COLLATERAL AGENT
Reel/Frame 046371/0420 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 29, 2015
From: THORUP, DAVID HAMMOND, JR.
To: MYLIO, LLC
Reel/Frame 036035/0143 →