IP Library › Granted Patent US 10,977,765
Granted Patent B2
US 10,977,765 · App. 16/380,690 · Granted Apr 13, 2021

Hierarchical neural network image registration

Inventors: Derek J. Walvoord (Rochester, NY); Doug W. Couwenhoven (Rochester, NY)
Assignee: Eagle Technology, LLC
G06T3/0068G06N3/0454G06N3/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,977,765
App. No.
16/380,690
Granted
Apr 13, 2021
Kind
B2
Abstract

One or more neural networks generate a first vector field from an input image and a reference image. The first vector field is applied to the input image to generate a first warped image. The training of the neural networks is evaluated via one or more objective functions. The neural networks are updated in response to the evaluating. The neural networks generate a second vector field from the input image and the reference image. A number of degrees of freedom in the first vector field is less than a number of degrees of freedom in the second vector field. The second vector field is applied to the input image to generate a second warped image. The neural networks are evaluated via the one or more objective functions, the reference image and the second warped image. The networks are updated in response to the evaluating.

Claims (58)

1. A method comprising:

generating, via one or more neural networks from a reference image and first image data of an input image, a first vector field;

applying the first vector field to second image data of the input image to generate a first warped image of the input image;

evaluating the training of the one or more neural networks via one or more objective functions, the reference image and the first warped image;

updating the one or more neural networks in response to the evaluating via the one or more objective functions, the reference image and the first warped image;

generating, via the one or more neural networks from the reference image and third image data of the input image, a second vector field, wherein a number of degrees of freedom in the first vector field is less than a number of degrees of freedom in the second vector field;

applying the second vector field to fourth image data of the input image to generate a second warped image of the input image;

evaluating the one or more neural networks via the one or more objective functions, the reference image and the second warped image; and

updating the one or more neural networks in response to the evaluating via the one or more objective functions, the reference image and the second warped image.

2. The method of claim 1 , wherein generating the first vector field comprises generating the first vector field via a first neural network; and

wherein generating the second vector field comprises generating the second vector field via a second neural network that is different from the first neural network.

3. The method of claim 1 , wherein the first image data comprises the input image.

4. The method of claim 1 , wherein the second image data comprises the input image.

5. The method of claim 1 , wherein the third image data comprises the first warped image.

6. The method of claim 1 , wherein the third image data comprises the first image data.

7. The method of claim 1 , wherein the fourth image data comprises the first image data.

8. The method of claim 1 , wherein the fourth image data comprises the first warped image.

9. The method of claim 1 , wherein applying the second vector field to the fourth image data comprises:

summing the first vector field with the second vector field to generate a combined vector field; and

applying the combined vector field to the fourth image data.

10. The method of claim 1 , further comprising upsampling the first vector field to generate an upsampled first vector field, and wherein applying the first vector field to the second image data comprises applying the upsampled first vector field to the second image data.

11. The method of claim 1 , further comprising repeating the operations of claim 1 for one or more additional input images.

12. The method of claim 1 , further comprising:

generating, via the one or more neural networks from the reference image and fifth image data of the input image, a third vector field, wherein the number of degrees of freedom in the second vector field is less than a number of degrees of freedom in the third vector field;

applying the third vector field to sixth image data of the input image to generate a third warped image; and

evaluating the one or more neural networks via the one or more objective functions, the reference image and the third warped image.

13. The method of claim 12 , wherein the number of degrees of freedom in the first vector field is two;

wherein the number of degrees of freedom in the second vector field is eight; and

wherein the number of degrees of freedom in the third vector field is thirty two.

14. The method of claim 1 , further comprising registering a plurality of images using the one or more neural networks.

15. An apparatus comprising:

a memory figured to store image data; and

a processor, wherein the processor is configured to:

generate, via one or more neural networks from a reference image and first image data of an input image, a first vector field;

apply the first vector field to second image data of the input image to generate a first warped image of the input image;

evaluate the training of the one or more neural networks via one or more objective functions, the reference image and the first warped image;

update the one or more neural networks in response to the evaluating via the one or more objective functions, the reference image and the first warped image;

generate, via the one or more neural networks from the reference image and third image data of the input image, a second vector field, wherein a number of degrees of freedom in the first vector field is less than a number of degrees of freedom in the second vector field;

apply the second vector field to fourth image data of the input image to generate a second warped image of the input image;

evaluate the one or more neural networks via the one or more objective functions, the reference image and the second warped image; and

update the one or more neural networks in response to the evaluating via the one or more objective functions, the reference image and the second warped image.

16. The apparatus of claim 15 , wherein the processor is further configured to:

generate the first vector field by generating the first vector field via a first neural network; and

generate the second vector field by generating the second vector field via a second neural network that is different from the first neural network.

17. The apparatus of claim 15 , wherein the processor is configured to register a plurality of images using the one or more neural networks.

18. One or more tangible, non-transitory computer readable media encoded with instructions, wherein the instructions, when executed by a processor, are operable to:

generate, via one or more neural networks from a reference image and first image data of an input image, a first vector field;

apply the first vector field to second image data of the input image to generate a first warped image of the input image;

evaluate the training of the one or more neural networks via one or more objective functions, the reference image and the first warped image;

update the one or more neural networks in response to the evaluating via the one or more objective functions, the reference image and the first warped image;

generate, via the one or more neural networks from the reference image and third image data of the input image, a second vector field, wherein a number of degrees of freedom in the first vector field is less than a number of degrees of freedom in the second vector field;

apply the second vector field to fourth image data of the input image to generate a second warped image of the input image;

evaluate the one or more neural networks via the one or more objective functions, the reference image and the second warped image; and

update the one or more neural networks in response to the evaluating via the one or more objective functions, the reference image and the second warped image.

19. The one or more tangible, non-transitory computer readable media of claim 18 , wherein the instructions are further operable to:

generate the first vector field by generating the first vector field via a first neural network; and

generate the second vector field by generating the second vector field via a second neural network that is different from the first neural network.

20. The one or more tangible, non-transitory computer readable media of claim 18 , wherein the instructions are further operable to register a plurality of images using the one or more neural networks.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 7, 2019
From: WALVOORD, DEREK J.; COUWENHOVEN, DOUG W.
To: EAGLE TECHNOLOGY, LLC
Reel/Frame 049100/0295 →
Continuity (1)
Related Publication 20200327639A1 · Oct 15, 2020