IP Library Granted Patent US 10,486,596
Granted Patent B2
US 10,486,596 · App. 16/246,624 · Granted Nov 26, 2019

Multi-camera dynamic top view vision system

Inventors: Ghanshyam Rathi (Mississauga, CA); Hilda Faraji (Toronto, CA); Nikhil Gupta (Brampton, CA); Christian Traub (Windach, DE); Michael Schaffner (Gross-Gerau, DE); Goerg Pflug (Weil der Stadt, DE)
Assignee: MAGNA ELECTRONICS INC.
B60R1/00G06K9/00791G06K9/00798G06K9/00805G06T3/4038H04N5/2258H04N5/23229B60R2300/105B60R2300/303B60R2300/607
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,486,596
App. No.
16/246,624
Granted
Nov 26, 2019
Kind
B2
Abstract

A dynamic image stitching system for stitching images captured by cameras of a vision system for a vehicle includes a plurality of cameras disposed at the vehicle and having respective fields of view exterior the vehicle. The field of view of each side camera partially overlaps the field of view of a front camera and partially overlaps the field of view of a rear camera. Image data captured by the plurality of cameras is processed at a processor to detect an object present in an overlapping portion of the fields of view of two cameras. The processor stitches, via a stitching algorithm, image data captured by the cameras. The processor adjusts the stitching algorithm responsive to a determination of a difference between a characteristic of the feature as captured by one of the two cameras and the characteristic of the feature as captured by another of the two cameras.

Claims (50)

1. A dynamic image stitching system for stitching images captured by multiple cameras of a vision system for a vehicle, said dynamic image stitching system comprising:

a plurality of cameras disposed at a vehicle and having respective fields of view exterior the vehicle;

wherein said plurality of cameras comprises (i) a rear camera disposed at a rear portion of the vehicle and having a field of view at least rearward of the vehicle, (ii) a left-side camera disposed at a left side portion of the vehicle and having a field of view at least sideward of the vehicle, (iii) a right-side camera disposed at a right side portion of the vehicle and having a field of view at least sideward of the vehicle and (iv) a front camera disposed at a front portion of the vehicle and having a field of view at least forward of the vehicle;

wherein the field of view of said left-side camera partially overlaps the field of view of said front camera and partially overlaps the field of view of said rear camera;

wherein the field of view of said right-side camera partially overlaps the field of view of said front camera and partially overlaps the field of view of said rear camera;

wherein image data captured by said plurality of cameras is processed at a processor to detect an object present in an overlapping portion of the fields of view of two cameras of said plurality of cameras and to determine a characteristic of a feature of the detected object present in the overlapping portion of the fields of view of the two cameras;

wherein said processor stitches, via a stitching algorithm, image data captured by said plurality of cameras to form stitched images;

wherein said processor adjusts said stitching algorithm responsive to a determination of a difference between the characteristic of the feature of the detected object as captured by a first camera of the two cameras and the characteristic of the feature of the detected object as captured by a second camera of the two cameras; and

wherein said processor, via adjusting the stitching algorithm, mitigates misalignment of the stitching of image data due to vehicle loading.

2. The dynamic image stitching system of claim 1 , wherein said processor stitches, via said stitching algorithm, captured image data to form a seamless stitched image.

3. The dynamic image stitching system of claim 1 , wherein the first camera comprises said front camera or said rear camera and the second camera comprises said left-side camera or said right-side camera.

4. The dynamic image stitching system of claim 3 , wherein said processor processes captured image data over multiple frames of captured image data to track the feature of the detected object as it moves through the overlapping portion of the fields of view of the first camera and the second camera, and wherein said processor determines differences in the characteristic of the feature of the detected object over multiple frames of captured image data.

5. The dynamic image stitching system of claim 1 , wherein the characteristic of the feature of the detected object comprises one selected from the group consisting of (i) a vertical characteristic of the detected object and (ii) a horizontal characteristic of the detected object.

6. The dynamic image stitching system of claim 1 , wherein the characteristic of the feature of the detected object comprises at least one selected from the group consisting of (i) a curb at a side of a road on which the vehicle is being driven, (ii) a mailbox being approached and passed by the vehicle and (iii) a post at a side of a road on which the vehicle is being driven.

7. The dynamic image stitching system of claim 1 , wherein said processor adjusts said stitching algorithm at least in part responsive to a determination of a change in pitch or roll of the vehicle relative to the ground at the vehicle.

8. The dynamic image stitching system of claim 7 , wherein said processor determines a change in distance to the detected object exterior the vehicle that is indicative of a change in pitch or roll of the vehicle.

9. The dynamic image stitching system of claim 7 , wherein the determination of a change in pitch or roll of the vehicle is made at least in part responsive to a suspension system of the vehicle.

10. The dynamic image stitching system of claim 1 , wherein said processor adjusts said stitching algorithm at least in part responsive to a non-image-based sensor of the vehicle.

11. The dynamic image stitching system of claim 10 , wherein said non-image-based sensor determines a distance to the detected object exterior the vehicle, and wherein said processor, responsive to an output of said non-image-based sensor, determines a change in distance to the detected object that is indicative of a change in pitch or roll of the vehicle.

12. A dynamic image stitching system for stitching images captured by multiple cameras of a vision system for a vehicle, said dynamic image stitching system comprising:

a plurality of cameras disposed at a vehicle and having respective fields of view exterior the vehicle;

wherein said plurality of cameras comprises (i) a rear camera disposed at a rear portion of the vehicle and having a field of view at least rearward of the vehicle, (ii) a left-side camera disposed at a left side portion of the vehicle and having a field of view at least sideward of the vehicle, (iii) a right-side camera disposed at a right side portion of the vehicle and having a field of view at least sideward of the vehicle and (iv) a front camera disposed at a front portion of the vehicle and having a field of view at least forward of the vehicle;

wherein the field of view of said left-side camera partially overlaps the field of view of said front camera and partially overlaps the field of view of said rear camera;

wherein the field of view of said right-side camera partially overlaps the field of view of said front camera and partially overlaps the field of view of said rear camera;

wherein image data captured by said plurality of cameras is processed at a processor to detect an object present in an overlapping portion of the fields of view of two cameras of said plurality of cameras and to determine a characteristic of a feature of the detected object present in the overlapping portion of the fields of view of the two cameras;

wherein said processor stitches, via a stitching algorithm, image data captured by said plurality of cameras to form stitched images;

wherein said processor adjusts said stitching algorithm responsive to a determination of a difference between the characteristic of the feature of the detected object as captured by a first camera of the two cameras and the characteristic of the feature of the detected object as captured by a second camera of the two cameras; and

wherein said processor, via adjusting the stitching algorithm, mitigates misalignment of stitching of image data due to due to road surface out-of-flatness.

13. The dynamic image stitching system of claim 12 , wherein said processor stitches, via said stitching algorithm, captured image data to form a seamless stitched image.

14. The dynamic image stitching system of claim 12 , wherein the first camera comprises said front camera or said rear camera and the second camera comprises said left-side camera or said right-side camera.

15. The dynamic image stitching system of claim 14 , wherein said processor processes captured image data over multiple frames of captured image data to track the feature of the detected object as it moves through the overlapping portion of the fields of view of the first camera and the second camera, and wherein said processor determines differences in the characteristic of the feature of the detected object over multiple frames of captured image data.

16. The dynamic image stitching system of claim 12 , wherein the characteristic of the feature of the detected object comprises one selected from the group consisting of (i) a vertical characteristic of the detected object and (ii) a horizontal characteristic of the detected object.

17. The dynamic image stitching system of claim 12 , wherein the characteristic of the feature of the detected object comprises at least one selected from the group consisting of (i) a curb at a side of a road on which the vehicle is being driven, (ii) a mailbox being approached and passed by the vehicle and (iii) a post at a side of a road on which the vehicle is being driven.

18. The dynamic image stitching system of claim 12 , wherein said processor adjusts said stitching algorithm at least in part responsive to a determination of a change in pitch or roll of the vehicle relative to the ground at the vehicle.

19. The dynamic image stitching system of claim 18 , wherein said processor determines a change in distance to the detected object exterior the vehicle that is indicative of a change in pitch or roll of the vehicle.

20. The dynamic image stitching system of claim 18 , wherein the determination of a change in pitch or roll of the vehicle is made at least in part responsive to a suspension system of the vehicle.

21. The dynamic image stitching system of claim 12 , wherein said processor adjusts said stitching algorithm at least in part responsive to a non-image-based sensor of the vehicle.

22. The dynamic image stitching system of claim 21 , wherein said non-image-based sensor determines a distance to the detected object exterior the vehicle, and wherein said processor, responsive to an output of said non-image-based sensor, determines a change in distance to the detected object that is indicative of a change in pitch or roll of the vehicle.

23. A dynamic image stitching system for stitching images captured by multiple cameras of a vision system for a vehicle, said dynamic image stitching system comprising:

a plurality of cameras disposed at a vehicle and having respective fields of view exterior the vehicle;

wherein said plurality of cameras comprises (i) a rear camera disposed at a rear portion of the vehicle and having a field of view at least rearward of the vehicle, (ii) a left-side camera disposed at a left side portion of the vehicle and having a field of view at least sideward of the vehicle, (iii) a right-side camera disposed at a right side portion of the vehicle and having a field of view at least sideward of the vehicle and (iv) a front camera disposed at a front portion of the vehicle and having a field of view at least forward of the vehicle;

wherein the field of view of said left-side camera partially overlaps the field of view of said front camera and partially overlaps the field of view of said rear camera;

wherein the field of view of said right-side camera partially overlaps the field of view of said front camera and partially overlaps the field of view of said rear camera;

wherein image data captured by said plurality of cameras is processed at a processor to detect an object present in an overlapping portion of the fields of view of two cameras of said plurality of cameras and to determine a characteristic of a feature of the detected object present in the overlapping portion of the fields of view of the two cameras;

wherein said processor stitches, via a stitching algorithm, image data captured by said plurality of cameras to form seamless stitched images;

wherein said processor adjusts said stitching algorithm responsive to a determination of a difference between the characteristic of the feature of the detected object as captured by a first camera of the two cameras and the characteristic of the feature of the detected object as captured by a second camera of the two cameras;

wherein the first camera comprises said front camera or said rear camera and the second camera comprises said left-side camera or said right-side camera; and

wherein said processor processes captured image data over multiple frames of captured image data to track the feature of the detected object as it moves through the overlapping portion of the fields of view of the first camera and the second camera, and wherein said processor determines differences in the characteristic of the feature of the detected object over multiple frames of captured image data.

24. The dynamic image stitching system of claim 23 , wherein said processor adjusts said stitching algorithm at least in part responsive to a determination of a change in pitch or roll of the vehicle relative to the ground at the vehicle.

25. The dynamic image stitching system of claim 23 , wherein said processor adjusts said stitching algorithm at least in part responsive to a non-image-based sensor of the vehicle.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 21, 2019
From: RATHI, GHANSHYAM; GUPTA, NIKHIL
To: MAGNA ELECTRONICS INC.
Reel/Frame 050118/0651 →
Continuity (4)
Continuation 14191512 · Feb 27, 2014
Provisional Application 61793592 · Mar 15, 2013
Provisional Application 61770051 · Feb 27, 2013
Related Publication 20190143896A1 · May 16, 2019
Cited By (3)
US 12,225,320 US 12,242,998 US 12,469,300