Granted Patent B1

US 10,666,856 · App. 15/211,618 · Granted May 26, 2020

Inventor: Devon Rueckner (San Diego, CA)

Assignee: Basil Gang LLC

H04N5/23219G02B27/0172G02B27/0179G06T17/00G06T19/006H04N5/2256H04N5/23212H04N5/23216H04N5/247G02B2027/0138G02B2027/0147G02B2027/0187

Abstract

Claims (59)

1. A method for providing augmented reality feedback from a physical hand-held camera system having a dynamic camera field of view, the method comprising:

providing a head-mounted augmented reality display at least partially overlaying a dynamic field of view of a user, the dynamic camera field of view being independent of the field of view of the user;

geometrically mapping the camera field of view to the field of view of the user, the mapping-using an estimation of relative poses that defines an epipolar geometry relationship between the camera field of view and the field of view of the user; and

augmenting the field of view of the user via the display by highlighting a representation of a boundary of the dynamic camera field of view over the field of view of the user.

2. The method according to claim 1 , wherein the augmenting the field of view of the user further comprises superimposing at least one of a contrasting transparency plane, a reticle with crop marks, a bounding box, and a blurred highlight.

3. The method according to claim 1 , further comprising distinguishing regions of the field of view of the user that are occluded in the camera field of view from regions of the field of view of the user that are visible in the camera field of view.

4. The method according to claim 1 , further comprising:

mapping a first and a second region of the camera field of view to a first and a second region of the field of view of the user, wherein the first region of the camera field of view represents a front plane of a depth of field of the camera system and the second region of the camera field of view represents a back plane of the depth of field of the camera system; and

augmenting the field of view of the user via the display by highlighting a space between the first and the second region of the field of view of the user.

5. The method according to claim 1 , further comprising:

identifying one or more objects in the field of view of the camera; and

augmenting the field of view of the user via the display by highlighting a region representing the one or more objects.

6. The method according to claim 5 , wherein the one or more objects are one or more target objects of a gaze of the user determined via one or more gaze tracking devices.

7. The method according to claim 1 , further comprising:

given that a region of the camera field of view of the camera system is in focus, mapping said region of the camera field of view to a region of the field of view of the user; and

augmenting the field of view of the user via the display by highlighting the mapped region of the field of view of the user over the field of view of the user.

8. The method according to claim 1 , further comprising:

given that a region of the camera field of view of the camera system is under-exposed, mapping said region of the camera field of view to a region of the field of view of the user; and

augmenting the field of view of the user via the display by highlighting the mapped region of the field of view of the user over the field of view of the user.

9. The method according to claim 1 , further comprising:

given that a region of the camera field of view of the camera system is over-exposed, mapping said region of the camera field of view to a region of the field of view of the user; and

augmenting the field of view of the user via the display by highlighting the mapped region of the field of view of the user over the field of view of the user.

10. The method according to claim 1 , wherein the augmenting the field of view of the user includes displaying at least one of an image captured by the camera system and a live view feed of the camera system.

11. The method according to claim 1 , wherein the augmenting the field of view of the user includes displaying at least one state of the camera system.

12. The method according to claim 1 , further comprising:

determining the camera system is within a predetermined proximity of a point of view of the user; and

disabling the display.

13. The method according to claim 1 , further comprising:

providing a plurality of camera systems, each of the plurality of camera systems having a respective camera field of view;

mapping a plurality of second regions of plurality of respective camera fields of view to a plurality of second regions of the field of view of the user; and

selectively augmenting the field of view of the user via the display by highlighting at least one of the plurality of second regions of the plurality of respective camera fields of view over the field of view of the user.

14. The method according to claim 1 , further comprising:

given that a region of the camera field of view of the camera system is occluded, mapping said region of the camera field of view to a second region of the field of view of the user; and

augmenting the field of view of the user via the display by highlighting the mapped region of the field of view of the user over the field of view of the user.

15. The method according to claim 1 , further comprising:

given that a region of the camera field of view of the camera system is out of frame, mapping said region of the camera field of view to a second region of the field of view of the user; and

augmenting the field of view of the user via the display by highlighting the mapped region of the field of view of the user over the field of view of the user.

16. The method according to claim 1 , further comprising:

tracking a target on the display based on a secondary user input from the user that is independent of a gaze location of the user and a head position of the user.

17. The method according to claim 1 , further comprising:

tracking multiple targets on the display based on a secondary user input from the user that is independent of a gaze location of the user and a head position of the user, the tracking multiple targets comprising racking focus between the multiple targets.

18. The method according to claim 1 , further comprising:

tracking multiple targets on the display based on a secondary user input from the user that is independent of a gaze location of the user and a head position of the user; and

framing a selected target of the multiple targets based on a combination of one or more of aesthetic cropping heuristics, using weighted gaze location over a period of time, and inertial measurement unit data for horizon leveling.

19. An augmented reality camera feedback system, the system comprising:

a physical hand-held camera system having a camera field of view, the camera field of view independent of a field of view of a user of the physical hand-held camera system;

a head-mounted augmented reality display at least partially overlaying the field of view of the user, wherein the head-mounted augmented reality display highlights a subset of the field of view of the user;

at least one processor; and

a memory coupled to the at least one processor, the memory storing instructions which are executable by the processor to perform a method comprising:

geometrically mapping the camera field of view to the field of view of the user, the mapping using an estimation of relative poses that defines an epipolar geometry relationship between the camera field of view and the field of view of the user; and

augmenting the field of view of the user via the head-mounted augmented reality display by highlighting a representation of a boundary of the camera field of view over the field of view of the user.

20. The augmented reality camera feedback system according to claim 19 , wherein the display comprises at least one of a semi-translucent surface, a holographic display, a light-field display, a waveguide display, a curved mirror display, a projector, and a visual prosthetic.

21. The augmented reality camera feedback system according to claim 19 , further comprising at least one sensor, the at least one sensor being at least one of a motion sensor, a depth rangefinder, a structured-light projector, and a lidar.

22. The augmented reality camera feedback system according to claim 21 , wherein the at least one sensor is disposed on at least one of the camera system and the display.

23. The augmented reality camera feedback system according to claim 19 , wherein the mapping includes generating a three-dimensional model based on information from at least one sensor.

24. A non-transitory computer readable medium storing computer-executable instructions that, when executed, cause at least one processor to:

determine a dynamic camera field of view of a physical hand-held camera system and a dynamic field of view of a user, the dynamic camera field of view of the camera system being independent of the field of view of the user;

geometrically map the camera field of view to the field of view of the user, the mapping using an estimation of relative poses that defines an epipolar geometry relationship between the camera field of view and the field of view of the user; and

augment the field of view of the user via a head-mounted augmented reality display at least partially overlaying the field of view of the user, the display highlighting a representation of a boundary of the dynamic camera field of view over the field of view of the user.

Assignments (2)

Continuity (2)

Cited By (13)

US 12,201,048 US 12,241,755 US 12,314,553 US 12,347,175 US 12,369,513 US 12,373,999 US 12,401,889 US 12,439,840 US 12,450,854 US 12,495,204 US 12,627,883 US 12,659,569 US 12,707,145