IP Library › Granted Patent US 12,707,023
Granted Patent B2
US 12,707,023 · App. 18/463,994 · Granted Aug 11, 2026

Interpolation based camera motion for transitioning between best overview frames in live video

Inventors: Kacper Wysocki (Oslo, NO); Mats Gabriel Love Johansen (Oslo, NO); Kai Alexander Wig (Oslo, NO); Mona Kleven Lauritzen (Oslo, NO)
Assignee: Huddly AS
H04N7/147G06F18/214G06V10/764G06V10/82G06V20/49G06V20/53G06V40/20H04N5/2628H04N7/15H04N7/183H04N23/61H04N23/69H04N23/695H04L65/403
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,707,023
App. No.
18/463,994
Filed
Sep 8, 2023
Granted
Aug 11, 2026
Kind
B2
Art Unit
2488
USPC
348/169
Abstract

The present invention is a method and a processing device that allows smoothing of all camera parameters that are changeable while the camera is running through a tasklist component that schedules transitions synchronized with the output video frame rate. A device-native thread runs the scheduler which picks smoothing tasks off a list to meet real-time requirements. Different strategies for smoothing based on Bézier curves, splines and linear interpolation are scheduled on the runtime. The camera, which is automatically able to detect people and frame them based on where they are, uses this module to smoothly change the frame when people move in the camera field of view, among other functions.

Claims (51)

1 . A camera system, comprising:

at least one image sensor configured to capture an overview video stream; and

a video processing unit configured to:

output at least a portion of the overview video stream as a primary video stream,

wherein the primary video stream includes an initial image frame associated with a first set of camera parameter values, a target image frame associated with a second set of camera parameter values, and at least one intermediate image frame between the initial image frame and the target image frame,

wherein a transition from the initial image frame to the target image frame, in the primary video stream, involves a non-linear change of at least one camera parameter value between the first set of camera parameter values and the second set of camera parameter values,

wherein the at least one camera parameter value includes a zoom level, and

wherein a transition time for the transition from the initial image frame to the target image frame is selected based on:

whether a change from the initial image frame to the target image frame involves zooming in or zooming out, and

a magnitude of a difference in zoom between the first set of camera parameter values and the second set of camera parameter values.

2 . The camera system of claim 1 , wherein the at least one camera parameter value further includes at least one of a pan or tilt, or zoom level.

3 . The camera system of claim 1 , wherein the at least one camera parameter value further includes white balance.

4 . The camera system of claim 1 , wherein the initial image frame is associated with a current view of the primary video stream, and the target image frame is associated with a new desired view of the primary video stream.

5 . The camera system of claim 1 , wherein the transition from the initial image frame to the target image frame through the at least one intermediate image frame runs in lockstep with an output video frame rate of the at least one image sensor.

6 . The camera system of claim 1 , wherein the non-linear change of the at least one camera parameter value tracks a parametric equation.

7 . The camera system of claim 6 , wherein the parametric equation is associated with a Bezier curve.

8 . The camera system of claim 1 , wherein the non-linear change of the at least one camera parameter value determines a smoothness of the transition.

9 . The camera system of claim 1 , wherein the video processing unit and the at least one image sensor are located on a camera.

10 . A camera system, comprising:

at least one image sensor configured to capture an overview video stream; and

a video processing unit configured to:

output at least a portion of the overview video stream as a primary video stream,

wherein the primary video stream includes an initial image frame associated with a first set of camera parameter values, a target image frame associated with a second set of camera parameter values, and a number of intermediate image frames between the initial image frame and the target image frame,

wherein a transition time for a transition from the initial image frame to the target image frame is selected based on:

whether a change from the initial image frame to the target image frame involves zooming in or zooming out, and

a magnitude of a difference in zoom between the first set of camera parameter values and the second set of camera parameter values,

wherein the transition runs in lockstep with an output video frame rate of the at least one image sensor, and

wherein the number of intermediate image frames corresponds to the transition time and the output video frame rate.

11 . The camera system of claim 10 , wherein a rate of change in at least one camera parameter value between the initial image frame and the target image frame is nonlinear.

12 . The camera system of claim 10 , wherein the video processing unit and the at least one image sensor are located on a camera.

13 . A camera system, comprising:

at least one image sensor configured to capture an overview video stream; and

a video processing unit configured to output at least a portion of the overview video stream as a primary video stream, wherein the primary video stream includes at least an initial image frame associated with a first set of camera parameter values, and wherein the video processing unit is further configured to:

automatically determine for the primary video stream, and based on a detected first frame trigger, a first target image frame associated with a second set of camera parameter values;

initiate a first frame transition from the initial image frame to the first target image frame, via a set of intermediate image frames, wherein the first frame transition includes a change in at least one camera parameter value between the initial image frame and the first target image frame;

automatically determine for the primary video stream, and based on a detected second frame trigger, a second target image frame associated with a third set of camera parameter values;

alter at least one aspect of the first frame transition; and

initiate, prior to completion of the first frame transition, a second frame transition from a current image frame, among the set of intermediate image frames, to the second target image frame, wherein the second frame transition includes a change in at least one camera parameter value between the current image frame and the second target image frame.

14 . The camera system of claim 13 , wherein the first frame trigger includes a first detected face.

15 . The camera system of claim 14 , wherein the second frame trigger includes a second detected face, and wherein the second target image frame is different from the first target image frame.

16 . The camera system of claim 13 , wherein the first frame transition is altered by foregoing at least a portion of the first frame transition.

17 . The camera system of claim 13 , wherein at least one of the first frame transition or the second frame transition involves a change in pan, tilt, or zoom level.

18 . The camera system of claim 13 , wherein an initial transition speed associated with the second frame transition is selected to match a current transition speed associated with the first frame transition.

19 . The camera system of claim 13 , wherein the video processing unit and the at least one image sensor are located on a camera.

20 . A camera system, comprising:

at least one image sensor configured to capture an overview video stream; and

a video processing unit configured to output at least a portion of the overview video stream as a primary video stream, wherein the primary video stream includes at least an initial image frame associated with a first set of camera parameter values, and wherein the video processing unit is further configured to:

automatically determine a target image frame for the primary video stream, wherein the target image frame is has a second set of camera parameter values, and wherein determination of the target image frame is based on a detected frame trigger, including an appearance of a partially occluded subject in one or more frames of the overview video stream; and

initiate a frame transition from the initial image frame to the target image frame, via a set of intermediate image frames, wherein the frame transition includes a change in at least one camera parameter value between the initial image frame and the target image frame.

21 . The camera system of claim 20 , wherein the video processing unit includes a hardware accelerated convolutional neural network.

22 . The camera system of claim 20 , wherein the video processing unit and the at least one image sensor are located on a camera.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 28, 2024
From: WYSOCKI, KACPER; WIG, KAI ALEXANDER; LOVE JOHANSEN, MATS GABRIEL; LAURITZEN, MONA KLEVEN
To: HUDDLY AS
Reel/Frame 067870/0248 →
Priority Claims (1)
NO 20190477 · Apr 8, 2019 · national
Continuity (2)
Continuation 17602121 · Apr 7, 2020
Related Publication 20230421723A1 · Dec 28, 2023
References Cited (35)
US 6297846B1 · Edanami · 2001 [cited by applicant]
US 10659731B2 · Harrison et al. · 2020 [cited by applicant]
US 20090015658A1 · Enstad et al. · 2009 [cited by applicant]
US 20090160936A1 · McCormack · 2009 [cited by examiner]
US 20100128145A1 · Pitts et al. · 2010 [cited by applicant]
US 20100238262A1 · Kurtz et al. · 2010 [cited by applicant]
US 20120169842A1 · Chuang · 2012 [cited by examiner]
US 20140049595A1 · Feng et al. · 2014 [cited by applicant]
US 20160277712A1 · Michot · 2016 [cited by applicant]
US 20170125064A1 · Aggarwal et al. · 2017 [cited by applicant]
US 20180063482A1 · Goesnar · 2018 [cited by examiner]
US 20180091727A1 · Pell et al. · 2018 [cited by applicant]
US 20180098026A1 · Gadnir et al. · 2018 [cited by applicant]
US 20180376108A1 · Bright-Thomas et al. · 2018 [cited by applicant]
US 20190007623A1 · Wang et al. · 2019 [cited by applicant]
US 20190313058A1 · Harrison et al. · 2019 [cited by applicant]
CN 1908962A · 2007 [cited by applicant]
CN 108605087A · 2018 [cited by applicant]
EP 3287947A1 · 2018 [cited by applicant]
Yokoi T, Fujiyoshi H. Virtual camerawork for generating lecture video from high resolution images. In2005 IEEE International Conference on Multimedia and Expo Jul. 6, 2005 (p. 4-pp). IEEE. (Year: 2005). [cited by examiner]
Shanmugasundaram, M. and Irani, P., 2008. The effect of animated transitions in zooming interfaces. In: Proceedings of the Working Conference on Advanced Visual Interfaces [Internet]. (Year: 2008). [cited by examiner]
Buckley C. Bezier curves for camera motion. Trinity College Dublin, Department of Computer Science; 1994. (Year: 1994). [cited by applicant]
Office Action from the European Patent Office in Application No. 20722224.1 dated Oct. 5, 2023 (8 pages). [cited by applicant]
International Search Report and Written Opinion of the International Search Authority in International Application No. PCT/EP2020/059928 dated Jul. 14, 2020 (12 pages). [cited by applicant]
Office Action and Search Report from the Norwegian Patent Office for Application No. 20190477 dated Nov. 7, 2019 (9 pages). [cited by applicant]
Kaipeng Zhang et al., “Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks”., 2016 (5 pages). [cited by applicant]
T. Yokoi et al., “Virtual Camerawork for Generating Lecture Video from High Resolution Images”., 2005, IEEE International Conference On Multimedia and Expo, pp. 751-754 (4 pages). [cited by applicant]
Degeler Andrii., “Norwegian intelligent camera company Huddly acquires AI startup Epigram”., 2018, Tech.eu, pp. 1-7 (2 pages). [cited by applicant]
“Convolutional neural network”., 2019, Wikipedia, pp. 1-25 (25 pages). [cited by applicant]
Saypadith Savath et al., “Real-Time Multiple Face Recognition using Deep Learning on Embedded GPU System”., 2018, APSIPA Annual Summit and Conference 2018, pp. 1318-1324 (7 pages). [cited by applicant]
Xinding Sun et al., “Region of interest extraction and virtual camera control based on panoramic video capturing”., 2005, IEEE Transactions On Multimedia, pp. 981-990 (10 pages). [cited by applicant]
Summons to Attend Oral Proceedings from the European Patent Office in Application No. 20722224.1 dated Aug. 23, 2024 (9 pages). [cited by applicant]
The First Office Action from the Chinese Patent Office in Application No. 202080037865.9 dated Dec. 31, 2024 (32 pages). [cited by applicant]
The Second Office Action of the China National Intellectual Property Administration in Application No. 202080037865.9 dated Jul. 10, 2025 (28 pages). [cited by applicant]
Rejection Decision of the China National Intellectual Property Administration in Application No. 202080037865.9 dated Mar. 11, 2026 (28 pages). [cited by applicant]