IP Library › Granted Patent US 11,743,444
Granted Patent B2
US 11,743,444 · App. 17/464,781 · Granted Aug 29, 2023

Electronic device and method for temporal synchronization of videos

Inventors: Brent Faust (San Diego, CA); Cheng-Yi Liu (San Jose, CA)
Assignee: SONY GROUP CORPORATION
H04N13/296H04N5/06H04N13/243H04N13/282H04N23/56
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,743,444
App. No.
17/464,781
Granted
Aug 29, 2023
Kind
B2
Abstract

Provided is an electronic device for a temporal synchronization, which determines a set of parameters associated with each imaging device of a plurality of imaging devices. The set of parameters include frame rate of each imaging device. The electronic device generates a synchronization signal that includes a preamble pulse of a first time duration set based on the frame rate and a sequence of alternating ON and OFF pulses. Each pulse of the sequence of alternating ON and OFF pulses is of a second time duration set based on the set of parameters. Based on the synchronization signal, lighting devices may be controlled to generate a pattern of alternating light pulses that is captured by each imaging device. The electronic device further acquires a plurality of images that includes information about the pattern of alternating light pulses. The electronic device further synchronizes the plurality of images, based on the information.

Claims (87)

1. An electronic device, comprising:

circuitry communicatively coupled to a plurality of imaging devices and one or more first lighting devices, wherein the circuitry is configured to:

determine a set of parameters associated with each imaging device of the plurality of imaging devices, wherein the set of parameters includes a frame rate of each imaging device of the plurality of imaging devices;

generate a synchronization signal that includes a preamble pulse of a first time duration and a sequence of alternating ON and OFF pulses, wherein

each pulse of the sequence of alternating ON and OFF pulses is of a second time duration,

the first time duration of the preamble pulse is based on the frame rate of a first imaging device of the plurality of imaging devices and the frame rate of a second imaging device of the plurality of imaging devices, and

the second time duration is based on one or more first parameters of the determined set of parameters associated with each imaging device of the plurality of imaging devices;

control the one or more first lighting devices based on the generated synchronization signal to generate a pattern of alternating light pulses, wherein the generated pattern of alternating light pulses is captured by the plurality of imaging devices;

acquire a plurality of images captured by the plurality of imaging devices, wherein the acquired plurality of images includes information about the captured pattern of alternating light pulses;

determine, for each imaging device of the plurality of imaging devices, a first frame in the acquired plurality of images, wherein the first frame corresponds to an end of a preamble pulse of the captured pattern of alternating light pulses;

determine, for each imaging device of the plurality of imaging devices, an amount of light intensity in each frame of a plurality of subsequent frames of the determined first frame;

generate, for each imaging device of the plurality of imaging devices, counter information based on the determined amount of light intensity in each frame of the plurality of subsequent frames, wherein

the generated counter information for each imaging device of the plurality of imaging devices includes:

a first number of frames of a first set of frames corresponding to an OFF pulse pattern of the captured pattern of alternating light pulses, and

a second number of frames of a second set of frames corresponding to an ON pulse pattern of the captured pattern of alternating light pulses, and

the plurality of subsequent frames includes the first set of frames and the second set of frames;

determine a framing offset for each imaging device of the plurality of imaging devices, based on the generated counter information and the determined set of parameters associated with each imaging device of the plurality of imaging devices; and

synchronize the acquired plurality of images, based on the determined framing offset of each imaging device of the plurality of imaging devices.

2. The electronic device according to claim 1 , wherein

the determined set of parameters further includes a first positive integer value and a second positive integer value corresponding to each imaging device of the plurality of imaging devices, and

the circuitry is further configured to determine the first positive integer value and the second positive integer value based on the frame rate of each imaging device of the plurality of imaging devices.

3. The electronic device according to claim 2 , wherein

the circuitry is further configured to determine the second time duration of each pulse of the synchronization signal based on the frame rate, the determined first positive integer value, and the determined second positive integer value associated with each imaging device of the plurality of imaging devices, and

each of the determined first positive integer value and the determined second positive integer value corresponds to the one or more first parameters.

4. The electronic device according to claim 2 , wherein a total duration of the sequence of alternating ON and OFF pulses of the synchronization signal is based on one or more second parameters of the determined set of parameters.

5. The electronic device according to claim 4 , wherein

the determined set of parameters further includes a third positive integer value that corresponds to the one or more second parameters, and

the circuitry is further configured to determine the total duration of the sequence of alternating ON and OFF pulses of the synchronization signal based on the determined first positive integer value and the third positive integer value.

6. The electronic device according to claim 4 , wherein the one or more first parameters of the determined set of parameters are different from the one or more second parameters of the determined set of parameters.

7. The electronic device according to claim 1 , wherein the circuitry is further configured to:

determine, for each imaging device of the plurality of imaging devices, a first OFF threshold value associated with presence of the OFF pulse pattern of the pattern of alternating light pulses;

determine, for each imaging device of the plurality of imaging devices, an amount of light intensity in the first frame of the plurality of images; and

select, for each imaging device of the plurality of imaging devices, the first frame in the plurality of images, based on a determination that the determined amount of light intensity in the first frame is less than the determined first OFF threshold value.

8. The electronic device according to claim 7 , wherein

the circuitry is further configured to determine, for each imaging device of the plurality of imaging devices, a set of incremental counter values included in the counter information,

a number of counter values of the set of incremental counter values is based on a first positive integer value corresponding to each imaging device of the plurality of imaging devices and a third positive integer value, and

each counter value in the set of incremental counter values corresponds to one of the first number of frames or the second number of frames of the plurality of subsequent frames.

9. The electronic device according to claim 8 , wherein the circuitry is further configured to determine the first number of frames of the first set of frames based on a determination that the determined amount of light intensity in the first set of frames is less than the determined first OFF threshold value and a determination that a counter index number of a corresponding counter value in the set of incremental counter values is an even number.

10. The electronic device according to claim 8 , wherein the circuitry is further configured to:

determine, for each imaging device of the plurality of imaging devices, a first ON threshold value associated with presence of the ON pulse pattern of the pattern of alternating light pulses; and

determine the second number of frames of the second set of frames based on a determination that the determined amount of light intensity in the second set of frames is less than the determined first ON threshold value and a determination that a counter index number of a corresponding counter value in the set of incremental counter values is an odd number.

11. The electronic device according to claim 8 , wherein the circuitry is further configured to:

determine a base index value of the set of incremental counter values, based on each counter value in the set of incremental counter values, the first positive integer value and the third positive integer value;

determine a resultant index value of the set of incremental counter values, based on the determined base index value of the set of incremental counter values and the determined amount of light intensity in each frame of the plurality of subsequent frames; and

determine the framing offset for each imaging device of the plurality of imaging devices, based on the resultant index value of the set of incremental counter values included in the generated counter information, the frame rate of each imaging device of the plurality of imaging devices, and the first positive integer value includcd in the set of parameters associated with each imaging device of the plurality of imaging devices.

12. The electronic device according to claim 1 , wherein the circuitry is further configured to temporally synchronize the plurality of images captured by the plurality of imaging devices, based on an alignment of the plurality of images corresponding to the framing offset determined for each imaging device of the plurality of imaging devices.

13. A method, comprising:

in an electronic device communicatively coupled to a plurality of imaging devices and one or more first lighting devices:

determining a set of parameters associated with each imaging device of the plurality of imaging devices, wherein the set of parameters includes a frame rate of each imaging device of the plurality of imaging devices;

generating a synchronization signal that includes a preamble pulse of a first time duration and a sequence of alternating ON and OFF pulses, wherein

each pulse of the sequence of alternating ON and OFF pulses is of a second time duration,

the first time duration of the preamble pulse is based on the frame rate of a first imaging device of the plurality of imaging devices and the frame rate of a second imaging device of the plurality of imaging devices, and

the second time duration is based on one or more first parameters of the determined set of parameters associated with each imaging device of the plurality of imaging devices;

controlling the one or more first lighting devices based on the generated synchronization signal to generate a pattern of alternating light pulses, wherein the generated pattern of alternating light pulses is captured by the plurality of imaging devices;

acquiring a plurality of images captured by the plurality of imaging devices, wherein the acquired plurality of images includes information about the captured pattern of alternating light pulses;

determining, for each imaging device of the plurality of imaging devices, a first frame in the acquired plurality of images, wherein the first frame corresponds to an end of a preamble pulse of the captured pattern of alternating light pulses;

determining, for each imaging device of the plurality of imaging devices, an amount of light intensity in each frame of a plurality of subsequent frames of the determined first frame;

generating, for each imaging device of the plurality of imaging devices, counter information based on the determined amount of light intensity in each frame of the plurality of subsequent frames, wherein

the generated counter information for each imaging device of the plurality of imaging devices includes:

a first number of frames of a first set of frames corresponding to an OFF pulse pattern of the captured pattern of alternating light pulses, and

a second number of frames of a second set of frames corresponding to an ON pulse pattern of the captured pattern of alternating light pulses, and

the plurality of subsequent frames includes the first set of frames and the second set of frames;

determining a framing offset for each imaging device of the plurality of imaging devices, based on the generated counter information and the determined set of parameters associated with each imaging device of the plurality of imaging devices; and

synchronizing the acquired plurality of images, based on the determined framing offset of each imaging device of the plurality of imaging devices.

14. The method according to claim 13 , wherein

the determined set of parameters further includes a first positive integer value and a second positive integer value corresponding to each imaging device of the plurality of imaging devices, and

the first positive integer value and the second positive integer value are determined based on the frame rate of each imaging device of the plurality of imaging devices.

15. The method according to claim 13 , wherein a total duration of the sequence of alternating ON and OFF pulses of the synchronization signal is based on one or more second parameters of the determined set of parameters.

16. The method according to claim 15 , wherein the one or more first parameters of the determined set of parameters are different from the one or more second parameters of the determined set of parameters.

17. The method according to claim 13 , wherein the plurality of images captured by the plurality of imaging devices is temporally synchronized, based on an alignment of the plurality of images corresponding to the framing offset determined for each imaging device of the plurality of imaging devices.

18. A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by an electronic device, causes the electronic device to execute operations, the operations comprising:

determining a set of parameters associated with each imaging device of a plurality of imaging devices, wherein the set of parameters includes a frame rate of each imaging device of the plurality of imaging devices;

generating a synchronization signal that includes a preamble pulse of a first time duration and a sequence of alternating ON and OFF pulses, wherein

each pulse of the sequence of alternating ON and OFF pulses is of a second time duration,

the first time duration of the preamble pulse is based on the frame rate of a first imaging device of the plurality of imaging devices and the frame rate of a second imaging device of the plurality of imaging devices, and

the second time duration is based on one or more first parameters of the determined set of parameters associated with each imaging device of the plurality of imaging devices;

controlling one or more first lighting devices based on the generated synchronization signal to generate a pattern of alternating light pulses, wherein the generated pattern of alternating light pulses is captured by the plurality of imaging devices;

acquiring a plurality of images captured by the plurality of imaging devices, wherein the acquired plurality of images includes information about the captured pattern of alternating light pulses;

determining, for each imaging device of the plurality of imaging devices, a first frame in the acquired plurality of images, wherein the first frame corresponds to an end of a preamble pulse of the captured pattern of alternating light pulses;

determining, for each imaging device of the plurality of imaging devices, an amount of light intensity in each frame of a plurality of subsequent frames of the determined first frame;

generating, for each imaging device of the plurality of imaging devices, counter information based on the determined amount of light intensity in each frame of the plurality of subsequent frames, wherein

the generated counter information for each imaging device of the plurality of imaging devices includes:

a first number of frames of a first set of frames corresponding to an OFF pulse pattern of the captured pattern of alternating light pulses, and

a second number of frames of a second set of frames corresponding to an ON pulse pattern of the captured pattern of alternating light pulses, and

the plurality of subsequent frames includes the first set of frames and the second set of frames;

determining a framing offset for each imaging device of the plurality of imaging devices, based on the generated counter information and the determined set of parameters associated with each imaging device of the plurality of imaging devices; and

synchronizing the acquired plurality of images, based on the determined framing offset of each imaging device of the plurality of imaging devices.

Assignments (2)
CORRECTIVE ASSIGNMENT TO CORRECT THE SECOND INVENTOR'S NAME PREVIOUSLY RECORDED AT REEL: 057572 FRAME: 0136. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT . Recorded Jan 17, 2022
From: FAUST, BRENT; LIU, CHENG-YI
To: SONY GROUP CORPORATION
Reel/Frame 058751/0198 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 23, 2021
From: FAUST, BRENT; LIU, CHENG-UI
To: SONY GROUP CORPORATION
Reel/Frame 057572/0136 →
Continuity (1)
Related Publication 20230072555A1 · Mar 9, 2023
Cited By (1)
US 12,342,026