System for wireless synchronized capturing, and smart device
The present disclosure provides a smart device. The smart device includes: single or multiple image sensors, a communication device, a processor; and a memory storing a computer program that, when executed by the processor, causes the processor to: synchronize a time of single or multiple timers of the smart device or the image sensors among the multiple smart devices, based on a protocol for time synchronization on wireless communication by using the communication device; and directly or indirectly control shutter opening and closing timing of each image sensor, in order to keep constancy of shooting time interval and phase, wherein the phase is a remainder obtained by dividing shooting time by the shooting time interval. The present disclosure improves the accuracy of synchronized capturing.
1 . A smart device for use in a system comprising a plurality of smart devices, the smart device comprising:
one or more image sensors,
a communication device,
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to:
synchronize a time of one or more timers of the smart device or the image sensors among the plurality of smart devices, based on a protocol for time synchronization on wireless communication by using the communication device; and
control a timing of shutter opening and closing timing of each image sensor based on the synchronized time, wherein the timing is adjusted to keep consistent shooting time interval and phase, wherein the phase is a remainder obtained by dividing shooting time by the shooting time interval.
2 . The smart device according to claim 1 , wherein shooting settings are adjusted by a user manually or adjusted by the user partially after auto adjustment shooting settings, comprising one or more of:
shooting time interval itself, target phase itself, image resolution, sensor type, exposure time, sensor sensitivity, focus distance, lens aperture, white balance, gamma correction, image un-distortion, image post-processing, or image recognition by additional machine learning system.
3 . The smart device according to claim 1 , wherein the instructions further cause the processor to:
attach a timestamp of the timer of the smart device or the image sensor to each image according to a consistent rule;
measure elapsed times between adjacent captured frame timestamps;
get an actual shooting time interval by averaging of measured elapsed times; and
confirm stability of shooting time interval by checking if variance of measured elapsed times is smaller than a predetermined threshold.
4 . The smart device according to claim 1 , wherein the instructions further cause the processor to:
perform injection shooting with some exposure time lengths and then measure an actual phase shift length;
test multiple injection shots with different exposure time length, and clarify a model of a relationship between injection length and phase shift length in run-time under some shooting settings; and
if the model of the relationship is fixed for some types of smart devices and some shooting settings, calculate inversely a best exposure time length of an injection shot from a current phase error.
5 . The smart device according to claim 4 , wherein when different elapsed time after the injection shot appears and the time lengths are A and B, A is smaller than constant shooting time interval, B is larger than the constant shooting time interval, both of A and B appear only once after the injection shot, A+B is equal to 2T+δ, and Tis a controllable injection shot time length and δ is device-specific constant time length, the instructions causing the processor to:
get the constant shooting time interval and check its stability by measuring the shooting time intervals multiple times;
measure A+B after the injection shot in which length of shooting time is T;
obtain an average value of δ by multiple values calculated by A+B−2T, and check the stability of δ;
obtain the current phase error δ equal to phase minus target phase which is half of the shooting time interval;
obtain T so that ε+2T+δ is equal to integral multiplication of shooting time interval; and
perform an injection shot in which length of shooting time is T, to make phase error zero.
6 . The smart device according to claim 1 , wherein when the smart device is selected by one or more other smart devices, the instructions further cause the processor to:
perform an automatic adjustment of shooting settings and wait for conversion of the shooting settings;
measure an adjustment result;
broadcast the adjustment result to one or more other devices, to capture with the same shooting settings by the plurality of smart devices;
periodically determine a phase error at each smart device; and
when a user starts still capturing or continuous capturing, capture images synchronously among all the smart devices, wherein shooting time intervals are the same and phase error is zero.
7 . The smart device according to claim 1 , wherein when the smart device selects other smart device, the instructions further cause the processor to:
receive a result of automatic adjustment of shooting settings from the other selected device, to capture with the same shooting settings by the plurality of smart devices;
periodically determine a phase error at each smart device; and
when a user starts still capturing or continuous capturing, capture images synchronously among all the smart devices, wherein shooting time intervals are the same and phase error is zero.
8 . The smart device according to claim 1 , wherein the instructions further cause the processor to:
display a user interface to let a user select “Auto”, “Manual”, or “Follow”, a following device, and related parameters, for each setting item and parameter of shooting settings; and
when “Auto” is selected, update related settings automatically and independently;
when “Manual” is selected, use user's manual settings independently; or
when “Follow” is selected, use following device's settings.
9 . The smart device according to claim 1 , wherein when the system can control shutter open/close timing directly in real-time, to realize synchronized capturing, the instructions further cause the processor to:
synchronize times between timers of devices or sensors based on a protocol for time synchronization on wireless communication by using the communication device;
if the image sensor does not have its own timer, use a smart device timer with time lag correction;
set the timer to trigger the image sensor to open its shutter at the time when (t % p) is equal to (φ−E/2) % p, and to close its shutter at the time when (t % p) is equal to (φ+E/2) % p, so that the image sensor sends captured image to memory with a timestamp of (t−t % p)*p+φ, wherein t is timer's time, p is shooting time interval, φ is target phase which is half of the shooting time interval, E is exposure time length, and % means MOD operation; and
when users or a program changes the settings, set the changed settings to those of the timer again.
10 . The smart device according to claim 1 , wherein the instructions further cause the processor to:
display a user interface to start still shooting and continuous shooting on all the smart devices;
when a user operates the user interface to start shooting, broadcast the shooting start commands with a target timestamp to all the smart devices participating in sync-capture;
retain recent captured images by any of the smart devices on its memory for a period of time, to save the captured images on its memory to its storage when necessary;
when the smart device receives shooting a start command or a shooting stop command, save target images to its storage; and
send the saved images to other devices or cloud for data sharing.
11 . A wireless synchronized image capturing system, comprising:
a plurality of smart devices, each smart device, comprising:
one or more image sensors,
a communication device,
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to:
synchronize a time of one or more timers of the smart device or the image sensors among the plurality of smart devices, based on a protocol for time synchronization on wireless communication by using the communication device; and
control a timing of shutter opening and closing timing of each image sensor, to keep constancy of shooting time interval and phase, wherein the phase is a remainder obtained by dividing shooting time by the shooting time interval.
12 . The system according to claim 11 , wherein shooting settings are adjusted by a user manually or adjusted by the user partially after auto adjustment shooting settings, comprising one or more of:
shooting time interval itself, target phase itself, image resolution, sensor type, exposure time, sensor sensitivity, focus distance, lens aperture, white balance, gamma correction, image un-distortion, image post-processing, or image recognition by additional machine learning system.
13 . The system according to claim 11 , wherein the instructions further cause the processor to:
attach a timestamp of the timer of the smart device or the image sensor to each image according to a consistent rule;
measure elapsed times between adjacent captured frame timestamps;
get an actual shooting time interval by averaging of measured elapsed times; and
confirm stability of shooting time interval by checking if variance of measured elapsed times is smaller than a predetermined threshold.
14 . The system according to claim 11 , wherein the instructions further cause the processor to:
perform injection shooting with some exposure time lengths and then measure an actual phase shift length;
test multiple injection shots with different exposure time length, and clarify a model of a relationship between injection length and phase shift length in run-time under some shooting settings; and
if the model of the relationship is fixed for some types of smart devices and some shooting settings, calculate inversely a best exposure time length of an injection shot from a current phase error.
15 . The system according to claim 14 , wherein when different elapsed time after the injection shot appears and the time lengths are A and B, A is smaller than constant shooting time interval, B is larger than the constant shooting time interval, both of A and B appear only once after the injection shot, A+B is equal to 2T+δ, and T is a controllable injection shot time length and δ is device-specific constant time length, the instructions causing the processor to:
get the constant shooting time interval and check its stability by measuring the shooting time intervals multiple times;
measure A+B after the injection shot in which length of shooting time is T;
obtain an average value of δ by multiple values calculated by A+B−2T, and check the stability of δ;
obtain the current phase error δ equal to phase minus target phase which is half of the shooting time interval;
obtain T so that ε+2T+δ is equal to integral multiplication of shooting time interval; and
perform an injection shot in which length of shooting time is T, to make phase error zero.
16 . The system according to claim 11 , wherein when the smart device is selected by one or more other smart devices, the instructions further cause the processor to:
perform an automatic adjustment of shooting settings and wait for conversion of the shooting settings;
measure an adjustment result;
broadcast the adjustment result to one or more other devices, to capture with the same shooting settings by the plurality of smart devices;
periodically determine a phase error at each smart device; and
when a user starts still capturing or continuous capturing, capture images synchronously among all the smart devices, wherein shooting time intervals are the same and phase error is zero.
17 . The system according to claim 11 , wherein when the smart device selects the other smart device, the instructions further cause the processor to:
receive a result of automatic adjustment of shooting settings from the other selected device, to capture with the same shooting settings by the plurality of smart devices;
periodically determine a phase error at each smart device; and
when a user starts still capturing or continuous capturing, capture images synchronously among all the smart devices, wherein shooting time intervals are the same and phase error is zero.
18 . The system according to claim 11 , wherein the instructions further cause the processor to:
display a user interface to let a user select “Auto”, “Manual”, or “Follow”, a following device, and related parameters, for each setting item and parameter of shooting settings; and
when “Auto” is selected, update related settings automatically and independently;
when “Manual” is selected, use user's manual settings independently; or
when “Follow” is selected, use following device's settings.
19 . A method for operating a smart device, comprising:
synchronizing, by a processor, a time of one or more timers of the smart device or image sensors among a plurality of smart devices, based on a protocol for time synchronization on wireless communication by using a communication device, wherein the smart device comprises the one or more image sensors, the communication device, and the processor; and
controlling, by the processor, shutter opening and closing timing of each image sensor based on the synchronized time, wherein the timing is adjusted to keep consistent shooting time interval and phase, wherein the phase is a remainder obtained by dividing shooting time by the shooting time interval.
20 . A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a processor, cause the processor to:
synchronize a time of one or more timers of a smart device or image sensors among a plurality of smart devices, based on a protocol for time synchronization on wireless communication by using a communication device; and
control shutter opening and closing timing of each image sensor, to keep constancy of shooting time interval and phase, wherein the phase is a remainder obtained by dividing shooting time by the shooting time interval.