Granted Patent B2

US 12,231,780 · App. 18/059,822 · Granted Feb 18, 2025

Inventors: Nguyen Thang Long Le (Richardson, TX); Zeeshan Nadir (Richardson, TX); John W. Glotzbach (Allen, TX); Hamid R. Sheikh (Allen, TX)

Assignee: Samsung Electronics Co., Ltd.

H04N23/745G06T5/50G06T7/11G06V10/44G06V10/761G06T2207/10144G06T2207/20224

Abstract

Claims (107)

1. A method comprising:

obtaining multiple image frames;

selecting, using at least one processing device of an electronic device, an asymmetrical image pair from the multiple image frames, the asymmetrical image pair comprising a first image frame and a second image frame, the first image frame having a shorter exposure than the second image frame;

identifying, using the at least one processing device, one or more features based on the asymmetrical image pair, wherein identifying the one or more features based on the asymmetrical image pair comprises:

generating a difference image based on the first image frame and the second image frame; and

dividing the difference image into patches;

determining, using the at least one processing device, whether the first image frame contains flicker based on the one or more features; and

enabling or disabling, using the at least one processing device, the first image frame as a reference candidate based on the determination whether the first image frame contains flicker.

2. The method of claim 1 , wherein identifying the one or more features based on the asymmetrical image pair further comprises, for each patch of the difference image:

identifying a signal representing an average brightness across the patch; and

identifying a frequency signal representative of frequency components in the signal representing the average brightness across the patch.

3. The method of claim 2 , wherein determining whether the first image frame contains flicker comprises:

detecting a dominant frequency across each patch of the difference image based on the frequency signal for that patch;

searching the dominant frequencies across all patches for a largest normalized energy;

comparing the largest normalized energy to a threshold;

determining that the first image frame contains flicker when the largest normalized energy is greater than the threshold; and

determining that the first image frame does not contain flicker when the largest normalized energy is less than the threshold.

4. The method of claim 2 , wherein determining whether the first image frame contains flicker comprises:

detecting a dominant frequency across each patch of the difference image based on the frequency signal for that patch;

determining a weighted average of normalized energy across frequencies and patches;

comparing the weighted average of normalized energy to a threshold;

determining that the first image frame contains flicker when the weighted average of normalized energy is greater than the threshold; and

determining that the first image frame does not contain flicker when the weighted average of normalized energy is less than the threshold.

5. The method of claim 2 , wherein determining whether the first image frame contains flicker comprises:

detecting a dominant frequency across each patch of the difference image based on the frequency signal for that patch;

determining a number of patches for which the dominant frequency is greater than a first threshold;

determining that the first image frame contains flicker when the number of patches is greater than a second threshold; and

determining that the first image frame does not contain flicker when the number of patches is less than the second threshold.

6. The method of claim 2 , wherein:

determining whether the first image frame contains flicker comprises:

providing the frequency signals for the patches to a trained machine learning model; and

determining, using the trained machine learning model, whether the first image frame contains flicker; and

the trained machine learning model has been trained using multiple pairs of frequency signals, each pair of frequency signals including a flicker-free frequency signal and a corresponding frequency signal containing flicker.

7. The method of claim 1 , wherein:

the first image frame has an exposure value of EV-4 or EV-2; and

the second image frame has an exposure value of EV-0.

8. An electronic device comprising:

at least one processing device configured to:

obtain multiple image frames;

select an asymmetrical image pair from the multiple image frames, the asymmetrical image pair comprising a first image frame and a second image frame, the first image frame having a shorter exposure than the second image frame;

identify one or more features based on the asymmetrical image pair, wherein, to identify the one or more features based on the asymmetrical image pair, the at least one processing device is configured to:

generate a difference image based on the first image frame and the second image frame; and

divide the difference image into patches;

determine whether the first image frame contains flicker based on the one or more features; and

enable or disable the first image frame as a reference candidate based on the determination whether the first image frame contains flicker.

9. The electronic device of claim 8 , wherein, to identify the one or more features based on the asymmetrical image pair, the at least one processing device is further configured, for each patch of the difference image, to:

identify a signal representing an average brightness across the patch; and

identify a frequency signal representative of frequency components in the signal representing the average brightness across the patch.

10. The electronic device of claim 9 , wherein, to determine whether the first image frame contains flicker, the at least one processing device is configured to:

detect a dominant frequency across each patch of the difference image based on the frequency signal for that patch;

search the dominant frequencies across all patches for a largest normalized energy;

compare the largest normalized energy to a threshold;

determine that the first image frame contains flicker when the largest normalized energy is greater than the threshold; and

determine that the first image frame does not contain flicker when the largest normalized energy is less than the threshold.

11. The electronic device of claim 9 , wherein to determine whether the first image frame contains flicker, the at least one processing device is configured to:

detect a dominant frequency across each patch of the difference image based on the frequency signal for that patch;

determine a weighted average of normalized energy across frequencies and patches;

compare the weighted average of normalized energy to a threshold;

determine that the first image frame contains flicker when the weighted average of normalized energy is greater than the threshold; and

determine that the first image frame does not contain flicker when the weighted average of normalized energy is less than the threshold.

12. The electronic device of claim 9 , wherein, to determine whether the first image frame contains flicker, the at least one processing device is configured to:

detect a dominant frequency across each patch of the difference image based on the frequency signal for that patch;

determine a number of patches for which the dominant frequency is greater than a first threshold;

determine that the first image frame contains flicker when the number of patches is greater than a second threshold; and

determine that the first image frame does not contain flicker when the number of patches is less than the second threshold.

13. The electronic device of claim 9 , wherein:

to determine whether the first image frame contains flicker, the at least one processing device is configured to:

provide the frequency signals for the patches to a trained machine learning model; and

determine, using the trained machine learning model, whether the first image frame contains flicker; and

the trained machine learning model has been trained using multiple pairs of frequency signals, each pair of frequency signals including a flicker-free frequency signal and a corresponding frequency signal containing flicker.

14. The electronic device of claim 8 , wherein:

the first image frame has an exposure value of EV-4 or EV-2; and

the second image frame has an exposure value of EV-0.

15. A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:

obtain multiple image frames;

select an asymmetrical image pair from the multiple image frames, the asymmetrical image pair comprising a first image frame and a second image frame, the first image frame having a shorter exposure than the second image frame;

identify one or more features based on the asymmetrical image pair, wherein the instructions that when executed cause the at least one processor to identify the one or more features based on the asymmetrical image pair comprise instructions that when executed cause the at least one processor to:

generate a difference image based on the first image frame and the second image frame; and

divide the difference image into patches;

determine whether the first image frame contains flicker based on the one or more features; and

enable or disable the first image frame as a reference candidate based on the determination whether the first image frame contains flicker.

16. The non-transitory machine-readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor to identify the one or more features based on the asymmetrical image pair further comprise instructions that when executed cause the at least one processor to:

for each patch of the difference image:

identify a signal representing an average brightness across the patch; and

identify a frequency signal representative of frequency components in the signal representing the average brightness across the patch.

17. The non-transitory machine-readable medium of claim 16 , wherein the instructions that when executed cause the at least one processor to determine whether the first image frame contains flicker comprise instructions that when executed cause the at least one processor to:

detect a dominant frequency across each patch of the difference image based on the frequency signal for that patch;

search the dominant frequencies across all patches for a largest normalized energy;

compare the largest normalized energy to a threshold;

determine that the first image frame contains flicker when the largest normalized energy is greater than the threshold; and

determine that the first image frame does not contain flicker when the largest normalized energy is less than the threshold.

18. The non-transitory machine-readable medium of claim 16 , wherein the instructions that when executed cause the at least one processor to determine whether the first image frame contains flicker comprise instructions that when executed cause the at least one processor to:

detect a dominant frequency across each patch of the difference image based on the frequency signal for that patch;

determine a weighted average of normalized energy across frequencies and patches;

compare the weighted average of normalized energy to a threshold;

determine that the first image frame contains flicker when the weighted average of normalized energy is greater than the threshold; and

determine that the first image frame does not contain flicker when the weighted average of normalized energy is less than the threshold.

19. The non-transitory machine-readable medium of claim 16 , wherein the instructions that when executed cause the at least one processor to determine whether the first image frame contains flicker comprise instructions that when executed cause the at least one processor to:

detect a dominant frequency across each patch of the difference image based on the frequency signal for that patch;

determine a number of patches for which the dominant frequency is greater than a first threshold;

determine that the first image frame contains flicker when the number of patches is greater than a second threshold; and

determine that the first image frame does not contain flicker when the number of patches is less than the second threshold.

20. The non-transitory machine-readable medium of claim 16 , wherein:

the instructions that when executed cause the at least one processor to determine whether the first image frame contains flicker comprise instructions that when executed cause the at least one processor to:

provide the frequency signals for the patches to a trained machine learning model; and

determine, using the trained machine learning model, whether the first image frame contains flicker; and

the trained machine learning model has been trained using multiple pairs of frequency signals, each pair of frequency signals including a flicker-free frequency signal and a corresponding frequency signal containing flicker.

Assignments (1)

Continuity (3)

References Cited (18)

Cited By (2)

US 12,713,141 US 12,745,011