Object extraction
View Patent ↗A video sequence is processed to extract one or more objects. The extracted objects are used to create a new video sequence. In particular, a user is able to select a background and/or an audio recording. Alternatively, the object extraction system automatically selects a background and/or audio recording. The extracted objects are integrated into the selected background and/or the audio recording is added to form a new video sequence.
1. A method for classifying an object in a non-static frame as foreground or background, comprising:
(a) classifying a pixel in the non-static frame as foreground when a value component of the pixel exceeds a mean value of the value component for the pixel across multiple frames by a first value;
(b) classifying the pixel as background when the pixel is in shadow;
(c) classifying the pixel as background when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value; and
(d) classifying the pixel as foreground when the pixel is not classified by acts (a), (b), or (c).
2. The method of claim 1 , wherein the first value comprises a first threshold.
3. The method of claim 2 , wherein the first threshold is an empirical value of 75.
4. The method of claim 1 , wherein the second value comprises a standard deviation of a component multiplied by a second threshold.
5. The method of claim 1 wherein the second value is dependent on illumination conditions.
6. The method of claim 1 , further comprising:
adjusting one or more pixels of the non-static frame by applying a filter core to the non-static frame.
7. The method of claim 6 , wherein the non-static frame is a first frame, and further comprising:
mapping each pixel in the first frame to a binary value;
generating a second frame comprised of the binary values of each pixel; and
applying the filter core to the second frame.
8. The method of claim 1 , wherein the non-static frame is part of a sequence of frames and wherein one of the other frames in the sequence is a static frame.
9. The method of claim 8 , further comprising identifying the static frame in the sequence.
10. The method of claim 1 , wherein a component is usable if the standard deviation of the component does not exceed a threshold.
11. The method of claim 10 , wherein the threshold comprises 25.
12. The method of claim 10 , wherein the component is a hue component.
13. The method of claim 10 , wherein the component is a saturation component.
14. The method of claim 1 , further comprising:
extracting a foreground object by selecting one or more pixels classified as foreground.
15. The method of claim 14 , further comprising:
extracting multiple foreground objects.
16. The method of claim 1 , further comprising:
providing a user interface, wherein the user interface enables a user to select a new background.
17. The method of claim 16 , further comprising:
extracting one or more foreground objects from the non-static frame; and
integrating the extracted foreground objects with the new background.
18. The method of claim 1 , further comprising:
providing a user interface, wherein the user interface enables a user to select an audio recording.
19. The method of claim 18 , further comprising:
extracting one or more foreground objects from the non-static frame; and
integrating the extracted foreground objects with the audio recording.
20. The method of claim 1 , wherein the non-static frame comprises pixels in a RGB color space and further comprising:
converting the RGB color space to a type of HSV color space.
21. A method for processing a sequence of frames, comprising:
extracting one or more objects from a frame by classifying each pixel of the frame as being background or foreground, said classifying each pixel comprising:
(a) classifying a pixel in the frame as foreground when a value component of the pixel exceeds a mean value of the value component for the pixel across multiple frames by a first value;
(b) classifying the pixel as background when the pixel is in shadow;
(c) classifying the pixel as background when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value; and
(d) classifying the pixel as foreground when the pixel is not classified by acts (a), (b), or (c);
selecting a new background;
selecting an audio recording; and
integrating the extracted objects with the new background and the audio recording.
22. The method of claim 21 , further comprising:
applying a filter core to the frame to confirm that each pixel has been accurately classified as background or foreground.
23. A system, comprising:
a computer including a memory;
a sequence of frames stored in the memory; and
a computer program executable by the computer, the computer program comprising instructions for classifying an object in a non-static frame from the sequence of frame as foreground or background, the instructions comprising:
(a) classifying in a pixel in the non-static frame as foreground when a value component of the pixel exceeds a mean value of the value component for the pixel across multiple frames in the sequence of frames by a first value;
(b) classifying the pixel as background when the pixel is in shadow;
(c) classifying the pixel as background when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value; and
(d) classifying the pixel as foreground when the pixel is not classified by acts (a), (b), or (c).
24. The system of claim 23 , wherein the first value comprises a first threshold.
25. The system of claim 24 , wherein the first threshold is an empirical value of 75.
26. The system of claim 23 , wherein the second value comprises a standard deviation of a component multiplied by a second threshold.
27. The system of claim 23 wherein the second value is dependent on illumination conditions.
28. The system of claim 23 , wherein the computer program further comprises instructions for:
adjusting one or more pixels of the non-static frame by applying a filter core to the frame.
29. The system of claim 28 , wherein the non-static frame is a first frame, and wherein the computer program further comprises instructions for:
mapping each pixel in the first frame to a binary value;
generating a second frame comprised of the binary values of each pixel; and
applying the filter core to the second frame.
30. The system of claim 23 , wherein one of the frames in the sequence of frames comprises a static frame.
31. The system of claim 30 , wherein the computer program further comprises instructions for:
identifying the static frame in the sequence of frames.
32. The system of claim 23 , wherein a component is usable if the standard deviation of the component does not exceed a threshold.
33. The system of claim 32 , wherein the threshold comprises 25.
34. The system of claim 32 , wherein the component is a hue component.
35. The system of claim 32 , wherein the component is a saturation component.
36. The system of claim 23 , wherein the computer program further comprises instructions for:
extracting a foreground object by selecting one or more pixels classified as foreground.
37. The system of claim 36 , wherein the computer program further comprises instructions for:
extracting multiple foreground objects.
38. The system of claim 23 , wherein the computer program further comprises instructions for:
providing a user interface, wherein the user interface enables a user to select a new background.
39. The system of claim 38 , wherein the computer program further comprises instructions for:
extracting one or more foreground objects from the non-static frame; and
integrating the extracted foreground objects with the new background.
40. The system of claim 23 , wherein the computer program further comprises instructions for:
providing a user interface, wherein the user interface enables a user to select an audio recording.
41. The system of claim 40 , wherein the computer program further comprises instructions for:
extracting one or more foreground objects from the non-static frame; and
integrating the extracted foreground objects with the audio recording.
42. The system of claim 23 , wherein the non-static frame comprises pixels in a RGB color space and wherein the computer program further comprises instructions for:
converting the RGB color space to a type of HSV color space.
43. A system, comprising:
means for extracting one or more objects from a frame by classifying each pixel of the frame as being background or foreground, said classifying comprising:
(a) classifying a pixel as foreground when a value component of the pixel exceeds a mean value of the value component for the pixel across multiple frames by a first value;
(b) classifying the pixel as background when the pixel is in shadow;
(c) classifying the pixel as background when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value; and
(d) classifying the pixel as foreground when the pixel is not classified by acts (a), (b), or (c);
means for selecting a new background;
means for selecting an audio recording; and
means for integrating the extracted objects with the new background and audio recording.
44. The system of claim 43 , further comprising:
means for applying a filter core to the frame to confirm that each pixel has been accurately classified as background or foreground.
45. A computer readable storage medium encoded with a computer program executable by a computer, the computer program comprising instructions for classifying an object in a non-static frame as foreground or background, wherein the instructions comprise:
(a) classifying a pixel in the non-static frame as foreground when a value component of the pixel exceeds a mean value of the value component for the pixel across multiple frames by a first value;
(b) classifying the pixel as background when the pixel is in shadow;
(c) classifying the pixel as background when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value; and
(d) classifying the pixel as foreground when the pixel is not classified by acts (a), (b), or (c).
46. The computer readable storage medium of claim 45 , wherein the first value comprises a first threshold.
47. The computer readable storage medium of claim 46 , wherein the first threshold is an empirical value of 75.
48. The computer readable storage medium of claim 45 , wherein the second value comprises a standard deviation of a component multiplied by a second threshold.
49. The computer readable storage medium of claim 45 , wherein the second value is dependent on illumination conditions.
50. The computer readable storage medium of claim 45 , wherein the computer program further comprises instructions for:
adjusting one or more pixels of the non-static frame by applying a filter core to the frame.
51. The computer readable storage medium of claim 50 , wherein the non-static frame is a first frame, and wherein the computer program further comprises instructions for:
mapping each pixel in the first frame to a binary value;
generating a second frame comprised of the binary values of each pixel; and
applying the filter core to the second frame.
52. The computer readable storage medium of claim 45 , wherein the non-static frame is part of a sequence of frames and wherein one of the frames is a static frame.
53. The computer readable storage medium of claim 52 , wherein the computer program further comprises instructions for:
identifying the static frame in the sequence.
54. The computer readable storage medium of claim 45 , wherein a component is usable if the standard deviation of the component does not exceed a threshold.
55. The computer readable storage medium of claim 54 , wherein the threshold comprises 25.
56. The computer readable storage medium of claim 54 , wherein the component is a hue component.
57. The computer readable storage medium of claim 54 , wherein the component is a saturation component.
58. The computer readable storage medium of claim 45 , wherein the computer program further comprises instructions for:
extracting a foreground object by selecting one or more pixels classified as foreground.
59. The computer readable storage medium of claim 58 , wherein the computer program further comprises instructions for:
extracting multiple foreground objects.
60. The computer readable storage medium of claim 45 , wherein the computer program further comprises instructions for:
providing a user interface, wherein the user interface enables a user to select a new background.
61. The computer readable storage medium of claim 60 , wherein the computer program further comprises instructions for:
extracting one or more foreground objects from the non-static frame; and
integrating the extracted foreground objects with the new background.
62. The computer readable storage medium of claims 45 , wherein the computer program further comprises instructions for:
providing a user interface, wherein the user interface enables a user to select an audio recording.
63. The computer readable storage medium of claim 62 , wherein the computer program further comprises instructions for:
extracting one or more foreground objects from the non-static frame; and
integrating the extracted foreground objects with the audio recording.
64. The computer readable storage medium of claim 45 , wherein the non-static frame comprises pixels in a RGB color space and wherein the computer program further comprises instructions for:
converting the RGB color space to a type of HSV color space.
65. A computer readable storage medium encoded with software instructions, wherein the the software instructions comprise:
extracting one or more objects from a frame by classifying each pixel of the frame as being background or foreground, said classifying comprising:
(a) classifying a pixel in the frame as foreground when a value component of the pixel exceeds a mean value of the value component for the pixel across multiple frames by a first value;
(b) classifying the pixel as background when the pixel is in shadow;
(c) classifying the pixel as background when the pixel is under different illumination and each usable component of the pixel exceeds a mean value of that component by a second value; and
(d) classifying the pixel as foreground when the pixel is not classified by acts (a), (b), or (c);
selecting a new background;
selecting an audio recording; and
integrating the extracted objects with the new background and audio recording.
66. The computer readable storage medium of claim 65 , wherein the computer program further comprises instructions for:
applying a filter core to the frame to confirm that each pixel has been accurately classified as background or foreground.