Coordination and combination of video sequences with spatial and temporal normalization
Given two video sequences, a composite video sequence can be generated ( 15 ) which includes visual elements from each of the given sequences, suitably synchronized ( 11 ) and represented in a chosen focal plane. A composite video sequence can be made also by similarly combining a video sequence with an audio sequence. In the composite video sequence, contestants, action figures or objects can be shown against a common background ( 12 ) even if the given video sequences differ as to background, with the common background taken from one or the other of the given sequences, for example. Alternatively, a different suitable background can be used, e.g. as derived from the given video sequences, as obtained from another video sequence or image, or as otherwise synthesized.
1. A computer method for generating a composite video sequence from a plurality of given video sequences for display on a display device in communication with a computer processor, wherein each of the given video sequences comprises a representation of a respective one of a plurality of elements with action relative to a respective environment, the method comprising the steps of:
(a) forming a background representation, using the computer processor, said background representation including a common environment;
(b) aligning said plurality of given video sequences, using the computer processor, based on a matching in size and position of elements, from a background and/or from a foreground, which are present in at least two of the plurality of given video sequences;
(c) forming the composite video sequence, using the computer processor, based on the formed background representation and the aligned video sequences; and
(d) displaying the composite video sequence on the display device.
2. The method of claim 1 , wherein step (a) comprises forming the background representation independent of any of the plurality of elements.
3. The method of claim 2 , wherein the background representation is a single image recorded by a camera.
4. The method of claim 3 , wherein forming the background representation comprises panoramically combining images from an image sequence.
5. The method of claim 4 , wherein the image sequence is one of the given video sequences.
6. The method of claim 5 , comprising updating the background representation so as to give preferential weight to more recent images of the given video sequence.
7. The method of claim 2 , wherein the background representation comprises a parametric environmental representation.
8. The method of claim 2 , wherein forming the background representation comprises associating a reliability measure to representational elements.
9. The method of claim 1 , further comprising estimating a geometrical relation between a given video frame and a corresponding background representation.
10. The method of claim 9 , wherein estimating comprises image analysis.
11. The method of claim 9 , wherein estimating comprises taking into account instrument data furnished by an instrumented camera used in generating one of the given video sequences.
12. The method of claim 9 , comprising (i) using the geometrical relation for aligning a given video frame and a corresponding background representation and (ii) associating with pixels a discrepancy measure between the aligned video frame and the background representation.
13. The method of claim 12 , comprising using the discrepancy measure in determining a probability for a pixel of a given video frame to belong to background.
14. The method of claim 13 , comprising using the discrepancy measure in classifying a pixel of a given video frame as belonging either to background or else to a foreground element.
15. The method of claim 14 , wherein classifying takes auxiliary information into account.
16. The method of claim 14 , comprising using the classification in inferring statistical data for the foreground element.
17. The method of claim 14 , comprising using the classification in replacing the background with a different background.
18. The method of claim 1 , wherein each of the given video sequences comprises representations of a respective plurality of elements with action relative to the respective environment.
19. The method of claim 1 , wherein the elements are canonical elements.
20. The method of claim 1 , comprising automated selection of the elements.
21. The method of claim 1 , comprising accepting input for the selection of the elements.
22. The method of claim 1 , further comprising identifying time of execution at least at one control point in the execution of an action performance.
23. The method of claim 22 , wherein the identification is based at least in part on visual information in one of the given video sequence.
24. The method of claim 22 , wherein the identification is based at least in part on an auxiliary signal.
25. The method of claim 24 , wherein the auxiliary signal is one of an audio signal, an ultrasound signal, a microwave signal, a radar signal and an optical detector signal.
26. The method of claim 22 , comprising storing identified time for later use along with a video sequence.
27. The method of claim 22 , wherein time identification is automated.
28. The method of claim 22 , comprising accepting input for time identification.
29. The method of claim 22 , comprising determining a temporal transformation from one set of control points onto another.
30. The method of claim 22 , comprising applying a temporal transformation to at least one of the given video sequences to generate a transformed sequence whose pace is adjusted for temporal matching at control points of given video sequences.
31. The method of claim 30 , wherein one of the given video sequences is a reference video sequence, and wherein the temporal transformation is applied only to another given video sequence.
32. The method of claim 30 , wherein the temporal transformation comprises offsetting an action performance in one given video sequences with an action performance in another given video sequence.
33. The method of claim 32 , wherein offsetting is for controlling playback of a recorded given video so that time of execution at control points corresponds to time of execution in a live-performance given video.
34. The method of claim 30 , comprising tagging/indexing/marking video frames of a given video sequence which represent a state of action at the control points.
35. The method of claim 22 , comprising applying a nonlinear temporal transformation to at least one of the video sequences.
36. The method of claim 35 , wherein the transformation minimizes a global error measure between the video sequences.
37. The method of claim 36 , comprising searching for minimization.
38. The method of claim 37 , wherein searching is based on a dynamic programming technique.
39. The method of claim 1 , further comprising choosing the respective environment of one of the given video sequences as the common environment.
40. The method of claim 1 , further comprising choosing the common environment to be different from the respective environments of each of the given video sequences.
41. The method of claim 1 , wherein step (c) comprises blending in forming the composite video sequence.
42. The method of claim 1 , wherein step (c) comprises split-screen arranging in forming the composite video sequence.
43. The method of claim 1 , wherein the given video sequences are from two sports performances.
44. The method of claim 43 , wherein the two sports performances are in different respective environments.
45. The method of claim 44 , wherein one of the sports performances is a reference performance.
46. The method of claim 45 , wherein the reference performance is one of a world record performance, a national record performance, and a best personal performance.
47. The method of claim 45 , wherein the performance other than the reference performance is one of a recent performance and a live performance.
48. The method of claim 43 , wherein the sport is one of long jump, triple jump, high jump, pole vault, 100-meter race, and golf swing.
49. The method of claim 19 , wherein the canonical elements are one of long-jump running pit lines, long-jump take-off board, 100-m race lanes, pole-vault poles, pole-vault bars, and high-jump bars.
50. The method of claim 1 , further comprising the step of recording by an image recording device the plurality of given video sequences.
51. A system for generating a composite video sequence from a plurality of given video sequences wherein each of the given video sequences comprises a representation of a respective one of a plurality of elements with action relative to a respective environment, the system comprising:
(a) a computer processor forming a background representation, said background representation including a common environment;
(b) said computer processor aligning said plurality of video sequences, based on a matching in size and position of elements from a background and/or from a foreground which are present in at least two of the plurality of given video sequences with a visual element in another of the given video sequences, wherein said visual elements are background elements;
(c) said computer processor further forming the composite video sequence based on the formed background representation and the aligned video sequences; and
(d) a display device in communication with the computer processor, said display device displaying the composite video sequence.
52. A non-transitory tangible medium comprising a representation of computer instructions for generating a non-transitory composite video sequence from a plurality of given video sequences and displaying the composite video sequence on a display device, wherein each of the given video sequences comprises a representation of a respective one of a plurality of elements with action relative to a respective environment, the computer instructions comprising:
(a) instructions for forming a background representation, said background representation including a common environment;
(b) instructions for aligning said plurality of video sequences, based on a matching in size and position of elements from a background and/or from a foreground which are present in at least two of given video sequences;
(c) instructions for forming the non-transitory composite video sequence based on the formed background representation and the aligned video sequences; and
(d) instructions for displaying the non-transitory composite video sequence on the display device.
53. A computer method for generating a composite video sequence from a plurality of given video sequences and displaying the composite video sequence on a display device in communication with a computer processor, wherein each of the given video sequences comprises a representation of a respective one of a plurality of elements with action relative to a respective environment, the method comprising the steps of:
(a) forming a background representation using the computer processor, said background representation including a common environment, wherein for each video sequence a weighted mask sequence is computed, said weighted mask sequence describing the probability of a pixel to belong to the background;
(c) aligning said plurality of given video sequences using the computer processor, based on a matching in size and position of elements from a background and/or from a foreground which are present in at least two of the plurality of given video sequences;
(d) forming the composite video sequence, using the computer processor, based on the formed background representation and the aligned video sequences; and
(e) displaying the composite video sequence on the display device.
54. A computer method for a computer processor to generate a composite video sequence from a plurality of given video sequences wherein each of the given video sequences comprises a representation of a respective one of a plurality of elements with action relative to a respective environment, the method comprising the steps of:
(a) forming a background representation including a common environment, using the computer processor;
(b) ascertaining the representation of the respective one of the plurality of elements in each of the given video sequences, using the computer processor:
(c) coordinating the actions of the ascertained representations, using the computer processor, wherein said coordinating includes a spatial normalization between the given video sequences, for matching as to size and position a first visual element in one of the given video sequences with a second visual element in another of the given video sequences, wherein said visual elements are background elements;
(d) forming the composite video sequence based on the formed background representation and the coordinated representations, using the computer processor, such that the coordinated representations appear in the common environment; and
(e) displaying the composite video sequence on a display device.