IP Library Granted Patent US 7,792,190
Granted Patent B2
US 7,792,190 · App. 10/938,972 · Granted Sep 7, 2010

Inserting a high resolution still image into a lower resolution video stream

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 7,792,190
App. No.
10/938,972
Granted
Sep 7, 2010
Kind
B2
Abstract

A high resolution still image is inserted into a stream of lower resolution video. The lower resolution video stream is a compressed video stream having I-frames. One of the I-frames is a lower resolution version of which a higher resolution image is desired. The I-frame is decoded and decompressed and zoomed up in size to the high resolution size. The zoomed up I-frame is subtracted from the original high resolution still image to generate a difference frame called an X-frame. The X-frame is compressed and inserted into the video stream. The high resolution image can be recovered by extracting the X-frame from the hybrid video stream, and reversing the process used to generate the X-frame. The functionality for regenerating the high resolution image can be realized in software as part of a viewer program, or can be realized as part of a camera, television, or other video display device.

Claims (65)

1. A method comprising:

(a) transforming a sequence of frames of image data into a video stream, wherein one of the frames of image data is a selected frame of a first pixel resolution, and wherein the video stream is of a second pixel resolution, the video stream including a sequence of I-frames, wherein the first pixel resolution is greater than the second pixel resolution, and wherein a reduced resolution version frame of said selected frame is present in the video stream in the form of one of the I-frames;

(b) transforming said selected frame from a first domain to a second domain to obtain a second frame of image data;

(c) quantizing the second frame of image data to obtain a third frame of image data;

(d) inverse quantizing the third frame of image data to obtain a fourth frame of image data;

(e) inverse transforming the fourth frame of image data from the second domain to the first domain and thereby obtaining a fifth frame of image data;

(f) zooming up the fifth frame of image data to the first pixel resolution to obtain a sixth frame of image data;

(g) differencing said selected frame and the sixth frame and thereby generating a frame of difference information, the frame of difference information being associated in one-to-one correspondence with said one of the I-frames;

(h) incorporating the frame of difference information into the video stream; and

(i) repeating steps (b) through (h) such that some of the I-frames have associated frames of difference information and such that others of the I-frames do not have associated frames of difference information.

2. The method of claim 1 , wherein step (a) involves MPEG2 encoding said selected frame to form said one of the I-frames, and wherein steps (b) and (c) are steps in said MPEG2 encoding of said selected frame to form said one of the I-frames.

3. The method of claim 1 , wherein the sequence of I-frames includes a first I-frame that has an associated frame of difference information that is incorporated into the video stream in step (h), wherein the sequence of I-frames includes a second I-frame that has no associated frame of difference information that is incorporated into the video stream in step (h), wherein the sequence of I-frames includes a third I-frame that has an associated frame of difference information that is incorporated into the video stream in step (h), and wherein the first, second and third I-frames occur in the video stream in the order first, second, third.

4. The method of claim 1 , wherein the frame of difference information incorporated into the video stream in (h) is compressed difference information.

5. The method of claim 1 , wherein the video stream is an MPEG2 video stream having I-frames and P-frames and B-frames, and wherein none of the P-frames and none of the B-frames has an associated frame of difference information in the video stream.

6. The method of claim 1 , wherein the video stream is a compressed video stream.

7. The method of claim 1 , wherein the reduced resolution version frame is an I-frame.

8. The method of claim 1 , wherein the frame of difference information includes a picture header portion and an image data portion, and wherein the frame of difference information includes no motion vectors.

9. The method of claim 1 , wherein the selected frame is zoomed down to the second pixel resolution prior to the transforming of step (b).

10. The method of claim 1 , wherein the transforming of step (b) is a discrete cosine transforming (DCT), and wherein the inverse transforming of step (e) is an inverse discrete cosine transforming (IDCT).

11. The method of claim 1 , wherein steps (a) through (i) are performed by a digital camera, wherein the frame of difference information incorporated into the video stream is a compressed frame, wherein the compressed frame is generated by a compression engine of the digital camera, wherein said compression engine also compresses other image frames, the other image frames being stored in the digital camera as discrete files.

12. A method for generating a final video stream of a first pixel resolution, the final video stream including a sequence of I-frames, the final video stream including a difference frame, the difference frame being a compressed frame of a second pixel resolution, the second pixel resolution being greater than the first pixel resolution, wherein the difference frame and one of the I-frames both correspond to an original image frame, the method comprising:

(a) zooming down the original image frame and thereby generating a first frame of image information;

(b) transforming and compressing the first frame of image information and thereby generating a second frame of image information;

(c) decompressing and inverse transforming the second frame of image information and thereby generating a third frame of image information;

(d) zooming up the third frame of image information and thereby generating a fourth frame of image information, the fourth frame of image information being of the second pixel resolution;

(e) differencing the fourth frame of image information and the original image frame and thereby generating a fifth frame of image information;

(f) transforming and compressing the fifth frame of image information and thereby generating the difference frame;

(g) incorporating the difference frame into a video stream;

(h) repeating steps (a) through (g) and thereby generating the final video stream such that the final video stream comprises multiple difference frames, wherein the final video stream includes a first difference frame associated with a first I-frame of the final video stream, wherein there is no difference frame associated with a second I-frame of the final video stream, wherein there is a second difference frame associated with a third I-frame of the final video stream, and wherein the first, second and third I-frames occur in the final video stream in the order first, second and third.

13. The method of claim 12 , further comprising:

pressing a button on a camera, wherein the pressing of the button causes a rapid sequence of difference frames to be generated and incorporated into the video stream thereby generating the final video stream.

14. A method comprising:

(a) receiving a video stream, wherein the video stream includes a stream of image frames of a firstpixel resolution and also includes a compressed difference image frame;

(b) decompressing the compressed difference image frame to generate a first frame of a second pixel resolution, wherein the second pixel resolution is greater than the first pixel resolution;

(c) decompressing an image frame of the stream of image frames of the first pixel resolution and thereby generating a decompressed image frame of the first pixel resolution;

(d) zooming up the decompressed image frame of the first pixel resolution to generate a second frame of the second pixel resolution; and

(e) adding the first frame of the second pixel resolution to the second frame of the second pixel resolution and thereby generating a third image frame of the second pixel resolution.

15. The method of claim 14 , wherein the video stream is an MPEG2 video stream comprising compressed I-frames and compressed P-frames, and wherein the decompressed image frame of the first pixel resolution is a decompressed I-frame of the MPEG2 video stream.

16. The method of claim 14 , wherein the decompressing of (b) is JPEG decompression.

17. The method of claim 14 , wherein the decompressing of step (b) includes a variable length decoding step, an inverse quantizing step, and an inverse discrete cosine transforming (IDCT) step.

18. The method of claim 14 , wherein the video stream is a compressed video stream, wherein the compressed video stream is decompressed and then output to a display device, the method further comprising:

outputting the third image frame to the display device.

19. The method of claim 18 , further comprising:

receiving a prompt from a user of a display device such that the display device switches from displaying video of the decompressed video stream and switches to displaying the third image frame.

20. The method of claim 14 , wherein the compressed difference image frame includes a picture header portion and an image data portion.

21. The method of claim 14 , wherein steps (a) through (e) are performed by one of the group consisting of: a television, a personal computer, a cable set-top box, a satellite set-top box, an optical disk player, a video recorder, a hybrid digital camera, and a stereo tuner/receiver with video controlling capabilities.

22. The method of claim 14 , wherein the third image frame contains advertising information.

23. The method of claim 14 , wherein steps (a) through (e) are performed by a software video viewer program executing on a personal computer.

24. An encoding apparatus, comprising:

a video encoder that outputs a compressed video stream of a first pixel resolution, the compressed video stream including a sequence of reference frames, and relative frames; and

circuitry that receives a high pixel resolution image frame that corresponds to one of the reference frames, the high pixel resolution image frame being of a second pixel resolution, the second pixel resolution being substantially greater than the first pixel resolution, the circuitry generating an X-frame that represents a difference between the high pixel resolution image frame and the said one reference frame, wherein said one reference frame is an I-frame, X-frame is not a reference frame, or a relative frame; and

multiplexing circuitry that inserts the X-frame into the compressed video stream.

25. The apparatus of claim 24 , wherein the compressed video stream as it is output from the multiplexing circuitry includes a substantially greater number of reference frames than X-frames.

26. The apparatus of claim 25 , wherein the multiplexing circuitry inserts fewer than three X-frames per second into the compressed video stream such that the compressed video stream as it is output from the multiplexing circuitry includes fewer than three X-frames per second of video.

27. A decoding apparatus comprising:

a video decoder that receives a compressed video stream of a first pixel resolution, the compressed video stream including a sequence of I-frames and P-frames, the video decoder outputting a decompressed video stream of the first pixel resolution; and

circuitry that decompresses an X-frame to generate a decompressed X -frame, the decompressed X-frame representing a difference between a high pixel resolution image frame and information in one of the I-frames of the compressed video stream, the high pixel resolution image frame being of a second pixel resolution, the second pixel resolution being substantially greater than the first pixel resolution, the circuitry combining the decompressed X-frame and information in said one I-frame to regenerate the high pixel resolution image frame.

28. The decoding apparatus of claim 27 , wherein the X-frame is received onto the decoding apparatus as part of the compressed video stream, the compressed video stream including a substantially greater number of I-frames than X-frames.

29. The decoding apparatus of claim 28 , wherein the X-frame includes a header that indicates that the X-frame is not an I-frame and is not a P-frame.

30. The decoding apparatus of claim 29 , wherein there are fewer than three X-frames embedded in each second of the compressed video stream.

31. A digital camera comprising:

a video encoder that outputs a compressed video stream of a first pixel resolution, the compressed video stream including a sequence of I-frames and P-frames; and

means for receiving a high pixel resolution image frame that corresponds to one of the I-frames, the high pixel resolution image frame being of a second pixel resolution, the second pixel resolution being substantially greater than the first pixel resolution, the means generating an X-frame that represents difference information between the high pixel resolution image frame and an uncompressed and zoomed-up version of the I-frame, wherein the means inserts the X-frame into the compressed video stream, the means inserting substantially fewer X-frames into the compressed video stream than there are I-frames in the compressed video stream.

32. The digital camera of claim 31 , wherein the uncompressed and zoomed-up version of the I-frame is of the second pixel resolution.

33. The digital camera of claim 31 , wherein each of the I-frames and each of the X-frames is compressed in accordance with a single compression method, and wherein there are fewer than three X-frames embedded in each second of the compressed video stream.

Assignments (5)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 11, 2021
From: MEDIATEK SINGAPORE PTE LTD.
To: XUESHAN TECHNOLOGIES INC.
Reel/Frame 055283/0048 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 21, 2009
From: MEDIATEK USA INC.
To: MEDIATEK SINGAPORE PTE LTD.
Reel/Frame 023546/0868 →
CHANGE OF NAME Recorded Feb 26, 2008
From: NUCORE TECHNOLOGY INC.
To: MEDIATEK USA INC.
Reel/Frame 020564/0064 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 13, 2004
From: WATANABE, SEIICHIRO; SEI, KYOJIRO
To: NUCORE TECHNOLOGY INC.
Reel/Frame 016064/0955 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 9, 2004
From: WATANABE, SEIICHIRO
To: NUCORE TECHNOLOGY INC.
Reel/Frame 015787/0776 →