IP Library Granted Patent US 9,154,799
Granted Patent B2
US 9,154,799 · App. 13/081,662 · Granted Oct 6, 2015

Encoding and decoding motion via image segmentation

Inventors: James Bankoski (Wynantskill, NY); Paul Wilkins (Cambridge, GB); Yaowu Xu (San Diego, CA)
Assignee: GOOGLE INC.
H04N19/17H04N19/105H04N19/137H04N19/186H04N19/46H04N19/51
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,154,799
App. No.
13/081,662
Granted
Oct 6, 2015
Kind
B2
Abstract

A system and method for encoding and decoding a video image having a plurality of frames is disclosed. The method includes a previously reconstructed frame that is segmented to create an alternate reference frame used to encode a current frame. The segments of the previously reconstructed frame are assigned motion parameters, which are used to compile the alternate reference frame. The current frame is encoded and the segmentation parameters and motion parameters are transmitted in the bitstream. The segmentation parameters and motion parameters can be used by a decoder to recreate the alternate reference frame in order to decode the encoded current frame.

Claims (37)

1. A method for encoding a current frame of a video signal having a plurality of frames, each frame having a plurality of pixels, the method comprising:

segmenting a previously reconstructed frame into a plurality of image segments using a segmentation parameter, at least one image segment from the plurality of image segments having a segment identifier;

determining, using a processor, a motion parameter for each respective image segment from the plurality of image segments;

creating an alternate reference frame from the previously reconstructed frame using the motion parameter for each respective image segment from the plurality of image segments; and

encoding the current frame using the alternate reference frame for motion prediction.

2. The method of claim 1 wherein the previously reconstructed frame is one of a golden frame, an original alternate reference frame, or a last frame to be encoded.

3. The method of claim 1 , further comprising:

identifying an association between the current frame and the previously reconstructed frame;

identifying an association between the current frame and the motion parameter for each respective image segment from the plurality of image segments; and

identifying an association between the current frame and the segmentation parameter.

4. The method of claim 3 , further comprising: including at least one of the identified associations in the encoded output.

5. The method of claim 1 wherein segmenting the previously reconstructed frame comprises using at least one of color segmentation or motion segmentation, and the segmentation parameter comprises at least one of a color or a motion vector for each pixel of the previously reconstructed frame.

6. The method of claim 1 wherein segmenting the previously reconstructed frame comprises performing at least one of k-means clustering, binary tree deconstruction, quad-tree deconstruction, or oct-tree deconstruction.

7. The method of claim 1 wherein determining the motion parameter for each respective image segment from the plurality of image segments comprises:

performing a motion vector search on pixels making up each respective image segment from the plurality of image segments; and

selecting a motion vector based on the motion vector search for each respective image segment from the plurality of image segments as the motion parameter.

8. The method of claim 1 wherein creating the alternate reference frame comprises:

calculating a new position for each pixel of each respective image segment from the plurality of image segments using the motion parameter of each respective image segment from the plurality of image segments;

comparing the new position to a range of positions of the alternate reference frame; and

on a condition that the new position is within the range of positions of the alternate reference frame, copying a value associated with the pixel in the new position in the alternate reference frame.

9. The method of claim 1 wherein creating the alternate reference frame comprises:

for each image segment in the plurality of image segments:

calculating a new position for each pixel of the image segment using the motion parameter of the image segment;

comparing the new position to a range of positions of the alternate reference frame; and

on a condition that the new position is within the range of positions of the alternate reference frame, copying a value associated with the pixel in the new position in the alternate reference frame.

10. The method of claim 9 wherein creating the alternate reference frame comprises:

selecting an unfilled pixel of the alternate reference frame that has no value copied to it after values for all pixels of the previously reconstructed frame having the new position within the range of positions of the alternate reference frame are copied into the new position in the alternate reference frame;

determining a desired value for the unfilled pixel; and

copying the desired value to the unfilled pixel.

11. The method of claim 10 wherein determining the desired value comprises selecting one of a value of a nearby filled pixel, an average of more than one nearby filled pixel, or a value of the pixel at the same location in the current frame to be encoded.

12. The method according to claim 9 , further comprising:

initializing values for each pixel in the alternate reference frame to zero and labeling each pixel in the new frame as unfilled before comparing the new position to the range of positions of the alternate reference frame; and

on a condition that the new position is within the range of positions of the alternate reference frame, changing a label for the pixel to filled.

13. The method of claim 1 wherein the motion parameter for each respective image segment from the plurality of image segments is at least one of a motion vector or a parameter reflecting at least one of a zoom, a rotation, or a shear of the image segment.

14. The method of claim 1 wherein creating the alternate reference frame from the previously reconstructed frame includes omitting an image segment and a respective motion parameter, wherein the image segment includes fewer than a predetermined number of pixels.

15. The method of claim 1 , wherein encoding the current frame includes using the alternate reference frame for motion prediction in response to determining whether to use the alternate reference frame or the previously reconstructed frame for motion prediction.

16. The method of claim 1 , wherein the alternate reference frame is not a reconstructed frame.

Assignments (2)
CHANGE OF NAME Recorded Oct 2, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044334/0466 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 8, 2011
From: BANKOSKI, JAMES; WILKINS, PAUL; XU, YAOWU
To: GOOGLE INC.
Reel/Frame 026095/0761 →
Continuity (1)
Related Publication 20120257677A1 · Oct 11, 2012