Conversion of 2-dimensional image data into 3-dimensional image data
View Patent ↗Two dimensional data is converted into three dimensional picture data in a method that can provide a real time high quality display during conversion. Pixels of a frame of picture data are segmented to create pixel segments by applying a k-means algorithm. The k-means algorithm groups pixels based on closeness of a combined value that includes luma, chroma, and motion information. By balancing this information the algorithm collects pixels into groups that are assigned relative depths to turn the two-dimensional information into three-dimensional information for display. Another method includes determining a depth map for the different pixel segments by determining an amount of motion of one of the pixel segments between two frames of a video and scaling the three-dimensional depth of one of the pixel segments based on the amount of motion between the two frames.
1. A method of converting two dimensional picture data into three dimensional picture data for real time display, the method comprising:
using a computing device to segment pixels of a frame of picture data to create pixel segments
comprising pixel clusters by applying a k-means algorithm, the applying the k-means algorithm comprising:
determining a clustering value for individual starting pixels wherein the clustering value comprises a combination of luma and chroma information of the individual starting pixels;
comparing the clustering values for the individual starting pixels to the clustering values for other pixels of the frame;
clustering pixels of the other pixels with pixels of the individual starting pixels based on closeness of the clustering values to create pixel clusters associated with the individual starting pixels;
re-defining a second starting pixel for individual ones of the pixel clusters, the second starting pixel for a pixel cluster being the pixel of the pixel cluster having a clustering value in the middle of the clustering values for the pixels of the pixel cluster;
determining whether clustering values for the second starting pixels differ from clustering values for corresponding starting pixels by more than a clustering value difference threshold, wherein the applying the k-means algorithm further comprises defining the clustering value difference threshold to be different values for different pixel clusters;
in response to determining that the clustering values for the second starting pixels differ from the clustering values for corresponding starting pixels by more than the cluster value difference threshold, repeating the following:
clustering pixels of the frame other than the second starting pixels based on closeness of the clustering values to create new pixel clusters associated with individual ones of the second starting pixels, and
re-defining new starting pixels for the new pixel clusters, the new starting pixel for a new pixel cluster being the pixel of the new pixel cluster having a clustering value in the middle of the clustering values for the pixels of the new pixel cluster,
until the clustering values for the new starting pixels differ from clustering values for previous starting pixels by less than the clustering value difference threshold;
determining a depth map for the pixel segments to define three dimensional depths of the pixel segments relative to each other; adjusting the pixels of the frame of picture data to provide a three-dimensional pixel set for the frame using the depth map;
sending a signal to effect a three-dimensional display of the three-dimensional pixel set via a display device.
2. The method of claim 1 wherein the determining the clustering value for individual starting pixels comprises determining the clustering value comprising a combination of luma, chroma, and motion information of the individual starting pixels, wherein the motion information comprises information relating to relative movement of pixels between frames of video data.
3. The method of claim 1 wherein the determining the depth map comprises:
determining an amount of motion of one of the pixel segments between two frames of picture data in a video;
scaling a three dimensional depth of one of the pixel segments based on the amount of motion between the two frames, wherein the three dimensional depth of the one of the pixel segments is a same depth as an earlier frame where there is a limited amount of motion between the two frames for the one of the pixel segments.
4. The method of claim 1 further comprising merging pixel clusters having at least one of luma, chroma, or motion information within a cluster merge value threshold.
5. The method of claim 1 further comprising applying a correction model to the three dimensional depths by varying the three dimensional depth of a pixel based on the pixel's vertical placement in the frame.
6. A method of converting two dimensional picture data into three dimensional picture data for real time display, the method comprising:
segmenting pixels of a frame of picture data to create pixel segments comprising pixel clusters by using a computing device to segment pixels comprises applying a k-means algorithm, the applying the k-means algorithm comprising:
determining a clustering value for individual starting pixels wherein the clustering value comprises a combination of luma, chroma, and motion information of the individual starting pixels;
comparing the clustering values for the individual starting pixels to the clustering values for other pixels of the frame;
clustering pixels of the other pixels with pixels of the individual starting pixels based on closeness of the clustering values to create pixel clusters associated with the individual starting pixels;
re-defining a second starting pixel for individual ones of the pixel clusters, the second starting pixel for a pixel cluster being the pixel of the pixel cluster having a clustering value in the middle of the clustering values for the pixels of the pixel cluster;
determining whether clustering values for the second starting pixels differ from clustering values for corresponding starting pixels by more than a clustering value difference threshold, wherein the applying the k-means algorithm further comprises defining the clustering value difference threshold to be different values for different pixel clusters;
in response to determining that the clustering values for the second starting pixels differ from the clustering values for corresponding starting pixels by more than the cluster value difference threshold, repeating the following:
clustering pixels of the frame other than the second starting pixels based on closeness of the clustering values to create new pixel clusters associated with individual ones of the second starting pixels, and
re-defining new starting pixels for the new pixel clusters, the new starting pixel for a new pixel cluster being the pixel of the new pixel cluster having a clustering value in the middle of the clustering values for the pixels of the new pixel cluster,
until the clustering values for the new starting pixels differ from clustering values for previous starting pixels by less than the clustering value difference threshold;
using the computing device to determine a depth map for the pixel segments to define three dimensional depth of the pixel segments relative to each other, the determining the depth map comprising:
determining an amount of motion of one of the pixel segments between two frames of picture data in a video, and
scaling the three dimensional depth of the one of the pixel segments based on the amount of motion between the two frames, wherein the three dimensional depth of the one of the pixel segments is a same depth as an earlier frame where there is a limited amount of motion between the two frames for the one of the pixel segments;
adjusting the pixels of the frame of picture data to provide a three-dimensional pixel set for the frame using the depth map; and
sending a signal to effect a three-dimensional display of the three-dimensional pixel set via a display device.
7. The method of claim 6 further comprising merging pixel clusters having at least one of luma, chroma, or motion information within a cluster merge value threshold.
8. The method of claim 6 further comprising applying a correction model to the three dimensional depths by varying the three dimensional depth of a pixel based on the pixel's vertical placement in the frame.
9. An apparatus for converting two dimensional picture data into three dimensional picture data for real time display, the apparatus comprising:
a processing device configured to segment pixels of a frame of picture data stored on a storage device to create pixel segments comprising pixel clusters by applying a k-means algorithm, the applying the k-means algorithm comprising:
determining a clustering value for individual starting pixels wherein the clustering value comprises a combination of luma and chroma information of the individual starting pixels;
comparing the clustering values for the individual starting pixels to the clustering values for other pixels of the frame;
clustering pixels of the other pixels with pixels of the individual starting pixels based on closeness of the clustering values to create pixel clusters associated with the individual starting pixels;
re-defining a second starting pixel for individual ones of the pixel clusters, the second starting pixel for a pixel cluster being the pixel of the pixel cluster having a clustering value in the middle of the clustering values for the pixels of the pixel cluster;
determining whether clustering values for the second starting pixels differ from clustering values for corresponding starting pixels by more than a clustering value difference threshold, wherein the applying the k-means algorithm further comprises defining the clustering value difference threshold to be different values for different pixel clusters;
in response to determining that the clustering values for the second starting pixels differ from the clustering values for corresponding starting pixels by more than the cluster value difference threshold, repeating the following:
clustering pixels of the frame other than the second starting pixels based on closeness of the clustering values to create new pixel clusters associated with individual ones of the second starting pixels, and
re-defining new starting pixels for the new pixel clusters, the new starting pixel for a new pixel cluster being the pixel of the new pixel cluster having a clustering value in the middle of the clustering values for the pixels of the new pixel cluster,
until the clustering values for the new starting pixels differ from clustering values for previous starting pixels by less than the clustering value difference threshold;
the processing device further configured to determine a depth map for the pixel segments to define three dimensional depths of the pixel segments relative to each other;
the processing device further configured to adjust the pixels of the frame of picture data to provide a three-dimensional pixel set for the frame using the depth map;
the processing device further configured to effect a three-dimensional display of the three-dimensional pixel set via a display device.
10. The apparatus of claim 9 wherein the determining the clustering value for individual starting pixels comprises determining the clustering value comprising a combination of luma, chroma, and motion information of the individual starting pixels, wherein the motion information comprises information relating to relative movement of pixels between frames of video data.
11. The apparatus of claim 9 wherein the determining the depth map comprises:
determining an amount of motion of one of the pixel segments between two frames of picture data in a video;
scaling a three dimensional depth of one of the pixel segments based on the amount of motion between the two frames, wherein the three dimensional depth of the one of the pixel segments is a same depth as an earlier frame where there is a limited amount of motion between the two frames for the one of the pixel segments.
12. The apparatus of claim 9 wherein the processing device is further configured to merge pixel clusters having at least one of luma, chroma, or motion information within a cluster merge value threshold.
13. The apparatus of claim 9 wherein the processing device is further configured to apply a correction model to the three dimensional depths by varying the three dimensional depth of a pixel based on the pixel's vertical placement in the frame.
14. An apparatus for converting two dimensional picture data into three dimensional picture data for real time display, the apparatus comprising:
a processing device configured to segment pixels of a frame of picture data stored on a storage device to create pixel segments comprising pixel clusters by applying a k-means algorithm, the applying the k-means algorithm comprising:
determining a clustering value for individual starting pixels wherein the clustering value comprises a combination of luma, chroma, and motion information of the individual starting pixels;
comparing the clustering values for the individual starting pixels to the clustering values for other pixels of the frame;
clustering pixels of the other pixels with pixels of the individual starting pixels based on closeness of the clustering values to create pixel clusters associated with the individual starting pixels;
re-defining a second starting pixel for individual ones of the pixel clusters, the second starting pixel for a pixel cluster being the pixel of the pixel cluster having a clustering value in the middle of the clustering values for the pixels of the pixel cluster;
determining whether clustering values for the second starting pixels differ from clustering values for corresponding starting pixels by more than a clustering value difference threshold, wherein the applying the k-means algorithm further comprises defining the clustering value difference threshold to be different values for different pixel clusters;
in response to determining that the clustering values for the second starting pixels differ from the clustering values for corresponding starting pixels by more than the cluster value difference threshold, repeating the following:
clustering pixels of the frame other than the second starting pixels based on closeness of the clustering values to create new pixel clusters associated with individual ones of the second starting pixels, and
re-defining new starting pixels for the new pixel clusters, the new starting pixel for a new pixel cluster being the pixel of the new pixel cluster having a clustering value in the middle of the clustering values for the pixels of the new pixel cluster,
until the clustering values for the new starting pixels differ from clustering values for previous starting pixels by less than the clustering value difference;
the processing device further configured to determine a depth map for the pixel segments to define three dimensional depth of the pixel segments relative to each other,
the determining the depth map comprising: determining an amount of motion of one of the pixel segments between two frames of picture data in a video, and
scaling the three dimensional depth of the one of the pixel segments based on the amount of motion between the two frames, wherein the three dimensional depth of the one of the pixel segments is a same depth as an earlier frame where there is a limited amount of motion between the two frames for the one of the pixel segments;
the processing device further configured to adjust the pixels of the frame of picture data to provide a three-dimensional pixel set for the frame using the depth map;
the processing device further configured to effect a three-dimensional display of the three-dimensional pixel set via a display device.
15. The apparatus of claim 14 wherein the processing device is further configured to merge pixel clusters having at least one of luma, chroma, or motion information within a cluster merge value threshold.
16. The apparatus of claim 14 wherein the processing device is further configured to apply a correction model to the three dimensional depths by varying the three dimensional depth of a pixel based on the pixel's vertical placement in the frame.
17. A non-transitory computer readable medium comprising instructions configured to cause a computing device to perform operations comprising:
segmenting pixels of a frame of picture data to create pixel
segments comprising pixel clusters by applying a k-means algorithm, the
applying the k-means algorithm comprising:
determining a clustering value for individual starting pixels wherein the clustering value comprises a combination of luma and chroma information of the individual starting pixels;
comparing the clustering values for the individual starting pixels to the clustering values for other pixels of the frame;
clustering pixels of the other pixels with pixels of the individual starting pixels based on closeness of the clustering values to create pixel clusters associated with the individual starting pixels;
re-defining a second starting pixel for individual ones of the pixel clusters, the second starting pixel for a pixel cluster being the pixel of the pixel cluster having a clustering value in the middle of the clustering values for the pixels of the pixel cluster;
determining whether clustering values for the second starting pixels differ from clustering values for corresponding starting pixels by more than a clustering value difference threshold, wherein the applying the k-means algorithm further comprises defining the clustering value difference threshold to be different values for different pixel clusters;
in response to determining that the clustering values for the second starting pixels differ from the clustering values for corresponding starting pixels by more than the cluster value difference threshold, repeating the following:
clustering pixels of the frame other than the second starting pixels based on closeness of the clustering values to create new pixel clusters associated with individual ones of the second starting pixels, and
re-defining new starting pixels for the new pixel clusters, the new starting pixel for a new pixel cluster being the pixel of the new pixel cluster having a clustering value in the middle of the clustering values for the pixels of the new pixel cluster,
until the clustering values for the new starting pixels differ from clustering values for previous starting pixels by less than the clustering value difference threshold;
determining a depth map for the pixel segments to define three dimensional depths of the pixel segments relative to each other;
adjusting the pixels of the frame of picture data to provide a three-dimensional pixel set for the frame using the depth map;
sending a signal to effect a three-dimensional display of the three-dimensional pixel set via a display device.
18. The non-transitory computer readable medium of claim 17 wherein the determining the clustering value for individual starting pixels comprises determining the clustering value comprising a combination of luma, chroma, and motion information of the individual starting pixels, wherein the motion information comprises information relating to relative movement of pixels between frames of video data.
19. The non-transitory computer readable medium of claim 17 wherein the determining the depth map comprises:
determining an amount of motion of one of the pixel segments between two frames of picture data in a video;
scaling a three dimensional depth of one of the pixel segments based on the amount of motion between the two frames, wherein the three dimensional depth of the one of the pixel segments is a same depth as an earlier frame where there is a limited amount of motion between the two frames for the one of the pixel segments.
20. The non-transitory computer readable medium of claim 17 further comprising instructions to cause the computing device to perform operations comprising merging pixel clusters having at least one of luma, chroma, or motion information within a cluster merge value threshold.
21. The non-transitory computer readable medium of claim 17 further comprising instructions to cause the computing device to perform operations comprising applying a correction model to the three dimensional depths by varying the three dimensional depth of a pixel based on the pixel's vertical placement in the frame.
22. A non-transitory computer readable medium comprising instructions configured to cause a computing device to perform operations comprising:
segmenting pixels of a frame of picture data to create pixel segments comprising pixel clusters by applying a k-means algorithm, the applying the k-means algorithm comprising:
determining a clustering value for individual starting pixels wherein the clustering value comprises a combination of luma, chroma, and motion information of the individual starting pixels;
comparing the clustering values for the individual starting pixels to the clustering values for other pixels of the frame;
clustering pixels of the other pixels with pixels of the individual starting pixels based on closeness of the clustering values to create pixel clusters associated with the individual starting pixels;
re-defining a second starting pixel for individual ones of the pixel clusters, the second starting pixel for a pixel cluster being the pixel of the pixel cluster having a clustering value in the middle of the clustering values for the pixels of the pixel cluster;
determining whether clustering values for the second starting pixels differ from clustering values for corresponding starting pixels by more than a clustering value difference threshold, wherein the applying the k-means algorithm further comprises defining the clustering value difference to be different values for different pixel clusters;
in response to determining that the clustering values for the second starting pixels differ from the clustering values for corresponding starting pixels by more than the cluster value difference threshold, repeating the following:
clustering pixels of the frame other than the second starting pixels based on closeness of the clustering values to create new pixel clusters associated with individual ones of the second starting pixels, and
re-defining new starting pixels for the new pixel clusters, the new starting pixel for a new pixel cluster being the pixel of the new pixel cluster having a clustering value in the middle of the clustering values for the pixels of the new pixel cluster,
until the clustering values for the new starting pixels differ from clustering values for previous starting pixels by less than the clustering value difference threshold;
causing the computing device to determine a depth map for the pixel segments to define three dimensional depth of the pixel segments relative to each other, the determining the depth map comprising:
determining an amount of motion of one of the pixel segments between two frames of picture data in a video, and
scaling the three dimensional depth of the one of the pixel segments based on the amount of motion between the two frames, wherein the three dimensional depth of the one of the pixel segments is a same depth as an earlier frame where there is a limited amount of motion between the two frames for the one of the pixel segments;
adjusting the pixels of the frame of picture data to provide a three-dimensional pixel set for the frame using the depth map;
sending a signal to effect a three-dimensional display of the three-dimensional pixel set via a display device.
23. The non-transitory computer readable medium of claim 22 further comprising instructions to cause the computing device to perform operations comprising merging pixel clusters having at least one of luma, chroma, or motion information within a cluster merge value threshold.
24. The non-transitory computer readable medium of claim 22 further comprising instructions to cause the computing device to perform operations comprising applying a correction model to the three dimensional depths by varying the three dimensional depth of a pixel based on the pixel's vertical placement in the frame.