Encoding device, encoding method, decoding device, and decoding method
There is provided an encoding device, an encoding method, a decoding device, and a decoding method capable of generating a more accurate three-dimensional model. A three-dimensional model generating unit generates three-dimensional model information representing a three-dimensional model of a subject on the basis of a plurality of captured images and active depth information, and a conversion processing unit converts the three-dimensional model represented by the three-dimensional model information into a plurality of two-dimensional images by projecting the three-dimensional model from a plurality of directions, and generates depth information representing a depth from an arbitrary viewpoint to the three-dimensional model by using the plurality of two-dimensional images. Then, transmit data including the plurality of two-dimensional images, the depth information, and the active depth information is transmitted to the decoding device. The present technology can be applied to, for example, a free viewpoint video transmission system.
1. An encoding device comprising:
a three-dimensional model generating unit configured to generate three-dimensional model information representing a three-dimensional model of a subject on a basis of a plurality of captured images obtained by imaging the subject from a plurality of viewpoints, and active depth information indicating a distance to the subject from a plurality of viewpoints different from the plurality of viewpoints from which the plurality of captured images are imaged;
a conversion processing unit configured to
convert the three-dimensional model represented by the three-dimensional model information into a plurality of two-dimensional images by projecting the three-dimensional model from a plurality of directions, and
generate depth information representing a depth from an arbitrary viewpoint to the three-dimensional model by using the plurality of two-dimensional images; and
a transmission unit configured to transmit transmit data including the plurality of two-dimensional images and the depth information to a decoding device,
wherein the three-dimensional model generating unit, the conversion processing unit, and the transmission unit are each implemented via at least one processor.
2. The encoding device according to claim 1 , wherein
the transmission unit transmits the transmit data further including the active depth information.
3. The encoding device according to claim 1 , wherein
the three-dimensional model generating unit includes:
a background difference generating unit configured to
determine a background difference from the plurality of captured images, and
generate a plurality of silhouette images each representing, in binary form, a silhouette indicating a range in which the subject is shown in each of the captured images; and
a silhouette processing unit configured to
project the active depth information onto the plurality of silhouette images, and
perform processing on a corruption occurring in the silhouette of the subject on the silhouette images.
4. The encoding device according to claim 3 , wherein
the silhouette processing unit is further configured to output processed area information indicating an area in which the silhouette of the subject on the silhouette images is subjected to processing, and
the transmission unit transmits the transmit data further including the processed area information.
5. The encoding device according to claim 3 , wherein
the silhouette processing unit performs processing that fills a hole appearing in the silhouette of the subject on the silhouette images.
6. The encoding device according to claim 3 , wherein
the silhouette processing unit performs processing that removes a shadow of the subject appearing as the silhouette in the silhouette images.
7. The encoding device according to claim 3 , further comprising:
a dark/whiteout area detecting unit configured to detect a dark area having a predetermined brightness value or lower and a whiteout area having a predetermined brightness value or higher in the plurality of captured images,
wherein the silhouette processing unit performs processing on the silhouette of the subject in the dark area or the whiteout area on the silhouette images, and
wherein the dark/whiteout area detecting unit is implemented via at least one processor.
8. The encoding device according to claim 1 , further comprising:
an object/substance detecting unit configured to detect an object or substance of the subject on a basis of the plurality of captured images and the active depth information; and
a correction value calculating unit configured to calculate a correction value used to correct the active depth information depending on the object or substance of the subject,
wherein the active depth information corrected on a basis of the correction value is used to generate the three-dimensional model information representing the three-dimensional model of the subject, and
wherein the object/substance detecting unit and the correction value detecting unit are each implemented via at least one processor.
9. The encoding device according to claim 8 , wherein
the transmission unit transmits the transmit data further including the correction value and reliability of the correction value.
10. The encoding device according to claim 8 , wherein the object/substance detecting unit is further configured to
divide the captured image into segments each corresponding to a subject in the captured image, and
detect an object or substance of the subject segment by segment.
11. An encoding method comprising:
generating three-dimensional model information representing a three-dimensional model of a subject on a basis of a plurality of captured images obtained by imaging the subject from a plurality of viewpoints, and active depth information indicating a distance to the subject from a plurality of viewpoints different from the plurality of viewpoints from which the plurality of captured images are imaged;
converting the three-dimensional model represented by the three-dimensional model information into a plurality of two-dimensional images by projecting the three-dimensional model from a plurality of directions;
generating depth information representing a depth from an arbitrary viewpoint to the three-dimensional model by using the plurality of two-dimensional images; and
transmitting transmit data including the plurality of two-dimensional images and the depth information to a decoding device,
wherein the active depth information is generated based on a detected substance of the subject.
12. A decoding device comprising:
a reception unit configured to receive transmit data including a plurality of two-dimensional images and depth information, the transmit data being transmitted after three-dimensional model information representing a three-dimensional model of a subject is generated on a basis of a plurality of captured images obtained by imaging the subject from a plurality of viewpoints and active depth information indicating a distance to the subject from a plurality of viewpoints different from the plurality of viewpoints from which the plurality of captured images are imaged, the three-dimensional model represented by the three-dimensional model information is converted into the plurality of two-dimensional images by projecting the three-dimensional model from a plurality of directions, and the depth information representing a depth from an arbitrary viewpoint to the three-dimensional model is generated by using the plurality of two-dimensional images; and
a three-dimensional data conversion processing unit configured to convert the plurality of two-dimensional images and the depth information into the three-dimensional model information representing the three-dimensional model of the subject,
wherein the active depth information is generated based on a detected substance of the subject, and
wherein the reception unit and the three-dimensional data conversion unit are each implemented via at least one processor.
13. The decoding device according to claim 12 , wherein
the reception unit receives the transmit data including the plurality of two-dimensional images, the depth information, and the active depth information, and
the three-dimensional data conversion processing unit converts the plurality of two-dimensional images and the depth information into the three-dimensional model information representing the three-dimensional model of the subject with reference to the active depth information.
14. The decoding device according to claim 12 , wherein
the reception unit receives the transmit data further including processed area information that indicates an area in which a corruption occurring in a silhouette of the subject on a plurality of silhouette images is subjected to processing, the processed area information being generated after a background difference is determined from the plurality of captured images, the silhouette images each representing, in binary form, the silhouette indicating a range in which the subject is shown in each of the captured images are generated, the active depth information is projected onto the silhouette images, and the corruption occurring in the silhouette of the subject on the silhouette images is subjected to the processing, and
the three-dimensional data conversion processing unit projects the active depth information onto the area indicated by the processed area information.
15. The decoding device according to claim 12 , wherein
the substance of the subject is detected on a basis of the plurality of captured images and the active depth information, then the reception unit receives the transmit data including a correction value used to correct the active depth information depending on the detected substance of the subject and reliability of the correction value, and
the three-dimensional data conversion processing unit refers to the active depth information corrected on a basis of the correction value and the reliability.
16. A decoding method comprising:
receiving transmit data including a plurality of two-dimensional images and depth information, the transmit data being transmitted after three-dimensional model information representing a three-dimensional model of a subject is generated on a basis of a plurality of captured images obtained by imaging the subject from a plurality of viewpoints and active depth information indicating a distance to the subject from a plurality of viewpoints different from the plurality of viewpoints from which the plurality of captured images are imaged, the three-dimensional model represented by the three-dimensional model information is converted into the plurality of two-dimensional images by projecting the three-dimensional model from a plurality of directions, and the depth information representing a depth from an arbitrary viewpoint to the three-dimensional model is generated by using the plurality of two-dimensional images; and
converting the plurality of two-dimensional images and the depth information into the three-dimensional model information representing the three-dimensional model of the subject,
wherein the active depth information is generated based on a detected substance of the subject.
17. The decoding device according to claim 11 , wherein
the substance of the subject is detected on a basis of the plurality of captured images and the active depth information, then a correction value is generated to correct the active depth information depending on the detected substance of the subject and reliability of the correction value, and
the correction value and the reliability are transmitted with the transmit data.