Image processing method and device
Image processing method and device are provided. The method includes: controlling a projector to project multi-stripe structured light and dot structured light onto an object, to form multiple light stripes and light spots between the light stripes on the surface of the object; acquiring a target image of the object having the light stripes and the light spots on the surface; obtaining a reference image including the light stripes and the light spots; calculating a first depth value at each light stripe in the target image based on feature information of the light stripes in the reference image and target image; calculating a second depth value at each light spot in the target image based on feature information of the light spots in the reference image and target image; and generating a depth image of the target image based on the first depth value and the second depth value.
1. An image processing method, which is applied to an electronic device, the electronic device comprising at least an image sensor and a light projector, wherein the method comprises:
controlling the light projector to project multi-stripe structured light and dot structured light onto a surface of an object to be captured, to form a plurality of light stripes and a plurality of light spots between the light stripes on the surface of the object to be captured;
acquiring, by the image sensor, a target image of the object to be captured having the light stripes and the light spots on the surface;
obtaining a reference image comprising the light stripes and the light spots;
calculating a first depth value at each of the light stripes in the target image based on feature information of the light stripes in the reference image and feature information of the light stripes in the target image;
calculating a second depth value at each of the light spots in the target image based on feature information of the light spots in the reference image and feature information of the light spots in the target image; and
generating a depth image of the target image based on the first depth value at the light stripe and the second depth value at the light spot in the target image.
2. The method according to claim 1 , wherein the calculating a first depth value at each of the light stripes in the target image comprises:
calculating the first depth value at a stripe center of each of the light stripes in the target image.
3. The method according to claim 1 , wherein the calculating a first depth value at each of the light stripes in the target image based on feature information of the light stripes in the reference image and feature information of the light stripes in the target image comprises:
matching the light stripes in the target image with the light stripes in the reference image, to obtain a plurality of stripe match pairs, wherein each of the stripe match pairs comprises a target light stripe in the target image and a reference light stripe in the reference image;
calculating a first parallax of the target light stripe with respect to the reference light stripe for each of the stripe match pairs based on feature information of the target light stripe and the reference light stripe in the stripe match pair; and
calculating the first depth value at the target light stripe in the target image based on the first parallax.
4. The method according to claim 1 , wherein the calculating a second depth value at each of the light spots in the target image based on feature information of the light spots in the reference image and feature information of the light spots in the target image comprises:
matching the light spots in the target image with the light spots in the reference image, to obtain a plurality of light spot match pairs, wherein each of the light spot match pairs comprises a first light spot in the target image and a second light spot in the reference image;
calculating a second parallax of the first light spot with respect to the second light spot for each of the light spot match pairs based on feature information of the first light spot and the second light spot in the light spot match pair; and
calculating the second depth value at the first light spot in the target image based on the second parallax.
5. The method according to claim 3 , wherein the calculating a second depth value at each of the light spots in the target image based on feature information of the light spots in the reference image and feature information of the light spots in the target image comprises:
matching the light spots in the target image with the light spots in the reference image, to obtain a plurality of light spot match pairs, wherein each of the light spot match pairs comprises a first light spot in the target image and a second light spot in the reference image;
calculating a second parallax of the first light spot with respect to the second light spot for each of the light spot match pairs based on feature information of the first light spot and the second light spot in the light spot match pair; and
calculating the second depth value at the first light spot in the target image based on the second parallax.
6. The method according to claim 4 , wherein the matching the light spots in the target image with the light spots in the reference image comprises:
determining, in the reference image, a first reference light stripe and a second reference light stripe respectively matching with a first target light stripe and a second target light stripe which are any two adjacent target light stripes in the target image; and
matching a target light spot between the first target light stripe and the second target light stripe in the target image with a reference light spot from light spots between the first reference light stripe and the second reference light stripe in the reference image, to obtain the light spot match pair.
7. The method according to claim 5 , wherein the matching the light spots in the target image with the light spots in the reference image comprises:
determining, in the reference image, a first reference light stripe and a second reference light stripe respectively matching with a first target light stripe and a second target light stripe which are any two adjacent target light stripes in the target image; and
matching a target light spot between the first target light stripe and the second target light stripe in the target image with a reference light spot from light spots between the first reference light stripe and the second reference light stripe in the reference image, to obtain the light spot match pair.
8. An image processing device, which is applied to an electronic device, the electronic device comprising at least an image sensor and a light projector, wherein the device comprises:
a processor, configured to control the light projector to project multi-stripe structured light and dot structured light onto a surface of an object to be captured, to form a plurality of light stripes and a plurality of light spots between the light stripes on the surface of the object to be captured;
the image sensor, configured to acquire a target image of the object to be captured having the light stripes and the light spots on the surface, wherein the processor is further configured to:
obtain a reference image comprising the light stripes and the light spots;
calculate a first depth value at each of the light stripes in the target image based on feature information of the light stripes in the reference image and feature information of the light stripes in the target image;
calculate a second depth value at each of the light spots in the target image based on feature information of the light spots in the reference image and feature information of the light spots in the target image; and
generate a depth image of the target image based on the first depth value at the light stripe and the second depth value at the light spot in the target image.
9. The device according to claim 8 , wherein the processor is further configured to:
calculate the first depth value at a stripe center of each of the light stripes in the target image based on the feature information of the light stripes in the reference image and the feature information of the light stripes in the target image.
10. The device according to claim 8 , wherein the processor is further configured to:
match the light stripes in the target image with the light stripes in the reference image, to obtain a plurality of stripe match pairs, wherein each of the stripe match pairs comprises a target light stripe in the target image and a reference light stripe in the reference image;
calculate a first parallax of the target light stripe with respect to the reference light stripe for each of the stripe match pairs based on feature information of the target light stripe and the reference light stripe in the stripe match pair; and
calculate the first depth value at the target light stripe in the target image based on the first parallax.
11. The device according to claim 8 , wherein the processor is further configured to:
match the light spots in the target image with the light spots in the reference image, to obtain a plurality of light spot match pairs, wherein each of the light spot match pairs comprises a first light spot in the target image and a second light spot in the reference image;
calculate a second parallax of the first light spot with respect to the second light spot for each of the light spot match pairs based on feature information of the first light spot and the second light spot in the light spot match pair; and
calculate the second depth value at the first light spot in the target image based on the second parallax.
12. The device according to claim 10 , wherein the processor is further configured to:
match the light spots in the target image with the light spots in the reference image, to obtain a plurality of light spot match pairs, wherein each of the light spot match pairs comprises a first light spot in the target image and a second light spot in the reference image;
calculate a second parallax of the first light spot with respect to the second light spot for each of the light spot match pairs based on feature information of the first light spot and the second light spot in the light spot match pair; and
calculate the second depth value at the first light spot in the target image based on the second parallax.
13. The device according to claim 11 , wherein the processor is further configured to:
determine, in the reference image, a first reference light stripe and a second reference light stripe respectively matching with a first target light stripe and a second target light stripe which are any two adjacent target light stripes in the target image; and
match a target light spot between the first target light stripe and the second target light stripe in the target image with a reference light spot from light spots between the first reference light stripe and the second reference light stripe in the reference image, to obtain the light spot match pair.
14. The device according to claim 12 , wherein the processor is further configured to:
determine, in the reference image, a first reference light stripe and a second reference light stripe respectively matching with a first target light stripe and a second target light stripe which are any two adjacent target light stripes in the target image; and
match a target light spot between the first target light stripe and the second target light stripe in the target image with a reference light spot from light spots between the first reference light stripe and the second reference light stripe in the reference image, to obtain the light spot match pair.