Systems and methods for obtaining a structured light reconstruction of a 3D surface
View Patent ↗Disclosed are systems and methods for obtaining a structured light reconstruction using a hybrid spatio-temporal pattern sequence projected on a surface. The method includes projecting a structured light pattern, such as a binary de Bruijn sequence, onto a 3D surface and acquiring an image set of at least a portion of this projected sequence with a camera system, and projecting a binary edge detection pattern onto the portion of the surface and acquiring an image set of the same portion of the projected binary pattern. The acquired image set of the binary pattern is processed to determine edge locations therein, and then employed to identify the locations of pattern edges within the acquired image set of the structured light pattern. The detected edges of the structured light pattern images are employed to decode the structured light pattern and calculate a disparity map, which is used to reconstruct the 3D surface.
1. A method for obtaining structured light reconstruction of a 3D surface using a structured light system comprising a projector, and a first camera and a second camera, the method comprising:
employing the projector to project a structured light pattern onto the 3D surface;
acquiring, with the first camera and the second camera, respective structured light images of the projected structured light pattern;
employing the projector to project a binary edge detection pattern onto the 3D surface, wherein the binary edge detection pattern is projected such that the elements of the projected binary edge detection pattern are spatially aligned with the elements of the projected structured light pattern, and such that a plurality of transition edges of the binary edge detection pattern are respectively aligned with non-transition edges of the projected structured light pattern, wherein transition edges correspond to boundaries between adjacent elements having different states, and wherein non-transition edges correspond to boundaries between adjacent elements having a common state;
acquiring, with the first camera and the second camera, respective binary edge detection images of the projected binary edge detection pattern;
processing each binary edge detection image to identify, within each binary edge detection image, locations of the transition edges;
determining edge locations of elements within the structured light images, wherein the locations of the transition edges of the binary edge detection images are employed to identify corresponding non-transition edges within the respective structured light images;
employing the edge locations within the structured light images to decode the structured light pattern within the structured light images;
performing stereo matching of the decoded structured light images and obtaining a disparity map; and
generating a reconstruction of the 3D surface based on the disparity map.
2. The method according to claim 1 wherein the structured light pattern is a binary structured light pattern.
3. The method according to claim 1 wherein the binary edge detection image comprise a binary comb pattern aligned with at least two adjacent elements of the structured light pattern that have a common state.
4. The method according to claim 1 wherein the binary edge detection pattern is a binary comb pattern.
5. The method according to claim 1 wherein the binary edge detection pattern includes transition edges that respectively correspond to each non-transition edge of the structured light pattern.
6. The method according to claim 1 wherein the binary edge detection pattern includes transition edges that respectively correspond to a subset of the non-transition edges of the structured light pattern.
7. The method according to claim 1 wherein the binary edge detection pattern is a first binary edge detection pattern and the binary edge detection images are first binary edge detection images, the method further comprising:
employing the projector to project an additional binary edge detection pattern onto the 3D surface and acquiring, with the first camera and the second camera, respective additional binary edge detection images of the projected additional binary edge detection pattern;
wherein the additional binary edge detection pattern is projected such that the elements of the projected additional binary edge detection pattern are spatially aligned with the elements of the projected structured light pattern, and such that a plurality of transition edges of the additional binary edge detection pattern are respectively aligned with non-transition edges of the projected structured light pattern; and
wherein locations of transition edges determined in the additional binary edge detection images are employed to identify corresponding non-transition edges within the respective structured light images.
8. The method according to claim 7 wherein the additional binary edge detection pattern is an inverse of the first binary edge detection pattern.
9. The method according to claim 8 wherein the first binary edge detection pattern comprises a binary comb pattern.
10. The method according to claim 8 wherein the first binary edge detection pattern is a binary comb pattern.
11. The method according to claim 1 wherein the structured light pattern is a de Bruijn pattern.
12. The method according to claim 11 wherein the de Bruijn pattern is a binary de Bruijn pattern.
13. The method according to claim 11 wherein the de Bruijn pattern has an alphabet greater than two.
14. The method according to claim 12 wherein the projected binary de Bruijn pattern comprises two de Bruijn patterns that are projected in a common row, the method further comprising:
employing the projector to project at least one Gray code the 3D surface such that the Gray code pattern is spatially aligned with the common row;
acquiring, with the first camera and the second camera, respective Gray code images of the projected Gray code pattern;
employing the Gray code images to subdivide the projected binary de Bruijn pattern into regions respectively corresponding to the two de Bruijn patterns.
15. A method for obtaining structured light reconstruction of a 3D surface using a structured light system comprising a projector and a camera, the method comprising:
employing the projector to project a structured light pattern onto the 3D surface;
acquiring a structured light image of the projected structured light pattern with the camera;
employing the projector to project a binary edge detection pattern onto the 3D surface, wherein the binary edge detection pattern is projected such that the elements of the projected binary edge detection pattern are spatially aligned with the elements of the projected structured light pattern, and such that a plurality of transition edges of the binary edge detection pattern are respectively aligned with non-transition edges of the projected structured light pattern, wherein the transition edges correspond to boundaries between adjacent elements having different states, and the non-transition edges correspond to boundaries between adjacent elements having a common state;
acquiring a binary edge detection image of the projected binary edge detection pattern with the camera;
processing the binary edge detection image to identify locations of the transition edges;
determining edge locations of elements within the structured light image, wherein the locations of the transition edges of the binary edge detection image are employed to identify corresponding non-transition edges within the structured light image;
employing the edge locations within the structured light image to decode the structured light pattern within the structured light image;
performing stereo matching of the decoded structured light image with a reference and obtaining a disparity map; and
generating a reconstruction of the 3D surface based on the disparity map.
16. The method according to claim 15 wherein the structured light pattern is a binary structured light pattern.
17. The method according to claim 15 wherein the binary edge detection images comprise a binary comb pattern aligned with at least two adjacent elements of the structured light pattern that have a common state.
18. The method according to claim 15 wherein the binary edge detection pattern is a binary comb pattern.
19. The method according to claim 15 wherein the binary edge detection pattern includes transition edges that respectively correspond to each non-transition edge of the structured light pattern.
20. The method according to claim 15 wherein the binary edge detection pattern includes transition edges that respectively correspond to a subset of the non-transition edges of the structured light pattern.
21. The method according to claim 15 wherein the binary edge detection pattern is a first binary edge detection pattern and the binary edge detection image is a first binary edge detection image, the method further comprising:
employing the projector to project an additional binary edge detection pattern onto the 3D surface and acquiring, with the camera, an additional binary edge detection image of the projected additional binary edge detection pattern;
wherein the additional binary edge detection pattern is projected such that the elements of the projected additional binary edge detection pattern are spatially aligned with the elements of the projected structured light pattern, and such that a plurality of transition edges of the additional binary edge detection pattern are respectively aligned with non-transition edges of the projected structured light pattern; and
wherein locations of transition edges determined in the additional binary edge detection image are employed to identify corresponding non-transition edges within the structured light image.
22. The method according to claim 21 wherein the additional binary edge detection pattern is an inverse of the first binary edge detection pattern.
23. The method according to claim 22 wherein the first binary edge detection pattern comprises a binary comb pattern.
24. The method according to claim 22 wherein the first binary edge detection pattern is a binary comb pattern.
25. The method according to claim 15 wherein the structured light pattern is a de Bruijn pattern.
26. The method according to claim 25 wherein the de Bruijn pattern is a binary de Bruijn pattern.
27. The method according to claim 25 wherein the de Bruijn pattern has an alphabet greater than two.
28. The method according to claim 26 wherein the projected binary de Bruijn pattern comprises two de Bruijn patterns that are projected in a common row, the method further comprising:
employing the projector to project at least one Gray code the 3D surface such that the Gray code pattern is spatially aligned with the common row;
acquiring, with the camera, respective Gray code images of the projected Gray code pattern;
employing the Gray code image to subdivide the projected binary de Bruijn pattern into regions respectively corresponding to the two de Bruijn patterns.
29. A system for obtaining structured light reconstruction of a 3D surface, the system comprising:
a projector configured to project patterns onto a surface;
a first camera and a second camera, wherein the first camera and the second camera are oriented to image the patterns projected by the projector; and
computer hardware operatively coupled to said projector, said first camera and said second camera, wherein said computer hardware comprises memory coupled with one or more processors to store instructions, which when executed by the one or more processors, causes the one or more processors to perform operations comprising:
employing the projector to project a structured light pattern onto the 3D surface;
acquiring, with the first camera and the second camera, respective structured light images of the projected structured light pattern;
employing the projector to project a binary edge detection pattern onto the 3D surface, wherein the binary edge detection pattern is projected such that the elements of the projected binary edge detection pattern are spatially aligned with the elements of the projected structured light pattern, and such that a plurality of transition edges of the binary edge detection pattern are respectively aligned with non-transition edges of the projected structured light pattern, wherein transition edges correspond to boundaries between adjacent elements having different states, and wherein non-transition edges correspond to boundaries between adjacent elements having a common state;
acquiring, with the first camera and the second camera, respective binary edge detection images of the projected binary edge detection pattern;
processing each binary edge detection image to identify, within each binary edge detection image, locations of the transition edges;
determining edge locations of elements within the structured light images, wherein the locations of the transition edges of the binary edge detection images are employed to identify corresponding non-transition edges within the respective structured light images;
employing the edge locations within the structured light images to decode the structured light pattern within the structured light images;
performing stereo matching of the decoded structured light images and obtaining a disparity map; and
generating a reconstruction of the 3D surface based on the disparity map.
30. The system according to claim 29 wherein said projector is controlled by said computer hardware such that the structured light pattern is a binary structured light pattern.
31. The system according to claim 29 wherein said projector is controlled by said computer hardware such that the binary edge detection images comprise a binary comb pattern aligned with at least two adjacent elements of the structured light pattern that have a common state.
32. The system according to claim 29 wherein said projector is controlled by said computer hardware such that the binary edge detection pattern is a binary comb pattern.
33. The system according to claim 29 wherein said projector is controlled by said computer hardware such that the binary edge detection pattern includes transition edges that respectively correspond to each non-transition edge of the structured light pattern.
34. The system according to claim 29 wherein said projector is controlled by said computer hardware such that the binary edge detection pattern includes transition edges that respectively correspond to a subset of the non-transition edges of the structured light pattern.
35. The system according to claim 29 wherein said projector is controlled by said computer hardware such that the binary edge detection pattern is a first binary edge detection pattern and the binary edge detection images are first binary edge detection images, and wherein said computer hardware is further configured to perform operations comprising:
employing the projector to project an additional binary edge detection pattern onto the 3D surface and acquiring, with the first camera and the second camera, respective additional binary edge detection images of the projected additional binary edge detection pattern;
wherein the additional binary edge detection pattern is projected such that the elements of the projected additional binary edge detection pattern are spatially aligned with the elements of the projected structured light pattern, and such that a plurality of transition edges of the additional binary edge detection pattern are respectively aligned with non-transition edges of the projected structured light pattern; and
wherein locations of transition edges determined in the additional binary edge detection images are employed to identify corresponding non-transition edges within the respective structured light images.
36. The system according to claim 35 wherein said projector is controlled by said computer hardware such that the additional binary edge detection pattern is an inverse of the first binary edge detection pattern.
37. The system according to claim 36 wherein said projector is controlled by said computer hardware such that the first binary edge detection pattern comprises a binary comb pattern.
38. The system according to claim 36 wherein said projector is controlled by said computer hardware such that the first binary edge detection pattern is a binary comb pattern.
39. The system according to claim 29 wherein said projector is controlled by said computer hardware such that the structured light pattern is a de Bruijn pattern.
40. The system according to claim 39 wherein said projector is controlled by said computer hardware such that the de Bruijn pattern is a binary de Bruijn pattern.
41. The system according to claim 39 wherein said projector is controlled by said computer hardware such that the de Bruijn pattern has an alphabet greater than two.
42. The system according to claim 40 wherein said projector is controlled by said computer hardware such that the projected binary de Bruijn pattern comprises two de Bruijn patterns that are projected in a common row, and wherein said computer hardware is further configured to perform operations comprising:
employing the projector to project at least one Gray code the 3D surface such that the Gray code pattern is spatially aligned with the common row;
acquiring, with the first camera and the second camera, respective Gray code images of the projected Gray code pattern;
employing the Gray code images to subdivide the projected binary de Bruijn pattern into regions respectively corresponding to the two de Bruijn patterns.
43. A system for obtaining structured light reconstruction of a 3D surface, the system comprising:
a projector configured to project patterns onto a surface;
a camera oriented to image the patterns projected by the projector; and
computer hardware operatively coupled to said projector and said camera, wherein said computer hardware comprises memory coupled with one or more processors to store instructions, which when executed by the one or more processors, causes the one or more processors to perform operations comprising:
employing the projector to project a structured light pattern onto the 3D surface;
acquiring a structured light image of the projected structured light pattern with the camera;
employing the projector to project a binary edge detection pattern onto the 3D surface, wherein the binary edge detection pattern is projected such that the elements of the projected binary edge detection pattern are spatially aligned with the elements of the projected structured light pattern, and such that a plurality of transition edges of the binary edge detection pattern are respectively aligned with non-transition edges of the projected structured light pattern, wherein the transition edges correspond to boundaries between adjacent elements having different states, and the non-transition edges correspond to boundaries between adjacent elements having a common state;
acquiring a binary edge detection image of the projected binary edge detection pattern with the camera;
processing the binary edge detection image to identify locations of the transition edges;
determining edge locations of elements within the structured light image, wherein the locations of the transition edges of the binary edge detection image are employed to identify corresponding non-transition edges within the structured light image;
employing the edge locations within the structured light image to decode the structured light pattern within the structured light image;
performing stereo matching of the decoded structured light image with a reference and obtaining a disparity map; and
generating a reconstruction of the 3D surface based on the disparity map.
44. The system according to claim 43 wherein said projector is controlled by said computer hardware such that wherein the structured light pattern is a binary structured light pattern.
45. The system according to claim 43 wherein said projector is controlled by said computer hardware such that the binary edge detection images comprise a binary comb pattern aligned with at least two adjacent elements of the structured light pattern that have a common state.
46. The system according to claim 43 wherein said projector is controlled by said computer hardware such that the binary edge detection pattern is a binary comb pattern.
47. The system according to claim 43 wherein said projector is controlled by said computer hardware such that the binary edge detection pattern includes transition edges that respectively correspond to each non-transition edge of the structured light pattern.
48. The system according to claim 43 wherein said projector is controlled by said computer hardware such that the binary edge detection pattern includes transition edges that respectively correspond to a subset of the non-transition edges of the structured light pattern.
49. The system according to claim 43 wherein said projector is controlled by said computer hardware such that the binary edge detection pattern is a first binary edge detection pattern and the binary edge detection image is a first binary edge detection image, wherein said computer hardware is further configured to perform operations comprising:
employing the projector to project an additional binary edge detection pattern onto the 3D surface and acquiring, with the camera, an additional binary edge detection image of the projected additional binary edge detection pattern;
wherein the additional binary edge detection pattern is projected such that the elements of the projected additional binary edge detection pattern are spatially aligned with the elements of the projected structured light pattern, and such that a plurality of transition edges of the additional binary edge detection pattern are respectively aligned with non-transition edges of the projected structured light pattern; and
wherein locations of transition edges determined in the additional binary edge detection image are employed to identify corresponding non-transition edges within the structured light image.
50. The system according to claim 49 wherein said projector is controlled by said computer hardware such that the additional binary edge detection pattern is an inverse of the first binary edge detection pattern.
51. The system according to claim 50 wherein said projector is controlled by said computer hardware such that the first binary edge detection pattern comprises a binary comb pattern.
52. The system according to claim 50 wherein said projector is controlled by said computer hardware such that the first binary edge detection pattern is a binary comb pattern.
53. The system according to claim 43 wherein said projector is controlled by said computer hardware such that the structured light pattern is a de Bruijn pattern.
54. The system according to claim 53 wherein said projector is controlled by said computer hardware such that the de Bruijn pattern is a binary de Bruijn pattern.
55. The system according to claim 53 wherein said projector is controlled by said computer hardware such that the de Bruijn pattern has an alphabet greater than two.
56. The system according to claim 54 wherein said projector is controlled by said computer hardware such that the projected binary de Bruijn pattern comprises two de Bruijn patterns that are projected in a common row, and wherein said computer hardware is further configured to perform operations comprising:
employing the projector to project at least one Gray code the 3D surface such that the Gray code pattern is spatially aligned with the common row;
acquiring, with the camera, respective Gray code images of the projected Gray code pattern;
employing the Gray code image to subdivide the projected binary de Bruijn pattern into regions respectively corresponding to the two de Bruijn patterns.