Room layout estimation methods and techniques
View Patent ↗Systems and methods for estimating a layout of a room are disclosed. The room layout can comprise the location of a floor, one or more walls, and a ceiling. In one aspect, a neural network can analyze an image of a portion of a room to determine the room layout. The neural network can comprise a convolutional neural network having an encoder sub-network, a decoder sub-network, and a side sub-network. The neural network can determine a three-dimensional room layout using two-dimensional ordered keypoints associated with a room type. The room layout can be used in applications such as augmented or mixed reality, robotics, autonomous indoor navigation, etc.
1. A system comprising:
non-transitory memory configured to store:
a room image for room layout estimation; and
a neural network for estimating a layout of a room, the neural network comprising:
an encoder-decoder sub-network configured to receive the digital room image as an input, wherein the encoder-decoder sub-network comprises an encoder and a decoder; and
a classifier sub-network in communication with the encoder-decoder sub-network configured to classify a room type associated with the room image;
a hardware processor in communication with the non-transitory memory, the hardware processor programmed to:
access the room image;
determine, using the encoder, decoder, and the room image, a plurality of predicted two-dimensional (2D) keypoint maps corresponding to a plurality of room types;
determine, using the encoder, the classifier sub-network and the room image, a predicted room type from the plurality of room types;
determine, using the plurality of predicted 2D keypoint maps and the predicted room type, a plurality of ordered keypoints associated with the predicted room type; and
determine, using the plurality of ordered keypoints, a predicted layout of the room in the room image.
2. The system of claim 1 , wherein each room type in the plurality of room types comprises an ordered set of room type keypoints.
3. The system of claim 2 , wherein each room type in the plurality of room types comprises a semantic segmentation for regions in the room type, the semantic segmentation comprising an identification as a floor, a ceiling, or a wall.
4. The system of claim 2 , wherein a first keypoint order is associated with a first room type of the plurality of room types and a second keypoint order is associated with a second room type of the plurality of room types, wherein the first keypoint order and the second keypoint order are different.
5. The system of claim 1 , wherein the room image comprises a monocular image.
6. The system of claim 1 , wherein the room image comprises a Red-Green-Blue (RGB) image.
7. The system of claim 1 , wherein a dimensionality of the room image is larger than a dimensionality of the predicted 2D keypoint maps.
8. The system of claim 1 , wherein the encoder-decoder sub-network comprises an encoder sub-network comprising a plurality of convolutional layers and a plurality of pooling layers.
9. The system of claim 1 , wherein the encoder-decoder sub-network comprises a decoder sub-network comprising a plurality of convolutional layers and a plurality of upsampling layers.
10. The system of claim 1 , wherein the encoder-decoder sub-network comprises a memory augmented recurrent encoder-decoder (MRED) network.
11. The system of claim 1 , wherein the encoder-decoder sub-network comprises a plurality of recurrent layers.
12. The system of claim 11 , wherein a number of recurrent iterations of the plurality of recurrent layers is two.
13. The system of claim 11 , wherein a number of recurrent iterations of the plurality of recurrent layers is at least three.
14. The system of claim 11 , wherein each of the plurality of recurrent layers has a weight matrix, and the weight matrix is the same for all of the plurality of recurrent layers.
15. The system of claim 1 , wherein the predicted two-dimensional (2D) keypoint maps comprise heat maps.
16. The system of claim 15 , wherein the hardware processor is programmed to extract keypoint locations from the heat maps as maxima of the heat map.
17. The system of claim 1 , wherein the hardware processor is programmed to:
access object information from an object recognizer that analyzes the room image; and
combine the object information with the predicted layout of the room.
18. The system of claim 17 , wherein the object recognizer is configured to detect cuboids in the room image.
19. A wearable display device comprising:
an outward-facing imaging system configured to capture the room image for room layout estimation; and
the system of claim 1 .
20. A system comprising:
non-transitory memory configured to store parameters for the neural network; and
a hardware processor in communication with the non-transitory memory, the hardware processor programmed to:
receive a training room image, wherein the training room image is associated with:
a reference room type from a plurality of room types, and
reference keypoints associated with a reference room layout;
generate a neural network for room layout estimation, wherein the neural network comprises:
an encoder-decoder sub-network configured to receive the training room image as an input and output predicted two-dimensional (2D) keypoints associated with a predicted room layout associated with each of the plurality of room types, wherein the encoder-decoder sub-network comprises an encoder and a decoder, and
a side sub-network in communication with the encoder-decoder network configured to output a predicted room type from the plurality of room types, wherein the predicted room type is determined at least in part by the encoder and side sub-network; and
optimize a loss function based on a first loss for the predicted 2D keypoints and a second loss for the predicted room type; and
update parameters of the neural network based on the optimized loss function.
21. The system of claim 20 , wherein a number of the plurality of room types is greater than 5.
22. The system of claim 20 , wherein the reference keypoints and the predicted 2D keypoints are associated with a keypoint order.
23. The system of claim 20 , wherein a first keypoint order is associated with a first room type of the plurality of room types and a second keypoint order is associated with a second room type of the plurality of room types, wherein the first keypoint order and the second keypoint order are different.
24. The system of claim 20 , wherein the training room image comprises a monocular image.
25. The system of claim 20 , wherein the training room image comprises a Red-Green-Blue (RGB) image.
26. The system of claim 20 , wherein a dimensionality of the training room image is larger than a dimensionality of a map associated with the predicted 2D keypoints.
27. The system of claim 20 , wherein the encoder sub-network and the decoder sub-network comprises a plurality of recurrent layers.
28. The system of claim 27 , wherein a number of recurrent iterations of the recurrent layers is two or three.
29. The system of claim 27 , wherein deep supervision is applied to the recurrent layers.
30. The system of claim 27 , wherein weights associated with a first recurrent iteration of the iterations of the recurrent layers are identical to weights associated with a second recurrent iteration of the current layers.
31. The system of claim 27 , wherein the plurality of recurrent layers are configured as a memory-augmented recurrent encoder-decoder (MRED) network.
32. The system of claim 20 , wherein the side sub-network comprises a room type classifier.
33. The system of claim 20 , wherein the first loss for the predicted 2D keypoints comprises a Euclidean loss between the plurality of reference keypoint locations and the predicted 2D keypoints.
34. The system of claim 20 , wherein the second loss for the predicted room type comprises a cross entropy loss.
35. The system of claim 20 , wherein the predicted 2D keypoints are extracted from a predicted heat map.
36. The system of claim 35 , wherein hardware processor is programmed to:
calculate a reference heat map associated with the reference keypoints of the training image; and
calculate the first loss for the predicted 2D keypoints based on a difference between the predicted heat map and the reference heat map.
37. The system of claim 36 , wherein the reference heat map comprises a two-dimensional distribution centered at a location for each reference keypoint.
38. The system of claim 36 , wherein the reference heat map comprises a background away from the reference keypoints and a foreground associated with the reference keypoints, and the hardware processor is programmed to weight gradients in the reference heat map based on a ratio between the foreground and the background.
39. The system claim 38 , wherein to weight the gradients in the reference heat map, the hardware processor is programmed to degrade values of pixels in the background by a degrading factor less than one.