Three-dimensional room analysis with audio input

1. A computer-implemented method for generating a representation of a physical environment comprising:

under control of a computer hardware processor configured with specific computer executable instructions,

receiving, from a user computing device, input data including (i) a first digital representation of a physical space, wherein each discrete unit extracted from the first digital representation represents a portion of the physical space and is associated with a first timestamp, and (ii) audio data, wherein at least some of the first digital representation and the audio data were synchronously captured by the user computing device;

generating a transcript from the audio data;

identifying a keyword from the transcript associated with a second timestamp;

selecting, from a plurality of machine learning models, a machine learning model based at least in part on the keyword, wherein the machine learning model is trained to locate an object that corresponds to the keyword in at least a portion of a digital representation; and

generating a three-dimensional model of the physical space from at least the first digital representation, wherein generating the three-dimensional model further includes:

identifying a first portion of the first digital representation based at least in part on matching the second timestamp with the first timestamp;

applying the machine learning model to the first portion of the first digital representation, wherein applying the machine learning model results in model output;

locating an object in the three-dimensional model based at least in part on the model output; and

creating metadata that locates the object in the three-dimensional model.

2. The computer-implemented method of claim 1 , wherein the metadata includes an annotation that describes the object.

3. The computer-implemented method of claim 1 , wherein (i) the first digital representation includes a video and (ii) the discrete unit includes a frame, and wherein identifying the first portion of the first digital representation further comprises:

identifying a plurality of frames within a threshold period of time of the video synchronized with the second timestamp.

4. The computer-implemented method of claim 1 , wherein (i) the first digital representation includes a point cloud and (ii) the discrete unit includes a point, and wherein the point cloud is generated from a lidar sensor of the user computing device.

5. The computer-implemented method of claim 1 , wherein identifying the keyword from the transcript further comprises:

comparing a first word from the transcript with a plurality of keywords from a keyword dictionary.

6. The computer-implemented method of claim 1 , wherein the machine learning model includes at least one of a neural network, a classifier, a detector, or a segmenter.

7. A system comprising:

a non-transitory data storage medium; and

a computer hardware processor in communication with the non-transitory data storage medium, wherein the computer hardware processor is configured to execute computer-executable instructions to at least:

receive, from a user computing device, input data including (i) a first digital representation of a physical space, wherein each discrete unit extracted from the first digital representation is associated with a first timestamp, and (ii) audio data;

obtain a transcript for the audio data;

identify a keyword from the transcript associated with a second timestamp;

select, from a plurality of machine learning models, a machine learning model based at least in part on the keyword, wherein the machine learning model is trained to identify an object that corresponds to the keyword in at least a portion of a digital representation; and

generate a first three-dimensional model of the physical space from at least the first digital representation, wherein to generate the first three-dimensional model, the computer hardware processor is configured to execute the computer-executable instructions to at least:

identify a first portion of the first digital representation based at least in part on matching the second timestamp with the first timestamp;

apply the machine learning model to the first portion of the first digital representation;

receive model output from the machine learning model; and

identify a first object in the first digital representation based at least in part on the model output.

8. The system of claim 7 , wherein the computer hardware processor is configured to execute further computer-executable instructions to at least:

determine, from at least some words from the transcript, a likely user intent associated with the keyword, wherein the likely user intent indicates a likely intent for removal or replacement of the first object.

9. The system of claim 8 , wherein to generate the first three-dimensional model, the computer hardware processor is configured to execute additional computer-executable instructions to at least:

create metadata that identifies the first object in the first three-dimensional model, wherein the metadata further includes an indication of the likely intent for the removal or replacement of the first object.

10. The system of claim 8 , wherein to generate the first three-dimensional model, the computer hardware processor is configured to execute additional computer-executable instructions to at least:

remove the first object from the first three-dimensional model.

11. The system of claim 10 , wherein the computer hardware processor is configured to execute supplemental computer-executable instructions to at least:

generate a second three-dimensional model that corresponds to the first three-dimensional model, wherein to generate the second three-dimensional model, the computer hardware processor is configured to execute the supplemental computer-executable instructions to at least:

insert, into the second three-dimensional model, a second object at a location where the first object was located.

12. The system of claim 7 , wherein (i) the first digital representation includes a video and (ii) the discrete unit includes a frame, wherein the input data includes a point cloud of the physical space, and wherein to generate the first three-dimensional model, the computer hardware processor is configured to execute further computer-executable instructions to at least:

provide the point cloud to a library; and

receive the three-dimensional model from the library based at least in part on the point cloud.

13. The system of claim 12 , wherein to identify the first portion of the first digital representation, the computer hardware processor is configured to execute further computer-executable instructions to at least:

identify a plurality of frames within a threshold period of time of the video synchronized with the second timestamp.

14. A system comprising:

a non-transitory data storage medium; and

a computer hardware processor in communication with the non-transitory data storage medium, wherein the computer hardware processor is configured to execute computer-executable instructions to at least:

receive, from a user computing device, input data including (i) a first digital representation of a physical space, wherein each discrete unit extracted from the first digital representation is associated with a first timestamp, and (ii) audio data;

obtain a transcript for the audio data;

identify a keyword from the transcript associated with a second timestamp; and

generate a three-dimensional model of the physical space from at least the first digital representation, wherein to generate the three-dimensional model, the computer hardware processor is configured to execute the computer-executable instructions to at least:

identify a first portion of the first digital representation based at least in part on matching the second timestamp with the first timestamp;

apply a machine learning model to the first portion of the first digital representation based at least in part on the keyword, wherein the machine learning model is trained to identify an object that corresponds to the keyword in at least a portion of a digital representation;

receive model output from the machine learning model; and

identify a first object in the first digital representation based at least in part on the model output.

15. The system of claim 14 , wherein to apply the machine learning model, the computer hardware processor are configured to execute further computer-executable instructions to at least:

convert the keyword to model input; and

provide (i) the model input and (ii) the first portion of the first digital representation to the machine learning model as input.

16. The system of claim 15 , wherein the model output includes a plurality of probabilities, each probability of the plurality of probabilities indicating a likelihood that a respective object is present in the first portion of the first digital representation, wherein to identify the first object, the computer hardware processor is configured to execute additional computer-executable instructions to at least:

select a highest probability from the plurality of probabilities, wherein the highest probability is associated with the first object.

17. The system of claim 15 , wherein (i) the first digital representation includes a video and (ii) the discrete unit includes a frame, wherein the model output includes a segmentation mask, the segmentation mask associates a label with a pixel from the frame.

18. The system of claim 14 , wherein the computer hardware processor is configured to execute further computer-executable instructions to at least:

determine, from at least some words from the transcript, a likely user intent associated with the keyword, wherein the likely user intent indicates a likely intent to keep the first object.

19. The system of claim 18 , wherein to generate the three-dimensional model, the computer hardware processor is configured to execute additional computer-executable instructions to at least:

create metadata that identifies the first object in the three-dimensional model, wherein the metadata further includes an indication of the likely intent to keep the first object.

20. The system of claim 18 , wherein to generate the three-dimensional model, the computer hardware processor is configured to execute additional computer-executable instructions to at least:

include the first object in the three-dimensional model.