Granted Patent B2

US 12,444,139 · App. 17/950,865 · Granted Oct 14, 2025

Inventors: Manjunath Narayana (Chino Hills, CA); Eric M. Penner (Centennial, CO); Ivaylo Boyadzhiev (Seattle, WA); Sing Bing Kang (Redmond, WA)

Assignee: MFTB Holdco, Inc.

G06T19/003G06F3/04842G06F16/587G06T19/006G06V20/20

Abstract

Claims (129)

1. A system comprising:

one or more hardware processors of one or more computing devices; and

one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations including at least:

obtaining data for a building with multiple rooms, including information about multiple objects at the building, and including information about respective positions at the building associated with the multiple objects, wherein the multiple objects in the building include at least appliances, fixtures, and structural elements, and wherein the multiple objects in the building further have colors and types of surface materials;

receiving a target image showing a current view from a camera of a mobile device at the building;

determining a subset of the building visible in the target image;

selecting one or more objects of the multiple objects that have respective positions in the subset of the building;

generating visual enhancement data that includes, for each of the selected objects, at least one visual indicator for that selected object for presentation at the respective position of that selected object, and information indicating the determined one or more utility values associated with that selected object; and

providing information to the mobile device, including providing the visual enhancement data for presentation on the mobile device overlaid on the current view from the camera, wherein the visual enhancement data for presentation includes at least one of a color or a type of surface material for one of the selected objects.

2. The system of claim 1 wherein the stored instructions include software instructions that, when executed, cause the one or more computing devices to perform the obtaining of the data for the building by performing at least one of:

determining, based at least in part on analyzing visual data of a plurality of images acquired in the multiple rooms of the building, the multiple objects and the respective positions of the multiple objects; or

generating, for each of the selected one or more objects, a textual description of the object, and wherein the generating of the visual enhancement data further includes, for each of the selected one or more objects, the generated textual description for that object.

3. The system of claim 2 wherein the one or more computing devices include the mobile device, and wherein the providing of the information to the mobile device is performed by the mobile device and includes overlaying the visual enhancement data on the target image.

4. The system of claim 2 wherein the one or more computing devices include at least one computing device remote from the building,

wherein the obtaining of the data for the building is performed by the at least one computing device remote from the building and includes performing the determining of the multiple objects and the respective positions of the multiple objects, including:

capturing the plurality of the images in the multiple rooms of the building using one or more mobile image acquisition devices separate from the mobile device; and

performing the analyzing of the visual data of the plurality of images using one or more trained machine learning models; and

wherein the providing of the visual enhancement data includes presenting the target image overlaid with the visual enhancement data including, for each of the selected one or more objects, displaying the at least one visual indicator for that selected object at a determined position in the target image that is based on the respective position for that selected object.

5. The system of claim 2 wherein the obtaining of the data for the building includes analyzing, for a floor plan showing a room layout for the building with at least two-dimensional room shapes and relative positions of the multiple rooms, the floor plan to determine one or more attributes of the building that each corresponds to a characteristic of the room layout, and

wherein the providing of the information to the mobile device at the building further includes selecting one of the attributes corresponding to a characteristic of the room layout to include in the generated visual enhancement data; and

wherein the providing of the visual enhancement data includes displaying overlaid information about the selected one attribute at a position in the current view from the camera that is determined for the selected one attribute.

6. The system of claim 1 wherein the providing of the visual enhancement data includes presenting on the mobile device the visual enhancement data overlaid on the current view from the camera as part of a graphical user interface (GUI) displayed on at least one display of the mobile device, and wherein the providing of the information to the mobile device further includes:

determining, by the mobile device and before the providing of the visual enhancement data, one or more GUI controls for which to include one or more respective user-selectable visual representations in the generated visual enhancement data;

receiving, by the mobile device and after the presenting of the visual enhancement data overlaid on the current view from the camera, a selection by the user of one of the respective user-selectable visual representations for one of the GUI controls; and

updating, by the mobile device and in response to the selection by the user, the at least one display of the mobile device to display at least one type of additional data.

7. The system of claim 1 wherein the target image shows a portion of a first room of the multiple rooms in which the mobile device is located, and wherein the automated operations further include:

determining additional information to present to a user in a second room of the multiple rooms that is different from the first room;

generating guidance instructions to direct the user from the first room to the second room;

displaying at least some of the generated guidance instructions overlaid on that target image to direct the user to the second room; and

displaying, after the mobile device reaches the second room, at least some of the determined additional information.

8. The system of claim 1 wherein the providing of the visual enhancement data further includes obtaining information about a user of the mobile device that includes one or more preferences of the user, wherein the selecting of the one or more objects is performed based on the one or more preferences of the user, and wherein the providing of the visual enhancement data includes presenting to the user the visual enhancement data in combination with the current view from the camera.

9. The system of claim 1 wherein the providing of the visual enhancement data includes presenting, on at least one display of the mobile device, the visual enhancement data overlaid on the target image, and further includes transmitting, to one or more additional devices separate from the mobile device and remote from the building, the enhancement data overlaid on the target image for concurrent presentation on the one or more additional devices.

10. A system comprising:

one or more hardware processors of one or more computing devices; and

one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations including at least:

obtaining data for a building with multiple rooms, including information about multiple objects at the building, and including information about respective positions at the building associated with the multiple objects, wherein the obtaining of the data for the building includes generating, for each of the selected one or more objects, a textual description of that object by using one or more language models that are trained to use, as input, information about that object and about a room of the multiple rooms in the building in which that object is located, wherein the one or more language models include at least one of a Vision and Language Model (VLM) that is trained using image/caption tuples, or a Knowledge Enhanced Natural Language Generation (VENLG) model that is trained using one or more defined knowledge sources;

receiving a target image showing a current view from a camera of a mobile device at the building;

determining a subset of the building visible in the target image;

selecting one or more objects of the multiple objects that have respective positions in the subset of the building;

generating visual enhancement data that includes, for each of the selected objects, at least one visual indicator for that selected object for presentation at the respective position of that selected object, and information indicating the determined one or more utility values associated with that selected object, wherein the generating of the visual enhancement data further includes, for each of the selected one or more objects, the generated textual description for that object; and

providing information to the mobile device, including providing the visual enhancement data for presentation on the mobile device overlaid on the current view from the camera.

11. A system comprising:

one or more hardware processors of one or more computing devices; and

one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations including at least:

obtaining data for a building with multiple rooms, including information about multiple objects at the building, and including information about respective positions at the building associated with the multiple objects, wherein the obtaining of the data further includes obtaining information about both objective attributes about the building that are able to be independently verified and subjective attributes for the building that are predicted by the one or more trained neural network models;

receiving a target image showing a current view from a camera of a mobile device at the building;

determining a subset of the building visible in the target image;

selecting one or more objects of the multiple objects that have respective positions in the subset of the building;

generating visual enhancement data that includes, for each of the selected objects, at least one visual indicator for that selected object for presentation at the respective position of that selected object, and information indicating the determined one or more utility values associated with that selected object; and

providing information to the mobile device, including providing the visual enhancement data for presentation on the mobile device overlaid on the current view from the camera, wherein the providing of the visual enhancement data includes presenting information about at least one objective attribute and at least one subjective attribute.

12. A system comprising:

one or more hardware processors of one or more computing devices; and

one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations including at least:

obtaining data for a building with multiple rooms, including information about multiple objects at the building, and including information about respective positions at the building associated with the multiple objects, wherein the obtaining of the data further includes obtaining information about one or more comparison attributes that each includes a comparison of at least one type of information about the building to that at least one type of information about one or more other buildings;

receiving a target image showing a current view from a camera of a mobile device at the building;

determining a subset of the building visible in the target image;

selecting one or more objects of the multiple objects that have respective positions in the subset of the building;

generating visual enhancement data that includes, for each of the selected objects, at least one visual indicator for that selected object for presentation at the respective position of that selected object, and information indicating the determined one or more utility values associated with that selected object; and

providing information to the mobile device, including providing the visual enhancement data for presentation on the mobile device overlaid on the current view from the camera, wherein the providing of the visual enhancement data includes presenting information that describes at least one of the comparison attributes.

13. A system comprising:

one or more hardware processors of one or more computing devices; and

one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations including at least:

obtaining data for a building with multiple rooms, including information about multiple objects at the building, and including information about respective positions at the building associated with the multiple objects;

receiving a target image showing a current view from a camera of a mobile device at the building;

determining a subset of the building visible in the target image, wherein the determined subset of the building visible in the target image includes a portion of a first room of the multiple rooms, and wherein the target image shows at least some of one or more walls of the first room;

selecting one or more objects of the multiple objects that have respective positions in the subset of the building;

generating visual enhancement data that includes, for each of the selected objects, at least one visual indicator for that selected object for presentation at the respective position of that selected object, and information indicating the determined one or more utility values associated with that selected object, and wherein the generating of the visual enhancement data includes:

determining one or more second rooms of the multiple rooms that are behind the at least some of the one or more walls and that are not visible in the target image; and

generating visual representations of the determined one or more second rooms for use as part of the generated visual enhancement data; and

providing information to the mobile device, including providing the visual enhancement data for presentation on the mobile device overlaid on the current view from the camera, and presenting, as part of the providing of the visual enhancement data, the generated visual representations of the determined one or more second rooms at locations on the current view from the camera that correspond to positions of the determined one or more second rooms behind the at least some of the one or more walls.

14. A system comprising:

one or more hardware processors of one or more computing devices; and

one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations including at least:

obtaining data for a building with multiple rooms, including information about multiple objects at the building, and including information about respective positions at the building associated with the multiple objects;

receiving a target image showing a current view from a camera of a mobile device at the building;

determining a subset of the building visible in the target image;

selecting one or more objects of the multiple objects that have respective positions in the subset of the building;

generating visual enhancement data that includes, for each of the selected objects, at least one visual indicator for that selected object for presentation at the respective position of that selected object, and information indicating the determined one or more utility values associated with that selected object, wherein the generating of the visual enhancement data includes determining, for each of the selected one or more objects, one or more utility values associated with the object, each utility value indicating one of a condition of that object or a monetary value associated with that object; and

providing information to the mobile device, including providing the visual enhancement data for presentation on the mobile device overlaid on the current view from the camera, and presenting, as part of the providing of the visual enhancement data, the visual enhancement data overlaid on the current view from the camera, including presenting information indicating the determined one or more utility values associated with each of the selected one or more objects.

15. A computer-implemented method comprising:

obtaining, by one or more computing devices, data for a building with multiple rooms, including information about multiple objects at the building, and including information about respective positions at the building associated with the multiple objects, wherein the multiple objects in the building include at least appliances, fixtures, and structural elements, and wherein the multiple objects in the building further have colors and types of surface materials;

receiving, by the one or more computing devices, a target image showing a current view from a camera of a mobile device at the building;

determining, by the one or more computing devices, a subset of the building visible in the target image;

selecting, by the one or more computing devices, one or more objects of the multiple objects that have respective positions in the subset of the building;

generating, by the one or more computing devices, visual enhancement data that includes, for each of the selected objects, at least one visual indicator for that selected object for presentation at the respective position of that selected object, and information indicating the determined one or more utility values associated with that selected object; and

providing, by the one or more computing devices, information to the mobile device, including providing the visual enhancement data for presentation on the mobile device overlaid on the current view from the camera, wherein the visual enhancement data for presentation includes at least one of a color or a type of surface material for one of the selected objects.

16. The computer-implemented method of claim 15 wherein the obtaining of the data for the building includes at least one of:

determining, based at least in part on analyzing visual data of a plurality of images acquired in the multiple rooms of the building, the multiple objects and the respective positions of the multiple objects; or

generating, for each of the selected one or more objects, a textual description of the object, and wherein the generating of the visual enhancement data further includes, for each of the selected one or more objects, the generated textual description for that object.

17. The computer-implemented method of claim 16 wherein the one or more computing devices include the mobile device, and wherein the providing of the information to the mobile device is performed by the mobile device and includes overlaying the visual enhancement data on the target image.

18. The computer-implemented method of claim 16 wherein the one or more computing devices include at least one computing device remote from the building,

wherein the obtaining of the data for the building is performed by the at least one computing device remote from the building and includes performing the determining of the multiple objects and the respective positions of the multiple objects, including:

capturing the plurality of the images in the multiple rooms of the building using one or more mobile image acquisition devices separate from the mobile device; and

performing the analyzing of the visual data of the plurality of images using one or more trained machine learning models; and

wherein the providing of the visual enhancement data includes presenting the target image overlaid with the visual enhancement data including, for each of the selected one or more objects, displaying the at least one visual indicator for that selected object at a determined position in the target image that is based on the respective position for that selected object.

19. The computer-implemented method of claim 16 wherein the obtaining of the data for the building includes the generating of the textual description of each of the selected one or more objects by using one or more language models that are trained to use, as input and for each of the selected one or more objects, information about that selected object and about a room in the building in which that selected object is located, and wherein the one or more language models include at least one of a Vision and Language Model (VLM) that is trained using image/caption tuples, or a Knowledge Enhanced Natural Language Generation (VENLG) model that is trained using one or more defined knowledge sources.

20. The computer-implemented method of claim 16 wherein the obtaining of the data for the building includes analyzing, for a floor plan showing a room layout for the building with at least two-dimensional room shapes and relative positions of the multiple rooms, the floor plan to determine one or more attributes of the building that each corresponds to a characteristic of the room layout, and

wherein the providing of the information to the mobile device at the building further includes selecting one of the attributes corresponding to a characteristic of the room layout to include in the generated visual enhancement data; and

wherein the providing of the visual enhancement data includes displaying overlaid information about the selected one attribute at a position in the current view from the camera that is determined for the selected one attribute.

21. The computer-implemented method of claim 15 wherein the providing of the visual enhancement data includes presenting on the mobile device the visual enhancement data overlaid on the current view from the camera as part of a graphical user interface (GUI) displayed on at least one display of the mobile device, and wherein the providing of the information to the mobile device further includes:

determining, by the mobile device and before the providing of the visual enhancement data, one or more GUI controls for which to include one or more respective user-selectable visual representations in the generated visual enhancement data;

receiving, by the mobile device and after the presenting of the visual enhancement data overlaid on the current view from the camera, a selection by the user of one of the respective user-selectable visual representations for one of the GUI controls; and

updating, by the mobile device and in response to the selection by the user, the at least one display of the mobile device to display at least one type of additional data.

22. The computer-implemented method of claim 15 wherein the obtaining of the data further includes obtaining information about both objective attributes about the building that are able to be independently verified and subjective attributes for the building that are predicted by the one or more trained neural network models, and wherein the providing of the visual enhancement data includes presenting information about at least one objective attribute and at least one subjective attribute.

23. The computer-implemented method of claim 15 wherein the target image shows a portion of a first room of the multiple rooms in which the mobile device is located, and wherein the automated operations further include:

determining additional information to present to a user in a second room of the multiple rooms that is different from the first room;

generating guidance instructions to direct the user from the first room to the second room;

displaying at least some of the generated guidance instructions overlaid on that target image to direct the user to the second room; and

displaying, after the mobile device reaches the second room, at least some of the determined additional information.

24. The computer-implemented method of claim 15 wherein the obtaining of the data further includes obtaining information about one or more comparison attributes that each includes a comparison of at least one type of information about the building to that at least one type of information about one or more other buildings, and wherein the providing of the visual enhancement data includes presenting information that describes at least one of the comparison attributes.

25. The computer-implemented method of claim 15 wherein the providing of the visual enhancement data further includes obtaining information about a user of the mobile device that includes one or more preferences of the user, wherein the selecting of the one or more objects is performed based on the one or more preferences of the user, and wherein the providing of the visual enhancement data includes presenting to the user the visual enhancement data in combination with the current view from the camera.

26. The computer-implemented method of claim 15 wherein the determined subset of the building visible in the target image includes a portion of a first room of the multiple rooms, wherein the target image shows at least some of one or more walls of the first room, and wherein the providing of the visual enhancement data includes:

determining one or more second rooms of the multiple rooms that are behind the at least some of the one or more walls and that are not visible in the target image;

generating visual representations of the determined one or more second rooms for use as part of the generated visual enhancement data; and

presenting, as part of the providing of the visual enhancement data, the generated visual representations of the determined one or more second rooms at locations on the current view from the camera that correspond to positions of the determined one or more second rooms behind the at least some of the one or more walls.

27. The computer-implemented method of claim 15 wherein the providing of the visual enhancement data includes:

determining, for each of the selected one or more objects, one or more utility values associated with the object, each utility value indicating one of a condition of that object or a value associated with that object; and

presenting, as part of the providing of the visual enhancement data, the visual enhancement data overlaid on the current view from the camera, including presenting information indicating the determined one or more utility values associated with each of the selected one or more objects.

28. The computer-implemented method of claim 15 wherein the providing of the visual enhancement data includes presenting, on at least one display of the mobile device, the visual enhancement data overlaid on the target image, and further includes transmitting, to one or more additional devices separate from the mobile device and remote from the building, the enhancement data overlaid on the target image for concurrent presentation on the one or more additional devices.

29. A non-transitory computer-readable medium having stored contents that cause one or more computing devices to perform automated operations, the automated operations including at least:

obtaining, by the one or more computing devices, data for a building with multiple rooms, including information about multiple objects at the building, and including information about respective positions at the building associated with the multiple objects, wherein the multiple objects in the building include at least appliances, fixtures, and structural elements, and wherein the multiple objects in the building further have colors and types of surface materials;

receiving, by the one or more computing devices, a target image showing a current view from a camera of a mobile device at the building;

determining, by the one or more computing devices, a subset of the building visible in the target image;

selecting, by the one or more computing devices, one or more objects of the multiple objects that have respective positions in the subset of the building;

generating, by the one or more computing devices, visual enhancement data that includes, for each of the selected objects, at least one visual indicator for that selected object for presentation at the respective position of that selected object, and information indicating the determined one or more utility values associated with that selected object; and

providing, by the one or more computing devices, information to the mobile device, including providing the visual enhancement data for presentation on the mobile device overlaid on the current view from the camera, wherein the visual enhancement data for presentation includes at least one of a color or a type of surface material for one of the selected objects.

30. The non-transitory computer-readable medium of claim 29 wherein the stored contents include software instructions that, when executed, cause the one or more computing devices to perform the obtaining of the data for the building by performing at least one of:

determining, based at least in part on analyzing visual data of a plurality of images acquired in the multiple rooms of the building, the multiple objects and the respective positions of the multiple objects; or

generating, for each of the selected one or more objects, a textual description of the object, and wherein the generating of the visual enhancement data further includes, for each of the selected one or more objects, the generated textual description for that object.

Assignments (4)

Continuity (1)

References Cited (285)