IP Library › Granted Patent US 12,373,495
Granted Patent B2
US 12,373,495 · App. 18/085,386 · Granted Jul 29, 2025

Active and passive querying of spatial recordings

Inventors: Karthik Mahadevan (Toronto, CA); Tovi Grossman (Toronto, CA); Fraser Anderson (Newmarket, CA); George William Fitzmaurice (Toronto, CA); Qian Zhou (Toronto, CA)
Assignee: Autodesk, Inc.
G06F16/90328G06T19/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,373,495
App. No.
18/085,386
Granted
Jul 29, 2025
Kind
B2
Abstract

Methods, systems, and apparatus, including medium-encoded computer program products, for searching through spatial recordings by constructing queries. One of the methods include constructing a query based on input received at a three-dimensional (3D) space displayed at a visual programming interface, wherein the input includes positioning of objects at 3D positions within the 3D space over time, wherein the input defines at least one spatial orientation between at least two objects from the objects in the 3D space; executing the query to search a database of 3D recordings to find at least one segment from at least one 3D recording that includes the at least two objects and matches the spatial orientation between the at least two objects, as defined in the input, at a specific point in time; and presenting the at least one segment from the at least one 3D recording via the visual programming interface.

Claims (134)

1. A method comprising:

receiving input for a three-dimensional (3D) space displayed at a visual programming interface, the input defining positioning of objects at 3D positions within the 3D space over a period of time, and the input defining at least one spatial orientation between at least two objects from the objects in the 3D space;

constructing a query based on the input including the positioning of the objects at the 3D positions within the 3D space over the period of time and including the at least one spatial orientation between the at least two objects from the objects in the 3D space;

executing the query to search a database of 3D recordings to find at least one segment from at least one 3D recording that includes the at least two objects and matches the spatial orientation between the at least two objects, as defined in the input, at a specific point in time; and

presenting the at least one segment from the at least one 3D recording via the visual programming interface.

2. The method of claim 1 , wherein the receiving comprises:

receiving the input through the visual programming interface as direct user interactions with the objects in the 3D space, wherein the 3D space is a search cube, wherein the direct user interactions are performed with each of the objects to position each object relative to the other object during construction of the query,

wherein the query is constructed based at least on extracting information for the 3D positions of the objects in the search cube.

3. The method of claim 1 , wherein the receiving comprises:

receiving the input as a copy of a 3D scene defined outside of the 3D space in a design environment.

4. The method of claim 1 , wherein the receiving comprises:

obtaining design operations performed by a user in a design space to be mirrored dynamically at the 3D space and to be used as the input for constructing the query.

5. The method of claim 4 , wherein the design operations are evaluated to identify a first viewpoint of the user in the design space to be mapped to a second viewpoint in the 3D space when mirroring the user design operations in the 3D space to construct the query, and

wherein constructing the query comprising constructing the query based on input including event information for objects added in the design 3D space that are mirrored in the 3D space, wherein the event information includes at least one of a spatial relationship between the objects, a proximity of an avatar to at least one of the objects, and event time data identifying when the user added a respective object into the 3D space.

6. The method of claim 1 , wherein the receiving comprises:

receiving the input at the 3D space during a collaborative design process where at least two users interact with respective sets of the objects, wherein each user of the at least two users interact with a respective set of objects from a respective user viewpoint of the 3D space to position and move the respective sets of objects in the 3D space over time;

wherein the query is defined from a viewpoint specified at the visual programming interface for constructing the query.

7. The method of claim 1 , wherein the receiving comprises:

receiving the input as a definition of a 3D scene in the 3D space, wherein the input comprises information for a respective relationship status of at least one of the objects with the 3D scene, wherein the relationship status defines either presence or absence of the object in the 3D scene.

8. The method of claim 1 , wherein constructing the query comprises:

identifying inclusion statuses for each of the objects provided in the 3D space, wherein at least one object has an inclusion status specifying that the at least one object is non-present in the 3D space;

categorizing the at least one spatial orientation between the at least two objects from the objects to an object-to-object relationship according to a predefined categorization mapping rule; and

wherein the query is constructed based on the identified inclusion statuses and object-to object relationships of objects identified with the input, and

wherein executing the query comprises:

filtering the 3D recording in the database to identify the at least one 3D recording that includes objects corresponding to objects provided with the input in the 3D space different from the at least one object that is with the non-present inclusion status, and

filtering the 3D recording in the database to identify the at least one 3D recording to include at least two objects corresponding to the at least two objects from the input, wherein the at least two objects in the at least one 3D recording are identified to match the object-to-object relationship categorizing the at least one spatial orientation between the at least two objects in the 3D space.

9. The method of claim 8 , wherein executing the query comprises:

determining ranks for recordings of a set of 3D recording from the 3D database, wherein the set of 3D recordings match the inclusion statuses for each of the objects provided in the 3D space, wherein each rank of a recording is based on a similarity between the recording and the constructed query.

10. The method of claim 1 , wherein constructing the query comprises:

identifying priority statuses for each of the objects provided in the 3D space to be applied during searching of the 3D recording in the database.

11. The method of claim 1 , wherein the receiving comprises:

receiving the input at the 3D space, wherein the input comprises a desired proximity between respective 3D positions of the at least two objects.

12. The method of claim 1 , comprising:

receiving a subsequent input from a user to select a first segment of a 3D recording from the at least one segment of the at least one 3D recording that comprise the at least two objects defined in the query and match the spatial orientation between the at least two objects as defined in the input;

mirroring the selected first segment into the 3D space of the visual programming interface;

receiving a modification instruction at the 3D space as new input to query the 3D database, wherein the modification instruction is for modifying a position of at least one object comprised in the mirrored first segment in the 3D space; and

constructing a new query using the new input received for the mirrored first segment into the 3D space, wherein the query is constructed based on considering i) spatial orientation and distance between objects from the first segment and ii) absolute positioning of the objects from the first segment in the 3D space; and

executing the new query to identify one or more segments from one or more 3D recordings in the 3D database that match the query according to a threshold similarity level.

13. The method of claim 1 , wherein the 3D space is displayed with a first viewpoint, and the input defines the spatial orientation between the at least two objects from a second viewpoint, the second viewpoint being of an avatar of a user providing the input,

wherein constructing the query comprises converting the spatial orientation between the at least to objects to a new spatial orientation with respect to the first viewpoint of the 3D space, and wherein presenting the at least one segment comprises presenting the at least one segment to match the first viewpoint of the 3D space.

14. The method of claim 13 ,

wherein the input identifies an avatar as a first object defined for the query that is associated with a second viewpoint and a second spatial orientation towards an object of the at least two objects, wherein the second spatial orientation is associated with a different time point in the query compared to a first time point matching the new spatial orientation of the at least two objects.

15. The method of claim 1 , wherein the receiving comprises:

receiving a voice instruction input specifying addition or exclusion of objects in the 3D space of the visual programming interface for constructing the query, wherein the voice instruction input specifies a 3D position of an object in the 3D space that is to be added; and

converting the voice instruction input into a transcription including word phrases that upon processing are used for constructing query statements to be included in the query, the voice instruction input comprising speech that defines an association between an object of the objects.

16. A system comprising:

one or more computers; and

one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising

receiving input for a three-dimensional (3D) space displayed at a visual programming interface, the input defining positioning of objects at 3D positions within the 3D space over a period of time, and the input defining at least one spatial orientation between at least two objects from the objects in the 3D space,

constructing a query based on the input including the positioning of the objects at the 3D positions within the 3D space over the period of time and including the at least one spatial orientation between the at least two objects from the objects in the 3D space,

executing the query to search a database of 3D recordings to find at least one segment from at least one 3D recording that includes the at least two objects and matches the spatial orientation between the at least two objects, as defined in the input, at a specific point in time, and

presenting the at least one segment from the at least one 3D recording via the visual programming interface.

17. The system of claim 16 , wherein the receiving comprises:

receiving the input through the visual programming interface as direct user interactions with the objects in the 3D space, wherein the 3D space is a search cube, wherein the direct user interactions are performed with each of the objects to position each object relative to the other object during construction of the query,

wherein the query is constructed based at least on extracting information for the 3D positions of the objects in the search cube.

18. The system of claim 16 , wherein the receiving comprises:

receiving the input as a copy of a 3D scene defined outside of the 3D space in a design environment.

19. The system of claim 16 , wherein the receiving comprises:

receiving the input as a definition of a 3D scene in the 3D space, wherein the input comprises information for a respective relationship status of at least one of the objects with the 3D scene, wherein the relationship status defines either presence or absence of the object in the 3D scene.

20. The system of claim 16 , wherein the receiving comprises:

obtaining design operations performed by a user in a design space to be mirrored dynamically at the 3D space and to be used as the input for constructing the query.

21. The system of claim 20 , wherein the design operations are evaluated to identify a first viewpoint of the user in the design space to be mapped to a second viewpoint in the 3D space when mirroring the user design operations in the 3D space to construct the query, and

wherein constructing the query comprising constructing the query based on input including event information for objects added in the design 3D space that are mirrored in the 3D space, wherein the event information includes at least one of a spatial relationship between the objects, a proximity of an avatar to at least one of the objects, and event time data identifying when the user added a respective object into the 3D space.

22. The system of claim 16 , wherein the receiving comprises:

receiving the input at the 3D space during a collaborative design process where at least two users interact with respective sets of the objects, wherein each user of the at least two users interact with a respective set of objects from a respective user viewpoint of the 3D space to position and move the respective sets of objects in the 3D space over time;

wherein the query is defined from a viewpoint specified at the visual programming interface for constructing the query.

23. The system of claim 16 , wherein constructing the query comprises:

identifying inclusion statuses for each of the objects provided in the 3D space, wherein at least one object has an inclusion status specifying that the at least one object is non-present in the 3D space;

categorizing the at least one spatial orientation between the at least two objects from the objects to an object-to-object relationship according to a predefined categorization mapping rule; and

wherein the query is constructed based on the identified inclusion statuses and object-to object relationships of objects identified with the input, and wherein executing the query comprises:

filtering the 3D recording in the database to identify the at least one 3D recording that includes objects corresponding to objects provided with the input in the 3D space different from the at least one object that is with the non-present inclusion status, and

filtering the 3D recording in the database to identify the at least one 3D recording to include at least two objects corresponding to the at least two objects from the input, wherein the at least two objects in the at least one 3D recording are identified to match the object-to- object relationship categorizing the at least one spatial orientation between the at least two objects in the 3D space.

24. The system of claim 23 , wherein executing the query comprises:

determining ranks for recordings of a set of 3D recording from the 3D database, wherein the set of 3D recordings match the inclusion statuses for each of the objects provided in the 3D space, wherein each rank of a recording is based on a similarity between the recording and the constructed query.

25. The system of claim 16 , wherein constructing the query comprises:

identifying priority statuses for each of the objects provided in the 3D space to be applied during searching of the 3D recording in the database.

26. The system of claim 16 , wherein the receiving comprises:

receiving the input at the 3D space, wherein the input comprises a desired proximity between respective 3D positions of the at least two objects.

27. The system of claim 16 , wherein the operations comprise:

receiving a subsequent input from a user to select a first segment of a 3D recording from the at least one segment of the at least one 3D recording that comprise the at least two objects defined in the query and match the spatial orientation between the at least two objects as defined in the input;

mirroring the selected first segment into the 3D space of the visual programming interface;

receiving a modification instruction at the 3D space as new input to query the 3D database, wherein the modification instruction is for modifying a position of at least one object comprised in the mirrored first segment in the 3D space; and

constructing a new query using the new input received for the mirrored first segment into the 3D space, wherein the query is constructed based on considering i) spatial orientation and distance between objects from the first segment and ii) absolute positioning of the objects from the first segment in the 3D space; and

executing the new query to identify one or more segments from one or more 3D recordings in the 3D database that match the query according to a threshold similarity level.

28. The system of claim 16 , wherein the 3D space is displayed with a first viewpoint, and the input defines the spatial orientation between the at least two objects from a second viewpoint, the second viewpoint being of an avatar of a user providing the input,

wherein constructing the query comprises converting the spatial orientation between the at least to objects to a new spatial orientation with respect to the first viewpoint of the 3D space, and wherein presenting the at least one segment comprises presenting the at least one segment to match the first viewpoint of the 3D space.

29. The system of claim 28 , wherein the input identifies an avatar as a first object defined for the query that is associated with a second viewpoint and a second spatial orientation towards an object of the at least two objects, wherein the second spatial orientation is associated with a different time point in the query compared to a first time point matching the new spatial orientation of the at least two objects.

30. The system of claim 16 , wherein the receiving comprises:

receiving a voice instruction input specifying addition or exclusion of objects in the 3D space of the visual programming interface for constructing the query, wherein the voice instruction input specifies a 3D position of an object in the 3D space that is to be added; and

converting the voice instruction input into a transcription including word phrases that upon processing are used for constructing query statements to be included in the query, the voice instruction input comprising speech that defines an association between an object of the objects.

31. A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving input for a three-dimensional (3D) space displayed at a visual programming interface, the input defining positioning of objects at 3D positions within the 3D space over a period of time, and the input defining at least one spatial orientation between at least two objects from the objects in the 3D space;

constructing a query based on the input including the positioning of the objects at the 3D positions within the 3D space over the period of time and including the at least one spatial orientation between the at least two objects from the objects in the 3D space;

executing the query to search a database of 3D recordings to find at least one segment from at least one 3D recording that includes the at least two objects and matches the spatial orientation between the at least two objects, as defined in the input, at a specific point in time; and

presenting the at least one segment from the at least one 3D recording via the visual programming interface.

32. The non-transitory computer-readable storage medium of claim 31 , wherein the receiving comprises:

receiving the input through the visual programming interface as direct user interactions with the objects in the 3D space, wherein the 3D space is a search cube, wherein the direct user interactions are performed with each of the objects to position each object relative to the other object during construction of the query,

wherein the query is constructed based at least on extracting information for the 3D positions of the objects in the search cube.

33. The non-transitory computer-readable storage medium of claim 31 , wherein the receiving comprises:

receiving the input as a copy of a 3D scene defined outside of the 3D space in a design environment.

34. The non-transitory computer-readable storage medium of claim 31 , wherein the receiving comprises:

obtaining design operations performed by a user in a design space to be mirrored dynamically at the 3D space and to be used as the input for constructing the query.

35. The non-transitory computer-readable storage medium of claim 34 , wherein the design operations are evaluated to identify a first viewpoint of the user in the design space to be mapped to a second viewpoint in the 3D space when mirroring the user design operations in the 3D space to construct the query, and

wherein constructing the query comprising constructing the query based on input including event information for objects added in the design 3D space that are mirrored in the 3D space, wherein the event information includes at least one of a spatial relationship between the objects, a proximity of an avatar to at least one of the objects, and event time data identifying when the user added a respective object into the 3D space.

36. The non-transitory computer-readable storage medium of claim 31 , wherein the receiving comprises:

receiving the input at the 3D space during a collaborative design process where at least two users interact with respective sets of the objects, wherein each user of the at least two users interact with a respective set of objects from a respective user viewpoint of the 3D space to position and move the respective sets of objects in the 3D space over time;

wherein the query is defined from a viewpoint specified at the visual programming interface for constructing the query.

37. The non-transitory computer-readable storage medium of claim 31 , wherein the receiving comprises:

receiving the input as a definition of a 3D scene in the 3D space, wherein the input comprises information for a respective relationship status of at least one of the objects with the 3D scene, wherein the relationship status defines either presence or absence of the object in the 3D scene.

38. The non-transitory computer-readable storage medium of claim 31 , wherein constructing the query comprises:

identifying inclusion statuses for each of the objects provided in the 3D space, wherein at least one object has an inclusion status specifying that the at least one object is non-present in the 3D space;

categorizing the at least one spatial orientation between the at least two objects from the objects to an object-to-object relationship according to a predefined categorization mapping rule; and

wherein the query is constructed based on the identified inclusion statuses and object-to object relationships of objects identified with the input, and

wherein executing the query comprises:

filtering the 3D recording in the database to identify the at least one 3D recording that includes objects corresponding to objects provided with the input in the 3D space different from the at least one object that is with the non-present inclusion status, and

filtering the 3D recording in the database to identify the at least one 3D recording to include at least two objects corresponding to the at least two objects from the input, wherein the at least two objects in the at least one 3D recording are identified to match the object-to- object relationship categorizing the at least one spatial orientation between the at least two objects in the 3D space.

39. The non-transitory computer-readable storage medium of claim 38 , wherein executing the query comprises:

determining ranks for recordings of a set of 3D recording from the 3D database, wherein the set of 3D recordings match the inclusion statuses for each of the objects provided in the 3D space, wherein each rank of a recording is based on a similarity between the recording and the constructed query.

40. The non-transitory computer-readable storage medium of claim 31 , wherein constructing the query comprises:

identifying priority statuses for each of the objects provided in the 3D space to be applied during searching of the 3D recording in the database.

41. The non-transitory computer-readable storage medium of claim 31 , wherein the receiving comprises:

receiving the input at the 3D space, wherein the input comprises a desired proximity between respective 3D positions of the at least two objects.

42. The non-transitory computer-readable storage medium of claim 31 , wherein the operations comprise:

receiving a subsequent input from a user to select a first segment of a 3D recording from the at least one segment of the at least one 3D recording that comprise the at least two objects defined in the query and match the spatial orientation between the at least two objects as defined in the input;

mirroring the selected first segment into the 3D space of the visual programming interface;

receiving a modification instruction at the 3D space as new input to query the 3D database, wherein the modification instruction is for modifying a position of at least one object comprised in the mirrored first segment in the 3D space; and

constructing a new query using the new input received for the mirrored first segment into the 3D space, wherein the query is constructed based on considering i) spatial orientation and distance between objects from the first segment and ii) absolute positioning of the objects from the first segment in the 3D space; and

executing the new query to identify one or more segments from one or more 3D recordings in the 3D database that match the query according to a threshold similarity level.

43. The non-transitory computer-readable storage medium of claim 31 , wherein the 3D space is displayed with a first viewpoint, and the input defines the spatial orientation between the at least two objects from a second viewpoint, the second viewpoint being of an avatar of a user providing the input,

wherein constructing the query comprises converting the spatial orientation between the at least to objects to a new spatial orientation with respect to the first viewpoint of the 3D space, and wherein presenting the at least one segment comprises presenting the at least one segment to match the first viewpoint of the 3D space.

44. The non-transitory computer-readable storage medium of claim 43 , wherein the input identifies an avatar as a first object defined for the query that is associated with a second viewpoint and a second spatial orientation towards an object of the at least two objects, wherein the second spatial orientation is associated with a different time point in the query compared to a first time point matching the new spatial orientation of the at least two objects.

45. The non-transitory computer-readable storage medium of claim 31 , wherein the receiving comprises:

receiving a voice instruction input specifying addition or exclusion of objects in the 3D space of the visual programming interface for constructing the query, wherein the voice instruction input specifies a 3D position of an object in the 3D space that is to be added; and

converting the voice instruction input into a transcription including word phrases that upon processing are used for constructing query statements to be included in the query, the voice instruction input comprising speech that defines an association between an object of the objects.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 4, 2023
From: MAHADEVAN, KARTHIK; GROSSMAN, TOVI; ANDERSON, FRASER; FITZMAURICE, GEORGE WILLIAM; ZHOU, QIAN
To: AUTODESK, INC.
Reel/Frame 063542/0034 →
Continuity (1)
Related Publication 20240202245A1 · Jun 20, 2024
References Cited (66)
US 11836869B1 · Alberico · 2023 [cited by examiner]
US 20070070066A1 · Bakhash · 2007 [cited by examiner]
US 20080238916A1 · Ghosh · 2008 [cited by examiner]
US 20090096790A1 · Wiedemann · 2009 [cited by examiner]
US 20100085358A1 · Wegbreit · 2010 [cited by examiner]
US 20210123717A1 · Wierda · 2021 [cited by examiner]
US 20230128422A1 · Li · 2023 [cited by examiner]
amazon.com [online], “Amazon Transcribe,” available on or before Mar. 2, 2022, via Internet Archive: Wayback Machine URL <http://web.archive.org/web/20220302023111/https://aws.amazon.com/transcribe/>, retrieved on Nov. … [cited by applicant]
Arkio.is [online], “Arkio,” available on or before Apr. 6, 2022, via Internet Archive: Wayback Machine URL <http://web.archive.org/web/20220406224358/https://www.arkio.is/>, retrieved on Nov. 16, 2022, URL <https://arki… [cited by applicant]
Bergman et al., “DocWizards: a system for authoring follow-me documentation wizards,” Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, Oct. 2005, pp. 191-200. [cited by applicant]
Brudy et al., “EagleView: A Video Analysis Tool for Visualising and Querying Spatial Interactions of People and Devices,” Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces, Tokyo, J… [cited by applicant]
Büschel et al., “Miria: A Mixed Reality Toolkit for the In-Situ Visualization and Analysis of Spatio-Temporal Interaction Data,” Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Ja… [cited by applicant]
Bustos et al., “Feature-based similarity search in 3D object databases,” ACM Computing Surveys, Dec. 2005, 37(4):345-387. [cited by applicant]
Chen et al., “Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings,” arXiv:1803.08495v1, submitted on Mar. 22, 2018, 35 pages. [cited by applicant]
Chow et al., “Challenges and Design Considerations for Multimodal Asynchronous Collaboration in VR,” Proceedings of the ACM on Human-Computer Interaction, Nov. 2019, 3(CSCW):40, 24 pages. [cited by applicant]
Danyluk et al., “A Design Space Exploration of Worlds in Miniature,” Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan, May 8-13, 2021; ACM, 2021, 21 pages. [cited by applicant]
Dragicevic et al., “Video browsing by direct manipulation,” CHI '08: CHI Conference on Human Factors in Computing System, Florence, Italy, Apr. 5-10, 2008; Proceedings of the SIGCHI Conference on Human Factors in Comput… [cited by applicant]
Fisher et al., “Characterizing structural relationships in scenes using graph kernels,” SIGGRAPH '11: Special Interest Group on Computer Graphics and Interactive Techniques Conference, Vancouver, British Columbia, Canad… [cited by applicant]
Fisher et al., “Example-based synthesis of 3D object arrangements,” ACM Transactions on Graphics, Nov. 2012, 31(6), 11 pages. [cited by applicant]
Funkhouser et al., “A search engine for 3D models,” ACM Transactions on Graphics, Jan. 2003, 22(1):83-105. [cited by applicant]
Giunchi et al., “3D sketching for interactive model retrieval in virtual reality,” Expressive '18: Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorea… [cited by applicant]
Goldfeder et al., “Autotagging to improve text search for 3D models,” Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries, Jun. 2008, 355-358. [cited by applicant]
Grabner et al., “3D Pose Estimation and 3D Model Retrieval for Objects in the Wild,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, Jun. 2018, 3022-3031. [cited by applicant]
Grabner et al., “Location Field Descriptors: Single Image 3D Model Retrieval in the Wild,” arXiv:1908.02853v1, submitted Aug. 7, 2019, 18 pages. [cited by applicant]
Grossman et al., “Chronicle: capture, exploration, and playback of document workflow histories,” UIST '10: Proceedings of the 23rd annual ACM symposium on User interface software and technology, Oct. 2010, pp. 143-152. [cited by applicant]
Guimbretiére et al., “ExperiScope: an analysis tool for interaction data,” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, San Jose, California, USA, Apr. 2007, 1333-1342. [cited by applicant]
Holz et al., “Data miming: inferring spatial object descriptions from human gesture,” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, Canada, May 2011, 811-820. [cited by applicant]
Hubenschmid, “ReLive: Bridging In-Situ and Ex-Situ Visual Analytics for Analyzing Mixed Reality User Studies,” Proceedings of the 2022 CHI Conference on Human Factors in Computing System, New Orleans, LA, USA, Apr. 29,-… [cited by applicant]
Igarashi et al., “Teddy: a sketching interface for 3D freeform design,” SIGGRAPH06: Special Interest Group on Computer Graphics and Interactive Techniques Conference, Boston, MA, USA, Jul. 30,-Aug. 3, 2006; ACM SIGGRAPH… [cited by applicant]
Karrer et al., “Dragon: a direct manipulation interface for frame-accurate in-scene video navigation,” CHI '08: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Apr. 2008, pp. 247-250. [cited by applicant]
Kloiber et al., “Immersive analysis of user motion in VR applications,” The Visual Computer, Oct. 2020, 36:1937-1949. [cited by applicant]
Krauss et al., “Nonverbal Behavior and Nonverbal Communication: What do Conversational Hand Gestures Tell Us?” Advances in Experimental Social Psychology, Elsevier, 1996, 389-450. [cited by applicant]
Krosnick et al., “Think-Aloud Computing: Supporting Rich and Low-Effort Knowledge Capture,” Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan, May 2021, 14 pages. [cited by applicant]
Lee et al., “An Empirical Study of How Socio-Spatial Formations are Influenced by Interior Elements and Displays in an Office Context,” Proceedings of the ACM on Human-Computer Interaction, Nov. 2019, 3(CSCW):58, 26 pag… [cited by applicant]
Lilija et al., “Who Put That There? Temporal Navigation of Spatial Recordings by Direct Manipulation,” Paper 477, 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, Apr. 25-30, 2020; CHI '20: … [cited by applicant]
Lobo et al., “Flex-ER: A Platform to Evaluate Interaction Techniques for Immersive Visualizations,” Proceedings of the ACM on Human-Computer Interaction, Nov. 2020, 4(ISS):1-20. [cited by applicant]
Lun et al., “3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks,”arXiv:1707.06375v3, revised Sep. 29, 2017, 11 pages. [cited by applicant]
Luo et al., “Towards 3D VR-Sketch to 3D Shape Retrieval,” 2020 International Conference on 3D Vision (3DV), Fukuoka, Japan, Nov. 2020, 81-90. [cited by applicant]
Matejka et al., “Ambient help,” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, May 2011, pp. 2751-2760. [cited by applicant]
Merrell et al., “Interactive furniture layout using interior design guidelines,” SIGGRAPH '11: Special Interest Group on Computer Graphics and Interactive Techniques Conference, Vancouver, British Columbia, Canada, Aug.… [cited by applicant]
Mulloni et al., “Indoor navigation with mixed reality world-in-miniature views and sparse localization on mobile devices,” Proceedings of the International Working Conference on Advanced Visual Interfaces—AVI '12, Capri… [cited by applicant]
Nam et al., “Worlds-in-Wedges: Combining Worlds-in-Miniature and Portals to Support Comparative Immersive Visualization of Forestry Data,” 26th IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2019—Proceedi… [cited by applicant]
Nebeling et al., “MRAT: The Mixed Reality Analytics Toolkit,” Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, Apr. 2020, 12 pages. [cited by applicant]
Nguyen et al., “Direct manipulation video navigation in 3D,” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France, Apr. 2013, pp. 1169-1172. [cited by applicant]
nkba.org [online], “NKBA Kitchen & Bathroom Planning Guidelines Handbook,” available on or before Dec. 13, 2021, via Internet Archive: Wayback Machine URL <http://web.archive.org/web/20211213204024/https://store.nkba.or… [cited by applicant]
Ohbuchi et al., “Shape-similarity search of 3D models by using enhanced shape functions,” International Journal of Computer Applications in Technology, Jan. 2005, 23(2-4):70-85. [cited by applicant]
Pausch et al., “Navigation and locomotion in virtual worlds via flight into hand-held miniatures,” Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques—SIGGRAPH '95, Sep. 1995, pp. 3… [cited by applicant]
Pu et al., “A 2D Sketch-Based User Interface for 3D CAD Model Retrieval,” Computer-Aided Design and Applications, Jan. 2005, 2(6):717-725. [cited by applicant]
reddit.com [online], “Reddit—Dive into anything,” available on or before Apr. 7, 2022, via Internet Archive: Wayback Machine URL <http://web.archive.org/web/20220407064911/https://www.reddit.com/>, retrieved on Nov. 17,… [cited by applicant]
Reimers et al., “Sentence-BERT: Sentence Embeddings using Siamese BERT—Networks,” arXiv:1908.10084, submitted Aug. 27, 2019, 11 pages. [cited by applicant]
Savva et al., “PiGraphs: learning interaction snapshots from observations,” ACM Transactions on Graphics, Jul. 2016, 35(4):139, 12 pages. [cited by applicant]
Savva et al., “SceneGrok: inferring action maps in 3D environments,” ACM Transactions on Graphics, Nov. 2014, 33(6):1-10. [cited by applicant]
secondspectrum.com [online], “Second Spectrum,” available on or before Feb. 12, 2022, via Internet Archive: Wayback Machine URL <http://web.archive.org/web/20220212025924/https://www.secondspectrum.com/index.html>, retr… [cited by applicant]
Spatial.io [online], “Spatial—Metaverse Spaces That Bring Us Together,” available on or before Dec. 16, 2021, via Internet Archive: Wayback Machine URL <http://web.archive.org/web/20211216035157/https://spatial.io/>, re… [cited by applicant]
Stemasov et al., “ShapeFindAR: Exploring In-Situ Spatial Search for Physical Artifact Retrieval using Mixed Reality,” CHI Conference on Human Factors in Computing Systems (CHI '22), New Orleans, LA, USA, Apr. 29-May 5, … [cited by applicant]
Stoakley et al., “Virtual reality on a WIM: interactive worlds in miniature,” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems—CHI '95, Denver, Colorado, United States, May 1995, 265-272. [cited by applicant]
strivr.com [online], “Strivr | Enterprise Virtual Reality Training Solutions,” available on or before Dec. 19, 2021, via Internet Archive: Wayback Machine URL <http://web.archive.org/web/20211219225015/https://www.striv… [cited by applicant]
thewild.com [online], “The Wild—VR Collaboration for Architecture & Design Teams,” available on or before Dec. 7, 2021, via Internet Archive: Wayback Machine URL <http://web.archive.org/web/20211207031120/https://thewil… [cited by applicant]
virti.com [online], “Virti | Improving Human Performance,” available on or before Feb. 16, 2022, via Internet Archive: Wayback Machine URL <http://web.archive.org/web/20220216020855/https://www.virti.com/>, retrieved on… [cited by applicant]
vrchat.com [online], “VRChat,” available on or before Mar. 2, 2022, via Internet Archive: Wayback Machine URL <http://web.archive.org/web/20220302075700/https://hello.vrchat.com/>, retrieved on Nov. 17, 2022, URL <https… [cited by applicant]
Walkerdine et al., “WorldFinder: A tool for finding Virtual Worlds,” Working Paper (Unpublished), 2000, 14 pages. [cited by applicant]
Wang et al. “Again, Together: Socially Reliving Virtual Reality Experiences When Separated,” Paper 513, Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, Apr. 2020, pp. 1-1… [cited by applicant]
Wang et al., “Sketch-based 3D Shape Retrieval using Convolutional Neural Networks,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, Jun. 7-12, 2015; arXiv:1504.03504v1, submitted… [cited by applicant]
Xia et al., “Spacetime: Enabling Fluid Individual and Collaborative Editing in Virtual Reality,” Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, Oct. 2018, pp. 853-866. [cited by applicant]
xrdr.org [online], “XRDRN—XR Distributed Research Network,” available on or before Dec. 5, 2021, via Internet Archive: Wayback Machine URL <http://web.archive.org/web/20211205161223/http://www.xrdr.org/>, retrieved on N… [cited by applicant]
Yoon et al., “Sketch-based 3D model retrieval using diffusion tensor fields of suggestive contours,” Proceedings of the International Conference on Multimedia—MM '10, Firenze, Italy, Oct. 25-29, 2010; ACM, 2010, pp. 193… [cited by applicant]
Cited By (1)
US 12,633,049