Granted Patent B2

US 12,658,186 · App. 17/916,986 · Granted Jun 16, 2026

Inventors: Tanya Makker (Milpitas, CA); Dhairya Shrivastava (Los Altos, CA); Mark David Mendenhall (Fremont, CA); Stephen Clark Brown (San Mateo, CA); Nitesh Trikha (Pleasanton, CA); Anurag Gupta (San Jose, CA); Ajay Malik (Milpitas, CA); Siyao Sui (San Jose, CA); Chuqing Wang (San Jose, CA); Rao P. Mulpuri (Saratoga, CA)

Assignee: View Operating Corporation

G10L15/22E06B3/6722E06B9/24G02F1/163G06F3/04883G06F3/04886G06F3/16G06F3/167G06F21/32G10L15/26E06B2009/2464G06F2203/0381G10L2015/223

Abstract

Claims (37)

1 . A method for controlling an interactive target of a facility, the method comprising:

monitoring a location of a mobile circuitry relative to a digital twin that comprises a virtual three dimensional representation of a structural feature of the facility having a real interactive target, wherein the mobile circuitry (I) is movable by a user, (II) has a known location relative to at least a portion of the structural feature, and (III) is coupled to a virtual representation of the real interactive target in the digital twin;

relating a gesture imparted on the mobile circuitry to the digital twin and generating a result, wherein the gesture (i) is imparted by the user, (ii) is signifies a remote input to the real interactive target to cause an alteration of the real interactive target, and (iii) is during coupling with the real interactive target, wherein generating the result comprises determining an action by the virtual representation of the real interactive target that would result from the gesture; and

using the result to alter a current state of the real interactive target in the facility, based at least in part on the determined action.

2 . The method of claim 1 , wherein the facility comprises a control network that is communicatively coupled to the real interactive target to support monitoring location of the mobile circuitry, and/or to support altering the current state of the real interactive target.

3 . The method of claim 1 , wherein the mobile circuitry:

is included in a handheld pointing device, in a mobile phone, in a handheld gaming controller having a motion function and a clicking/select function, in a laptop computer, in a tablet computer, in a virtual reality (V/R) interface that comprises a display headset, a handheld controller with a motion function, or a select function;

includes, or is coupled to, a motion sensor; and/or

is not using an electromagnetic beam or a sonic beam.

4 . The method of claim 1 , wherein the gesture comprises movement, and wherein the location relative to the structural feature of the facility is established at a first time, and the relative location is maintained as the mobile circuitry moves locally in the facility or remotely from the facility.

5 . The method of claim 1 , wherein coupling of the mobile circuitry to the virtual representation of the real interactive target in the digital twin is comprised of (i) a spatial relationship between the mobile circuitry and the at least the portion of the structural feature identified in at least two dimensions and (ii) a relative pointing direction of the mobile circuitry to the real interactive target.

6 . The method of claim 1 , wherein the current state being altered comprises a command setting of an environmental control unit.

7 . The method of claim 6 , wherein the command setting comprises a tint density of a tintable window, a temperature setting of an HVAC unit, a fan setting of an HVAC unit, or an on/off setting of a lighting unit.

8 . The method of claim 6 , wherein the environmental control unit controls an environment of the facility.

9 . The method of claim 1 , further comprising exchanging one or more messages between the digital twin and the mobile circuitry to provide an analysis to the mobile circuitry corresponding to an initial virtual location, and navigate virtually in the digital twin to interact with the virtual representation of the real interactive target in the digital twin.

10 . The method of claim 9 , wherein the user manipulating the mobile circuitry is located distant from the known location, and wherein the initial virtual location is aligned to a virtual representation of the known location in the digital twin.

11 . The method of claim 1 , wherein the digital twin comprises virtual three dimensional representations of a plurality of structural features of the facility including a plurality of real interactive targets, and wherein the virtual three dimensional representations are modified in response to addition and/or subtraction of the real interactive targets in the facility.

12 . The method of claim 1 , wherein the real interactive target comprises a media display, wherein coupling of the mobile circuitry to the virtual representation of the real interactive target in the digital twin includes a selection of an active media element in the media display.

13 . The method of claim 12 , wherein selection of the active media element comprises a pointing motion, a movement motion, or a clicking action.

14 . An apparatus for controlling an interactive target of a facility, the apparatus comprising one or more controllers comprising circuitry, wherein the one or more controllers are configured to:

communicatively couple to a digital twin that comprises a virtual three dimensional representation of a structural feature of the facility having a real interactive target, and to a mobile circuitry that (I) is movable by a user, (II) has a known location relative to at least a portion of the structural feature, and (III) is coupled to a virtual representation of the real interactive target in the digital twin;

monitor, or direct monitoring of, a location of the mobile circuitry relative to the digital twin;

relate, or direct relating, a gesture imparted on the mobile circuitry to the digital twin and generate a result, wherein the gesture (i) is imparted by the user, (ii) signifies a remote input to the real interactive target to cause an alteration of the real interactive target, and (iii) is during user coupling with the real interactive target, wherein generating the result comprises determining an action by the virtual representation of the real interactive target that would result from the gesture; and

use the result to alter, or direct alteration of, a current state of the real interactive target in the facility, based at least in part on the determined action.

15 . The apparatus of claim 14 , wherein:

the one or more controllers are interconnected in a network disposed in the facility, and

the network is disposed at least in part in an envelope of the facility, in an electrical shaft, communication shaft, elevator shaft, and/or in an electrical room.

16 . The apparatus of claim 14 , wherein the real interactive target comprises a media display, wherein the digital twin is configured to identify a selection of an active media element in the media display.

17 . The apparatus of claim 14 , wherein the one or more controllers are configured to alter the current state of a command setting of an environmental control unit that is configured to control an environment of the facility.

18 . The apparatus of claim 17 , wherein the command state is a (i) a tint density of a tintable window, (ii) a temperature setting of an HVAC unit, (iii) a fan setting of an HVAC unit, or (iv) an on/off setting of a lighting unit.

19 . The apparatus of claim 14 , wherein the mobile circuitry is included in a handheld pointing device, in a mobile phone, in a handheld gaming controller having a motion function and a clicking/select function, in a laptop computer, in a tablet computer, in a virtual reality (VR) interface that comprises a display headset, a handheld controller with a motion function, or a select function, a handheld controller with a motion function, or a select function, the mobile circuitry includes, or is coupled to, a motion sensor, or the mobile circuitry is not using an electromagnetic beam or a sonic beam.

20 . A method for controlling a real interactive target of a facility, the method comprising:

monitoring a location of a mobile circuitry relative to a digital twin that comprises a virtual three dimensional representation of a structural feature of the facility having the real interactive target, wherein the mobile circuitry (I) is movable by a user, (II) has a known location relative to at least a portion of the structural feature, and (III) is coupled to a virtual representation of the real interactive target in the digital twin;

relating a gesture imparted on the mobile circuitry to the digital twin and generating a result, wherein the gesture (i) is imparted by the user, (ii) is representative of an indication to remotely cause an alteration of the real interactive target, and (iii) is during coupling with the real interactive target; and

using the result to alter a current state of the real interactive target in the facility,

wherein the current state being altered comprises a tint of an optically tintable window.

21 . The method of claim 20 , wherein the optically tintable window comprises an electrochromic window.

Assignments (3)

Continuity (40)

References Cited (339)