Granted Patent B2

US 12,730,496 · App. 18/799,959 · Granted Sep 8, 2026

Inventors: Youjean Cho (New York, NY); Lei Zhang (Ann Arbor, MI); Daekun Kim (Santa Monica, CA); Ava Robinson (Solana Beach, CA); Yu Jiang Tham (Seattle, WA); Rajan Vaish (Beverly Hills, CA); Andrés Monroy-Hernández (Seattle, WA)

Assignee: Snap Inc.

G06F3/011G06F3/017G06F3/0334G06V10/255G06V40/20G16Y20/40H04N13/111

Abstract

Claims (78)

1 . A method of controlling an Internet of Things (IoT) product using an electronic eyewear device comprising a camera and a display, the method comprising:

capturing frames of video data using the camera;

detecting the IoT product at an IoT product location in a physical environment based on the frames of video data;

estimating a current eyewear device location relative to the IoT product location;

detecting a foot shape at a foot location relative to the physical environment based on the frames of video data;

establishing a virtual plane relative to the display based on the foot location and the current eyewear device location;

presenting on the display a virtual control interface along the virtual plane according to the IoT product location;

detecting the foot shape performing a foot action relative to the virtual control interface; and

sending a control signal to the IoT product in accordance with the foot action, wherein the control signal comprises instructions associated with an IoT action.

2 . The method of claim 1 , wherein presenting the virtual control interface comprises presenting a virtual surface on the display at a surface position, such that the virtual surface appears to support the virtual control interface, and

wherein detecting the foot shape comprises detecting an intersection between the foot location and the virtual surface.

3 . The method of claim 1 , further comprising:

pairing the IoT product with the electronic eyewear device using a product API, wherein the product API comprises a product type and a controllable feature;

associating the foot action with the controllable feature; and

adjusting the controllable feature in accordance with the foot action and the control signal.

4 . The method of claim 1 , wherein the IoT product is associated with a controllable feature comprising a set of selectable options,

wherein presenting the virtual control interface comprises presenting along the virtual control interface a plurality of indicia, each associated with an option from the set of selectable options, and

wherein sending the control signal comprises generating an instruction based on the foot action relative to at least one of the plurality of indicia.

5 . The method of claim 1 , wherein establishing the virtual plane comprises:

establishing the virtual plane at a background location relative to the foot location, such that the foot location lies between the virtual plane and the current eyewear device location.

6 . The method of claim 1 , wherein the physical environment is characterized by a reference plane, and wherein the method comprises:

establishing the virtual plane at a foreground location relative to the reference plane, such that the virtual plane lies between the reference plane and the current eyewear device location.

7 . The method of claim 1 , wherein the IoT product is associated with a plurality of controllable features, and wherein presenting the virtual control interface comprises:

presenting along the virtual control interface a plurality of other virtual surfaces;

associating each of the other virtual surfaces with each of the plurality of controllable features; and

generating an instruction based on the foot action relative to at least one of the plurality of virtual surfaces.

8 . The method of claim 1 , wherein detecting the foot shape performing the foot action comprises:

detecting a foot motion relative to the virtual control interface, wherein the foot motion is selected from a group consisting of touching, hovering above, tapping and lifting, tapping and holding, sliding and stopping, and sliding.

9 . A product control system, comprising:

an electronic eyewear device comprising a processor, a memory, a camera, and a display;

programming in the memory, wherein execution of the programming by the processor configures the electronic eyewear device to perform functions, including functions to:

capture frames of video data using the camera;

detect an Internet of Things (IoT) product at an IoT product location in a physical environment based on the frames of video data;

estimate a current eyewear device location relative to the IoT product location;

detect a foot shape at a foot location relative to the physical environment based on the frames of video data;

establish a virtual plane relative to the display based on the foot location and the current eyewear device location;

present on the display a virtual control interface along the virtual plane according to the IoT product location;

detect the foot shape performing a foot action relative to the virtual control interface; and

send a control signal to the IoT product in accordance with the foot action, wherein the control signal comprises instructions associated with an IoT action.

10 . The product control system of claim 9 , wherein the execution of the programming by the processor configures the electronic eyewear device to perform functions, including functions to:

present a virtual surface on the display at a surface position, such that the virtual surface appears to support the virtual control interface; and

detect an intersection between the foot location and the virtual surface.

11 . The product control system of claim 9 , wherein the execution of the programming by the processor configures the electronic eyewear device to perform functions, including functions to:

pair the IoT product with the electronic eyewear device using a product API, wherein the product API comprises a product type and a controllable feature;

associate the foot action with the controllable feature; and

adjust the controllable feature in accordance with the foot action and the control signal.

12 . The product control system of claim 9 , wherein the IoT product is associated with a controllable feature comprising a set of selectable options, and wherein the execution of the programming by the processor configures the electronic eyewear device to perform functions, including functions to:

present along the virtual control interface a plurality of indicia, each associated with an option from the set of selectable options; and

generate an instruction based on the foot action relative to at least one of the plurality of indicia.

13 . The product control system of claim 9 , wherein the execution of the programming by the processor configures the electronic eyewear device to perform functions, including functions to:

establish the virtual plane at a background location relative to the foot location, such that the foot location lies between the virtual plane and the current eyewear device location.

14 . The product control system of claim 9 , wherein the execution of the programming by the processor configures the electronic eyewear device to perform functions, including functions to:

establish the virtual plane at a foreground location relative to a reference plane associated with the physical environment, such that the virtual plane lies between the reference plane and the current eyewear device location.

15 . The product control system of claim 9 , wherein the IoT product is associated with a plurality of controllable features, and wherein the execution of the programming by the processor configures the electronic eyewear device to perform functions, including functions to:

present along the virtual control interface a plurality of other virtual surfaces;

associate each of the other virtual surfaces with each of the plurality of controllable features; and

generate an instruction based on the foot action relative to at least one of the plurality of virtual surfaces.

16 . The product control system of claim 9 , wherein the foot action comprises a foot motion relative to the virtual control interface, wherein the foot motion is selected from a group consisting of touching, hovering above, tapping and lifting, tapping and holding, sliding and stopping, and sliding.

17 . A non-transitory computer-readable medium storing program code which, when executed, is operative to cause an electronic processor to perform the steps of:

capturing frames of video data using a camera, wherein the camera is coupled to an electronic eyewear device comprising the electronic processor, a memory, and a display;

detect an Internet of Things (IoT) product at an IoT product location in a physical environment based on the frames of video data;

estimating a current eyewear device location relative to the IoT product location;

detecting a foot shape at a foot location relative to the physical environment based on the frames of video data;

establishing a virtual plane relative to the display based on the foot location and the current eyewear device location;

presenting on the display a virtual control interface along the virtual plane according to the IoT product location;

detecting the foot shape performing a foot action relative to the virtual control interface; and

sending a control signal to the IoT product in accordance with the foot action, wherein the control signal comprises instructions associated with an IoT action.

18 . The non-transitory computer-readable medium storing program code of claim 17 , wherein the program code, when executed, is operative to cause the electronic processor to perform the further steps of:

presenting a virtual surface on the display at a surface position, such that the virtual surface appears to support the virtual control interface, wherein detecting the foot shape comprises detecting an intersection between the foot location and the virtual surface;

establishing the virtual plane at a background location relative to the foot location, such that the foot location lies between the virtual plane and the current eyewear device location; and

establishing the virtual plane at a foreground location relative to a reference plane associated with the physical environment, such that the virtual plane lies between the reference plane and the current eyewear device location.

19 . The non-transitory computer-readable medium storing program code of claim 17 , wherein the program code, when executed, is operative to cause the electronic processor to perform the further steps of:

pairing the IoT product with the electronic eyewear device using a product API, wherein the product API comprises a product type and a controllable feature;

associating the foot action with the controllable feature; and

adjusting the controllable feature in accordance with the foot action and the control signal.

20 . The non-transitory computer-readable medium storing program code of claim 17 , wherein the IoT product is associated with a controllable feature comprising a set of selectable options, and wherein the program code, when executed, is operative to cause the electronic processor to perform the further steps of:

presenting along the virtual control interface a plurality of indicia, each associated with an option from the set of selectable options; and

generating an instruction based on the foot action relative to at least one of the plurality of indicia.

Assignments (1)

Continuity (2)

References Cited (82)