Granted Patent B2

US 12,429,950 · App. 18/462,077 · Granted Sep 30, 2025

Inventors: Jesse Chand (Los Angeles, CA); Krish Jayaram (Santa Monica, CA)

Assignee: Snap Inc.

G06F3/016G06F3/041G06F3/0414G06F3/017G06F3/0412G06F3/0425G06F3/0488G06F3/04883G06F2203/014G06F2203/04803

Abstract

Claims (76)

1. A method comprising:

modifying a portion of a first image displayed in a first region of a display, and captured by a camera, based on a plurality of haptic interactions, at least a portion of the plurality of haptic interactions being performed within the same first region of the display in which the first image is displayed; and

concurrently with displaying, in the first region of a display, the first image having the modified portion, causing display, in a second region of the display separate from the first image, of a visual representation of a history of haptic interactions, the visual representation of the history of haptic interactions comprising physical interaction with a device, the visual representation of the history of haptic interactions including the plurality of haptic interactions performed by user interactions with the device over a given time interval previous to a current time, at least two of the plurality of haptic interactions being detected by a different one of a plurality of types of sensors of the device, the visual representation comprising:

a first portion of the visual representation comprising a first visual indication of a first of the plurality of haptic interactions made during a first portion of the given time interval;

determining a type of component of the first of the plurality of haptic interactions performed by the user interactions with the device, the type of component corresponding to an individual one of the plurality of types of sensors of the device;

selecting a shape corresponding to the type of component of the first of the plurality of haptic interactions which has been performed by the user interactions with the device; and

the first visual indication having the selected shape with a size that is determined based on a duration of time corresponding to the first of the plurality of haptic interactions.

2. The method of claim 1 , the visual representation comprising a first portion with a first shape representing the first of the plurality of haptic interactions detected by a first of the plurality of types of sensors and comprising a second portion with a second shape representing a second of the plurality of haptic interactions detected by a second of the plurality of types of sensors, further comprising:

receiving haptic data from the device after capturing the first image, the haptic data indicating an interaction with a sensor included in the device; and

determining, based on the haptic data, the plurality of haptic interactions performed over the given time interval.

3. The method of claim 1 , wherein the plurality of haptic interactions is performed with respect to the first image displayed on a first portion of a touchscreen of a device and result in one or more modifications of the first image, further comprising:

determining first and second haptic characteristics;

determining that a combination of the first and second haptic characteristics match a predetermined combination of threshold characteristics; and

in response to determining that the combination of the first and second haptic characteristics match the predetermined combination of threshold characteristics, generating a second image that depicts the combination of the first and second haptic characteristics, the second image comprising the visual representation of the history of haptic interactions.

4. The method of claim 1 , further comprising:

causing display of a graph over a portion of the first image, wherein the visual representation includes a graph that corresponds to the plurality of haptic interactions, the graph indicating a length that corresponds to a duration of each of the plurality of interactions, the graph indicating an amplitude that corresponds to an amount of force of each of the plurality of interactions, the graph having a frequency that corresponds to a gesture performed by a user.

5. The method of claim 1 , further comprising:

receiving audio or video data;

determining characteristics of the audio or video data being played back;

determining that the characteristics of the audio or video data being played back trigger a haptic response; and

executing the haptic response during playback of the audio or video data that includes the plurality of haptic interactions.

6. The method of claim 1 , further comprising setting a color for the first visual indication based on a first type of haptic data included in a first of the plurality of haptic interactions and setting the shape for the first visual indication based on a second type of haptic data included in the first of the plurality of haptic interactions, and wherein an intensity of the color is determined based on an amount of force corresponding to the first of the plurality of haptic interactions.

7. The method of claim 1 , further comprising:

determining that a second device has a different download speed than a first device;

adjusting the plurality of haptic interactions based on a difference in download speed of the first device relative to the second device; and

triggering a response on the second device based on the adjusted plurality of haptic interactions.

8. The method of claim 1 , further comprising:

modifying a graphical element that assists with navigation of a user interface of a device based on the plurality of haptic interactions, wherein the visual representation of the history of haptic interactions is generated in response to determining that haptic characteristics match predetermined threshold characteristics.

9. The method of claim 1 , further comprising:

determining capabilities of a second device;

translating the plurality of haptic interactions into further haptic characteristics based on the capabilities of the second device;

generating an image that depicts the further haptic characteristics; and

causing display of the image that depicts the further haptic characteristics on the second device.

10. The method of claim 9 , wherein determining the capabilities of the second device includes receiving, from the second device, a signal that indicates the capabilities of the second device, and wherein translating comprises removing data from the plurality of haptic interactions in response to determining that the second device includes circuitry that results in slower processing speeds than a first device.

11. A system comprising:

one or more processors; and

a memory storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

modifying a portion of a first image displayed in a first region of a display, and captured by a camera, based on a plurality of haptic interactions, at least a portion of the plurality of haptic interactions being performed within the same first region of the display in which the first image is displayed; and

concurrently with displaying, in the first region of a display, the first image having the modified portion, causing display, in a second region of the display separate from the first image, of a visual representation of a history of haptic interactions, the visual representation of the history of haptic interactions comprising physical interaction with a device, the visual representation of the history of haptic interactions including the plurality of haptic interactions performed by user interactions with the device over a given time interval previous to a current time, at least two of the plurality of haptic interactions being detected by a different one of a plurality of types of sensors of the device, the visual representation comprising:

a first portion of the visual representation comprising a first visual indication of a first of the plurality of haptic interactions made during a first portion of the given time interval;

determining a type of component of the first of the plurality of haptic interactions performed by the user interactions with the device, the type of component corresponding to an individual one of the plurality of types of sensors of the device;

selecting a shape corresponding to the type of component of the first of the plurality of haptic interactions which has been performed by the user interactions with the device; and

the first visual indication having the selected shape with a size that is determined based on a duration of time corresponding to the first of the plurality of haptic interactions.

12. The system of claim 11 , wherein the plurality of haptic interactions is performed with respect to the first image displayed on a first portion of a touchscreen of a device and result in one or more modifications of the first image, and wherein the operations further comprise:

determining first and second haptic characteristics;

determining that a combination of the first and second haptic characteristics match a predetermined combination of threshold characteristics; and

in response to determining that the combination of the first and second haptic characteristics match the predetermined combination of threshold characteristics, generating a second image that depicts the determined combination of the first and second haptic characteristics, the second image comprising the visual representation of the history of haptic interactions.

13. The system of claim 11 , wherein the visual representation of the history of haptic interactions is displayed on top of a portion of the display in which one or more messages received from another user device are displayed.

14. The system of claim 11 , wherein the operations further comprise:

causing display of a graph over a portion of the first image, wherein the visual representation includes a graph that corresponds to the plurality of haptic interactions, the graph indicating a length that corresponds to a duration of each of the plurality of interactions, the graph indicating an amplitude that corresponds to an amount of force of each of the plurality of interactions.

15. The system of claim 11 , wherein the operations further comprise:

receiving audio or video data;

determining characteristics of the audio or video data being played back;

determining that the characteristics of the audio or video data being played back trigger a haptic response; and

executing the haptic response during playback of the audio or video data that includes the plurality of haptic interactions.

16. The system of claim 11 , wherein the operations further comprise:

setting a color for the first visual indication based on a first type of haptic data included in a first of the plurality of haptic interactions and setting the shape for the first visual indication based on a second type of haptic data included in the first of the plurality of haptic interactions, wherein an intensity of the color is determined based on an amount of force corresponding to the first of the plurality of haptic interactions.

17. The system of claim 11 , wherein the operations further comprise:

determining that a second device has a different download speed than a first device;

adjusting the plurality of haptic interactions based on a difference in download speed of the first device relative to the second device; and

triggering a response on the second device based on the adjusted plurality of haptic interactions.

18. The system of claim 11 , wherein the operations further comprise:

determining that a second device has a different screen size than a first device;

adjusting the plurality of haptic interactions based on a difference in screen size of the first device relative to the second device; and

triggering a response on the second device based on the adjusted plurality of haptic interactions.

19. The system of claim 11 , wherein the type of component comprises a first type of component comprising force, the selected shape being a first shape, further comprising:

determining a second type of component of a second of the plurality of haptic interactions;

selecting a second shape corresponding to the second type of component of the second of the plurality of haptic interactions; and

presenting a second visual indication having the second shape with another size that is determined based on a duration of time corresponding to the second of the plurality of haptic interactions.

20. A non-transitory machine readable medium storing instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

modifying a portion of a first image displayed in a first region of a display, and captured by a camera, based on a plurality of haptic interactions, at least a portion of the plurality of haptic interactions being performed within the same first region of the display in which the first image is displayed; and

concurrently with displaying, in the first region of a display, the first image having the modified portion, causing display, in a second region of the display separate from the first image, of a visual representation of a history of haptic interactions, the visual representation of the history of haptic interactions comprising physical interaction with a device, the visual representation of the history of haptic interactions including the plurality of haptic interactions performed by user interactions with the device over a given time interval previous to a current time, at least two of the plurality of haptic interactions being detected by a different one of a plurality of types of sensors of the device, the visual representation comprising:

a first portion of the visual representation comprising a first visual indication of a first of the plurality of haptic interactions made during a first portion of the given time interval;

determining a type of component of the first of the plurality of haptic interactions performed by the user interactions with the device, the type of component corresponding to an individual one of the plurality of types of sensors of the device;

selecting a shape corresponding to the type of component of the first of the plurality of haptic interactions which has been performed by the user interactions with the device; and

the first visual indication having the selected shape with a size that is determined based on a duration of time corresponding to the first of the plurality of haptic interactions.

Assignments (1)

Continuity (3)

References Cited (76)