User comfort monitoring and notification
Some implementations disclosed herein present multimedia content to a user, identify a comfort state of the user (e.g., happy, unhappy, distressed, etc.) based on body data of the user (e.g., facial expressions, hand movements, physiological data, etc.), and update the multimedia content based on the identified comfort state.
1. A method of updating a multimedia environment, the method comprising:
at a device comprising a processor, a display, and a sensor:
presenting a multimedia environment;
obtaining data regarding a body via the sensor, wherein the obtained data corresponds to a facial expression and a change in at least one of a position, an orientation, and a shape of a portion of the body;
determining a comfort state while presenting the multimedia environment based on: i) decoding facial expressions, and ii) identifying a change in at least one of the position, the orientation, and the shape of the portion of the body as being associated with the comfort state; and
updating the multimedia environment based on determining the comfort state.
2. The method of claim 1 , wherein determining that the user is in the comfort state is based on at least two of:
an image captured by a camera sensor;
a motion captured by a motion sensor;
a heart rate captured by a heart rate sensor;
a blood oxygen level captured by a pulse oximeter sensor;
a blood pressure captured by a blood pressure sensor;
a temperature captured by a thermometer sensor; or
an electro-cardiogram (EKG) captured by an EKG sensor.
3. The method of claim 1 , wherein determining the comfort state is based on a machine learning algorithm.
4. The method of claim 1 , wherein determining the comfort state includes classifying the comfort state as a particular comfort state from amongst a plurality of predefined classes of comfort states.
5. The method of claim 1 , wherein determining the comfort state is based on a context selected from a group consisting of a setting of a physical environment or the multimedia environment, a sensitivity, a history of comfort states, and an activity.
6. The method of claim 1 , wherein updating the multimedia environment includes displaying a message.
7. The method of claim 6 , wherein the message includes a notification of the comfort state.
8. The method of claim 6 , wherein the message suggests to take a break from the multimedia environment.
9. The method of claim 6 , wherein the message provides an option to report content or another user in the multimedia environment to an authority, guardian, or moderator.
10. The method of claim 6 , wherein the message provides an option to block content or another user in the multimedia environment.
11. The method of claim 1 , further comprising recording information associated with the data regarding the body, the comfort state, or the updating of the multimedia environment.
12. The method of claim 1 , wherein the data regarding the body and data regarding the comfort state of the user is stored in a secure enclave on the device to restrict access and transmission to other devices.
13. The method of claim 1 , wherein the device is a head-mounted device (HMD) and the multimedia environment comprises a computer-generated reality (CGR) environment.
14. A system comprising:
a device with a display and a sensor;
a processor; and
a computer-readable storage medium comprising instructions that upon execution by the processor cause the system to perform operations, the operations comprising:
presenting a multimedia environment;
obtaining data regarding a body via the sensor, wherein the obtained data corresponds to a facial expression and a change in at least one of a position, an orientation, and a shape of a portion of the body;
determining a comfort state while presenting the multimedia environment based on: i) decoding facial expressions, and ii) identifying a change in at least one of the position, the orientation, and the shape of the portion of the body as being associated with the comfort state; and
updating the multimedia environment based on determining that the user is in the particular comfort state.
15. The system of claim 14 , wherein determining the comfort state includes classifying the comfort state as a particular comfort state from amongst a plurality of predefined classes of comfort states.
16. The system of claim 14 , wherein determining the comfort state is based on a context selected from a group consisting of a setting of a physical environment or the multimedia environment, a sensitivity, a history of comfort states, and an activity.
17. The system of claim 14 , wherein updating the multimedia environment includes displaying a message, wherein the message includes at least one of a notification of the comfort state and a suggestion to take a break from the multimedia environment.
18. The system of claim 14 , wherein updating the multimedia environment includes displaying a message, wherein the message provides at least one of: i) an option to report content or another user in the multimedia environment to an authority, guardian, or moderator, and ii) an option to block content or another user in the multimedia environment.
19. The system of claim 14 , wherein the data regarding the body and data regarding the comfort state is stored in a secure enclave on the device to restrict access and transmission to other devices.
20. A non-transitory computer-readable storage medium, storing computer-executable program instructions on a computer to perform operations comprising:
presenting a multimedia environment;
obtaining data regarding a body via a sensor, wherein the obtained data corresponds to a facial expression and a change in at least one of a position, an orientation, and a shape of a portion of the body;
determining a comfort state while presenting the multimedia environment based on: i) decoding facial expressions, and ii) identifying a change in at least one of the position, the orientation, and the shape of the portion of the body as being associated with the comfort state; and
updating the multimedia environment based on determining the comfort state.