Indexing presentation data using biometric sensor data streams
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for providing a presentation based on stored user experience data. In one aspect, a method includes receiving a data stream including a plurality of biometric data points that each correspond to a particular timestamp in a plurality of timestamps; determining, for a particular timestamp, that a particular biometric data point in the data stream satisfies a predetermined biometric response threshold; in response to determining that the particular biometric data point satisfies the predetermined biometric response threshold, identifying a time period spanning an interval before the particular timestamp through an interval after the particular timestamp; storing user experience data for the identified time period in a storage device; and providing a presentation based on the stored user experience data.
1 . A computer-implemented method comprising:
receiving, from a biometric sensor associated with a user and during presentation of a plurality of media content frames, a data stream including a plurality of biometric data points wherein each of the plurality biometric data points corresponds to each of a plurality of timestamps, wherein the plurality of timestamps are associated with the plurality of media content frames, and wherein the plurality of biometric data points represent measurements of each of a plurality of physiologic characteristics;
determining, by an index engine, for first timestamp among the plurality of time stamps, a first biometric data point among the plurality of data points in the data stream satisfies a predetermined biometric response threshold;
in response to determining that the first biometric data point satisfies the predetermined biometric response threshold, identifying a time period between a first interval before the first timestamp and a second interval after the first timestamp;
identifying a user experience during the presentation of the plurality of media content frames and storing user experience data corresponding to the identified user experience for the identified time period in a storage device, wherein the user experience data comprises:
a set of key media content frames in the plurality of media content frames corresponding to the plurality of timestamps in the identified time period; and
a set of key biometric data points in the plurality of biometric data points, wherein each of the set of key biometric data points corresponds to each of the plurality of timestamps in the identified time period, wherein each of the set of key media content frames is indexed with an index key wherein the index key corresponds to each of the set of key biometric data points; and
providing a presentation based on the stored user experience data.
2 . The method of claim 1 , wherein receiving the data stream comprises:
receiving the data stream while providing the plurality of media content frames to a user in a display of an augmented reality device, a virtual reality device, or a mixed reality device that is equipped with the biometric sensor.
3 . The method of claim 1 , wherein the presentation of a plurality of media content frames comprises one or more of: two-dimensional video data, three-dimensional video data, or audio data.
4 . The method of claim 1 , wherein the data stream comprises one of: a cortisol level data stream, a heartbeat data stream, an eye tracking data stream, or a brain signal data stream.
5 . The method of claim 1 , wherein determining that the first biometric data point satisfies the predetermined biometric response threshold comprises:
assigning different biometric response thresholds to each of the different emotional states;
receiving from the user a selection of a first emotional state among the different emotional states; and
selecting a first biometric response threshold among the different biometric response thresholds corresponding to the first emotional state as the predetermined biometric response threshold.
6 . The method of claim 5 , wherein the predetermined biometric response threshold comprises:
an upper threshold value of the first biometric data point, a lower threshold value of the first biometric data point, or both.
7 . The method of claim 1 , wherein the user experience data further comprises transcripts of the set of key media content frames.
8 . The method of claim 1 , wherein the user experience data further comprises facial gesture data, speech data, or both of the user while presenting the plurality of media content frames.
9 . The method of claim 1 , wherein storing the set of key media content frames in the storage device comprises discarding non-key media content frames in the plurality of media content frames corresponding to timestamps outside of the time period.
10 . The method of claim 1 , wherein the presentation comprises the stored user experience data, and a biological heat map of the plurality of biometric data points.
11 . A system comprising one or more computers and non-transitory computer storage medium for-storing computer readable instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving, from a biometric sensor and during presentation of a plurality of media content frames, a data stream including a plurality of biometric data points, wherein each biometric data point corresponds to each of a plurality of timestamps, wherein the plurality of timestamps are associated with the plurality of media content frames, and wherein the plurality of biometric data points represent measurements of a physiologic characteristics;
determining, by an index engine, for a particular first timestamp among the plurality of time stamps, that a particular first biometric data point among the plurality of data points in the data stream satisfies a predetermined biometric response threshold;
in response to determining that the particular first biometric data point satisfies the predetermined biometric response threshold, identifying a time period between a first interval before the particular first timestamp and a second interval after the particular first timestamp;
identifying a user experience during the presentation of the plurality of media content frames and storing user experience data corresponding to the identified user experience for the identified time period in a storage device, wherein the user experience data comprises:
a set of key media content frames in the plurality of media content frames corresponding to the plurality of timestamps in the identified time period; and
a set of key biometric data points in the plurality of biometric data points wherein, each of the set of key biometric data points corresponds to each of the plurality of timestamps in the identified time period, wherein each of the set of key media content frames is indexed with an index key wherein the index key corresponds to each of the set of key biometric data points; and
providing a presentation based on the stored user experience data.
12 . The system of claim 11 , wherein receiving the data stream comprises:
receiving the data stream while providing the plurality of media content frames to a user in a display of an augmented reality device, a virtual reality device, or a mixed reality device that is equipped with the biometric sensor.
13 . The system of claim 11 , wherein the presentation of a plurality of media content frames comprises one or more of: two-dimensional video data, three-dimensional video data, or audio data.
14 . The system of claim 11 , wherein the data stream comprises one of: a cortisol level data stream, a heartbeat data stream, an eye tracking data stream, or a brain signal data stream.
15 . The system of claim 11 , wherein determining that the first biometric data point satisfies the predetermined biometric response threshold comprises:
maintaining data assigning different biometric response thresholds corresponding respectively to each of the different emotional states;
receiving from the user a selection of a particular a first emotional state among the different emotional states; and
selecting a first biometric response threshold among the different biometric response thresholds corresponding to the first emotional state as the predetermined biometric response threshold.
16 . The system of claim 15 , wherein the predetermined biometric response threshold comprises:
an upper threshold value of the first biometric data point, a lower threshold value of the first biometric data point, or both.
17 . The system of claim 11 , wherein the user experience data further comprises transcripts of the set of key media content frames.
18 . The system of claim 11 , wherein the user experience data further comprises facial gesture data, speech data, or both of the user while presenting the plurality of media content frames.
19 . The system of claim 11 , wherein storing the set of key media content frames in the storage device comprises discarding non-key media content frames in the plurality of media content frames corresponding to timestamps outside of the time period.
20 . A non-transitory computer storage medium encoded with computer readable instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving, from a biometric sensor and during presentation of a plurality of media content frames, a data stream including a plurality of biometric data points, wherein each biometric data point corresponds to each of a plurality of timestamps, wherein the plurality of timestamps are associated with the plurality of media content frames, and wherein the plurality of biometric data points represent measurements of a physiologic characteristics;
determining, by an index engine, for a particular first timestamp among the plurality of time stamps, that a particular first biometric data point among the plurality of data points in the data stream satisfies a predetermined biometric response threshold;
in response to determining that the particular first biometric data point satisfies the predetermined biometric response threshold, identifying a time period between a first interval before the particular first timestamp and a second interval after the particular first timestamp;
identifying a user experience during the presentation of the plurality of media content frames and storing user experience data corresponding to the identified user experience for the identified time period in a storage device, wherein the user experience data comprises:
a set of key media content frames in the plurality of media content frames corresponds corresponding to the plurality of timestamps in the identified time period; and
a set of key biometric data points in the plurality of biometric data points wherein, each of the set of key biometric data points corresponds to each of the plurality of timestamps in the identified time period, wherein each of the set of key media content frames is indexed with an index key wherein the index key corresponds to each of the set of key biometric data points; and
providing a presentation based on the stored user experience data.