System to correlate video data and contextual data
In some embodiments, a method of processing image data may include receiving environmental data and associated capture time data from a sensor of a mobile computing device, the capture time data reflecting capture time of the environmental data; processing the environmental data to generate metadata; time stamping the metadata using the capture time data; receiving video data and video time data at a processor; correlating the metadata to the video data using the capture time data and the video time data; receiving a search query; and/or identifying a frame within the video data by performing a search of the metadata using the search criterion.
1. A method of processing image data, the method comprising:
receiving environmental data and associated capture time data from a sensor of a mobile computing device, the capture time data reflecting capture time of the environmental data, wherein the environmental data comprise audio data;
processing the environmental data to identify, based on the audio data, a particular physical location at which the environmental data was captured, wherein the identifying of the particular physical location comprises:
filtering the audio data using a low pass filter that excludes voice frequencies and isolates background audio data, thereby extracting non-voice environmental audio data;
accessing stored audio fingerprint data that comprises multiple audio fingerprints associated with respective known locations;
comparing the non-voice environmental audio data to the multiple audio fingerprints to identify the particular physical location, wherein predetermined first Global Positioning System (GPS) data is stored for the particular physical location in association with at least one of the multiple audio fingerprints; and
correlating the predetermined first GPS data stored for the particular physical location in association with the at least one of the multiple audio fingerprints with second GPS data for a location as received by the mobile computing device;
based on the processing of the environmental data, identifying in the audio data a verbal description, provided by a user of the mobile computing device during video capture, of subject matter contained in video data associated with the environmental data;
in an automated operation, automatically generating a transcription of the verbal description;
generating metadata based at least in part on:
the particular physical location, and
the transcription of the verbal description
time stamping the metadata using the capture time data;
receiving, at a processor, the video data and video time data associated with the environmental data, the video data comprising a plurality of video frames and the video time data reflecting record time of the video data;
correlating the metadata to the video data using the capture time data and the video time data;
receiving a search query, including a search criterion, at the processor;
identifying a video frame within the video data by performing a search of the metadata using the search criterion; and
including the identified video frame in a search result.
2. The method of claim 1 , wherein the processing of the environmental data further includes:
filtering the audio data using at least one of a band pass filter and a high pass filter, to extract voice-containing environmental audio data; and
performing voice recognition on the voice-containing environmental audio data to identify a word, and wherein the generating the metadata to include the word.
3. The method of claim 1 , wherein the environmental data comprises location data, the method further comprising processing of the environmental data to identify at least one place, object or event using the location data, and wherein the generating the metadata includes the at least one place, object or event.
4. The method of claim 1 , including performing image recognition with respect to the video frame, and tagging the video frame with a tag indicative of an object recognized within the video frame, the identifying of the video frame performed by matching the search criterion and the tag.
5. The method of claim 1 , including performing characteristic recognition with respect to the video frame, and tagging the video frame with a tag indicative of a characteristic recognized in the video frame, the identifying of the video frame being performed by matching the search criterion and the tag.
6. The method of claim 1 , comprising receiving segments of the video data by a messaging system as part of respective video messages, and combining the segments to constitute the video data.
7. The method of claim 6 , wherein the respective video messages are ephemeral messages.
8. The method of claim 1 , wherein the particular physical location comprises a museum exhibit inside of a museum building.
9. A system comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, configure the system to:
receive environmental data and associated capture time data from a sensor of a mobile computing device, the capture time data reflecting capture time of the environmental data, wherein the environmental data comprise audio data;
process the environmental data to identify, based on the audio data, a particular physical location at which the environmental data was captured, wherein the identifying of the particular physical location comprises:
filtering the audio data using a low pass filter that excludes voice frequencies and isolates background audio data, thereby extracting non-voice environmental audio data;
accessing stored audio fingerprint data that comprises multiple audio fingerprints associated with respective known locations;
comparing the non-voice environmental audio data to the multiple audio fingerprints to identify the particular physical location wherein predetermined first Global Positioning System (GPS) data is stored for the particular physical location in association with at least one of the multiple audio fingerprints; and
correlating the predetermined first GPS data stored for the particular physical location in association with the at least one of the multiple audio fingerprints with a second GPS data for a location as received by the mobile computing device;
based on the processing of the environmental data, identifying in the audio data a verbal description, provided by a user of the mobile computing device during video capture, of subject matter contained in video data associated with the environmental data;
in an automated operation, automatically generating a transcription of the verbal description;
generating metadata based at least in part on:
the particular physical location, and
the transcription of the verbal description
time stamping the metadata using the capture time data;
receiving, at a processor, the video data and video time data associated with the environmental data, the video data comprising a plurality of video frames and the video time data reflecting record time of the video data;
correlating the metadata to the video data using the capture time data and the video time data;
receiving a search query, including a search criterion, at the processor;
identifying a video frame within the video data by performing a search of the metadata using the search criterion; and
including the identified video frame in a search result.
10. The system of claim 9 , wherein the identifying the particular physical location further comprises generating an audio fingerprint using at least a portion of the non-voice environmental audio data, and comparing the audio fingerprint to the multiple stored audio fingerprints.
11. The system of claim 9 , wherein the identifying the particular physical location further comprises filtering the audio data to extract background audio data and the instructions, when executed by the at least one processor, configure the system to use the background audio data to identify the particular physical location.
12. The system of claim 9 , wherein the instructions, when executed by the at least one processor, further configure the system to perform operations comprising:
filter the audio data using at least one of a band pass filter and a high pass filter, to extract voice-containing environmental audio data;
performing voice recognition on the voice-containing environmental audio data to identify a word included in the audio data; and
and to said generate the metadata to include the identified word.
13. The system of claim 9 , wherein the environmental data comprises location data, and the instructions, when executed by the at least one processor, further configure the system to process the environmental data to identify at least one place, object or event using the location data, and to said generate the metadata to include the at least one place, object or event.
14. The system of claim 9 , wherein the instructions, when executed by the at least one processor, further configure the system to perform characteristic recognition with respect to the frame, and to tag the frame with a tag indicative of a characteristic recognized in the frame, the identifying of the frame being performed by matching the search criterion and the tag.
15. The system of claim 9 , wherein:
the instructions, when executed by the at least one processor, further configure the system to receive segments of the video data by a messaging system as part of respective video messages, and to combine the segments to constitute the video data; and
the respective video messages are ephemeral messages.
16. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
receiving environmental data and associated capture time data from a sensor of a mobile computing device, the capture time data reflecting capture time of the environmental data, wherein the environmental data comprise audio data;
processing the environmental data to identify, based on the audio data, a particular physical location at which the environmental data was captured, wherein the identifying of the particular physical location comprises:
filtering the audio data using a low pass filter that excludes voice frequencies and isolates background audio data, thereby extracting non-voice environmental audio data;
accessing stored audio fingerprint data that comprises multiple audio fingerprints associated with respective known locations;
comparing the non-voice environmental audio data to the multiple audio fingerprints to identify the particular physical location, wherein predetermined first Global Positioning System (GPS) data is stored for the particular physical location in association with at least one of the multiple audio fingerprints; and
correlating the predetermined first GPS data stored for the particular physical location in association with the at least one of the multiple audio fingerprints with second GPS data for a location as received by the mobile computing device;
based on the processing of the environmental data, identifying in the audio data a verbal description, provided by a user of the mobile computing device during video capture, of subject matter contained in video data associated with the environmental data;
in an automated operation, automatically generating a transcription of the verbal description;
generating metadata based at least in part on:
the particular physical location, and
the transcription of the verbal description
time stamping the metadata using the capture time data;
receiving, at a processor, the video data and video time data associated with the environmental data, the video data comprising a plurality of video frames and the video time data reflecting record time of the video data;
correlating the metadata to the video data using the capture time data and the video time data;
receiving a search query, including a search criterion, at the processor;
identifying a video frame within the video data by performing a search of the metadata using the search criterion; and
including the identified video frame in a search result.