Grouping variable media inputs to reflect a user session
Embodiments of the invention present a system and method for identifying relationships between different types of data captured by a pen-based computing system, such as a smart pen. The pen-based computing system generates one or more sessions including different types of data that are associated with each other. In one embodiment, the pen-based computing system generates an index file including captured audio data and written data, where the written data is associated with a temporal location of the audio data corresponding to the time the written data was captured. For example, the pen-based computing system applies one or more heuristic processes to the received data to identify relationships between various types of the received data, used to associated different types of data with each other.
1. A smart pen device for generating one or more sessions including a plurality of different types of data comprising:
a processor;
a gesture capture system coupled to the processor for receiving gesture data of the smart pen, wherein the received gesture data represents marks made on a writing surface by the smart pen;
one or more microphones coupled to the processor for receiving audio data;
an onboard memory coupled to the processor and configured to store the received gesture data and received audio data responsive to the processor identifying the input; and
a non-transitory computer-readable storage medium storing computer program code executable by the processor, the computer program code including instructions for:
storing the received gesture data and received audio data to the onboard memory;
generating a session identifier, which identifies a session of the smart pen;
associating a portion of the received gesture data with a temporal position of the received audio data and with the session identifier; and
generating an index file describing the session, the index file storing an association between (1) the session identifier, (2) the portion of the received gesture data, and (3) the temporal position within the received audio data associated with the portion of the received gesture data; and
a storage device coupled to the processor for storing the index file.
2. The smart-pen device of claim 1 , further comprising:
a control mechanism coupled to the gesture capture system and the one or more microphones for receiving an input; and
wherein the computer program code further includes instructions for:
detecting that the control mechanism receives the input; and
responsive to detecting that the control mechanism receives the input, removing the association between the temporal position of the received audio data and the received gesture data.
3. The smart-pen device of claim 1 , wherein the computer program code executable by the processor further includes instructions for:
applying one or more rules to the received gesture data to determine whether to associate the received gesture data with the temporal position of the received audio data corresponding to a time when the gesture capture system captured the gesture data.
4. The smart pen device of claim 3 , wherein the one or more rules account for the timing, frequency or location of the received gesture data.
5. The smart pen device of claim 1 , wherein the computer program code executable by the processor further includes instructions for:
including metadata in the index file.
6. The smart pen device of claim 5 , wherein the metadata is selected from a group consisting of a session name, an identification tag specifying a portion of the session and a comment associated with the session.
7. The smart pen device of claim 1 , wherein the temporal position of the received audio data corresponds to a time when the gesture capture system captured the portion of the received gesture data.
8. A method for generating a session including a plurality of data types using a pen-based computing system, the method comprising:
receiving audio data to a memory a smart pen;
receiving gesture data of the smart pen, wherein the received gesture data represents marks made on a writing surface by the smart pen;
storing the received gesture data and the received audio data;
generating a session identifier, which identifies a session of the smart pen;
associating a portion of the received gesture data with a temporal position of the received audio data and with the session identifier;
generating an index file describing the session, the index file storing an association between (1) the session identifier, (2) the portion of the received gesture data, and (3) the temporal position within the received audio data associated with the portion of the received gesture data; and
storing the index file in a storage device of the smart pen.
9. The method of claim 8 , wherein the temporal position within the received audio data comprises:
a temporal position corresponding to a time when the smart pen captured the portion of the received gesture data.
10. The method of claim 8 , wherein associating the portion of the received gesture data with a temporal position within the received audio data comprises:
applying one or more heuristic processes to the portion of the received gesture data and the received audio data using one or more parameters to identify a relationship between the portion of the received gesture data and the received audio data.
11. The method of claim 9 , wherein the one or more parameters to identify the relationship are selected from a group consisting of timing of the captured gesture data, timing of the received audio data, location of the received gesture data, location of the received audio data, frequency of the received gesture data, frequency of the received audio data, characteristics associated with editing of the received gesture data, source of the received gesture data, source of the received audio data, location of the received gesture data within a page and identification of a speaker associated with the received audio data.
12. The method of claim 8 , wherein associating the received gesture data with the temporal position within the received audio data comprises:
responsive to receiving a user input, removing the association between the temporal position of the received audio data and the received gesture data.
13. The method of claim 8 , wherein associating the portion of the received gesture data with the temporal position within the received audio data comprises:
responsive to receiving a user input pausing receipt of the audio data, associating received gesture data received after the user input pausing receipt of the audio data with a time when the user input pausing receipt of the audio data was received.
14. The method of claim 8 , further comprising:
modifying the index file responsive to receiving a command to access the index file.
15. The method of claim 14 wherein modifying the index file responsive to receiving the command to access the index file comprises:
associating additional received gesture content with a temporal position within the received audio data identified by the command.
16. The method of claim 13 , wherein modifying the index file responsive to receiving the command to access the index file comprises:
including additional audio data into the received audio data at a temporal position within the received audio data identified by the command.
17. A pen-based computing system for generating a session including a plurality of different types of data comprising:
a smart pen device configured to receive audio data and gesture data, wherein the gesture data represents marks made on a writing surface by the smart pen;
a non-transitory computer-readable storage medium storing computer program code executable by a processor coupled to the smart pen
device, the computer program code including instructions for: storing the received gesture data and the received audio data, generating a session identifier, which identifies a session of the smart pen;
associating a portion of the received gesture data with a temporal position of the received audio data and with the session identifier; and
generating an index file describing the session, the index file storing an association between (1) the session identifier, (2) the portion of the received gesture data, and (3) the temporal position within the received audio data associated with the portion of the received gesture data; and
a storage device coupled to the processor and the smart pen for storing the index file.
18. The pen-based computing system of claim 17 , further comprising:
an output module coupled to the processor and to the storage device, the output module presenting at least one of the audio data or the gesture data included in the index file.
19. The pen-based computing of claim 17 , wherein the computer program code further includes instructions for:
receiving a search query; and
searching the audio data and gesture data included in the index file for data matching the search query.
20. The pen-based computing system of claim 17 , wherein associating the portion of the gesture data with the temporal position within the audio data comprises:
applying one or more heuristic processes to the gesture data and the audio data using one or more parameters to identify a relationship between the gesture data and the audio data.
21. The pen-based computing system of claim 20 , wherein the one or more parameters to identify the relationship are selected from a group consisting of timing of the gesture data, timing of the audio data, location of the gesture data, location of the audio data, frequency of the gesture data, frequency of the audio data, characteristics associated with editing of the gesture data, source of the gesture data, source of the audio data, location of the gesture data within a page and identification of a speaker associated with the audio data.