Media contextual information from buffered media data

1. A computer-implemented method, the method comprising:

storing in a buffer, on a user device, media data as buffered media data, the buffered media data being a most recent portion of media data received at the user device, the most recent portion inclusive of the media data received from a present time to a prior time that is fixed relative to the present time, wherein storing in the buffer the buffered media data comprises:

determining, for a media frame of the received media data, whether the media frame has a threshold change in content from a previously received media frame; and

storing, in the buffer on the user device, the media frame in response to determining the media frame has a threshold change in content from a previously received media frame;

responsive to a search operation invocation at the present time, sending the buffered media data to a data processing system that is separate from the user device, wherein the search operation invocation is determined by an assistant process executing on the user device that a user has requested a search operation; and

sending the buffered media data comprises:

receiving, by the assistant process executing on the user device, a search query;

determining that the search query is determined to be an ambiguous query; and

only in response to determining that the search query is determined to be an ambiguous query, sending the buffered media data with the search query to the data processing system; and

wherein the buffered media data is not sent when the search query is not determined to be ambiguous; and

receiving, from the data processing system and in response to the buffered media data, contextual information regarding an entity that the data processing system identified from processing the buffered media data.

2. The computer-implemented method of claim 1 , wherein receiving contextual information comprises receiving a user interface element at the user device.

3. The computer-implemented method of claim 1 , wherein:

sending buffered media data to a data processing system that is remote from the user device comprises iteratively sending, according to a predefined time period, buffered media data to the data processing system; and

receiving contextual information comprises receiving contextual information regarding the entity for each portion of buffer media data sent for each iteration.

4. The computer-implemented method of claim 1 , wherein:

storing in the buffer the buffered media data comprises:

receiving media data from multiple applications; and

storing the media data for each of the multiple applications as separate respective instances of buffered media data; and

sending buffered media data to a data processing system that is remote from the user device comprises:

determining, at the present time, an executing application for which the user device has a primary focus; and

selecting as selected buffered media data only the buffered media data for the executing application for which the user device has a primary focus; and

sending the selected buffered media to the data processing system.

5. The computer-implemented method of claim 1 , wherein:

the user device is a first user device; and

receiving, from the data processing system and in response to the buffered media data, contextual information regarding an entity that the data processing system identified from processing the buffered media data comprises receiving the contextual information at a second user device that is separate from the first user device.

6. The computer-implemented method of claim 1 , wherein receiving, from the data processing system and in response to the buffered media data, contextual information regarding an entity that the data processing system identified from processing the buffered media data comprises received the contextual information at the user device.

7. A system, comprising:

a data processing apparatus; and

a non-transitory computer readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and upon such execution cause the data processing apparatus to perform operations comprising:

storing in a buffer, on a user device, media data as buffered media data, the buffered media data being a most recent portion of media data received at the user device, the most recent portion inclusive of the media data received from a present time to a prior time that is fixed relative to the present time, wherein the buffered media data comprises media frames that each are determined to have a threshold change of content from a respectively previous media frame;

responsive to a search operation invocation at the present time, sending the buffered media data to a data processing system that is remote from the user device, wherein the search operation invocation is determined by an assistant process executing on the user device that a user has requested a search operation; and

sending the buffered media data comprises:

receiving, by the assistant process executing on the user device, a search query;

determining that the search query is determined to be an ambiguous query; and

only in response to determining that the search query is determined to be an ambiguous query, sending the buffered media data with the search query to the data processing system; and

wherein the buffered media data is not sent when the search query is not determined to be ambiguous; and

receiving, from the data processing system and in response to the buffered media data, contextual information regarding an entity that the data processing system identified from processing the buffered media data.

8. The system of claim 7 , wherein receiving contextual information comprises receiving a user interface element at the user device.

9. The system of claim 7 , wherein the search operation invocation is determined by an assistant process executing on the user device that a user has requested a search operation.

10. The system of claim 7 , wherein:

storing in the buffer the buffered media data comprises:

receiving media data from multiple applications; and

storing the media data for each of the multiple applications as separate respective instances of buffered media data; and

sending buffered media data to a data processing system that is remote from the user device comprises:

determining, at the present time, an executing application for which the user device has a primary focus; and

selecting as selected buffered media data only the buffered media data for the executing application for which the user device has a primary focus; and

sending the selected buffered media to the data processing system.