Visual and text search interface for text-based video editing
Embodiments of the present invention provide systems, methods, and computer storage media for a visual and text search interface used to navigate a video transcript. In an example embodiment, a freeform text query triggers a visual search for frames of a loaded video that match the freeform text query (e.g., frame embeddings that match a corresponding embedding of the freeform query), and triggers a text search for matching words from a corresponding transcript or from tags of detected features from the loaded video. Visual search results are displayed (e.g., in a row of tiles that can be scrolled to the left and right), and textual search results are displayed (e.g., in a row of tiles that can be scrolled up and down). Selecting (e.g., clicking or tapping on) a search result tile navigates a transcript interface to a corresponding portion of the transcript.
1. One or more computer storage media storing computer-useable instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:
responsive to receiving a freeform text query via a search interface of a video editing interface, triggering: both (i) a visual search for matching video frames, of a loaded video, that match the freeform text query by encoding the freeform text query into a query embedding of a text modality in a common embedding space with frame embeddings of a visual modality of video frames of the loaded video, and identifying the matching video frames based on a measure of similarity between corresponding frame embeddings of the visual modality and the query embedding of the text modality, and (ii) a text search for matching words in a transcript of the loaded video or in tags of detected features from the loaded video based on the freeform text query; and
causing the search interface to present both (i) visual search result tiles representing the matching video frames in a portion of the search interface and (ii) text search result tiles representing video segments corresponding to the matching words in a different portion of the search interface.
2. The one or more computer storage media of claim 1 , wherein the visual search comprises performing a nearest neighbor search to identify the matching video frames.
3. The one or more computer storage media of claim 1 , the operations further comprising causing the search interface to present the visual search result tiles in a row that can be scrolled left and right in the portion of the search interface, and to present the text search result tiles in a column that can be scrolled up and down in the different portion of the search interface.
4. The one or more computer storage media of claim 1 , the operations further comprising causing the search interface to present each of the visual search result tiles in the portion of the search interface with a video thumbnail of a matching video frame, a timestamp of the matching video frame in the loaded video, and a match score for the matching video frame with respect to the freeform text query.
5. The one or more computer storage media of claim 1 , the operations further comprising causing the search interface to present each of the text search result tiles in the different portion of the search interface with a video thumbnail of a video segment corresponding to a matching word in the transcript, a speaker thumbnail representing a diarized speaker detected from the video segment, and transcript text corresponding to the video segment and highlighting the matching word.
6. The one or more computer storage media of claim 1 , the operations further comprising causing the search interface to present each of the text search result tiles in the different portion of the search interface with a video thumbnail of a video segment from which a detected feature with a matched tag was detected, a speaker thumbnail representing a diarized speaker detected from the video segment, and transcript text corresponding to the video segment.
7. The one or more computer storage media of claim 1 , the operations further comprising, responsive to receiving input identifying a selected diarized speaker, triggering a search for parts of the transcript spoken by the selected diarized speaker.
8. The one or more computer storage media of claim 1 , the operations further comprising, responsive to receiving input identifying a selected sound class, triggering a search for parts of the transcript corresponding to parts of an audio track of the loaded video where the selected sound class was detected.
9. The one or more computer storage media of claim 1 , wherein the search interface is configured to navigate, responsive to selection of one of the visual search result tiles or the text search result tiles, to a corresponding portion of the transcript.
10. A method comprising:
responsive to receiving a freeform text query via a search interface of a video editing interface, triggering: both (i) a visual search for matching video frames of a loaded video by encoding the freeform text query into a query embedding of a text modality in a common embedding space with frame embeddings of a visual modality of video frames of the loaded video, and identifying the matching video frames based on a measure of similarity between corresponding frame embeddings of the visual modality and the query embedding of the text modality, and (ii) a text search for matching transcript text, or matching tags of detected features, of the loaded video; and
causing the search interface to present both (i) visual search result tiles representing the matching video frames in a portion of the search interface and (ii) text search result tiles representing video segments corresponding to the matching transcript text or the matching tags in a different portion of the search interface.
11. The method of claim 10 , wherein the visual search comprises performing a nearest neighbor search to identify the matching video frames.
12. The method of claim 10 , further comprising causing the search interface to present the visual search result tiles in a row that can be scrolled left and right in the portion of the search interface, and to present the text search result tiles in a column that can be scrolled up and down in the different portion of the search interface.
13. The method of claim 10 , further comprising causing the search interface to present each of the visual search result tiles in the portion of the search interface with a video thumbnail of a matching video frame, a timestamp of the matching video frame in the loaded video, and a match score for the matching video frame with respect to the freeform text query.
14. The method of claim 10 , further comprising causing the search interface to present each of the text search result tiles in the different portion of the search interface with a video thumbnail of a video segment corresponding to the matching transcript text, a speaker thumbnail representing a diarized speaker detected from the video segment, and transcript text corresponding to the video segment, including the matching transcript text.
15. The method of claim 10 , further comprising causing the search interface to present each of the text search result tiles in the different portion of the search interface with a video thumbnail of a video segment from which a detected feature with a matched tag was detected, a speaker thumbnail representing a diarized speaker detected from the video segment, and transcript text corresponding to the video segment, including the matching transcript text.
16. The method of claim 10 , further comprising, responsive to receiving input identifying a selected diarized speaker, triggering a search for parts of a transcript of the loaded video spoken by the selected diarized speaker.
17. The method of claim 10 , further comprising, responsive to receiving input identifying a selected sound class, triggering a search for parts of a transcript of the loaded video corresponding to parts of an audio track of the loaded video where the selected sound class was detected.
18. The method of claim 10 , wherein the search interface is configured to navigate, responsive to selection of one of the visual search result tiles or the text search result tiles, to a corresponding portion of a transcript of the loaded video.
19. A computer system comprising one or more processors and memory configured to provide computer program instructions to the one or more processors, the computer program instructions comprising:
a search component configured to trigger, responsive to receiving a freeform text query: both (i) a visual search for matching video frames of a loaded video by encoding the freeform text query into a query embedding of a text modality in a common embedding space with frame embeddings of a visual modality of video frames of the loaded video, and performing a nearest neighbor search to identify the matching video frames based on a measure of similarity between corresponding frame embeddings of the visual modality and the query embedding of the text modality, and (ii) a text search for matching transcript text, or matching tags of detected features, of the loaded video; and
a search result layout component configured to present both (i) visual search result tiles representing the matching video frames in a portion of a search interface and (ii) text search result tiles representing video segments corresponding to the matching transcript text or the matching tags in a different portion of the search interface.
20. The computer system of claim 19 , wherein the search result layout component is configured to present each of the visual search result tiles in the portion of the search interface with a video thumbnail of a matching video frame, a timestamp of the matching video frame in the loaded video, and a match score for the matching video frame with respect to the freeform text query.