IP Library Granted Patent US 10,430,661
Granted Patent B2
US 10,430,661 · App. 15/384,831 · Granted Oct 1, 2019

Generating a compact video feature representation in a digital medium environment

Inventors: Hao Hu (Orlando, FL); Zhaowen Wang (San Jose, CA); Joon-Young Lee (Milpitas, CA); Zhe Lin (Fremont, CA)
Assignee: Adobe Inc.
G06K9/00744G06K9/4628G06K9/6271G06N3/0445G06N3/0454G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,430,661
App. No.
15/384,831
Granted
Oct 1, 2019
Kind
B2
Abstract

Techniques and systems are described to generate a compact video feature representation for sequences of frames in a video. In one example, values of features are extracted from each frame of a plurality of frames of a video using machine learning, e.g., through use of a convolutional neural network. A video feature representation is generated of temporal order dynamics of the video, e.g., through use of a recurrent neural network. For example, a maximum value is maintained of each feature of the plurality of features that has been reached for the plurality of frames in the video. A timestamp is also maintained as indicative of when the maximum value is reached for each feature of the plurality of features. The video feature representation is then output as a basis to determine similarity of the video with at least one other video based on the video feature representation.

Claims (33)

1. In a digital medium environment to learn a compact feature representation of video, a method implemented by at least one computing device, the method comprising:

determining, by the at least one computing device, temporal order dynamics exhibited by a plurality of frames of the video through machine learning, the temporal order dynamics describing changes to the plurality of frames, one to another, over time;

generating, by the at least one computing device, a video feature representation from the temporal order dynamics, the generating based on:

a value of each feature of a plurality of features that has been reached for the plurality of frames in the video from the machine learning; and

a timestamp indicating when the value is reached for each feature of the plurality of features; and

outputting, by the at least one computing device, the video feature representation as a basis to determine similarity of the video with at least one other video.

2. The method as described in claim 1 , wherein the value describes a relative magnitude of a difference between values for respective features for respective frames of said plurality of frames.

3. The method as described in claim 1 , wherein the determining the temporal order dynamics is performed using the machine learning as part of a convolutional neural network.

4. The method as described in claim 3 , wherein the convolutional neural network learns changes to latent features of each frame of the plurality of frames to describe the temporal order dynamics.

5. The method as described in claim 1 , wherein the generating is performed using machine learning as part of a recurrent neural network.

6. The method as described in claim 1 , wherein the generating of the video feature representation is based at least in part on changes to values of features learned using machine learning.

7. The method as described in claim 1 , further comprising determining the similarity of the video with the at least one other video based on the video feature representation of the video and a video feature representation of the at least one other video.

8. The method as described in claim 7 , wherein the determining of the similarity is based on a Euclidean or cosine distance between the video feature representation of the video and the video feature representation of the at least one other video.

9. In a digital medium environment to learn a compact feature representation of video, a system comprising:

a first machine learning module implemented at least partially in hardware of at least one computing device to extract values of features from each frame of a plurality of frames of a video; and

a second machine learning module implemented at least partially in hardware of at least one computing device to generate a video feature representation of temporal order dynamics of the video, the second machine learning module including:

a max-pooling layer to determine a maximum value, for inclusion as part of the video feature representation, of each feature of the features for the plurality of frames in the video; and

a temporal preserving layer to generate a timestamp, for inclusion as part of the video feature representation, indicating when the maximum value is reached for each feature of the plurality of features in the video.

10. The system as described in claim 9 , wherein the maximum value describes a relative magnitude of a difference between values for respective features for respective frames of said plurality of frames.

11. The system as described in claim 9 , wherein the first machine learning module implements a convolutional neural network.

12. The system as described in claim 11 , wherein the features are latent features.

13. The system as described in claim 9 , wherein the second machine learning module implements a recurrent neural network.

14. The system as described in claim 9 , further comprising a similarity determination module implemented at least partially in hardware of the at least one computing device to determine a similarity of the video with at least one other video based on the video feature representation of the video and a video feature representation of the at least one other video.

15. The system as described in claim 14 , wherein the similarity is based on a Euclidean or cosine distance between the video feature representation of the video and the video feature representation of the at least one other video.

16. In a digital medium environment to learn a compact feature representation of video, a system comprising:

means for determining temporal order dynamics exhibited by a plurality of frames of the video through machine learning, the temporal order dynamics described using:

a value of each feature of a plurality of features for the plurality of frames in the video learned through the machine learning; and

a timestamp indicating when the value is reached for each feature of the plurality of features in the video;

means for generating a video feature representation from the temporal order dynamics; and

means for outputting the video feature representation as a basis to determine similarity of the video with at least one other video.

17. The system as described in claim 16 , further comprising means for determining the similarity of the video with the at least one other video based on the video feature representation of the video and a video feature representation of the at least one other video.

18. The system as described in claim 17 , wherein the determining similarity means is configured to determine the similarity based on a Euclidean or cosine distance between the video feature representation of the video and the video feature representation of the at least one other video.

19. The system as described in claim 16 , wherein the value describes a relative magnitude of a difference between values for respective features for respective frames of said plurality of frames.

Assignments (2)
CHANGE OF NAME Recorded Jan 21, 2019
From: ADOBE SYSTEMS INCORPORATED
To: ADOBE INC.
Reel/Frame 048103/0226 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 27, 2016
From: HU, HAO; WANG, ZHAOWEN; LEE, JOON-YOUNG; LIN, ZHE
To: ADOBE SYSTEMS INCORPORATED
Reel/Frame 040776/0687 →
Continuity (1)
Related Publication 20180173958A1 · Jun 21, 2018
Cited By (3)
US 12,293,575 US 12,387,242 US 12,387,243