IP Library Granted Patent US 10,283,162
Granted Patent B2
US 10,283,162 · App. 14/314,343 · Granted May 7, 2019

Method for triggering events in a video

Inventors: Elena Shaburova (Castro Valley, CA); Victor Shaburov (Castro Valley, CA)
Assignee: Avatar Merger Sub II, LLC
G11B27/031G06F3/011G06K9/00261G06K9/00268G06K9/00281G06K9/00302G06K9/00335G06K9/00711G06K9/00744G06K9/46G06K9/4652G06K9/4661G06K9/52G06N99/005G06T5/005G06T7/246G06T7/251G06T7/33G06T7/90G06T11/001G06T11/60G06T13/40G06T17/20G06T19/20H04N5/262H04N9/79G06K2009/4666G06T2207/10016G06T2207/10021G06T2207/10024G06T2207/20036G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,283,162
App. No.
14/314,343
Granted
May 7, 2019
Kind
B2
Abstract

A computer implemented method of triggering events in a video, the method comprising: providing a list of objects with their states and corresponding events in video such that each state from the list triggers at least one event of the corresponding events, wherein each object from the list has at least one state triggering at least one event of the corresponding events from the list in video; detecting at least one object from the list that at least partially and at least occasionally is presented in frames of the video; tracking the at least one object and its state; triggering at least one event of the corresponding events from the list in video in case the state of the at least one object matches with one of its states from the list.

Claims (62)

1. A computer-implemented method for triggering events in a video, the method being performed in connection with a computerized system comprising a processing unit and a memory, the method comprising:

providing a list comprising a set of objects, a set of object states associated with the set of objects, and a set of events, wherein the set of objects are associated with one or more images of a user and each object from the set of objects has at least one object state triggering at least one event from the set of events in the video and each object state is associated with at least one point of an object of the set of objects;

detecting a face of the user within frames of the video, the face including a set of landmark points corresponding to facial features;

aligning a mesh with the face of the user, the mesh containing a set of feature reference points, each feature reference point corresponding to a landmark point of the set of landmark points;

detecting at least one object from the list that at least partially and at least occasionally is presented in frames of the video, the at least one object including at least a portion of the set of feature reference points, the portion of the set of feature reference points aligned with one or more points of the at least one object;

tracking the at least one object across two or more frames, the at least one object having a first object state;

identifying a change in the first object state of the at least one object to a second object state, the change from the first object state to the second object state corresponding to movement of a first portion of landmark points relative to one or more feature reference points of the mesh, indicating movement of the first portion of landmark points on the face of the user, while a second portion of landmark points remain aligned with corresponding feature reference points of the mesh;

determining that the second object state of the at least one object matches a state from the set of object states; and

in response to determining the match, triggering at least one event of the set of events in the video, the at least one event modifying the one or more images of the user by:

selecting a visualization from a plurality of visualizations associated with the at least one event; and

replacing at least a portion of the face, associated with the feature reference point within the frames of the video, with the selected visualization to modify the video, the feature reference point corresponding to the at least one point moved by the change from the first object state to the second object state.

2. The computer-implemented method of claim 1 , wherein the second object state comprises a facial gesture, and wherein the plurality of visualizations includes a plurality of animal depictions.

3. The computer-implemented method of claim 1 , wherein the set of object states comprises a plurality of facial gestures, wherein each of the plurality of facial gestures is associated with a different animal depiction of a plurality of animal depictions.

4. The computer-implemented method of claim 1 , further comprising:

tracking the at least one object and its state after triggering one event of the set of events in the video; and

removing the triggered event from the video in case of detecting change of the state of the at least one object.

5. The computer-implemented method of claim 4 , wherein the triggered event is removed from the video in case of detecting a state of the at least one object, such that the state of the at least one object corresponds to its pre-triggering state, the pre-triggering state being defined as an average state based on at least one state prior to the state that triggered the event.

6. The computer-implemented method of claim 4 , wherein the triggered event is removed from the video in case of detecting change of the state of the at least one object from a current state to another state opposite to the current state.

7. The computer-implemented method of claim 1 , further comprising:

after triggering the at least one event, removing the selected visualization from the video after an amount of time.

8. The computer-implemented method of claim 1 , wherein the objects from the list include a human's face and parts of a human body.

9. The computer implemented method of claim 1 , wherein the video comprises a real-time video stream.

10. A computer implemented method for triggering events in a video, the method being performed in connection with a computerized system comprising a processing unit and a memory, the method comprising:

providing a list comprising a set of objects, a set of object states associated with the set of objects, and a set of events, wherein the set of objects are associated with one or more images of a user and each object from the set of objects has at least one object state triggering at least one event from the set of events in the video and each object state is associated with at least one point of an object of the set of objects;

detecting a face of the user within frames of the video, the face including a set of landmark points corresponding to facial features;

aligning a mesh with the face of the user, the mesh containing a set of feature reference points, each feature reference point corresponding to a landmark point of the set of landmark points;

detecting at least one object from the list that at least partially and at least occasionally is presented in frames of the video, the at least one object including at least a portion of the set of feature reference points, the portion of the set of feature reference points aligned with one or more points of the at least one object;

tracking the at least one object having a first object state across two or more frames;

identifying a change in the first object state of the at least one object to a second object state, the change from the first object state to the second object state corresponding to movement of a first portion of landmark points relative to one or more feature reference points of the mesh, indicating movement of the first portion of landmark points on the face of the user, while a second portion of landmark points remain aligned with corresponding feature reference points of the mesh;

determining the second object state of the at least one object matches a state from the set of object states; and

in response to determining the match, triggering one random event from the list in video, the random event modifying the one or more images of the user and the mesh aligned with the face of the user by:

selecting a visualization from a plurality of visualizations associated with the random event; and

replacing at least a portion of the face, associated with at least one feature reference point of the set of feature reference points within the frames of the video, with the selected visualization to modify the video.

11. A computer-implemented method of triggering events in a video which frames comprise images of a user and one or more objects associated with the images of the user, the method being performed in connection with a computerized system comprising a processing unit and a memory, the method comprising:

providing a list comprising a set of pieces of information and a set of events, wherein each piece of information from the set of pieces of information triggers at least one event from the set of events in the video, and wherein each event from the set of events modifies an object of the one or more objects associated with the images of the user;

detecting a face of the user within frames of the video, the face including a set of landmark points corresponding to facial features;

aligning a mesh with the face of the user, the mesh containing a set of feature reference points, each feature reference point corresponding to a landmark point of the set of landmark points;

identifying a change in the first object state of the at least one object to a second object state, the change from the first object state to the second object state corresponding to movement of a first portion of landmark points relative to one or more feature reference points of the mesh, indicating movement of the first portion of landmark points on the face of the user, while a second portion of landmark points remain aligned with corresponding feature reference points of the mesh;

determining the information relating to the user matches at least one of the pieces of information from the set of pieces of information; and

in response to identifying the information, triggering at least one event from the set of events in video, the at least one event modifying at least one object of the one or more objects associated with the images of the user by:

selecting a visualization from a plurality of visualizations associated with the at least one event; and

replacing at least a portion of the face, associated with at least one feature reference point of the set of feature reference points within the frames of the video, with the selected visualization to modify the video.

12. A mobile computerized system comprising a central processing unit and a memory, the memory storing instructions for:

providing a list comprising a set of objects, a set of object states associated with the set of objects, and a set of events, wherein the set of objects are associated with one or more images of a user and each object from the set of objects has at least one object state triggering at least one event from the set of events in the video and each object state is associated with at least one point of an object of the set of objects;

detecting a face of the user within frames of the video, the face including a set of landmark points corresponding to facial features;

aligning a mesh with the face of the user, the mesh containing a set of feature reference points, each feature reference point corresponding to a landmark point of the set of landmark points;

detecting at least one object from the list that at least partially and at least occasionally is presented in frames of the video, the at least one object including at least a portion of the set of feature reference points, the portion of the set of feature reference points aligned with one or more points of the at least one object;

tracking the at least one object having a first object state across two or more frames;

identifying a change in the first object state of the at least one object to a second object state, the change from the first object state to the second object state corresponding to movement of a first portion of landmark points relative to one or more feature reference points of the mesh, indicating movement of the first portion of landmark points on the face of the user, while a second portion of landmark points remain aligned with corresponding feature reference points of the mesh;

determining the second object state of the at least one object matches a state from the set of object states; and

in response to determining the match, triggering at least one event of the set of events in the video, the at least one event modifying the one or more images of the user by:

selecting a visualization from a plurality of visualizations associated with the at least one event; and

replacing at least a portion of the face, associated with the feature reference point within the frames of the video, with the selected visualization to modify the video, the feature reference point corresponding to the at least one point moved by the change from the first object state to the second object state.

13. The mobile computerized system of claim 12 , wherein the events in the set of events are selected from a group including modification of the at least one object, modification of at least one part of the at least one object, masking of the at least one object, adding at least one new object into the frames of the video, and using visual filters for the at least one object.

14. The mobile computerized system of claim 12 , wherein the at least one event of the set of events is triggered in the video after a time delay.

15. The mobile computerized system of claim 12 , wherein the memory further stores instructions for:

tracking the at least one object and its state after triggering one event of the set of events in the video; and

removing the triggered event from the video in case of detecting change of the state of the at least one object.

16. The mobile computerized system of claim 15 , wherein the triggered event is removed from the video in case of detecting a state of the at least one object, such that the state of the at least one object corresponds to its pre-triggering state, the pre-triggering state being defined as an average state based on at least one state prior to the state that triggered the event.

17. The mobile computerized system of claim 15 , wherein the triggered event is removed from the video in case of detecting change of the state of the at least one object from the current state to another state opposite to the current state.

18. The mobile computerized system of claim 12 , wherein the memory further stores instructions for:

after triggering the at least one event, removing the selected visualization from the video after an amount of time.

Assignments (3)
MERGER Recorded Jun 15, 2015
From: AVATAR ACQUISITION CORP.
To: AVATAR MERGER SUB II, LLC.
Reel/Frame 035913/0978 →
MERGER Recorded Jun 15, 2015
From: LOOKSERY, INC.
To: AVATAR ACQUISITION CORP.
Reel/Frame 035949/0506 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 28, 2015
From: SHABUROVA, ELENA; SHABUROV, VICTOR
To: LOOKSERY INC.
Reel/Frame 035280/0604 →
Continuity (2)
Provisional Application 61936016 · Feb 5, 2014
Related Publication 20150221338A1 · Aug 6, 2015