Object detection from visual search queries

1. A method of detecting an object in video and matching the object to one or more products comprising the steps of:

a) obtaining video and automatically extracting metadata and attributes of objects in frames and/or portions of frames in the video;

b) segmenting the video based on depicted settings and/or events by comparing contents of consecutive frames for similarities and differences, wherein the video is traversed sequentially to detect a pair of frames or sequence of frames which breach a similarity threshold, wherein a key frame is identified for each segment;

c) compiling segments of same or similar settings and/or events, wherein each segment is tagged with a segment identifier;

d) analysing one or more segments to detect one or more objects, wherein frames and/or portions of frames are compared with defined content in a database populated by aligning known objects and metadata clusters, wherein the metadata is linked to the frames and/or portions of frames by the segment identifier, wherein the location of the detected one or more objects are obtained in each key frame, wherein an object-wise feature vector is generated for each segment;

e) comparing the one or more objects to products;

f) identifying products associated with the one or more objects, wherein a convolutional neural network (CNN) is used;

g) notifying one or more viewers of the products;

wherein a second screen augmentation is used for live or streaming video.

2. The method of claim 1 , wherein the database is populated with defined content using a web crawler.

3. The method of claim 1 , wherein the step of notifying one or more viewers of the products includes displaying an advertisement.

4. The method of claim 1 , wherein the step of notifying one or more viewers of the products includes providing a hyperlink to a website or video.

5. A method of detecting one or more objects in a digital screen shot from a video and matching the one or more objects to promotional material comprising the steps of:

a) receiving an inquiry from a viewer in the form of a digital screen shot;

b) identifying one or more objects in the digital screen shot by comparing the digital screen shot and/or portions of the digital screen shot with defined content in a database populated by aligning known objects and metadata clusters, wherein the digital screen shot is matched with a segment identifier from the video to retrieve a list of objects linked with a segment of the video, wherein a bag of visual words approach is used and top N candidate images are filtered and added to the database;

c) comparing the one or more objects to products;

d) matching products associated with the one or more objects, wherein a convolutional neural network (CNN) is used, wherein spatial verification is performed on the matching products for verification; and

e) contacting the viewer with promotional material related to matched products;

wherein after receiving the inquiry, visual word clusters are assigned.

6. The method of claim 5 , wherein the database is populated with defined content using a web crawler.

7. The method of claim 5 , wherein second screen content augmentation is used for live or streaming video.

8. The method of claim 5 , wherein the step of contacting the viewer with promotional material related to the identified products includes displaying an advertisement and/or providing a hyperlink to a website or video.

9. A system for generating relationships between objects in video to products in a database of products comprising:

a computerized network and system to be locally or remotely exposed to a user or groups of users via a user interface application;

a module for detecting and storing media content locally or on a server;

a module for transmitting the media content to a processor, remote or server-based, for ingestion of metadata and/or visual features;

a module for transmitting the media content to a processor, remote or server-based, for extraction of metadata and visual features;

a means for receiving input from one or more users in the form of a digital image derived from the video and that includes visual features;

a module configured to implement the following:

identify the visual features in the digital image and correlate the visual features with objects and/or groups of related products in the database, wherein the database is populated with defined content by aligning known objects and/or groups of related products and metadata clusters, wherein a bag of visual words approach is used to identify visual features and correlate the visual features with objects and/or groups of related products in the database;

wherein the digital image is matched with a segment identifier to retrieve a list of objects and/or groups of related products linked with a segment of the video; and

analyse visual features and metadata of the digital image to correlate the visual features with objects and/or groups of related products using a convolutional neural network (CNN), wherein spatial verification is performed on the matching objects and/or groups of related products for verification, and

a networked service that distributes information on the objects and/or groups of related products to a user and/or groups of users;

wherein after receiving the input, visual word clusters are assigned.

10. The system of claim 9 , wherein the information on the objects and/or groups of related products includes advertisements.

11. The system of claim 9 , wherein the information on the objects and/or groups of related products includes a hyperlink or content accessible through the internet.