Search apparatus, training apparatus, search method, training method, and program
A search apparatus for searching media data for a target region that matches query data includes a first feature extraction unit configured to extract a first feature vector from the query data using a first trained neural network; a second feature extraction unit configured to obtain a first region from the media data and extract a second feature vector from the first region using a second trained neural network; a localization unit configured to determine a candidate for the target region using a third trained neural network, based on the first feature vector, the second feature vector, and the first region or a location of the first region; and a control unit configured to repeat the operations of the second feature extraction unit and the localization unit until a predetermined condition is satisfied, by using the determined candidate for the target region as the first region to be used by the second feature extraction unit.
1. An apparatus, comprising:
a processor; and
a memory that stores instructions, that when executed by the processor, cause the processor to
extract a first feature vector from query data by a first neural network;
obtain a first region from media data and extract a second feature vector from the first region by a second neural network;
determine a candidate for a target region by a third neural network, based on the first feature vector, the second feature vector, and the first region or a location of the first region;
repeat the operation of obtaining the first region and the operation of determining the candidate for the target region until a predetermined condition is satisfied, by using the determined candidate for the target region as the first region to be used in the operation of obtaining the first region, wherein the third neural network has been trained to minimize a number of executing the repeat operation according to reinforcement learning, and the candidate for the target region depends at least on a previously determined candidate during a previous iteration of the repeat operation;
determine whether the determined candidate for the target region captures the target region, and update parameters of the first neural network, the second neural network, and the third neural network, based on a result of the determination of the candidate capturing the target region;
extracting result media data from the media data according to the determined candidate for the target region to remove; and
presenting the result media data as a result of a query according to the query data.
2. The apparatus as claimed in claim 1 , wherein the first neural network, the second neural network, and the third neural network are trained using training query data and training media data based on a result of determining whether the candidate for the target region determined captures the target region, for the first feature vector extracted from the training query data and the first region obtained from the training media data.
3. The apparatus as claimed in claim 1 , wherein a parameter of the first neural network is the same as a parameter of the second neural network.
4. The apparatus as claimed in claim 1 , wherein the instructions further cause the processor to
down-sample the media data into down-sampled media data and obtain the first region based on the down-sampled media data using a fourth neural network.
5. The apparatus as claimed in claim 1 , wherein the updating parameters of the first neural network, the second neural network, and the third neural network, causes an increase of similarity between the first feature vector and the second feature vector.
6. The apparatus as claimed in claim 4 , wherein a parameter of the fourth neural network is updated together with the parameters of the first neural network, the second neural network, and the third neural network, based on the result of determining whether the candidate for the target region captures the target region.
7. A computer-implemented method, comprising:
extracting a first feature vector from query data by a first neural network;
obtaining a first region from media data and extracting a second feature vector from the first region by a second neural network;
determining a candidate for a target region using a third neural network, based on the first feature vector, the second feature vector, and the first region or a location of the first region;
repeating the obtaining step and the determining step until a predetermined condition is satisfied, by using the determined candidate for the target region as the first region to be used in the obtaining step, wherein the third neural network has been trained to minimize a number of executing the repeat operation according to reinforcement learning, and the candidate for the target region depends at least on a previously determined candidate during a previous iteration of the repeat operation;
determine whether the determined candidate for the target region captures the target region, and update parameters of the first neural network, the second neural network, and the third neural network, based on a result of the determination of the candidate capturing the target region;
extracting result media data from the media data according to the determined candidate for the target region to remove; and
presenting the result media data as a result of a query according to the query data.
8. The computer-implemented method according to claim 7 ,
wherein the updating parameters of the first neural network, the second neural network, and the third neural network causes an increase of similarity between the first feature vector and the second feature vector.
9. A non-transitory computer-readable storage medium that stores therein a program that causes a computer to function as an apparatus as claimed in claim 1 .