Computer-implemented method, data processing apparatus and computer program for object detection
A computer-implemented method of training an object detector, the method comprising: training an embedding neural network using, as an input, cropped images from an image dataset, wherein training the embedding neural network is performed using a self-supervised learning approach and the trained embedding neural network translates input images into a lower dimensional representation; and training an object detector neural network by, for images of the image dataset, repeatedly: passing an image through the object detector neural network to obtain proposed coordinates of an object within the image, cropping the image to the proposed coordinates to obtain a cropped image, passing the cropped image through the trained embedding neural network to obtain a cropped image representation, passing an exemplar through the trained embedding neural network to obtain an exemplar representation, wherein the exemplar is a cropped manually labelled image bounding a known object, computing a distance in embedding space between the cropped image representation and the exemplar representation, computing a gradient of the cropped image representation and the exemplar representation with respect to the distance, and passing the gradient into the object detector neural network for use in backpropagation to optimise the object detector neural network.
1 . A computer-implemented method of training an object detector, the method comprising:
training an embedding neural network using, as an input, cropped images from an image dataset, wherein training the embedding neural network is performed using a self-supervised learning approach and the trained embedding neural network translates input images into a lower dimensional representation; and
training an object detector neural network by, for images of the image dataset, repeatedly:
passing an image through the object detector neural network to obtain proposed coordinates of an object within the image,
cropping the image to the proposed coordinates to obtain a cropped image,
passing the cropped image through the trained embedding neural network to obtain a cropped image representation,
passing an exemplar through the trained embedding neural network to obtain an exemplar representation, wherein the exemplar is a cropped manually labelled image bounding a known object,
computing a distance in embedding space between the cropped image representation and the exemplar representation,
computing a gradient of the cropped image representation and the exemplar representation with respect to the distance using a finite difference method, wherein the finite difference method includes:
cropping the image to the proposed coordinates with a shift to obtain a shifted cropped image,
passing the shifted cropped image through the trained embedding neural network to obtain a shifted cropped image representation,
computing a second distance in embedding space between the shifted cropped image representation and the exemplar representation, and
computing the gradient as the difference between the distance and the second distance, and
passing the gradient into the object detector neural network for use in backpropagation to optimise the object detector neural network.
2 . The method of claim 1 , further comprising: optimising the object detector neural network by minimising the distance between the cropped image representation and the exemplar representation using the gradient in backpropagation.
3 . The method of claim 2 , wherein optimising the object detector neural network comprises minimising a distance-based loss function for each cropped image representation of the images and the exemplar representation.
4 . The method of claim 3 , wherein the distance-based loss function corresponds to a sum of L 1 loss and focal loss for each cropped image representation and the exemplar representation.
5 . The method of claim 1 , wherein training the object detector neural network further comprises scaling each cropped image such that all scaled cropped images are of the same size.
6 . The method of claim 5 , further comprising scaling the exemplar such that the scaled exemplar is the same size as the scaled cropped images.
7 . The method of claim 1 , wherein the method uses a plurality of exemplars for repeatedly training the object detector neural network, the method further comprising:
obtaining an exemplar representation for each exemplar, and
computing the distance and the gradient for each cropped image with respect to each exemplar representation.
8 . The method of claim 7 , wherein the method uses at least the same number of exemplars as there are classes of objects to be detected.
9 . The method of claim 1 , further comprising randomly initializing weights of a target embedding neural network, the target embedding neural network comprising the same structure as the embedding neural network, and wherein training the embedding neural network comprises, for images of the image dataset, repeatedly:
augmenting a cropped image to generate a first augmented view and a second augmented view;
passing the first augmented view through the embedding neural network to obtain a lower dimensional representation of the first augmented view;
passing the second augmented view through the target embedding network to obtain a lower dimensional representation of the second augmented view;
minimising a similarity loss between the embedding neural network and the target embedding network using stochastic gradient descent optimisation with respect to the weights of the embedding neural network.
10 . The method of claim 9 , wherein the stochastic gradient descent optimisation comprises updating the weights of the target embedding neural network as a moving average of the weights of the embedding neural network.
11 . The method of claim 9 , wherein augmenting the cropped image comprises applying at least one of the following augmentations:
colour jittering;
greyscale conversion;
Gaussian blurring;
horizontal flipping;
vertical flipping; and
random crop and resizing, optionally wherein
augmenting the cropped image comprises probabilistically applying a plurality of augmentations to the cropped image, each augmentation applied with a corresponding probability.
12 . The method of claim 1 , wherein the method is for detecting an object in an image enhancement or analysis process.
13 . The method of claim 12 , wherein the method is for detecting an object in an autonomous vehicle image analysis process.
14 . The method of claim 12 , wherein the method is for detecting an object in a railway mapping image analysis process.
15 . A computer-implemented method of object detection, the method comprising:
training an embedding neural network using, as an input, cropped images from an image dataset, wherein training the embedding neural network is performed using a self-supervised learning approach and the trained embedding neural network translates input images into a lower dimensional representation; and
training an object detector neural network by, for images of the image dataset, repeatedly:
passing an image through the object detector neural network to obtain proposed coordinates of an object within the image,
cropping the image to the proposed coordinates to obtain a cropped image,
passing the cropped image through the trained embedding neural network to obtain a cropped image representation,
passing an exemplar through the trained embedding neural network to obtain an exemplar representation, wherein the exemplar is a cropped manually labelled image bounding a known object,
computing a distance in embedding space between the cropped image representation and the exemplar representation,
computing a gradient of the cropped image representation and the exemplar representation with respect to the distance using a finite difference method, wherein the finite difference method includes:
cropping the image to the proposed coordinates with a shift to obtain a shifted cropped image,
passing the shifted cropped image through the trained embedding neural network to obtain a shifted cropped image representation,
computing a second distance in embedding space between the shifted cropped image representation and the exemplar representation, and
computing the gradient as the difference between the distance and the second distance, and
passing the gradient into the object detector neural network for use in backpropagation to optimise the object detector neural network;
receiving an input image;
passing the input image into the trained object detector neural network; and
outputting coordinates and object class of any objects detected within the input image.
16 . A data processing apparatus comprising a memory and a processor, the memory comprising instructions which, when executed by the processor:
train an embedding neural network using, as an input, cropped images from an image dataset, wherein training the embedding neural network is performed using a self-supervised learning approach and the trained embedding neural network translates input images into a lower dimensional representation; and
train an object detector neural network by, for images of the image dataset, repeatedly:
passing an image through the object detector neural network to obtain proposed coordinates of an object within the image,
cropping the image to the proposed coordinates to obtain a cropped image,
passing the cropped image through the trained embedding neural network to obtain a cropped image representation,
passing an exemplar through the trained embedding neural network to obtain an exemplar representation, wherein the exemplar is a cropped manually labelled image bounding a known object,
computing a distance in embedding space between the cropped image representation and the exemplar representation,
computing a gradient of the cropped image representation and the exemplar representation with respect to the distance using a finite difference method, wherein the finite difference method includes:
cropping the image to the proposed coordinates with a shift to obtain a shifted cropped image,
passing the shifted cropped image through the trained embedding neural network to obtain a shifted cropped image representation,
computing a second distance in embedding space between the shifted cropped image representation and the exemplar representation, and
computing the gradient as the difference between the distance and the second distance, and
passing the gradient into the object detector neural network for use in backpropagation to optimise the object detector neural network.
17 . A non-transitory computer readable medium having a computer program stored thereon, wherein the computer program, when executed by a computer, cause the computer to:
train an embedding neural network using, as an input, cropped images from an image dataset, wherein training the embedding neural network is performed using a self-supervised learning approach and the trained embedding neural network translates input images into a lower dimensional representation; and
train an object detector neural network by, for images of the image dataset, repeatedly:
passing an image through the object detector neural network to obtain proposed coordinates of an object within the image,
cropping the image to the proposed coordinates to obtain a cropped image,
passing the cropped image through the trained embedding neural network to obtain a cropped image representation,
passing an exemplar through the trained embedding neural network to obtain an exemplar representation, wherein the exemplar is a cropped manually labelled image bounding a known object,
computing a distance in embedding space between the cropped image representation and the exemplar representation,
computing a gradient of the cropped image representation and the exemplar representation with respect to the distance using a finite difference method, wherein the finite difference method includes:
cropping the image to the proposed coordinates with a shift to obtain a shifted cropped image,
passing the shifted cropped image through the trained embedding neural network to obtain a shifted cropped image representation,
computing a second distance in embedding space between the shifted cropped image representation and the exemplar representation, and
computing the gradient as the difference between the distance and the second distance, and
passing the gradient into the object detector neural network for use in backpropagation to optimise the object detector neural network.
18 . A computer-implemented method of training an object detector, the method comprising:
training an embedding neural network using, as an input, cropped images from an image dataset, wherein training the embedding neural network is performed using a self-supervised learning approach and the trained embedding neural network translates input images into a lower dimensional representation; and
training an object detector neural network by, for images of the image dataset, repeatedly:
passing an image through the object detector neural network to obtain proposed coordinates of an object within the image,
cropping the image to the proposed coordinates to obtain a cropped image,
passing the cropped image through the trained embedding neural network to obtain a cropped image representation,
passing a plurality of exemplars through the trained embedding neural network to obtain a corresponding plurality of exemplar representations, wherein each exemplar is a cropped manually labelled image bounding a known object, and wherein the method uses at least the same number of exemplars as there are classes of objects to be detected,
computing a distance in embedding space between the cropped image representation and the plurality of exemplar representations,
computing a gradient of the cropped image representation and the exemplar representations with respect to the distance, and
passing the gradient into the object detector neural network for use in backpropagation to optimise the object detector neural network.
19 . A computer-implemented method of training an object detector, the method comprising:
training an embedding neural network using, as an input, cropped images from an image dataset, wherein training the embedding neural network is performed using a self-supervised learning approach and the trained embedding neural network translates input images into a lower dimensional representation; and
training an object detector neural network by, for images of the image dataset, repeatedly:
passing an image through the object detector neural network to obtain proposed coordinates of an object within the image,
cropping the image to the proposed coordinates to obtain a cropped image,
passing the cropped image through the trained embedding neural network to obtain a cropped image representation,
passing an exemplar through the trained embedding neural network to obtain an exemplar representation, wherein the exemplar is a cropped manually labelled image bounding a known object,
computing a distance in embedding space between the cropped image representation and the exemplar representation,
computing a gradient of the cropped image representation and the exemplar representation with respect to the distance, and
passing the gradient into the object detector neural network for use in backpropagation to optimise the object detector neural network,
the method further comprising randomly initializing weights of a target embedding neural network, the target embedding neural network comprising the same structure as the embedding neural network, and wherein training the embedding neural network comprises, for images of the image dataset, repeatedly:
augmenting a cropped image to generate a first augmented view and a second augmented view;
passing the first augmented view through the embedding neural network to obtain a lower dimensional representation of the first augmented view;
passing the second augmented view through the target embedding network to obtain a lower dimensional representation of the second augmented view; and
minimising a similarity loss between the embedding neural network and the target embedding network using stochastic gradient descent optimisation with respect to the weights of the embedding neural network.
20 . The method of claim 19 , wherein the stochastic gradient descent optimisation comprises updating the weights of the target embedding neural network as a moving average of the weights of the embedding neural network.