Generating labeled training images for use in training a computational neural network for object or action recognition

1. A system for training a computational neural network to recognise objects or actions from images, the system comprising:

a training unit, comprising:

an input interface configured to receive: a plurality of images captured from one or more cameras, each image having an associated timestamp indicating a time the image was captured; and a data stream containing a plurality of timestamps each associated with an object or action, the data stream being generated by a system in an operative field of view of the one or more cameras;

an image identification unit configured to identify from the plurality of images a set of images that each have a timestamp that correlates to a timestamp associated with an object or action from the data stream;

a categorising unit configured to categorise each object into an object class of a set of one or more specified object classes using information in the data stream on that object, wherein an image label, for each image of the set of images, forms a classification label indicating a probability for each object class that the image depicts an object of that object class;

a data-labelling unit configured to determine, for each image of the set of images, the image label that indicates the probability the image depicts: (i) an object of each of the set of one or more specified object classes; or (ii) a specified human action in dependence on the correlation between the timestamp for the image and the timestamp associated with the object or action from the data stream, wherein the data-labelling unit is configured to determine the classification label for each image from the categorisation of the object having a timestamp in the data stream that correlates to the timestamp for that image; and

an output interface configured to output the image labels for use in training a computational neural network to identify from images objects of the object classes or the specified actions.

2. A system as claimed in claim 1 , wherein the image identification unit comprises a synchronising unit configured to synchronise the timestamps associated with the images with the timestamps associated with the objects or actions from the data stream.

3. A system as claimed in claim 1 , wherein each image of the set of images used to train the computational neural network has a timestamp within a specified time interval of a timestamp associated with an object or action from the data stream.

4. A system as claimed in claim 1 , wherein the image identification unit is configured to identify a set of primary images each having an associated timestamp within a specified time interval of a timestamp associated with an object or action from the data stream.

5. A system as claimed in claim 4 , wherein the data-labelling unit is configured to assign, to each primary image, a probability the image depicts: (i) an object of each of a set of one or more specified object classes; or (ii) a specified human action.

6. A system as claimed in claim 4 , wherein the image identification unit is further configured to identify, for each primary image, a subset of secondary images having a timestamp within a specified time interval of the timestamp for the primary image.

7. A system as claimed in claim 6 , wherein the data-labelling unit is further configured to assign, for each subset of secondary images, a probability to each image in the subset that the image depicts: (i) an object of each of a set of one or more specified object classes; or (ii) a specified human action determined from the probabilities assigned to the primary image for that subset.

8. A system as claimed in claim 7 , wherein the data-labelling unit is configured to assign the probabilities to each image in the subset of secondary images further from the difference between the timestamp of the secondary image and the timestamp of the primary image for the subset.

9. A system as claimed in claim 1 , wherein: the data stream contains information on a plurality of objects and timestamps associated with each object; each of the plurality of objects is depicted in at least one image of the plurality of received images; and the set of identified images each have timestamps that correlate to timestamps associated with an object from the data stream.

10. A system as claimed in claim 1 , wherein the image identification unit is configured to identify a set of primary images each having an associated timestamp within a specified time interval of a timestamp associated with an object or action from the data stream, wherein the data-labelling unit is configured to assign, to each primary image, a first probability that the image depicts an object of the object class into which the object, having a timestamp within the specified time interval of the image, has been categorised.

11. A system as claimed in claim 10 , wherein the image identification unit is further configured to identify, for each primary image, a subset of secondary images having a timestamp within a specified time interval of the timestamp for the primary image, wherein the data-labelling unit is configured to, for each subset of secondary images, assign to each image in the subset a probability that the image depicts an object of the same object class as the primary image for that subset.

12. A system as claimed in claim 1 , wherein the system further comprises an object-recognition unit adapted to use a computational neural network to identify objects from images and configured to: receive the set of images and the image labels output from the training unit; and use the set of images and their image labels to train the computational neural network to identify from images objects belonging to the object classes.

13. A system as claimed in claim 1 , wherein the image label for each image in the set of images comprises a value indicating a probability that the image depicts a person performing the specified action.

14. A system as claimed in claim 13 , wherein the image identification unit is configured to identify a set of primary images each having an associated timestamp within a specified time interval of a timestamp associated with an object or action from the data stream, wherein the data-labelling unit is configured to assign, to each primary image, a first probability that the image depicts a person performing a specified action.

15. A system as claimed in claim 14 , wherein the system further comprises an image-data labelling unit adapted to use a computational neural network to identify actions depicted in images and configured to: receive the set of images and the image labels output from the training unit; and use the set of images and their image labels to train the computational neural network to identify from images the specified actions.

16. A system as claimed in claim 1 , wherein the system comprises a data-generating system configured to generate the data stream, wherein the data-generating system is in an operative field of view of the one or more cameras.

17. A method of training a computational neural network used by an object or action recognition system, the method comprising:

receiving from one or more cameras a plurality of images each having an associated timestamp indicating a time the image was captured;

receiving a data stream containing a plurality of timestamps each associated with an object or action, the data stream being generated by a system in an operative field of view of the one or more cameras;

identifying from the plurality of images a set of images that each have a timestamp that correlates to a timestamp associated with an object or action from the data stream;

categorising each object into an object class of a set of one or more specified object classes using information in the data stream on that object, wherein an image label, for each image of the set of images, forms a classification label indicating a probability for each object class that the image depicts an object of that object class; and

determining, for each image of the set of images, an image label that indicates the probability the image depicts: (i) an object of each of the set of one or more specified object classes; or (ii) a specified human action in dependence on the correlation between the timestamp for the image and the timestamp associated with the object or action from the data stream, wherein the classification label is determined for each image from the categorisation of the object having a timestamp in the data stream that correlates to the timestamp for that image.

18. A non-transitory computer-readable storage medium having stored thereon program instructions that, when executed at a computer system, cause the computer system to perform a method of training a computational neural network to identify from images objects or actions, the method comprising:

receiving from one or more cameras a plurality of images each having an associated timestamp indicating a time the image was captured;

receiving a data stream containing a plurality of timing indications each associated with an object or action;

identifying from the plurality of images a set of images that each have a timestamp that correlates to a timestamp associated with an object or action from the data stream;

categorising each object into an object class of a set of one or more specified object classes using information in the data stream on that object, wherein an image label, for each image of the set of images, forms a classification label indicating a probability for each object class that the image depicts an object of that object class; and

determining, for each image of the set of images, an image label that indicates the probability the image depicts: (i) an object of each of the set of one or more specified object classes; or (ii) a specified human action in dependence on the correlation between the timestamp for the image and the timestamp associated with the object or action from the data stream, wherein the classification label is determined for each image from the categorisation of the object having a timestamp in the data stream that correlates to the timestamp for that image.