Extracting features from sensor data
A computer implemented method of training an encoder to extract features from sensor data comprises training a machine learning (ML) system based on a self-supervised loss function applied to a training set, the ML system comprising the encoder. The training set comprises first data representations and corresponding second data representations, wherein the encoder extracts features from each first and second data representation, and wherein the self-supervised loss function encourages the ML system to associate each first data representation with its corresponding second data representation based on their respective features. Each first data representation and its corresponding second data representation represent a common set of sensor data, and at least the second data representation is generated by: applying a 2D object detector to an image other than the first and second data representations, wherein the image contains or is associated with the common set of sensor data, and transforming the common set of sensor data based one or more objects detected in the image, the second data representation representing the transformed sensor data.
1 . A computer implemented method comprising:
training a machine learning (ML) system based on a self-supervised loss function applied to a training set, the ML system comprising an encoder, wherein the training set comprises first data representations and corresponding second data representations, wherein the encoder extracts features from each first and second data representation, and wherein the self-supervised loss function is configured to cause the ML system to learn feature representations such that each first data representation is more similar to its corresponding second data representation, based on their respective features, extracted by the encoder, than to non-corresponding data representations, wherein each first data representation and its corresponding second data representation represent a common set of sensor data; and
generating at least the second data representation, including
applying a two-dimensional (2D) object detector to an image other than the first and second data representations, wherein the image contains or is associated with the common set of sensor data, and
transforming the common set of sensor data based on one or more objects detected in the image, the common set of sensor data comprising point cloud data or other non-image data, the second data representation representing the transformed common set of sensor data, wherein the common set of sensor data comprises a point cloud encoded in a depth channel of the image and thus represented in a 2D image plane of the image, wherein the first and second data representations represent the point cloud in a 2D plane other than the 2D image plane of the image, and wherein the 2D plane is a bird's-eye view plane lying substantially perpendicular to the 2D image plane.
2 . The method of claim 1 , wherein the first and second data representations are discretised image representation of the point cloud in the 2D plane that optionally include respective height channels.
3 . The method of claim 1 , wherein the common set of sensor data comprises the point cloud encoded in the depth channel of the image and thus represented in the 2D image plane of the image, wherein the first and second data representations represent the point cloud in three-dimensional (3D) space.
4 . The method of claim 3 , wherein the first and second data representations are discretised voxel representations of the point cloud in the 3D space, or non-discretised representations of the point cloud in the 3D space.
5 . The method of claim 1 , wherein the image has been captured substantially simultaneously with the common set of sensor data, the sensor data of a non-image modality; wherein each detected object is matched with a corresponding subset of the common set of sensor data in order to transform the common set of sensor data.
6 . The method of claim 5 , wherein the common set of sensor data comprises a point cloud not encoded in the image.
7 . The method of claim 6 , wherein the point cloud has the non-image modality.
8 . The method of claim 1 , wherein the common set of sensor data is transformed by removing or distorting background sensor data that does not belong to any detected object.
9 . The method of claim 8 , wherein the 2D object detector computes a 2D bounding box for each detected object, wherein the background sensor data is identified as sensor data contained in or associated with a background region of the image outside of any 2D bounding box.
10 . The method of claim 8 , wherein the background sensor data is fully or partially removed and replaced with random noise.
11 . The method of claim 1 , wherein the ML system comprises a trainable projection component which projects the features from a feature space into a projection space, the self-supervised loss defined on the projected features, wherein the trainable projection component is trained simultaneously with the encoder.
12 . The method of claim 1 , wherein each set of sensor data captures a static or dynamic driving scene.
13 . The method of claim 1 , wherein the common set of sensor data comprises:
three-dimensional (3D) spatial data, or
2D spatial data in the 2D plane other than an image plane of the image.
14 . The method of claim 1 , wherein the 2D object detector is a trained machine learning (ML) 2D object detector, whereby knowledge learned in the training of the 2D ML object detector is transferred to the encoder during the training based on the self-supervised loss function.
15 . A computer system comprising:
at least one memory configured to store computer-readable instructions;
at least one hardware processor coupled to the at least one memory and configured to execute the computer-readable instructions, which upon execution cause the at least one hardware processor to perform operations including:
training a machine learning (ML) system based on a self-supervised loss function applied to a training set, the ML system comprising an encoder, wherein the training set comprises first data representations and corresponding second data representations, wherein the encoder is configured to extract features from each first and second data representation, and wherein the self-supervised loss function is configured to cause the ML system to learn feature representations such that each first data representation is more similar to its corresponding second data representation, based on their respective features extracted by the encoder, than to non-corresponding data representations, wherein each first data representation and its corresponding second data representation represent a common set of sensor data; and
generating at least the second data representation, including
applying a two-dimensional (2D) object detector to an image other than the first and second data representations, wherein the image contains or is associated with the common set of sensor data, and
transforming the common set of sensor data based one or more objects detected in the image, the common set of sensor data comprising point cloud data or other non-image data, the second data representation representing the transformed common set of sensor data, wherein the common set of sensor data comprises a point cloud encoded in a depth channel of the image and thus represented in a 2D image plane of the image, wherein the first and second data representations represent the point cloud in a 2D plane other than the image plane of the image, and wherein the 2D plane is a bird's-eye view plane lying substantially perpendicular to the 2D image plane.
16 . The computer system of claim 15 , wherein the at least one hardware processor is configured to implement a perception component, wherein the encoder is configured to receive an input sensor data representation and extract features therefrom, and wherein the perception component is configured to use the extracted features to interpret the input sensor data representation.
17 . A non-transitory medium embodying computer-readable instructions configured, when executed on one or more hardware processors, to cause the one or more hardware processors to perform operations including:
training a machine learning (ML) system based on a self-supervised loss function applied to a training set, the ML system comprising an encoder, wherein the training set comprises first data representations and corresponding second data representations, wherein the encoder extracts features from each first and second data representation, and wherein the self-supervised loss function encourages the ML system to associate each first data representation with its corresponding second data representation based on their respective features, wherein each first data representation and its corresponding second data representation represent a common set of sensor data; and
generating at least the second data representation including
applying a two-dimensional (2D) object detector to an image other than the first and second data representations, wherein the image contains or is associated with the common set of sensor data, and
transforming the common set of sensor data based one or more objects detected in the image, the common set of sensor data comprising point cloud data or other non-image data, the second data representation representing the transformed common set of sensor data, wherein the common set of sensor data comprises a point cloud encoded in a depth channel of the image and thus represented in a 2D image plane of the image, wherein the first and second data representations represent the point cloud in a 2D plane other than the image plane of the image, and wherein the 2D plane is a bird's-eye view plane lying substantially perpendicular to the 2D image plane.