Object detection and tracking
A method and a computing device for object detection and tracking from a video input are described. The method and the computing device may be used to, for example, track objects of interest, such as lane markings, in traffic. A plurality of frames corresponding to a video may be analyzed in a spatiotemporal domain by a neural network. The neural network may be trained using data synthesized in the spatiotemporal domain.
1 . A method, comprising:
obtaining a plurality of frames, corresponding to a video, comprising features of interest;
forming, based on the plurality of frames comprising features of interest, a spatiotemporal data volume, wherein two dimensions of the spatiotemporal data volume correspond to spatial dimensions of the plurality of frames, and wherein one dimension of the spatiotemporal data volume corresponds to a temporal dimension of the plurality of frames;
slicing the spatiotemporal data volume along a plurality of surfaces, producing a plurality of spatiotemporal images, wherein each spatiotemporal image in the plurality of spatiotemporal images corresponds to the spatiotemporal data volume along a corresponding surface in the plurality of surfaces;
enhancing the features of interest, of the plurality of frames, in the plurality of spatiotemporal images using a neural network, thereby producing enhanced features of interest of a processed plurality of spatiotemporal images of the plurality of frames, wherein the features of interest comprise at least one geometric shape in the plurality of spatiotemporal images, and wherein the enhancing the features of interest in the plurality of spatiotemporal images to produce the enhanced features of interest comprises performing at least one operation taken from the group consisting of:
connecting disconnected parts of the at least one geometrical shape in the plurality of spatiotemporal images;
extracting the at least one geometrical shape in the plurality of spatiotemporal images; and
classifying the at least one geometrical shape in the plurality of spatiotemporal images; and
projecting the enhanced features of interest of the processed plurality of spatiotemporal images onto the plurality of frames.
2 . The method according to claim 1 , wherein the obtaining the plurality of frames that comprises features of interest comprises:
obtaining a plurality of input frames corresponding to the video; and
performing feature extraction on the plurality of input frames, thereby producing the plurality of frames and the features of interest in the plurality of frames.
3 . The method according to claim 1 , wherein the features of interest correspond to objects of interest in vehicle traffic.
4 . The method according to claim 3 , wherein the objects of interest in vehicle traffic comprise at least one object taken from the group consisting of:
a lane marking;
a segment of road; and
an object in traffic to be tracked.
5 . The method according to claim 1 , wherein the neural network comprises a convolutional neural network.
6 . The method according to claim 1 , wherein the neural network is configured to be trained using synthesized spatiotemporal images comprising synthesized features of interest.
7 . A non-transitory computer readable medium comprising program code configured to perform a method comprising:
obtaining a plurality of frames, corresponding to a video, comprising features of interest;
forming, based on the plurality of frames comprising features of interest, a spatiotemporal data volume, wherein two dimensions of the spatiotemporal data volume correspond to spatial dimensions of the plurality of frames, and wherein one dimension of the spatiotemporal data volume corresponds to a temporal dimension of the plurality of frames;
slicing the spatiotemporal data volume along a plurality of surfaces, producing a plurality of spatiotemporal images, wherein each spatiotemporal image in the plurality of spatiotemporal images corresponds to the spatiotemporal data volume along a corresponding surface in the plurality of surfaces;
enhancing the features of interest, of the plurality of frames, in the plurality of spatiotemporal images using a neural network, thereby producing enhanced features of interest of a processed plurality of spatiotemporal images of the plurality of frames, wherein the features of interest comprise at least one geometric shape in the plurality of spatiotemporal images, and wherein the enhancing the features of interest in the plurality of spatiotemporal images to produce the enhanced features of interest comprises performing at least one operation taken from the group consisting of:
connecting disconnected parts of the at least one geometrical shape in the plurality of spatiotemporal images;
extracting the at least one geometrical shape in the plurality of spatiotemporal images; and
classifying the at least one geometrical shape in the plurality of spatiotemporal images; and
projecting the enhanced features of interest of the processed plurality of spatiotemporal images onto the plurality of frames.
8 . The non-transitory computer-readable medium according to claim 7 , wherein the obtaining the plurality of frames that comprises features of interest comprises:
obtaining a plurality of input frames corresponding to the video; and
performing feature extraction on the plurality of input frames, thereby producing the plurality of frames and the features of interest in the plurality of frames.
9 . The non-transitory computer-readable medium according to claim 7 , wherein the features of interest correspond to objects of interest in vehicle traffic.
10 . The non-transitory computer-readable medium according to claim 9 , wherein the objects of interest in vehicle traffic comprise at least one object taken from the group consisting of:
a lane marking;
a segment of road; and
an object in traffic to be tracked.
11 . The non-transitory computer-readable medium according to claim 7 , wherein the neural network comprises a convolutional neural network.
12 . The non-transitory computer-readable medium according to claim 7 , wherein the neural network is configured to be trained using synthesized spatiotemporal images comprising synthesized features of interest.
13 . A computing device comprising:
a processor; and
a non-transitory computer-readable medium including computer-executable instructions that, when executed by the processor, facilitate carrying out a method comprising:
obtaining a plurality of frames, corresponding to a video, comprising features of interest;
forming, based on the plurality of frames comprising features of interest, a spatiotemporal data volume, wherein two dimensions of the spatiotemporal data volume correspond to spatial dimensions of the plurality of frames, and wherein one dimension of the spatiotemporal data volume corresponds to a temporal dimension of the plurality of frames;
slicing the spatiotemporal data volume along a plurality of surfaces, producing a plurality of spatiotemporal images, wherein each spatiotemporal image in the plurality of spatiotemporal images corresponds to the spatiotemporal data volume along a corresponding surface in the plurality of surfaces;
enhancing the features of interest, of the plurality of frames, in the plurality of spatiotemporal images using a neural network, thereby producing enhanced features of interest of a processed plurality of spatiotemporal images of the plurality of frames, wherein the features of interest comprise at least one geometric shape in the plurality of spatiotemporal images, and wherein the enhancing the features of interest in the plurality of spatiotemporal images to produce the enhanced features of interest comprises performing at least one operation taken from the group consisting of:
connecting disconnected parts of the at least one geometrical shape in the plurality of spatiotemporal images;
extracting the at least one geometrical shape in the plurality of spatiotemporal images; and
classifying the at least one geometrical shape in the plurality of spatiotemporal images; and
projecting the enhanced features of interest of the processed plurality of spatiotemporal images onto the plurality of frames.
14 . The computing device according to claim 13 , wherein the obtaining the plurality of frames that comprises features of interest comprises:
obtaining a plurality of input frames corresponding to the video; and
performing feature extraction on the plurality of input frames, thereby producing the plurality of frames and the features of interest in the plurality of frames.
15 . The computing device according to claim 13 , wherein the features of interest correspond to objects of interest in vehicle traffic.
16 . The computing device according to claim 15 , wherein the objects of interest in vehicle traffic comprise at least one object taken from the group consisting of:
a lane marking;
a segment of road; and
an object in traffic to be tracked.
17 . The computing device according to claim 13 , wherein the neural network comprises a convolutional neural network.
18 . The computing device according to claim 13 , wherein the neural network is configured to be trained using synthesized spatiotemporal images comprising synthesized features of interest.
19 . A vehicle comprising the computing device according to claim 13 .
20 . The vehicle according to claim 19 , wherein the obtaining the plurality of frames that comprises features of interest comprises:
obtaining a plurality of input frames corresponding to the video; and
performing feature extraction on the plurality of input frames, thereby producing the plurality of frames and the features of interest in the plurality of frames.