Real-time system for generating 4D spatio-temporal model of a real world environment

1 . A method for deriving a 3D data from image data, comprising:

receiving, from at least one camera, image data comprising a time-ordered sequence of frames, the time-ordered sequence of frames comprising frames collected from each camera of the at least one cameras, the image data representing an environment;

detecting, from the image data, at least one object within the environment comprising detecting said at least one object in each frame of the time-ordered sequence of frames;

classifying the at least one detected object comprising classifying the at least one detected object in each frame of the sequence of frames, wherein the method comprises, for each classified object of the classified at least one objects:

determining a 2D skeleton of the classified object by implementing a neural network to identify features of the classified object in the image data corresponding to the classified object; and

constructing a 3D skeleton for the classified object, comprising a step of mapping the determined 2D skeleton to 3D, wherein that step comprises:

determining a plurality of 3D skeletons to form a time-varying 3D skeleton, and

applying a recursive estimator across the time-ordered sequence of frames to determine a time-varying 3D skeleton for the classified object, wherein a derived 3D skeleton at a first time instance is used as a further input to the derivation of a second 3D skeleton at a second time instance.

2 . The method of claim 1 , wherein each of said at least one object comprises a plurality of associated sub-objects, and wherein, for each of said at least one object:

detecting at least one object comprises detecting each of the plurality of associated sub-objects;

classifying at least one object comprises classifying each of the plurality of associated sub-objects;

determining the 2D skeleton of the classified object comprises identifying features of each of the plurality of classified sub-objects in the image data corresponding to the classified sub-objects.

3 . The method of claim 1 , wherein, for each classified object, mapping the determined 2D skeleton to 3D comprises:

implementation of a neural network; and/or

applying statistical and/or probabilistic methods to the determined 2D skeleton and applying holonomic constraints to the statistical and/or probabilistic methods.

4 . The method of claim 3 , wherein classifying the at least one detected object comprises classifying a first detected object of the at least one detected object as a human object, and, when statistical and/or probabilistic methods are applied to the determined 2D skeleton, the holonomic constraints comprise human anatomical holonomic constraints.

5 . The method of claim 3 , wherein the 3D skeleton comprises an anchor point and a plurality of child points, each child point is defined by applying a transformation to the anchor point or another child point of the plurality of child points, wherein the holonomic constraints define the degrees of freedom of the transformation defining each child point, and/or the range of possible values for each degree of freedom of the transformation.

6 . The method of claim 1 , wherein classifying the at least one detected object comprises classifying a second detected object of the at least one detected object as a non-human object.

7 . The method claim 1 , wherein the at least one camera comprises a plurality of cameras;

wherein detecting at least one object comprises detecting said at least one object in image data collected from each camera;

classifying the at least one object comprises classifying said at least one object detected in the image data from each camera;

determining the 2D skeleton for each classified object comprises determining a plurality of 2D skeletons, each of said plurality of 2D skeletons determined from identifying features of the classified object in image data collected from a different camera of the plurality of cameras; and

wherein constructing the 3D skeleton comprises combining the determined plurality of determined 2D skeletons.

8 . The method of claim 1 , wherein the application of recursive estimators comprises applying time-dependent holonomic constraints.

9 . The method of claim 8 , wherein:

the 3D skeleton comprises an anchor point and a plurality of child points, each child point is defined by applying a transformation to the anchor point or another child point of the plurality of child points;

the time-varying 3D skeleton is defined by time-variable transformations for each child point, and the time-dependent holonomic constraints define, for each point in time, the degrees of freedom of transformation defining each child point and/or the range of possible values for each degree of freedom of the transformation.

10 . The method of claim 1 , wherein the at least one camera comprises at least one first type of camera and at least one second type of camera, wherein each of the first type of camera captures image data at a first framerate, and each second type of camera captures image data at a second framerate that is different to the first framerate.

11 . The method of claim 1 , further comprising constructing, from the image data, a 3D model of the environment.

12 . The method of claim 11 , wherein the 3D environment model is constructed by

applying Simultaneous Localisation and Mapping techniques to the image data to estimate a position and an orientation of each camera of the at least one camera and to construct an environment map; and

constructing the 3D model by mapping the environment map to a pre-determined 3D model of the environment.

13 . The method of claim 11 further comprising integrating the time-varying 3D skeleton with the 3D environment model to construct a time-varying integrated environment.

14 . The method of claim 11 , further comprising:

for each classified object of the classified at least one objects: constructing a 3D avatar for the classified object comprising integrating the constructed 3D skeleton with a 3D model corresponding to the classified object, and

integrating the determined 3D environment model with the constructed 3D avatar for each of the classified objects to construct a 3D model of an integrated environment, said integrated environment comprising said environment and said objects within the environment.

15 . The method of claim 14 , further comprising refining the construction of the integrated environment by applying time-dependent smoothing to the constructed 3D avatar, using filtering methods.

16 . The method of claim 14 , further comprising refining the construction of the integrated environment by constraining interactions between each constructed 3D avatar and the 3D environment model.

17 . The method of claims 14 , wherein constructing a 3D avatar for each of the classified objects comprises constructing a plurality of 3D avatars, and the method further comprises refining the construction of the integrated environment by constraining interactions between the plurality of 3D avatars.

18 . The method of claim 1 , the method further comprising, for each classified object of the classified at least one objects:

constructing a 3D avatar for the classified object comprising integrating the constructed 3D skeleton with a 3D model corresponding to the classified object.

19 . The method of claim 18 , wherein the method further comprises capturing the 3D model corresponding to the classified object at a higher resolution than the resolution of the image data.

20 . The method of claim 1 , wherein the image data comprising a time-ordered sequence of frames is received from only a single camera.

21 . The method of claim 1 , wherein, for each classified object, mapping the determined 2D skeleton to 3D is based on holonomic constraints using a hidden Markov model.

22 . The method of claim 21 , wherein the hidden Markov model uses data from a single frame.

23 . The method of claim 22 , wherein in the hidden Markov model, a measurement vector Yt of the classified object is obtained from the single frame, from which a vector Xt of joints of the human skeleton in 3D space is determined.

24 . A computer comprising a processor configured to carry out the method of claim 1 .

25 . A non-transitory computer readable medium comprising executable instructions configured to cause a processor to carry out the method of claim 1 .