World-space segmentation

1. A computer-implemented method comprising:

processing a plurality of image frames obtained from at least one image capture device to establish a world-space map;

determining a segmentation for a first image frame of the plurality of image frames;

iteratively repeating the determination of the segmentation for each image frame following the first image frame until determining a sufficiency condition is met;

based on determining that the sufficiency condition is met, ceasing execution of determination of segmentation of each frame following the first image frame and continuing to use a last determined segmentation in subsequent iterations of image frames in the plurality of image frames; and

generating user interface data for display using the world-space map and the segmentation.

2. The computer-implemented method of claim 1 , wherein generating user interface data for display comprises:

generating a real-time virtual effect in screen-space using the world-space map and the segmentation; and

displaying the real-time virtual effect in a current image.

3. The computer-implemented method of claim 1 further comprising:

initializing a segmentation texture for storing segmentation information prior to determining the segmentation.

4. The computer-implemented method of claim 3 , wherein the segmentation information relates to a fixed area around at least one image capture device.

5. The computer-implemented method of claim 3 , wherein the segmentation texture is a two-dimensional data structure, each entry in the segmentation texture including at least one value indicating a likelihood that a corresponding pixel in an image frame fulfils a segmentation criterion.

6. The computer-implemented method of claim 5 , wherein the segmentation texture is a ground/floor segmentation texture and the segmentation criterion is a criterion for assigning a pixel to a ground/floor object class.

7. The computer-implemented method of claim 1 , wherein the world-space map is established by applying a surface tracker to the plurality of image frames.

8. The computer-implemented method of claim 1 , wherein determining the segmentation comprises:

obtaining segmentation in screen-space from at least one image frame of the plurality of image frames;

projecting the screen-space segmentation into world-space; and

integrating the projected screen-space segmentation into the world-space map using a temporal filter.

9. The computer-implemented method of claim 8 , wherein segmentation is obtained in screen space by executing a semantic segmentation neural network with the at least one image frame of the plurality of image frames as input.

10. The computer-implemented method of claim 8 , wherein determining the segmentation further comprises:

storing the screen-space segmentation in a segmentation texture.

11. The computer-implemented method of claim 8 , wherein the screen-space segmentation is projected into world-space by storing the projected screen-space segmentation in a temporary world-space segmentation texture and then integrating the temporary world-space segmentation texture from this frame into the world-space map.

12. The computer-implemented method of claim 8 , wherein the temporal filter is an infinite impulse response (IIR) filter or a finite impulse response (FIR) filter.

13. The computer-implemented method of claim 1 , further comprising:

running at least one world-space post-processing filter.

14. The computer-implemented method of claim 1 , wherein the sufficiency condition is a requirement that a variance for a pixel is less than a variance threshold.

15. The computer-implemented method of claim 1 , wherein the plurality of image frames are obtained from a forward and rear camera of at least one image capture device.

16. A computing apparatus, the computing apparatus comprising:

a processor; and

a memory storing instructions that, when executed by the processor, configure the computing apparatus to perform operations comprising:

processing a plurality of image frames obtained from at least one image capture device to establish a world-space map;

determining a segmentation for a first image frame of the plurality of image frames;

iteratively repeating the determination of the segmentation for each image frame following the first image frame until determining a sufficiency condition is met;

based on determining that the sufficiency condition is met, ceasing execution of determination of segmentation of each frame following the first image frame and continuing to use a last determined segmentation in subsequent iterations of image frames in the plurality of image frames; and

generating user interface data for display using the world-space map and the segmentation.

17. The computing apparatus of claim 16 , wherein generating user interface data for display comprises:

generating a real-time virtual effect in screen-space using the world-space map and the segmentation; and

displaying the real-time virtual effect in a current image.

18. The computing apparatus of claim 16 , the operations further comprising:

initializing a segmentation texture for storing segmentation information prior to determining the segmentation, wherein the segmentation information relates to a fixed area around at least one image capture device.

19. The computing apparatus of claim 16 , the operations further comprising:

initializing a segmentation texture for storing segmentation information prior to determining the segmentation,

wherein the segmentation texture is a two-dimensional data structure, each entry in the segmentation texture including at least one value indicating a likelihood that a corresponding pixel in an image frame fulfils a segmentation criterion, or

wherein the segmentation texture is a ground/floor segmentation texture and the segmentation criterion is a criterion for assigning a pixel to a ground/floor object class.

20. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:

processing a plurality of image frames obtained from at least one image capture device to establish a world-space map;

determining a segmentation for a first image frame of the plurality of image frames;

iteratively repeating the determination of the segmentation for each image frame following the first image frame until determining a sufficiency condition is met;

based on determining that the sufficiency condition is met, ceasing execution of determination of segmentation of each frame following the first image frame and continuing to use a last determined segmentation in subsequent iterations of image frames in the plurality of image frames; and

generating user interface data for display using the world-space map and the segmentation.