Method and apparatus for removing privacy sensitive objects in an augmented reality system

1. A method, comprising:

receiving, at an intermediary layer from a camera of a head mounted display (HMD), image data of a real-world operating environment of the HMD at a first time;

identifying, at the intermediary layer, one or more real-world objects in the image data of the real-world operating environment;

determining, at the intermediary layer, coordinates of a region of the image data comprising the one or more identified real-world objects;

determining a display permission for the one or more identified real-world objects;

modifying the image data in the region of the image data comprising the one or more identified real-world objects according to the display permission; and

outputting the modified image data to an application executing outside of the intermediary layer to be displayed according to the display permission.

2. The method of claim 1 , wherein the application executing outside of the intermediary layer is configured to only obtain image data of the real-world operating environment from the camera of the HMD via the intermediary layer.

3. The method of claim 1 , further comprising:

receiving, at the intermediary layer, context information of the real-world operating environment;

wherein the display permission for the one or more identified real-world objects is based at least in part on the context information.

4. The method of claim 3 , wherein the context information comprises one or more of; a location coordinate of the HMD, a combination of identified real-world objects in the image data, a network connection, or a received privacy signal.

5. The method of claim 1 , wherein determining the display permission for the one or more identified real-world objects comprises:

searching for the one or more identified real-world objects on a deny list; and

responsive to finding the one or more identified real-world objects on the deny list, obtaining the display permission associated with the one or more identified real-world objects.

6. The method of claim 1 , further comprising:

caching the modified image data of the real-world operating environment in a frame cache maintained at the intermediary layer;

receiving, at the intermediary layer from the camera, image data of the real-world operating environment of the HMD at a second time subsequent to the first time;

determining a similarity between the modified image data of the real-world operating environment in the frame cache and the image data of the real-world operating environment from the second time;

responsive to the modified image data of the real-world operating environment in the frame cache and the image data of the real-world operating environment from the second time satisfying a similarity threshold, modifying the image data of the real-world operating environment from the second time in the region comprising the one or more identified real-world objects according to the display permission; and

outputting the modified image data of the real-world operating environment from the second time to the application executing outside of the intermediary layer to be displayed according to the display permission.

7. The method of claim 6 , further comprising:

determining a change in camera perspective between the first time and the second time;

updating the coordinates of the region comprising the one or more identified real-world objects based on the change in camera perspective; and

storing the updated coordinates of the region comprising the one or more identified real-world objects in the frame cache.

8. An apparatus comprising:

at least one processor; and

at least one memory containing instructions that, when executed by the at least one processor, cause the apparatus to:

receive, at an intermediary layer from a camera of a head mounted display (HMD), image data of a real-world operating environment of the HMD at a first time,

identify, at the intermediary layer, one or more real-world objects in the image data of the real-world operating environment,

determine, at the intermediary layer, coordinates of a region of the image data comprising the one or more identified real-world objects,

determine a display permission for the one or more identified real-world objects,

modify the image data in the region of the image data comprising the one or more identified real-world objects according to the display permission, and

output the modified image data to an application executing outside of the intermediary layer to be displayed according to the display permission.

9. The apparatus of claim 8 , wherein the application executing outside of the intermediary layer is configured to only obtain image data of the real-world operating environment from the camera of the HMD via the intermediary layer.

10. The apparatus of claim 8 , wherein:

the at least one memory further contains instructions that, when executed by the at least one processor, cause the apparatus to receive, at the intermediary layer, context information of the real-world operating environment, and

the display permission for the one or more identified real-world objects is based at least in part on the context information.

11. The apparatus of claim 10 , wherein the context information comprises one or more of; a location coordinate of the HMD, a combination of identified real-world objects in the image data, a network connection, or a received privacy signal.

12. The apparatus of claim 8 , wherein the instructions that when executed cause the apparatus to determine the display permission for the one or more identified real-world objects comprise instructions that when executed cause the apparatus to:

search for the one or more identified real-world objects on a deny list, and

responsive to finding the one or more identified real-world objects on the deny list, obtain the display permission associated with the one or more identified real-world objects.

13. The apparatus of claim 8 , wherein the at least one memory further contains instructions that, when executed by the at least one processor, cause the apparatus to:

cache the modified image data of the real-world operating environment in a frame cache maintained at the intermediary layer,

receive, at the intermediary layer from the camera, image data of the real-world operating environment of the HMD at a second time subsequent to the first time,

determine a similarity between the modified image data of the real-world operating environment in the frame cache and the image data of the real-world operating environment from the second time,

responsive to the modified image data of the real-world operating environment in the frame cache and the image data of the real-world operating environment from the second time satisfying a similarity threshold, modify the image data of the real-world operating environment from the second time in the region comprising the one or more identified real-world objects according to the display permission, and

output the modified image data of the real-world operating environment from the second time to the application executing outside of the intermediary layer to be displayed according to the display permission.

14. The apparatus of claim 13 , wherein the at least one memory further contains instructions that, when executed by the at least one processor, cause the apparatus to:

determine a change in camera perspective between the first time and the second time,

update the coordinates of the region comprising the one or more identified real-world objects based on the change in camera perspective, and

store the updated coordinates of the region comprising the one or more identified real-world objects in the frame cache.

15. A non-transitory, computer-readable medium containing instructions that, when executed by at least one processor, cause an apparatus to:

receive, at an intermediary layer from a camera of a head mounted display (HMD), image data of a real-world operating environment of the HMD at a first time,

identify, at the intermediary layer, one or more real-world objects in the image data of the real-world operating environment,

determine, at the intermediary layer, coordinates of a region of the image data comprising the one or more identified real-world objects,

determine a display permission for the one or more identified real-world objects,

modify the image data in the region of the image data comprising the one or more identified real-world objects according to the display permission, and

output the modified image data to an application executing outside of the intermediary layer to be displayed according to the display permission.

16. The non-transitory, computer-readable medium of claim 15 , wherein the application executing outside of the intermediary layer is configured to only obtain image data of the real-world operating environment from the camera of the HMD via the intermediary layer.

17. The non-transitory, computer-readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the apparatus to receive, at the intermediary layer, context information of the real-world operating environment,

wherein the display permission for the one or more identified real-world objects is based at least in part on the context information.

18. The non-transitory, computer-readable medium of claim 17 , wherein the context information comprises one or more of; a location coordinate of the HMD, a combination of identified real-world objects in the image data, a network connection, or a received privacy signal.

19. The non-transitory, computer-readable medium of claim 15 , wherein the instructions that when executed cause the apparatus to determine the display permission for the one or more real-world objects comprise instructions that when executed cause the apparatus to:

search for the one or more identified real-world objects on a deny list, and

responsive to finding the one or more identified real-world objects on the deny list, obtain the display permission associated with the one or more identified real-world objects.

20. The non-transitory, computer-readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the apparatus to:

cache the modified image data of the real-world operating environment in a frame cache maintained at the intermediary layer,

receive, at the intermediary layer from the camera, image data of the real-world operating environment of the HMD at a second time subsequent to the first time,

determine a similarity between the modified image data of the real-world operating environment in the frame cache and the image data of the real-world operating environment from the second time,

responsive to the modified image data of the real-world operating environment in the frame cache and the image data of the real-world operating environment from the second time satisfying a similarity threshold, modify the image data of the real-world operating environment from the second time in the region comprising the one or more identified real-world objects according to the display permission, and

output the modified image data of the real-world operating environment from the second time to the application executing outside of the intermediary layer to be displayed according to the display permission.