Wearable device, method, and non-transitory computer-readable storage medium for setting depth value of pixels
A wearable device includes a display assembly comprising one or more displays, cameras, sensors, memory comprising one or more storage media and storing instructions, and at least one processor comprising processing circuitry. The instructions, when executed by the at least one processor individually or collectively, cause the wearable device to obtain an image, identify depth values of pixels using obtained depth information, identify the number of pixels, identify a reference depth value among the depth values of the pixels, identify a first portion of the pixels, identify a second portion of the pixels, and based on setting the depth values of the first portion of the pixels to the reference depth value and based on setting depth values of the second portion of the pixels to a predetermined depth value, display, via the display assembly, a screen corresponding to the image.
1 . A wearable device comprising:
a display assembly comprising one or more displays;
one or more cameras;
one or more sensors;
memory, comprising one or more storage media, configured to store instructions; and
at least one processor comprising processing circuitry,
wherein the instructions, when executed by the at least one processor individually or collectively, cause the wearable device to:
obtain an image via the one or more cameras,
identify depth values of pixels in the image using depth information obtained via at least one of the one or more cameras or the one or more sensors,
identify the number of one or more pixels having each of the depth values of the pixels,
identify a reference depth value using a depth value associated with the number corresponding to a peak value among the depth values,
identify a first portion of the pixels having first depth values that are within a reference range with respect to the reference depth value,
identify a second portion of the pixels that are different from the first portion of the pixels, and
based on setting the first depth values of the first portion of the pixels to the reference depth value and setting second depth values of the second portion of the pixels to a predetermined depth value, display, via the display assembly, a screen corresponding to the image having the first portion of the pixels and the second portion of the pixels.
2 . The wearable device of claim 1 , wherein the instructions, when executed by the at least one processor individually or collectively, further cause the wearable device to:
identify another reference depth value using another depth value associated to the number corresponding to another peak value among the depth values of the pixels in the image,
identify a third portion of the pixels that are different from the first portion of the pixels, wherein the third portion of the pixels has third depth values that are within the reference range with respect to the other reference depth value, and
based on setting the first depth values of the first portion of the pixels to the reference depth value, setting the second depth values of the second portion of the pixels to the predetermined depth value, and setting the third depth values of the third portion of the pixels to the other reference depth value, display, via the display assembly, a screen corresponding to the image having the first portion of the pixels, the second portion of the pixels, and the third portion of the pixels.
3 . The wearable device of claim 1 , further comprising one or more other sensors configured to obtain eye tracking data regarding a gaze of one or more eyes,
wherein the predetermined depth value is a first predetermined depth value, and
wherein the instructions, when executed by the at least one processor individually or collectively, further cause the wearable device to:
determine a foveated area of the image and a peripheral area of the image surrounding the foveated area of the image, using the eye tracking data obtained using the one or more other sensors,
identify the first portion of the pixels located within the foveated area of the image, wherein the first portion of the pixels has the first depth values that are within the reference range with respect to the reference depth value,
identify the second portion of the pixels located within the foveated area of the image, wherein the second portion of the pixels has the second depth values outside the reference range with respect to the reference depth value,
identify a third portion of the pixels located within the peripheral area of the image, and
based on setting the first depth values of the first portion of the pixels to the reference depth value, setting the second depth values of the second portion of the pixels to the first predetermined depth value, and setting third depth values of the third portion of the pixels to a second predetermined depth value, display, via the display assembly, a screen corresponding to the image having the first portion of the pixels, the second portion of the pixels, and the third portion of the pixels.
4 . The wearable device of claim 1 , further comprising:
one or more other sensors configured to obtain eye tracking data regarding a gaze of one or more eyes,
wherein the predetermined depth value is a first predetermined depth value, and
wherein the instructions, when executed by the at least one processor individually or collectively, further cause the wearable device to:
determine a foveated area of the image and a peripheral area of the image surrounding the foveated area of the image, using the eye tracking data obtained using the one or more other sensors,
identify the first portion of the pixels located within the foveated area of the image, wherein the first portion of the pixels has the first depth values that are outside another reference range and are within the reference range with respect to the reference depth value,
identify the second portion of the pixels located within the foveated area of the image, wherein the second portion of the pixels has the second depth values that are outside the other reference range and are outside the reference range with respect to the reference depth value,
identify a third portion of the pixels located within the foveated area of the image, wherein the third portion of the pixels has third depth values that are within the other reference range,
identify a fourth portion of the pixels located within the peripheral area of the image, and
based on setting the first depth values of the first portion of the pixels to the reference depth value, setting the second depth values of the second portion of the pixels to the first predetermined depth value, and setting fourth depth values of the fourth portion of the pixels to the second predetermined depth value, display, via the display assembly, a screen corresponding to the image having the first portion of the pixels, the second portion of the pixels and the fourth portion of the pixels, and
wherein depth values of an area within the screen corresponding to the third portion of the pixels correspond to the third depth values that are within the other reference range of the third portion of the pixels.
5 . The wearable device of claim 4 , wherein the one or more sensors comprise a direct time of flight (D-TOF) sensor and an indirect time of flight (I-TOF) sensor,
wherein the area within the screen comprises a first partial area and a second partial area,
wherein the first partial area in the area within the screen have depth values corresponding to depth values identified using the D-TOF sensor among the D-TOF sensor and the I-TOF sensor, and
wherein the second partial area in the area within the screen have depth values corresponding to depth values identified using the I-TOF sensor among the D-TOF sensor and the I-TOF sensor.
6 . The wearable device of claim 1 , wherein the predetermined depth value is obtained using an average value of the depth values of the pixels in the image.
7 . The wearable device of claim 1 , wherein the instructions, when executed by the at least one processor individually or collectively, further cause the wearable device to identify the reference depth value, using the depth value associated with the number and a difference between a position of the one or more cameras attached to the wearable device and a position of eyes of a user wearing the wearable device.
8 . A wearable device comprising:
a display assembly, including one or more displays;
one or more cameras;
one or more sensors;
one or more other sensors configured to obtain eye tracking data regarding a gaze of one or more eyes;
memory, comprising one or more storage media, configured to store instructions; and
at least one processor comprising processing circuitry,
wherein the instructions, when executed by the at least one processor individually or collectively, cause the wearable device to:
obtain an image via the one or more cameras,
determine a foveated area of the image and a peripheral area of the image surrounding the foveated area of the image, using the eye tracking data obtained using the one or more other sensors,
identify depth values of pixels within the foveated area of the image using depth information obtained via at least one of the one or more cameras or the one or more sensors,
identify the number of one or more pixels having each of the depth values of the pixels within the foveated area of the image,
identify a reference depth value using a depth value associated with the number corresponding to a peak value among the depth values,
identify a first portion of the pixels within the foveated area of the image having first depth values that are within a reference range with respect to the reference depth value,
identify a second portion of the pixels within the foveated area of the image that are different from the first portion of the pixels within the foveated area of the image, and
based on setting the first depth values of the first portion of the pixels within the foveated area of the image to the reference depth value, setting second depth values of the second portion of the pixels within the foveated area of the image to a first predetermined depth value, and setting depth values of pixels within the peripheral area of the image to a second predetermined depth value, display, via the display assembly, a screen corresponding to the image having the first portion of the pixels, the second portion of the pixels, and the pixels within the peripheral area.
9 . The wearable device of claim 8 , wherein the instructions, when executed by the at least one processor individually or collectively, further cause the wearable device to:
identify another reference depth value using another depth value associated to the number corresponding to another peak value among the depth values of the pixels within the foveated area of the image,
identify a third portion of the pixels within the foveated area of the image that are different from the first portion of the pixels within the foveated area of the image, wherein the third portion of the pixels within the foveated area of the image has third depth values that are within the reference range with respect to the other reference depth value, and
based on setting the first depth values of the first portion of the pixel within the foveated area of the image to the reference depth value, setting the second depth values of the second portion of the pixels within the foveated area of the image to the first predetermined depth value, setting the third depth values of the third portion of the pixels within the foveated area of the image to the other reference depth value, and setting the depth values of the pixels within the peripheral area of the image to the second predetermined depth value, display, via the display assembly, a screen corresponding to the image having the first portion of the pixels, the second portion of the pixels, the third portion of the pixels, and the pixels within the peripheral area.
10 . The wearable device of claim 8 , wherein the instructions, when executed by the at least one processor individually or collectively, further cause the wearable device to:
identify the first portion of the pixels within the foveated area of the image, wherein the first portion of the pixels has the first depth values that are outside another reference range and are within the reference range with respect to the reference depth value,
identify the second portion of the pixels within the foveated area of the image, wherein the second portion of the pixels has the second depth values that are outside the other reference range and are outside the reference range with respect to the reference depth value,
identify a third portion of the pixels within the foveated area of the image, wherein the third portion of the pixels has third depth values that are within the other reference range, and
based on setting the first depth values of the first portion of the pixels within the foveated area of the image to the reference depth value, setting the second depth values of the second portion of the pixels within the foveated area of the image to the predetermined depth value, and setting the depth values of the pixels within the peripheral area of the image to the second predetermined depth value, display, via the display assembly, a screen corresponding to the image having the first portion of the pixels, the second portion of the pixels, and the pixels within the peripheral area, and
wherein depth values of an area within the screen corresponding to the third portion of the pixels within the foveated area of the image correspond to the third depth values that are within the other reference range of the third portion of the pixels within the foveated area of the image.
11 . The wearable device of claim 10 , wherein the one or more sensors comprise a direct time of flight (D-TOF) sensor and an indirect time of flight (I-TOF) sensor,
wherein the area within the screen comprises a first partial area and a second partial area,
wherein the first partial area in the area within the screen have depth values corresponding to depth values identified using the D-TOF sensor among the D-TOF sensor and the I-TOF sensor, and
wherein the second partial area in the area within the screen have depth values corresponding to depth values identified using the I-TOF sensor among the D-TOF sensor and the I-TOF sensor.
12 . The wearable device of claim 8 , wherein the first predetermined depth value is obtained using an average value of the depth values of the pixels within the foveated area of the image.
13 . The wearable device of claim 8 , wherein the second predetermined depth value is obtained using an average value of the depth values of the pixels within the peripheral area of the image.
14 . The wearable device of claim 8 , wherein the instructions, when executed by the at least one processor individually or collectively, further cause the wearable device to:
identify the reference depth value, using the depth value associated with the number and a difference between a position of the one or more cameras attached to the wearable device and a position of eyes of a user wearing the wearable device.
15 . A wearable device comprising:
a display assembly including one or more displays;
one or more cameras;
one or more sensors;
memory, comprising one or more storage media, configured to store instructions; and
at least one processor comprising processing circuitry,
wherein the instructions, when executed by the at least one processor individually or collectively, cause the wearable device to:
obtain an image via the one or more cameras,
identify an object within the image,
identify first depth values of a first portion of pixels within an area within the image occupied by the object, using depth information obtained via at least one of the one or more cameras or the one or more sensors,
identify a reference depth value using at least portion of the first depth values of the first portion of the pixels,
identify a second portion of the pixels that are different from the first portion of the pixels, and
based on setting the first depth values of the first portion of the pixels to the reference depth value and setting second depth values of the second portion of the pixels to a predetermined depth value, display, via the display assembly, a screen corresponding to the image having the first portion of the pixels and the second portion of the pixels.
16 . The wearable device of claim 15 , wherein the reference depth value is a first reference depth value,
wherein the instructions, when executed by the at least one processor individually or collectively, further cause the wearable device to:
identify another object within the image,
identify third depth values of a third portion of the pixels within an area within the image occupied by the other object, using depth information,
identify a second reference depth value using the third depth values of the third portion of the pixels,
based on setting the first depth values of the first portion of the pixels to the first reference depth value, setting the second depth values of the second portion of the pixels to the predetermined depth value, and setting the third depth values of the third portion of the pixels to the second reference depth value, display, via the display assembly, a screen corresponding to the image having the first portion of the pixels, the second portion of the pixels, and the third portion of the pixels.
17 . The wearable device of claim 15 , wherein the instructions, when executed by the at least one processor individually or collectively, further cause the wearable device to:
identify the number of one or more pixels within the area of the image, wherein the one or more pixels have each of the depth values of the first portion of the pixels,
identify the reference depth value using a depth value associated with the number corresponding to a peak value among the depth values of the first portion of the pixels, and
based on setting the first depth values of the first portion of the pixels to the reference depth value and setting the second depth values of the second portion of the pixels to the predetermined depth value, display, via the display assembly, a screen corresponding to the image having the first portion of the pixels and the second portion of the pixels.
18 . The wearable device of claim 15 , further comprising:
one or more other sensors configured to obtain eye tracking data regarding a gaze of one or more eyes,
wherein the reference depth value is a first reference depth value, and
wherein the instructions, when executed by the at least one processor individually or collectively, further cause the wearable device to:
determine a foveated area of the image and a peripheral area of the image surrounding the foveated area of the image, using the eye tracking data obtained using the one or more other sensors,
based on identifying that the object within the image is located within the foveated area within the image:
identify depth values of the foveated area of the image,
identify the number of one or more pixels within the foveated area of the image, wherein the one or more pixels have each of the depth values of the foveated area of the image,
identify a second reference depth value using a depth value associated with the number corresponding to a peak value among the depth values of the foveated area of the image, and
based on setting the depth values of the foveated area of the image to the second reference depth value and setting depth values of the peripheral area of the image to the predetermined depth value, display, via the display assembly, a screen corresponding to the image, and
based on identifying that at least a portion of the object within the image is located within the peripheral area within the image, based on setting the depth values of the first portion of the pixels to the first reference depth value and setting the depth values of the second portion of the pixels to the predetermined depth value, display, via the display assembly, a screen corresponding to the image.
19 . The wearable device of claim 15 , wherein the reference depth value corresponds to an average value of the depth values of the first portion of the pixels.
20 . The wearable device of claim 15 , wherein the instructions, when executed by the at least one processor individually or collectively, further cause the wearable device to:
identify the reference depth value, using at least portion of the depth values of the first portion of the pixels and a difference between a position of the one or more cameras attached to the wearable device and a position of eyes of a user wearing the wearable device.