Electronic apparatus for real-time human detection and tracking system and controlling method thereof
An electronic apparatus is provided. The electronic apparatus includes a first sensor configured to obtain a color image, a second sensor configured to obtain a depth image, a memory storing a neural network model, and a processor configured to, based on a first color image being received from the first sensor, obtain a first region of interest by inputting the first color image to the neural network model, and identify whether a distance between an object included in the first region of interest and the electronic apparatus is less than a threshold distance.
1. An electronic apparatus comprising:
a first sensor configured to obtain a color image;
a second sensor configured to obtain a depth image;
memory storing a neural network model and instructions; and
one or more processors communicatively coupled to the first sensor, second sensor and the memory,
wherein the instructions, when executed by the one or more processors individually or collectively, cause the electronic apparatus to:
based on a first color image being obtained via the first sensor, obtain a first region of interest from the first color image by inputting the first color image to the neural network model,
identify whether a distance between an object included in the first region of interest and the electronic apparatus is less than a threshold distance,
based on the identified distance being less than the threshold distance, identify a first region including pixels including depth information less than the threshold distance among a plurality of pixels included in a first depth image corresponding to the first color image,
obtain intersection information between the first region of interest and the first region,
based on the obtained intersection information being a threshold value or more, identify that the first region includes the object and merge the first region of interest and the first region to obtain a merged region, and
track a position of the object based on the merged region.
2. The electronic apparatus of claim 1 , wherein the instructions that, when executed by the one or more processors individually or collectively, further cause the electronic apparatus to:
identify a second region including pixels including depth information less than the threshold distance in a second depth image corresponding to a second color image received from the second sensor,
obtain second intersection information between the first region and the second region,
based on the obtained second intersection information being the threshold value or more, identify that the second region includes the object, and
track the position of the object based on the first region and the second region.
3. The electronic apparatus of claim 2 , wherein the instructions that, when executed by the one or more processors individually or collectively, further cause the electronic apparatus to:
input a second color image received from the first sensor to the neural network model, and
based on a region of interest being not identified in the second color image based on an output of the neural network model, identify the second region including the pixels including the depth information less than the threshold distance in the second depth image corresponding to the second color image.
4. The electronic apparatus of claim 2 , wherein the instructions that, when executed by the one or more processors individually or collectively, further cause the electronic apparatus to:
based on a proportion of pixels including the depth information less than the threshold distance in the second depth image being less than a threshold proportion, obtain a second region of interest including the object from the first color image by inputting the second color image to the neural network model,
identify whether a distance between the object included in the second region of interest and the electronic apparatus is less than a threshold distance,
based on the identified distance being less than the threshold distance, identify the second region including the pixels including the depth information less than the threshold distance among a plurality of pixels included in the second depth image,
obtain third intersection information between the second region of interest and the second region, and
based on the obtained third intersection information being a threshold value or more, identify that the second region includes the object.
5. The electronic apparatus of claim 4 , wherein the instructions that, when executed by the one or more processors individually or collectively, further cause the electronic apparatus to, based on the identified distance being the threshold distance or more, track the position of the object based on the first region of interest, the first region, and the second region of interest.
6. The electronic apparatus of claim 1 , wherein the instructions that, when executed by the one or more processors individually or collectively, further cause the electronic apparatus to:
based on the obtained intersection information being the threshold value or more, obtain a first merged region in which the first region of interest and the first region are merged,
identify a second region including pixels including depth information less than the threshold distance in a second depth image corresponding to a second color image received from the second sensor,
obtain second intersection information between the first merged region and the second region,
based on the obtained second intersection information being the threshold value or more, identify that the second region includes the object, and
track the position of the object based on the first merged region and the second region.
7. The electronic apparatus of claim 6 , wherein the instructions that, when executed by the one or more processors individually or collectively, further cause the electronic apparatus to, based on a proportion of pixels including depth information less than the threshold distance in the second depth image being a threshold proportion or more, identify the second region including pixels including depth information less than the threshold distance among a plurality of pixels included in the second depth image.
8. The electronic apparatus of claim 1 , wherein the processor is instructions that, when executed by the one or more processors individually or collectively, further cause the electronic apparatus to, based on the identified distance being the threshold distance or more, identify the position of the object based on the first region of interest.
9. The electronic apparatus of claim 1 ,
wherein the first sensor comprises at least one of a camera or a red, green, and blue (RGB) color sensor, and
wherein the second sensor comprises at least one of a stereo vision sensor, a Time-of-Flight (ToF) sensor, or a light detection and ranging (LiDAR) sensor.
10. A method an electronic apparatus, the method comprising:
based on a first color image being obtained via a first sensor, obtaining, by the electronic apparatus, a first region of interest from the first color image by inputting the first color image to a neural network model; and
identifying, by the electronic apparatus, whether a distance between an object included in the first region of interest and the electronic apparatus is less than a threshold distance,
wherein the identifying comprises:
based on the identified distance being less than the threshold distance, identifying, by the electronic apparatus, a first region including pixels including depth information less than the threshold distance among a plurality of pixels included in a first depth image corresponding to the first color image,
obtaining, by the electronic apparatus, intersection information between the first region of interest and the first region,
based on the obtained intersection information being a threshold value or more, identifying, by the electronic apparatus, that the first region includes the object and merging the first region of interest and the first region to obtain a merged region, and
tracking a position of the object based on the merged region.
11. The method of claim 10 , further comprising:
identifying a second region including pixels including depth information less than the threshold distance in a second depth image corresponding to a second color image received from a second sensor;
obtaining second intersection information between the first region and the second region;
based on the obtained second intersection information being the threshold value or more, identifying that the second region includes the object; and
tracking the position of the object based on the first region and the second region.
12. The method of claim 11 , further comprising:
inputting a second color image received from the first sensor to the neural network model,
wherein the identifying of the second region comprises, based on a region of interest being not identified in the second color image based on an output of the neural network model, identifying the second region including the pixels including the depth information less than the threshold distance in the second depth image corresponding to the second color image.
13. The method of claim 11 , further comprising:
based on a proportion of pixels including the depth information less than the threshold distance in the second depth image being less than a threshold proportion, obtaining a second region of interest including the object from the first color image by inputting the second color image to the neural network model; and
identifying whether a distance between the object included in the second region of interest and the electronic apparatus is less than a threshold distance,
wherein the identifying of the second region comprises, based on the identified distance being less than the threshold distance, identifying the second region including the pixels including the depth information less than the threshold distance among a plurality of pixels included in the second depth image, and
wherein the method further comprises:
obtaining intersection information between the second region of interest and the second region, and
based on the obtained intersection information being a threshold value or more, identifying that the second region includes the object.
14. The method of claim 11 , wherein the neural network model comprises at least one of: a trained convolution neural network (CNN), a deep neural network (DNN) model of a Recurrent Neural Network (RNN), a Long Short-Term Memory Network (LSTM), a Gated Recurrent Units (GRU), or a Generative Adversarial Networks (GAN).
15. The method of claim 11 , wherein the neural network model comprises a neural network model trained to identify, based on the first color image, a region predicted to include the object in the first color image as the first region of interest.
16. The method of claim 15 , wherein the neural network model comprises a neural network model trained to identify a region in which a pixel value is changed by comparing the first color image preceding in chronological sequence and a following second color image, and determine the identified region as the first region of interest predicted to include the object.
17. The method of claim 10 , further comprising:
based on the obtained intersection information being the threshold value or more, obtaining a first merged region in which the first region of interest and the first region are merged;
identifying a second region including pixels including depth information less than the threshold distance in a second depth image corresponding to a second color image received from a second sensor;
obtaining third intersection information between the first merged region and the second region;
based on the obtained third intersection information being the threshold value or more, identifying that the second region includes the object; and
tracking the position of the object based on the first merged region and the second region.
18. The method of claim 17 , wherein the identifying of the second region comprises, based on a proportion of pixels including depth information less than the threshold distance in the second depth image being a threshold proportion or more, identifying the second region including pixels including depth information less than the threshold distance among a plurality of pixels included in the second depth image.