Information processing apparatus, image sensor, and control method using on-sensor image recognition
The present technology relates to an information processing apparatus and an information processing method which are capable of reducing power consumption that impairs convenience. An image of a predetermined detection target is included in a captured image captured by an image sensor, an output image output from the image sensor is acquired, predetermined processing is executed on the output image, and the output image is output from the image sensor in a case where it is detected that the image of the detection target is included.
1 . An information processing apparatus, comprising:
an image sensor configured to capture an image in an image frame, wherein the image sensor includes image recognition circuitry configured to:
execute image recognition on image data, obtained by a light receiving circuitry of the image sensor, based on an inference model having a Deep Neural Network (DNN) structure;
detect that an image of a specific detection target is included in the captured image captured by the image sensor; and
output a detection result including a detection trigger; and
processing circuitry, switchable between an activated state and a power-saving state, configured to:
in the power-saving state, cause the image sensor to operate in a sensing mode in which the image sensor executes the image recognition by the image recognition circuitry while suppressing transfer of the image data to the processing circuitry,
based on reception of the detection trigger from the image sensor,
transition from the power-saving state to the activated state, acquire an output image output from the image sensor, execute a specific processing on the output image, and
cause the image sensor to stop or change the sensing mode after completion of the specific processing.
2 . The information processing apparatus according to claim 1 , wherein
the image recognition circuitry includes
DNN processing circuitry configured to execute person detection based on the inference model, and
motion detection circuitry configured to detect a motion in the image frame based on a temporal change in the captured image, and
the image sensor is configured such that the DNN processing circuitry executes the person detection based on the motion detection by the motion detection circuitry.
3 . The information processing apparatus according to claim 2 , wherein the processing circuitry is further configured to execute a process to detect an image of a face included in the output image.
4 . The information processing apparatus according to claim 2 , wherein the processing circuitry is further configured to execute face authentication to authenticate that an image of a face included in the output image is a face of a specific person.
5 . The information processing apparatus according to claim 4 , wherein the processing circuitry is further configured to unlock an own device in a case where the face authentication succeeds.
6 . The information processing apparatus according to claim 4 , wherein the processing circuitry is further configured to enable lock of an own device in a case where the image recognition circuitry does not detect that the image of the specific person is included.
7 . The information processing apparatus according to claim 4 , wherein the processing circuitry is further configured to unlock opening and closing circuitry in a case where the face authentication succeeds.
8 . The information processing apparatus according to claim 2 , wherein the processing circuitry is further configured to detect an orientation of an image of a face included in the output image.
9 . The information processing apparatus according to claim 8 , wherein the processing circuitry is further configured to rotate an image displayed on a display screen based on the orientation of the image of the face.
10 . The information processing apparatus according to claim 9 , wherein the processing circuitry is further configured to rotate the image displayed on the display screen based on a posture of an own device in a case where the image recognition circuitry does not detect that an image of a specific person is included.
11 . The information processing apparatus according to claim 2 , wherein the processing circuitry is further configured to execute a process to prevent peeping in a case where the image recognition circuitry detects that images of a plurality of persons are included.
12 . The information processing apparatus according to claim 2 , wherein the processing circuitry is further configured to transmit the output image to a party of a video call.
13 . The information processing apparatus according to claim 1 , wherein
the image recognition circuitry is further configured to set the specific detection target as an animal, and
the processing circuitry is further configured to store the output image in a storage unit.
14 . The information processing apparatus according to claim 1 , wherein
the image recognition circuitry is further configured to set the specific detection target as at least one code of one-dimensional code or a two-dimensional code, and
the processing circuitry is further configured to execute a process corresponding to the at least one code.
15 . The information processing apparatus according to claim 14 , wherein the processing circuitry is further configured to activate an application corresponding to the at least one code.
16 . An information processing apparatus, comprising:
an image sensor configured to capture an image in an image frame, wherein the image sensor includes image recognition circuitry configured to:
execute image recognition on image data, obtained by light receiving circuitry of the image sensor, based on an inference model having a deep neural network structure;
detect whether an image of a specific detection target is included in the captured image captured by the image sensor; and
processing circuitry, switchable between an activated state and a power-saving state, configured to:
in the power-saving state, cause the image sensor to operate in a sensing mode in which the image sensor executes the image recognition by the image recognition circuitry while suppressing transfer of the image data to the processing circuitry,
execute a specific processing based on a result that is output from the image sensor and is detected by the image recognition circuitry, and
cause the image sensor to stop or change the sensing mode after completion of the specific processing.
17 . The information processing apparatus according to claim 1 , wherein the image recognition circuitry is further configured to execute a process of image recognition based on an image with lower resolution than resolution of the output image.
18 . An information processing method, comprising:
in an information processing apparatus having an image sensor:
executing, by image recognition circuitry of the image sensor, image recognition on image data, obtained by light receiving circuitry of the image sensor, based on an inference model having a deep neural network structure;
detecting, by the image recognition circuitry, that an image of a specific detection target is included in a captured image captured by the image sensor;
outputting, by the image recognition circuitry, a detection result including a detection trigger; and
based on reception of the detection trigger from the image sensor,
transitioning, by processing circuitry of the information processing apparatus, from a power-saving state to an activated state, wherein
the processing circuitry is switchable between the activated state and the power-saving state, and
in the power-saving state, the processing circuitry is configured to cause the image sensor to operate in a sensing mode in which the image sensor executes the image recognition by the image recognition circuitry while suppressing transfer of the image data to the processing circuitry,
acquiring, by the processing circuitry, an output image output from the image sensor,
executing, by the processing circuitry, a specific processing on the output image, and
causing, by the processing circuitry, the image sensor to stop or change the sensing mode after completion of the specific processing.
19 . A non-transitory computer readable medium having stored thereon, computer executable instructions, which when executed by a computer, cause the computer to execute operations:
executing, by image recognition circuitry of an image sensor included in an information processing apparatus, image recognition on image data, obtained by light receiving circuitry of the image sensor, based on an inference model having a deep neural network structure;
detecting, by the image recognition circuitry, that an image of a specific detection target is included in a captured image by the image sensor;
outputting, by the image recognition circuitry, a detection result including a detection trigger; and
based on reception of the detection trigger from the image sensor,
transitioning, by processing circuitry of the information processing apparatus, from a power-saving state to an activated state, wherein
the processing circuitry is switchable between the activated state and the power-saving state, and
in the power-saving state, the processing circuitry is configured to cause the image sensor to operate in a sensing mode in which the image sensor executes the image recognition by the image recognition circuitry while suppressing transfer of the image data to the processing circuitry,
acquiring, by the processing circuitry, an output image output from the image sensor,
executes predetermined executing, by the processing circuitry, a specific processing on the output image, and
causing, by the processing circuitry, the image sensor to stop or change the sensing mode after completion of the specific processing.