Object classification and related applications based on frame and event camera processing

1. A system, comprising:

circuitry configured to:

acquire an output of image sensor circuitry;

determine a first object classification result based on feature points associated with one or more first objects in a first frame of the acquired output;

execute an event-camera signal processing operation on at least two frames of the acquired output to generate an event frame, wherein

the generated event frame comprises pixel-level movement information associated with one or more second objects;

determine a second object classification result based on the pixel-level movement information;

determine whether there is a difference between the determined first object classification result and the determined second object classification result;

determine one or more object classification tags based on the determination that there is no difference between the determined first object classification result and the determined second object classification result, wherein

the determined one or more object classification tags correspond to one or more objects included in at least the one or more first objects and the one or more second objects;

determine, based on the determination that there is the difference between the determined first object classification result and the determined second object classification result, an errored object from one of the one or more first objects or the one or more second objects, wherein

the determined errored object is one of misclassified, unclassified, detected but unclassified, mis-detected, or mis-detected and mis-classified;

execute a pattern matching operation for the determined errored object based on the determined difference, the determined first object classification result, and the determined second object classification result; and

classify the determined errored object into a first object class based on the execution of the pattern matching operation.

2. The system according to claim 1 , wherein

the one or more first objects are in one of a stationary state or are about to reach the stationary state, and

the one or more second objects are in motion.

3. The system according to claim 1 , wherein the circuitry is further configured to:

detect the feature points in the first frame of the acquired output; and

determine a first pattern match between the detected feature points with first reference features associated with known object classes, wherein

the first object classification result is determined based on the determined first pattern match.

4. The system according to claim 1 , wherein the circuitry is further configured to determine the second object classification result based on pattern matching the pixel-level movement information with second reference features associated with known object classes.

5. The system according to claim 1 , wherein

the first object classification result comprises the first frame overlaid with first bounding box information associated with the one or more first objects, and

the second object classification result comprises the event frame overlaid with second bounding box information associated with the one or more second objects.

6. The system according to claim 1 , wherein the circuitry is further configured to determine, for the errored object, a first object classification tag of the one or more object classification tags based on a determination that the first object class is correct.

7. The system according to claim 1 , wherein the pattern matching operation is one of a machine learning-based pattern matching operation or a deep learning-based pattern matching operation.

8. The system according to claim 1 , wherein the circuitry is further configured to:

transmit one of the event frame or the first frame to a server, based on a determination that the first object class is incorrect;

receive a second object class for the determined errored object from the server based on the transmission; and

determine, for the errored object, a first object classification tag of the one or more object classification tags based on a determination that the second object class is correct.

9. The system according to claim 1 , wherein each of the first frame and the event frame capture scene information of an environment outside a vehicle.

10. The system according to claim 9 , wherein the circuitry is further configured to:

control an electronic control system of the vehicle to detect an event comprising one of an accident or a near miss in the environment; and

determine a behavior pattern of a driver of the vehicle based on the determined one or more object classification tags and the detected event.

11. The system according to claim 10 , wherein the electronic control system is one of Advanced Driver Assistance System (ADAS) or Autonomous Driving (AD) system.

12. The system according to claim 10 , wherein the circuitry is further configured to control one or more operations of the vehicle based on the determined behavior pattern.

13. The system according to claim 10 , wherein the circuitry is further configured to:

generate notification information based on the determined behavior pattern; and

transmit the generated notification information to an electronic device which is associated with one of the vehicle or the driver of the vehicle.

14. The system according to claim 1 , wherein the circuitry is further configured to:

control, inside a vehicle, an image projecting device to project an image of a user interface (UI) onto a projection surface inside the vehicle, wherein

each of the first frame and the event frame capture scene information of an in-vehicle environment inside the vehicle;

detect a finger movement over a virtual UI element of the UI included in the projected image, based on a second object classification tag of the determined one or more object classification tags;

select the virtual UI element based on the detection; and

control one or more vehicle functions based on the selection.

15. A method, comprising:

acquiring an output of image sensor circuitry;

determining a first object classification result based on feature points associated with one or more first objects in a first frame of the acquired output;

executing an event-camera signal processing operation on at least two frames of the acquired output to generate an event frame, wherein

the generated event frame comprises pixel-level movement information associated with one or more second objects;

determining a second object classification result based on the pixel-level movement information;

determining whether there is a difference between the determined first object classification result and the determined second object classification result;

determining one or more object classification tags based on the determination that there is no difference between the determined first object classification result and the determined second object classification result, wherein

the determined one or more object classification tags correspond to one or more objects included in at least the one or more first objects and the one or more second objects;

determining, based on the determination that there is the difference between the determined first object classification result and the determined second object classification result, an errored object from one of the one or more first objects or the one or more second objects, wherein

the determined errored object is one of misclassified, unclassified, detected but unclassified, mis-detected, or mis-detected and mis-classified;

executing a pattern matching operation for the determined errored object based on the determined difference, the determined first object classification result, and the determined second object classification result; and

classifying the determined errored object into a first object class based on the execution of the pattern matching operation.

16. The method according to claim 15 , further comprising:

detecting the feature points in the first frame of the acquired output; and

determining a first pattern match between the detected feature points with first reference features associated with known object classes, wherein

the first object classification result is determined based on the determined first pattern match.

17. The method according to claim 15 , further comprising determining the second object classification result based on pattern matching the pixel-level movement information with second reference features associated with known object classes.

18. A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by a system, causes the system to execute operations, the operations comprising:

acquiring an output of image sensor circuitry;

determining a first object classification result based on feature points associated with one or more first objects in a first frame of the acquired output;

executing an event-camera signal processing operation on at least two frames of the acquired output to generate an event frame, wherein

the generated event frame comprises pixel-level movement information associated with one or more second objects;

determining a second object classification result based on the pixel-level movement information;

determining whether there is a difference between the determined first object classification result and the determined second object classification result;

determining one or more object classification tags based on the determination that there is no difference between the determined first object classification result and the determined second object classification result, wherein

the determined one or more object classification tags correspond to one or more objects included in at least the one or more first objects and the one or more second objects;

determining, based on the determination that there is the difference between the determined first object classification result and the determined second object classification result, an errored object from one of the one or more first objects or the one or more second objects, wherein

the determined errored object is one of misclassified, unclassified, detected but unclassified, mis-detected, or mis-detected and mis-classified;

executing a pattern matching operation for the determined errored object based on the determined difference, the determined first object classification result, and the determined second object classification result; and

classifying the determined errored object into a first object class based on the execution of the pattern matching operation.