Surveillance system and surveillance method using multi-dimensional sensor data

1. A surveillance method using multi-dimensional sensor data, for use in a surveillance system, wherein the surveillance system comprises a plurality of sensors installed within a scene, and the plurality of sensor are classified into a plurality of types, the surveillance method comprising:

obtaining each type of sensor data about the scene using the plurality of sensors;

performing a local-object process on each type of the sensor data to generate local-object-feature information for each type;

performing a global-object process according to the local-object-feature information for each type to generate global-object-feature information; and

performing a global-object recognition process on the global-object-feature information to generate a global-recognition result,

wherein the performing the local-object process on each type of sensor data to generate local-object-feature information comprises:

performing a local-object detection and correspondence process on each type of sensor data to obtain a local-object-identifier (LOID) list and a corresponding local-rough-feature set (LRFS) for each type;

performing a local-feature-extraction-and-fusion process on each type of sensor data according to the LOID list and the corresponding LRFS for each type to obtain a plurality of local-detail-feature sets (LDFS), and fusing the plurality of LDFSs into a local-fusion feature; and

inputting each type of the local-fusion feature into a local-object-recognition model to obtain a local-identity-identifier (LIID) list for each type.

2. The surveillance method as claimed in claim 1 , wherein the sensors comprise: cameras, microphones, taste sensors, smell sensors, tactile sensors, or a combination thereof.

3. The surveillance method as claimed in claim 1 , wherein the LIID list for each type comprises one or more local objects of each type, and each object has a corresponding local-object identifier (LOID),

wherein the LRFS comprises direction, distance, and roughness of each type of local object.

4. The surveillance method as claimed in claim 3 , wherein the local-object-feature extraction and fusion process comprises:

extracting a plurality of local-detail features to build a local-detail-feature set (LDFS) according to the LIID list and corresponding LRFS of each type; and

fusing each type of LDFS into the local-fusion feature of each local object.

5. The surveillance method as claimed in claim 1 , wherein the local-object-feature information comprises the LOID list, the LRFS, the LFF, and the LIID list for each type, and the performing the global-object process according to the local-object-feature information of each type to generate the global-object-feature information comprises:

performing a global-object-correspondence process according to the local-object-feature information to generate a global-object-identifier (GOID) list and a corresponding global-rough-feature set (GRFS); and

performing a global-detail-feature-correspondence process according to the local-object-feature information, wherein the GOID list and the corresponding GRFS generate a global-detail-feature set (GDFS) for each of one or more global objects in the GOID list.

6. The surveillance method as claimed in claim 5 , wherein the performing the global-object recognition process on the global-object-feature information to generate the global-recognition result comprises:

performing a local-context-analysis process on each type of sensor data to generate a local-context-analysis result for each type, and merging the local-context-analysis result of each type to generate a local-context-merged result;

selecting an adjacent-differential-weighting coefficient (ADWC) or an adaptive-weighting coefficient (AWC) according to the local-context-merged result; and

performing a global-detail-feature-fusion process according to the selected ADWC or AWC to perform a global-detail-feature-fusion process to generate a global-fusion feature corresponding to each global object.

7. The surveillance method as claimed in claim 6 , wherein the global-recognition result comprises a confidence level, and the surveillance method further comprises:

decomposing the global-fusion feature into a plurality of local-detail features of different types; and

feeding each type of local-detail feature and its corresponding confidence level back to the local-object-recognition model for each type.

8. The surveillance method as claimed in claim 6 , wherein the local-context-analysis process comprises:

performing a context-retrieving process on each type of sensor data to obtain a context-region for each type;

performing a context-merging process on each type of context region to obtain a merged-context region for each type;

performing the local-context-analysis process on each type of merged-context region to generate the local-context-analysis result; and

combining each of the local-context-analysis results to generate the combined local-context result.

9. The surveillance method as claimed in claim 1 , further comprising:

when time stamps, the LRFS, and world-positioning-coordinate information of the local object of each type match each other, determining that the local objects successfully match each other; and

assigning a global-object-identifier to the successfully matched local objects.

10. A surveillance system, comprising:

a plurality of sensors, wherein the plurality of sensors are classified into a plurality of types and are configured to monitor a scene to obtain sensor data of each type; and

a computation apparatus, configured to perform a local-object process on each type of sensor data to generate local-object-feature information;

wherein the computation apparatus is configured to perform a global-object process according to the local-object-feature information of each type to generate global-object-feature information, and perform a global-object recognition process on the global-object-feature information to generate a global-recognition result,

wherein the computation apparatus is further configured to:

perform a local-object detection and correspondence process on each type of sensor data to obtain a local-object-identifier (LOID) list and a corresponding local-rough-feature set (LRFS) for each type;

perform a local-feature-extraction-and-fusion process on each type of sensor data according to the LOID list and the corresponding LRFS of each type to obtain a plurality of local-detail-feature sets (LDFS), and fuse the plurality of LDFSs into a local-fusion feature; and

input each type of local-fusion feature into a local-object-recognition model to obtain a local-identity-identifier (LIID) list for each type.

11. The surveillance system as claimed in claim 10 , wherein the sensors comprise: cameras, microphones, taste sensors, smell sensors, tactile sensors, or a combination thereof.

12. The surveillance system as claimed in claim 10 , wherein the LIID list in each type comprises one or more local objects of each type, and each object has a corresponding local-object identifier (LOID),

wherein the LRFS comprises direction, distance, and roughness of each local object of each type.

13. The surveillance system as claimed in claim 12 , wherein the computation apparatus is further configured to:

extract a plurality of local-detail features to build a local-detail-feature set (LDFS) according to the LIID list and corresponding LRFS of each type, and fuse the LDFS of each type into the local-fusion feature of each local object.

14. The surveillance system as claimed in claim 10 , wherein the local-object-feature information comprises the LOID list, the LRFS, the LFF, and the LIID list in each type, and the computation apparatus is further configured to:

perform a global-object-correspondence process according to the local-object-feature information to generate a global-object-identifier (GOID) list and a corresponding global-rough-feature set (GRFS), and perform a global-detail-feature-correspondence process according to the local-object-feature information, and the GOID list and corresponding GRFS generate a global-detail-feature set (GDFS) for each of one or more global objects in the GOID list.

15. The surveillance system as claimed in claim 14 , wherein the computation apparatus is further configured to:

perform a local-context-analysis process on each type of sensor data to generate a local-context-analysis result for each type, and merge the local-context-analysis result of each type to generate a local-context-merged result, and select an adjacent-differential-weighting coefficient (ADWC) or an adaptive-weighting coefficient (AWC) according to the local-context-merged result; and

perform a global-detail-feature-fusion process according to the selected ADWC or AWC to perform a global-detail-feature-fusion process to generate a global-fusion feature corresponding for each global object.

16. The surveillance system as claimed in claim 15 , wherein the global-recognition result comprises a confidence level, and the computation apparatus is further configured to:

decompose the global-fusion feature into a plurality of local-detail features of different types, and feed each type of local-detail feature and its corresponding confidence level back to the local-object-recognition model for each type.

17. The surveillance system as claimed in claim 15 , wherein the computation apparatus is further configured to:

perform a context-retrieving process on each type of sensor data to obtain a context-region for each type, and perform a context-merging process on each type of context region to obtain a merged-context region for each type; and

perform the local-context-analysis process on each type of merged-context region to generate the local-context-analysis result, and combine each of the local-context-analysis results to generate the combined local-context result.

18. The surveillance system as claimed in claim 10 , wherein when time stamps, the LRFS, and world-positioning-coordinate information of the local object in each type match, the computation apparatus is further configured to:

determine that the local objects are successfully matched, and assign a global-object-identifier to the successfully matched local objects.