Method for event-based semantic classification
View Patent ↗A method of automatically classifying images in a consumer digital image collection, includes generating an event representation of the image collection; computing global time-based features for each event within the hierarchical event representation; computing content-based features for each image in an event within the hierarchical event representation; combining content-based features for each image in an event to generate event-level content-based features; and using time-based features and content-based features for each event to classify an event into one of a pre-determined set of semantic categories.
1. A method comprising:
generating, using a processor, an event representation of an image collection;
computing global time-based features for each event within a hierarchical event representation;
computing image-level content-based features for each image in an event within the hierarchical event representation;
combining image-level content-based features for each image in an event to generate event-level content-based features;
classifying, using time-based features including at least one of a duration of event, inter-event duration, or number of images in the event and event-level content-based features, each event into a category from a pre-determined set of top-level event categories; and
validating, using a rule-based system, each event belongs to an associated category based on the event-level content-based features; and
refining the category of at least one event into a sub-category based on the event-level content-based features.
2. The method of claim 1 , wherein at least one of the pre-determined set of top-level event categories includes vacation, sports, family moments or social gathering.
3. The method of claim 1 , wherein the image-level content-based features include people present, ages of people present, indoor or outdoor, scene type, or materials present.
4. The method of claim 1 , wherein the hierarchical event representation includes at least two levels.
5. The method of claim 1 , wherein classifying each event comprises using a probabilistic classifier, a rule-based classifier or a combination thereof.
6. The method of claim 1 , wherein the validating each event comprises referencing a database of auxiliary factual information associated with subjects identified in the image collection.
7. The method of claim 1 , wherein the refining the category of at least one event comprises referencing a database of auxiliary factual information associated with subjects identified in the image collection.
8. The method of claim 1 , further comprising calculating a subject distance for each image in an event within the hierarchical event representation, wherein the classifying each event is based upon the subject distances.
9. The method of claim 8 , further comprising determining a location for each image in an event within the hierarchical event representation, wherein the classifying each event is based upon the locations.
10. The method of claim 1 , further comprising determining sub-events for each event.
11. A system comprising:
one or more electronic processors configured to:
generate an event representation of an image collection;
compute global time-based features for each event within a hierarchical event representation;
compute image-level content-based features for each image in an event within the hierarchical event representation;
combine image-level content-based features for each image in an event to generate event-level content-based features;
classify, using time-based features including at least one of a duration of event, inter-event duration, or number of images in the event and event-level content-based features, each event into a category from a pre-determined set of top-level event categories;
validate, using a rule-based system, each event belongs to an associated category based on the event-level content-based features; and
refine the category of at least one event into a sub-category based on the event-level content-based features.
12. The system of claim 11 , wherein the one or more processors are further configured to reference a database of auxiliary factual information associated with subjects identified in the image collection to validate each event.
13. The system of claim 11 , wherein the one or more processors are further configured to reference a database of auxiliary factual information associated with subjects identified in the image collection to refine the category of at least one event.
14. The system of claim 11 , wherein the one or more processors are further configured to calculate a subject distance for each image in an event within the hierarchical event representation, wherein the classifying each event is based upon the subject distances.
15. The system of claim 14 , wherein the one or more processors are further configured to determine a location for each image in an event within the hierarchical event representation, wherein the classifying each event is based upon the locations.
16. A non-transitory computer-readable medium having instructions stored thereon, the instructions comprising:
instructions to generate an event representation of an image collection;
instructions to compute global time-based features for each event within a hierarchical event representation;
instructions to compute image-level content-based features for each image in an event within the hierarchical event representation;
instructions to combine image-level content-based features for each image in an event to generate event-level content-based features;
instructions to classify, using time-based features including at least one of a duration of event, inter-event duration, or number of images in the event and event-level content-based features, each event into a category from a pre-determined set of top-level event categories;
instructions to validate, using a rule-based system, each event belongs to an associated category based on the event-level content-based features; and
instructions to refine the category of at least one event into a sub-category based on the event-level content-based features.
17. The non-transitory computer-readable medium of claim 16 , wherein the instructions to validate each event comprises instructions to reference a database of auxiliary factual information associated with subjects identified in the image collection.
18. The non-transitory computer-readable medium of claim 16 , wherein the instructions to refine the category of at least one event comprises instructions to reference a database of auxiliary factual information associated with subjects identified in the image collection.
19. The non-transitory computer-readable medium of claim 16 , wherein the instructions to classify each event comprises instructions to calculate a subject distance for each image in an event within the hierarchical event representation, wherein the classifying each event is based upon the subject distances.
20. The non-transitory computer-readable medium of claim 19 , wherein the instructions to classify each event comprises instructions to determine a location for each image in an event within the hierarchical event representation, wherein the classifying each event is based upon the locations.