Feature analysis in computer-based reasoning models
Techniques are provided herein for creating well-balanced computer-based reasoning systems and using those to control systems. The techniques include receiving a request to determine whether to use one or more particular features, cases, etc. in a computer-based reasoning model (e.g., as cases or features are being added, or as part of pruning existing features or cases). Conviction measures (such as targeted or untargeted conviction, contribution, surprisal, etc.) are determined and inclusivity conditions are tested. The result of comparing the conviction measure can be used to determine whether to include or exclude the feature, case, etc. in the computer-based reasoning model. A controllable system may then be controlled using the computer-based reasoning model. Examples controllable systems include self-driving cars, image labeling systems, manufacturing and assembly controls, federated systems, smart voice controls, automated control of experiments, energy transfer systems, and the like.
1. A method comprising:
receiving, at a training and analysis system, a request to determine whether to use one or more particular features of a computer-based reasoning model, wherein the training and analysis system executes on one or more computing devices, and is configured to execute training and analysis instructions;
determining, at the training and analysis system, one or more feature conviction measures for the one or more particular features of the computer-based reasoning model; wherein:
determining the one or more feature conviction measures for the one or more particular features comprises determining one of feature prediction conviction or feature prediction contribution for the one or more particular features,
feature prediction contribution is a measure of the relative information in the model with a given feature and without the given feature, and
feature prediction conviction is a measure of expected information for the given feature;
determine whether the one or more feature conviction measures for the one or more particular features meets inclusivity conditions;
in response to determining that the one or more feature conviction measures for the one or more particular features meets the inclusivity conditions, including the one or more particular features in the computer-based reasoning model;
in response to determining that the one or more feature conviction measures meet the inclusivity conditions:
including the one or more particular features in the computer-based reasoning model when the inclusivity conditions comprise an inclusion condition;
excluding the one or more particular features in the computer-based reasoning model when the inclusivity conditions comprise an exclusion condition;
causing, with a control system, control of a controllable system with the computer-based reasoning model,
wherein determining the one or more feature conviction measures for the one or more particular features comprises determining the feature prediction conviction and a familiarity conviction for the one or more particular features,
wherein determining whether the one or more feature conviction measures for the one or more particular features meets the inclusivity conditions comprises determining whether the feature prediction conviction for the one or more particular features is above a particular threshold and whether the familiarity conviction is below a certain threshold,
wherein the inclusivity conditions comprise an exclusion condition,
wherein the method is performed on one or more computing devices.
2. The method of claim 1 ,
wherein the one or more particular features relate to at least one label on at least one training image;
wherein causing control of the controllable system comprises causing control of an imagining system that identifies elements of an image using the computer-based reasoning model by:
receiving an input image for labelling;
determining one or more labels for the input image based on the image and the computer-based reasoning model;
labelling the input image based on the one or more determined labels.
3. The method of claim 1 ,
wherein causing control of the controllable system comprises causing control of a vehicle using the computer-based reasoning model by:
receiving a current context for the vehicle, wherein the vehicle can be controlled by the control system;
determining an action to take for the vehicle based on the current context for the vehicle and the computer-based reasoning model;
causing the vehicle to perform the determined action.
4. The method of claim 1 ,
wherein determining the one or more feature conviction measures for the one or more particular features comprises determining the feature prediction contribution for the one or more particular features and
wherein determining whether the one or more feature conviction measures for the one or more particular features meets the inclusivity conditions comprises determining whether the feature prediction contribution for the one or more particular features is above a particular threshold;
wherein the inclusivity conditions comprise the inclusion condition.
5. The method of claim 1 ,
wherein determining the one or more feature conviction measures for the one or more particular features comprises determining a targeted feature conviction for the one or more particular features and
wherein determining whether the one or more feature conviction measures for the one or more particular features meets the inclusivity conditions comprises determining whether the targeted feature conviction for the one or more particular features is below a particular threshold;
wherein the inclusivity conditions comprise an exclusion condition.
6. The method of claim 1 , wherein the inclusivity conditions comprise an exclusion condition and the method further comprises:
receiving a request for model reduction of the computer-based reasoning model;
determining one or more feature conviction measures for each feature in the one or more particular features, wherein the one or more particular features comprise all of features in the computer-based reasoning model;
determining a subset of the one or more particular features based at least in part on the exclusion condition and the one or more feature conviction measures for each feature in the one or more particular features, wherein the subset contains only features from the one or more particular features for which the feature conviction measures meet the exclusion condition;
removing the one or more particular features from the computer-based reasoning model.
7. The method of claim 1 , wherein the inclusivity conditions comprise the inclusion condition and the method further comprises:
receiving a request for model reduction of the computer-based reasoning model;
determining one or more feature conviction measures for each feature in the one or more particular features, wherein the one or more particular features comprise all of features in the computer-based reasoning model;
determining a subset of the one or more particular features based at least in part on the exclusion condition and the one or more feature conviction measures for each feature in the one or more particular features, wherein the subset contains only features from the one or more particular features for which the feature conviction measures meet the inclusion condition;
including only the subset of the one or more particular features in the computer-based reasoning model.
8. The method of claim 1 , further comprising:
initially receiving the one or more particular features as part of training for the computer-based reasoning model;
in response to determining that the one or more feature conviction measures for the one or more particular features meet the inclusion condition, sending an indication to continue to collect the one or more particular features as part of the training;
in response to determining that the one or more feature conviction measures for the one or more particular features meets an exclusion condition, sending an indication to no longer collect the one or more particular features.
9. A system for performing a machine-executed operation involving instructions, wherein said instructions are instructions which, when executed by one or more computing devices, cause performance of a process comprising:
receiving, at a training and analysis system, a request to determine whether to use one or more particular features of a computer-based reasoning model, wherein the training and analysis system executes on one or more computing devices, and is configured to execute training and analysis instructions;
determining, at the training and analysis system, one or more feature conviction measures for the one or more particular features of the computer-based reasoning model, wherein:
determining the one or more feature conviction measures for the one or more particular features comprises determining one of feature prediction conviction or feature prediction contribution for the one or more particular features,
feature prediction contribution is a measure of the relative information in the model with a given feature and without the given feature, and
feature prediction conviction is a measure of expected information for the given feature;
determine whether the one or more feature conviction measures for the one or more particular features meets inclusivity conditions;
in response to determining that the one or more feature conviction measures for the one or more particular features meets the inclusivity conditions, including the one or more particular features in the computer-based reasoning model;
in response to determining that the one or more feature conviction measures meet the inclusivity conditions:
including the one or more particular features in the computer-based reasoning model when the inclusivity conditions comprise an inclusion condition;
excluding the one or more particular features in the computer-based reasoning model when the inclusivity conditions comprise an exclusion condition;
causing, with a control system, control of a controllable system with the computer-based reasoning model,
wherein determining the one or more feature conviction measures for the one or more particular features comprises determining the feature prediction conviction and a familiarity conviction for the one or more particular features,
wherein determining whether the one or more feature conviction measures for the one or more particular features meets the inclusivity conditions comprises determining whether the feature prediction conviction for the one or more particular features is above a particular threshold and whether the familiarity conviction is below a certain threshold,
wherein the inclusivity conditions comprise an exclusion condition.
10. The system of claim 9 , the process further comprising:
wherein the one or more particular features relate to at least one label on at least one training image;
wherein causing control of the controllable system comprises causing control of an imagining system that identifies elements of an image using the computer-based reasoning model by:
receiving an input image for labelling;
determining one or more labels for the input image based on the image and the computer-based reasoning model;
labelling the input image based on the one or more determined labels.
11. The system of claim 9 ,
wherein causing control of the controllable system comprises causing control of a vehicle using the computer-based reasoning model by:
receiving a current context for the vehicle, wherein the vehicle can be controlled by the control system;
determining an action to take for the vehicle based on the current context for the vehicle and the computer-based reasoning model;
causing the vehicle to perform the determined action.
12. The system of claim 9 ,
wherein determining the one or more feature conviction measures for the one or more particular features comprises determining the feature prediction contribution for the one or more particular features and
wherein determining whether the one or more feature conviction measures for the one or more particular features meets the inclusivity conditions comprises determining whether the feature prediction contribution for the one or more particular features is above a particular threshold;
wherein the inclusivity conditions comprise the inclusion condition.
13. The system of claim 9 ,
wherein determining the one or more feature conviction measures for the one or more particular features comprises determining a targeted feature conviction for the one or more particular features and
wherein determining whether the one or more feature conviction measures for the one or more particular features meets the inclusivity conditions comprises determining whether the targeted feature conviction for the one or more particular features is below a particular threshold;
wherein the inclusivity conditions comprise an exclusion condition.
14. The system of claim 9 , wherein the inclusivity conditions comprise an exclusion condition and the process further comprises:
receiving a request for model reduction of the computer-based reasoning model;
determining one or more feature conviction measures for each feature in the one or more particular features, wherein the one or more particular features comprise all of features in the computer-based reasoning model;
determining a subset of the one or more particular features based at least in part on the exclusion condition and the one or more feature conviction measures for each feature in the one or more particular features, wherein the subset contains only features from the one or more particular features for which the feature conviction measures meet the exclusion condition;
removing the one or more particular features from the computer-based reasoning model.
15. The system of claim 9 , wherein the inclusivity conditions comprise the inclusion condition and the process further comprises:
receiving a request for model reduction of the computer-based reasoning model;
determining one or more feature conviction measures for each feature in the one or more particular features, wherein the one or more particular features comprise all of features in the computer-based reasoning model;
determining a subset of the one or more particular features based at least in part on the exclusion condition and the one or more feature conviction measures for each feature in the one or more particular features, wherein the subset contains only features from the one or more particular features for which the feature conviction measures meet the inclusion condition;
including only the subset of the one or more particular features in the computer-based reasoning model.
16. The system of claim 9 , the process further comprising:
initially receiving the one or more particular features as part of training for the computer-based reasoning model;
in response to determining that the one or more feature conviction measures for the one or more particular features meet the inclusion condition, sending an indication to continue to collect the one or more particular features as part of the training;
in response to determining that the one or more feature conviction measures for the one or more particular features meets an exclusion condition, sending an indication to no longer collect the one or more particular features.
17. A non-transitory computer readable medium storing instructions which, when executed by one or more computing devices, cause the one or more computing devices to perform a process of:
receiving; at a training and analysis system, a request to determine whether to use one or more particular features of a computer-based reasoning model, wherein the training and analysis system executes on one or more computing devices, and is configured to execute training and analysis instructions;
determining, at the training and analysis system, one or more feature conviction measures for the one or more particular features of the computer-based reasoning model, wherein:
determining the one or more feature conviction measures for the one or more particular features comprises determining one of feature prediction conviction or feature prediction contribution for the one or more particular features,
feature prediction contribution is a measure of the relative information in the model with a given feature and without the given feature, and
feature prediction conviction is a measure of expected information for the given feature;
determine whether the one or more feature conviction measures for the one or more particular features meets inclusivity conditions;
in response to determining that the one or more feature conviction measures for the one or more particular features meets the inclusivity conditions, including the one or more particular features in the computer-based reasoning model;
in response to determining that the one or more feature conviction measures meet the inclusivity conditions:
including the one or more particular features in the computer-based reasoning model when the inclusivity conditions comprise an inclusion condition;
excluding the one or more particular features in the computer-based reasoning model when the inclusivity conditions comprise an exclusion condition;
causing, with a control system, control of a controllable system with the computer-based reasoning model,
wherein determining the one or more feature conviction measures for the one or more particular features comprises determining the feature prediction conviction and a familiarity conviction for the one or more particular features and
wherein determining whether the one or more feature conviction measures for the one or more particular features meets the inclusivity conditions comprises determining whether the feature prediction conviction for the one or more particular features is above a particular threshold and whether the familiarity conviction is below a certain threshold;
wherein the inclusivity conditions comprise an exclusion condition.
18. The non-transitory computer readable medium of claim 17 , the process further comprising
wherein the one or more particular features relate to at least one label on at least one training image;
wherein causing control of the controllable system comprises causing control of an imagining system that identifies elements of an image using the computer-based reasoning model by:
receiving an input image for labelling;
determining one or more labels for the input image based on the image and the computer-based reasoning model;
labelling the input image based on the one or more determined labels.
19. The non-transitory computer readable medium of claim 17 ,
wherein causing control of the controllable system comprises causing control of a vehicle using the computer-based reasoning model by:
receiving a current context for the vehicle, wherein the vehicle can be controlled by the control system;
determining an action to take for the vehicle based on the current context for the vehicle and the computer-based reasoning model;
causing the vehicle to perform the determined action.
20. The non-transitory computer readable medium of claim 17 ,
wherein determining the one or more feature conviction measures for the one or more particular features comprises determining the feature prediction contribution for the one or more particular features and
wherein determining whether the one or more feature conviction measures for the one or more particular features meets the inclusivity conditions comprises determining whether the feature prediction contribution for the one or more particular features is above a particular threshold;
wherein the inclusivity conditions comprise the inclusion condition.
21. The non-transitory computer readable medium of claim 17 , wherein the inclusivity conditions comprise an exclusion condition and the process further comprises:
receiving a request for model reduction of the computer-based reasoning model;
determining one or more feature conviction measures for each feature in the one or more particular features, wherein the one or more particular features comprise all of features in the computer-based reasoning model;
determining a subset of the one or more particular features based at least in part on the exclusion condition and the one or more feature conviction measures for each feature in the one or more particular features, wherein the subset contains only features from the one or more particular features for which the feature conviction measures meet the exclusion condition;
removing the one or more particular features from the computer-based reasoning model.
22. The non-transitory computer readable medium of claim 17 , wherein the inclusivity conditions comprise the inclusion condition and the process further comprises:
receiving a request for model reduction of the computer-based reasoning model;
determining one or more feature conviction measures for each feature in the one or more particular features, wherein the one or more particular features comprise all of features in the computer-based reasoning model;
determining a subset of the one or more particular features based at least in part on the exclusion condition and the one or more feature conviction measures for each feature in the one or more particular features, wherein the subset contains only features from the one or more particular features for which the feature conviction measures meet the inclusion condition;
including only the subset of the one or more particular features in the computer-based reasoning model.
23. The non-transitory computer readable medium of claim 17 , the process further comprising:
initially receiving the one or more particular features as part of training for the computer-based reasoning model;
in response to determining that the one or more feature conviction measures for the one or more particular features meet the inclusion condition, sending an indication to continue to collect the one or more particular features as part of the training;
in response to determining that the one or more feature conviction measures for the one or more particular features meets an exclusion condition, sending an indication to no longer collect the one or more particular features.