System for discovering semantic relationships in computer programs
A system for discovering semantic relationships in computer programs is disclosed. In particular, the system may synergistically identify and validate semantic relationships, concepts, and groupings associated with data elements within a static or dynamic, time varying, source input. The system may utilize feature extractors to extract features from the input and reasoners to develop associations using data from multiple feature set types, and, can thus generate reliable, robust, and complete sets of semantic relationships from the input. The system may generate hypotheses associated with the relationships, concepts, and groupings, and validate the hypotheses by testing an application under evaluation by the system and observing the outputs generated from the testing. Information pertaining to validated or invalidated hypotheses may be provided to a learning engine to maximize reasoning and performance in subsequent discovery processes by adjusting models, vocabularies, dictionaries, parameters utilized by the system in identifying the relationships, concepts, and groupings.
1. A method comprising:
analyzing, for a semantic relationship discovery process, information provided by a source, wherein the information is associated with an application under evaluation by a system and is actively extracted from interactions conducted with the application under evaluation while the application under evaluation is under evaluation by the system;
determining, based on the analyzing of the information provided by the source, a concept associated with data elements in the information, a relationship between the data elements in the information, a grouping associated with the data elements in the information, or a combination thereof;
generating, based on the analyzing of the information, a constraint suggestion for the application under evaluation;
generating, based on the constraint suggestion and by utilizing instructions from a memory that are executed by a processor of the system, a hypothesis associated with the concept, the relationship, the grouping, or a combination thereof, wherein the hypothesis is ranked relative to other hypotheses generated by the system based on a confidence level of the hypothesis and confidence levels of the other hypotheses to form a ranked plurality of hypotheses;
filtering, by applying a confidence level threshold to the ranked plurality of hypotheses, a subset of hypotheses from the ranked plurality of hypotheses that satisfy the confidence level threshold;
outputting the subset of hypotheses satisfying the confidence level threshold:
validating the hypothesis, the other hypotheses, the subset of hypotheses, or a combination thereof, by testing the application under evaluation and observing outputs generated based on the testing; and
training, based on the validating of the hypothesis, the other hypotheses, the subset of hypotheses, or a combination thereof, and based on a confirmation or a rejection of the hypothesis, the other hypotheses, the subset of hypotheses, or a combination thereof, a model to enhance the determination of the concept, the relationship, the grouping, or a combination thereof, for a subsequent semantic relationship discovery process.
2. The method of claim 1 , further comprising adjusting a vocabulary, a dictionary, a parameter, a model, a confidence, or a combination thereof, utilized in determining the concept, the relationship, the grouping, or a combination thereof, based on the validating of the hypothesis, the other hypotheses, the subset of hypotheses, or a combination thereof.
3. The method of claim 1 , further comprising outputting the concept, the relationship, the grouping, or a combination thereof, if the concept, the relationship, the grouping, or a combination thereof, have confidence levels exceeding a selected threshold confidence level.
4. The method of claim 1 , further comprising training a reasoner utilizing to determine the concept, the relationship, the grouping, or a combination thereof, based on the outputs generated based on the testing.
5. The method of claim 1 , further comprising accepting an input from a user, a device, a program, or a combination thereof, that confirms or rejects the hypothesis, the other hypotheses, the subset of hypotheses, or a combination thereof.
6. The method of claim 1 , further comprising utilizing a cross-domain reasoner to determine the concept, the relationship, the grouping, or a combination thereof, from two or more component reasoners.
7. The method of claim 1 , wherein the confidence level of the hypothesis is based on a type of the hypothesis, content of the hypothesis, a complexity of the hypothesis, a source of the hypothesis, or a combination thereof.
8. The method of claim 1 , further comprising labeling features extracted from the information provided by the source and outputs associated with the features.
9. A system comprising:
a memory that stores instructions; and
a processor that executes the instructions to perform operations, the operations comprising:
analyzing, for a semantic relationship discovery process, information provided by a source, wherein the information is associated with an application under evaluation by the system and is actively extracted from interactions conducted with the application under evaluation while the application under evaluation is under evaluation by the system;
determining, based on the analyzing of the information provided by the source, a concept associated with data elements in the information, a relationship between the data elements in the information, a grouping associated with the data elements in the information, or a combination thereof;
generating, based on the analyzing of the information, a constraint suggestion for the application under evaluation;
generating, based on the constraint suggestion, a hypothesis associated with the concept, the relationship, the grouping, or a combination thereof;
ranking the hypothesis relative to other hypotheses generated by the system based on a confidence level of the hypothesis and confidence levels of the other hypotheses to form a ranked plurality of hypotheses;
filtering, by applying a confidence level threshold to the hypothesis, a subset of hypotheses from the ranked plurality of hypotheses that satisfy the confidence level threshold;
outputting the subset of hypotheses satisfying the confidence level threshold; and
testing the subset of hypotheses against the application under evaluation to confirm or reject the subset of hypotheses; and
training, based on the testing of the subset of hypotheses and based on a confirmation or a rejection of the subset of hypotheses, a model to enhance the determination of the concept, the relationship, the grouping, or a combination thereof, for a subsequent semantic relationship discovery process.
10. The system of claim 9 , wherein the operations further comprise testing the subset of hypotheses against a different application under evaluation.
11. The system of claim 10 , wherein the operations further comprise training the model based on the testing of the subset of hypotheses against the different application under evaluation.
12. The system of claim 9 , wherein the operations further comprise validating the hypothesis, the other hypotheses, or a combination thereof, by testing, based on the hypothesis, the other hypotheses, or a combination thereof, the concept associated with the data elements, the relationship between the data elements, the grouping associated with the data elements, or a combination thereof, with the application under evaluation, and wherein the operations further comprise observing an output of the application under evaluation generated in response to the testing in order to validate the hypothesis, the other hypotheses, the subset of hypotheses, or a combination thereof.
13. The system of claim 9 , wherein the operations further comprise extracting a structural feature of an organizational structure of a document included in the information that is associated with the application under evaluation, and wherein the operations further comprise determining the concept associated with the data elements in the information, the relationship between the data elements in the information, the grouping associated with the data elements in the information, or a combination thereof, based on the structural feature of the document.
14. The system of claim 9 , wherein the operations further comprise extracting a geometric feature from a rendering of the information associated with the application under evaluation, and wherein the operations further comprise determining the concept associated with the data elements in the information, the relationship between the data elements in the information, the grouping associated with the data elements in the information, or a combination thereof, based on the geometric feature.
15. The system of claim 9 , wherein the operations further comprise extracting a natural language processing feature from text extracted from the information associated with the application under evaluation, and wherein the operations further comprise determining the concept associated with the data elements in the information, the relationship between the data elements in the information, the grouping associated with the data elements in the information, or a combination thereof, based on the natural language processing feature.
16. The system of claim 9 , wherein the operations further comprise extracting a domain feature from a domain or an area associated with the source of the information associated with the application under evaluation, and wherein the operations further comprise determining the concept associated with the data elements in the information, the relationship between the data elements in the information, the grouping associated with the data elements in the information, or a combination thereof, based on the domain feature.
17. The system of claim 9 , wherein the operations further comprise determining the concept associated with the data elements in the information, the relationship between the data elements in the information, the grouping associated with the data elements in the information, or a combination thereof, by utilizing a cross-type reasoner that processes features extracted from the information associated with the application under evaluation by two or more feature extractors.
18. The system of claim 9 , wherein the operations further comprise testing other hypotheses against the application under evaluation, and wherein the operations further comprise confirming or rejecting the hypothesis, the other hypotheses, the subset of hypotheses, or a combination thereof, based on outputs generated based on the testing performed against the application under evaluation.
19. The system of claim 18 , wherein the operations further comprise providing the outputs indicating a confirmation or a rejection of the hypothesis, the other hypotheses, the subset of hypotheses, or a combination thereof, to a learning engine for processing, and wherein the operations further comprise training or improving a model, parameter, weight, a dictionary, a threshold, a confidence, a feature extractor, a reasoner, or a filter associated with generating a future hypothesis, wherein the training or the improving is based on the confirmation or the rejection of the hypothesis, the subset of hypotheses, or a combination thereof.
20. The system of claim 9 , wherein the operations further comprise processing, by utilizing an agglomerated model, the information to extract a feature for processing by a reasoner of the system.
21. The system of claim 9 , wherein the operations further comprise excluding a feature from being extracted from the information for subsequent concept, relationship, and grouping determinations if the feature does not correlate with a validated result obtained from testing the hypothesis, other hypotheses, the subset of hypotheses, or a combination thereof.
22. The system of claim 9 , wherein the operations further comprise excluding features from being extracted from the information for subsequent concept, relationship, and grouping determinations if the features indicate a collinear relationship, if the features indicate a lack of independent discriminatory effect, or a combination thereof.
23. A non-transitory computer-readable device comprising instructions, which when loaded and executed by a processor, cause the processor to perform operations comprising:
analyzing, for a semantic relationship discovery process, information provided by a source, wherein the information is associated with an application under evaluation by a system and is actively extracted from interactions conducted with the application under evaluation while the application under evaluation is under evaluation by the system;
determining, based on the analyzing of the information provided by the source, a concept associated with data elements in the information, a relationship between the data elements in the information, a grouping associated with the data elements in the information, or a combination thereof;
determining, based on the analyzing of the information, a constraint suggestion for the application under evaluation;
generating, based on the constraint suggestion, a hypothesis associated with the concept, the relationship, the grouping, or a combination thereof, wherein the hypothesis is ranked relative to other hypotheses generated by the system based on a confidence level of the hypothesis and confidence levels of the other hypotheses to form a ranked plurality of hypotheses;
filtering, by applying a confidence level threshold to the ranked plurality of hypotheses, a subset of hypotheses from the ranked plurality of hypotheses satisfying the confidence level threshold;
utilizing the subset of hypotheses filtered from the ranked plurality of hypothesis when testing the application under evaluation;
testing the hypothesis against the application under evaluation to confirm or reject the hypothesis; and
training, based on the testing of the hypothesis and based on a confirmation or a rejection of the hypothesis, a model to enhance the determination of the concept, the relationship, the grouping, or a combination thereof, for a subsequent semantic relationship discovery process.
24. The non-transitory computer-readable device of claim 23 , wherein the operations further comprise providing an output generated based on the testing to a learning engine for processing, wherein the learning engine determines, based on the output, a pattern or confidence associated with the hypothesis, the subset of hypotheses, the other hypotheses, or a combination thereof, to adjust generation of a future hypothesis, adjust the confidence level, adjust the confidence levels of the other hypotheses, or a combination thereof, over time.