Systems and methods to optimize testing using machine learning
A machine learning (ML) model is created via training or re-training one or more ML algorithms using past release(s) data (e.g., data comprising of requirements and corresponding test cases). The ML model comprises various clusters and these clusters are dynamically created every time when the ML model is trained (or retrained). One or more requirements exist in each cluster, and each requirement has one or more test cases associated with it. New requirements are mapped to a particular cluster and then test cases are compared against a universe of other test cases to determine whether to add a particular test case to a list of test cases that test the new requirement.
1. A method, the method comprising:
creating and storing an Machine Learning (ML) model in an electronic memory device using one or more Unsupervised Machine Learning (ML) algorithms, the machine learning model indicating requirements which describe features of electronic devices or electronic services, wherein the requirements are grouped into a plurality of clusters in the machine learning model, wherein each of the requirements have an associated requirements vector created using Natural Language Processing (NLP), and wherein each of the requirements in each of the clusters are associated with one or more test cases;
subsequent to creating the machine learning model, electronically receiving a new requirement from a user;
vectorizing the new requirement to form a vectorized new requirement;
using the one or more unsupervised machine learning algorithms, mapping the vectorized new requirement to a selected one of the plurality of clusters based upon the vectorized new requirement being a closest match with the selected one of the plurality of clusters;
determining all test cases associated with all of the requirements of the selected one of the plurality of clusters;
subsequently and for each of the test cases in the selected one of the plurality of clusters and using a similarity algorithm:
perform a closeness comparison of each test case in the selected one of the plurality of clusters to all other test cases associated with all other clusters of the plurality of clusters;
when the comparison determines a similarity that is within a predetermined threshold similarity, adding the other test case from the other cluster to a list;
when all comparisons are complete, executing each of the test cases on the list, the executing being one or more of:
automatically causing an automatic execution of the test case on the list, the automatic execution of the test case causing one or more control signals to be sent to a selected electronic device or selected electronic service, the one or more control signals causing actuation of components, members, or features of the selected electronic device or selected electronic service, and automatically monitoring the results of execution of the test case; and
manually causing a manual execution of the test case on the list, wherein a user physically interacts with the device or service to actuate a component or feature of the device or service;
wherein the ML model is subsequently retrained automatically in real-time using a detected defect, the detected defect being a leaked or slipped defect occurring during a next phase or during production and being associated with a new feature being added to the electronic device or electronic service that causes failure of an existing feature, the ML model retraining being effective to automatically change the predetermined threshold similarity or manually create one or more new test cases.
2. The method of claim 1 , wherein the electronic devices comprise one or more of a smart phone, a lap top, a tablet, a cellular phone, or a personal computer.
3. The method of claim 1 , wherein the one or more test cases comprise executable computer instructions.
4. The method of claim 1 , wherein the threshold is dynamic and adjustable manually by user or automatically by control circuit.
5. The method of claim 1 , further comprising utilizing, by a user, a testing knowledge service to gain further knowledge of the one or more test cases for the corresponding new incoming requirements.
6. The method of claim 1 , wherein a user estimates the regression testing effort needed for any new requirements, defects, or enhancements via knowing an exact count of regression test cases to be executed.
7. The method of claim 1 , further comprising automatically or by a user identifying the suitable manual regression test cases to be automated based on business criticality and repeatability of the regression test cases.
8. The method of claim 1 , where the electronic services comprise electronic telecommunication services.
9. The method of claim 1 , wherein the method is performed at a central control center.
10. The method of claim 1 , wherein the method is performed at a factory.
11. The method of claim 1 , wherein the closeness comparison between selected test cases is determined by using any similarity algorithms.
12. The method of claim 1 , wherein the electronic model is further refined and physically changed according to the new requirement.
13. The method of claim 1 , wherein the vectorized new requirement comprises an angle and a magnitude.
14. The method of claim 1 , wherein the vectorized new requirement is created using NLP.
15. A system, the system comprising:
an electronic model;
one or more electronic devices and electronic services;
an electronic memory device that stores the Machine Learning model, the Machine Learning model indicating requirements which describe features of the electronic devices or the electronic services, wherein the Machine Learning model is created using one or more Unsupervised Machine Learning (ML) algorithms, wherein the requirements are grouped into a plurality of clusters in the electronic model, wherein each of the requirements have an associated requirements vector, and wherein each of the requirements in each of the clusters are associated with one or more test cases;
a control circuit created using Natural Language Processing (NLP), the control circuit being coupled to the electronic memory device and the one or more electronic devices and electronic services, wherein the control circuit is configured to:
subsequent to creating the Machine Learning model, electronically receive a new requirement from a user;
vectorize the new requirement to form a vectorized new requirement;
using the one or more unsupervised machine learning algorithms, map the vectorized new requirement to a selected one of the plurality of clusters based upon the vectorized new requirement being a closest match with the selected one of the plurality of clusters;
determine all test cases associated with all of the requirements of the selected one of the plurality of clusters;
subsequently and for each of the test cases in the selected one of the plurality of clusters and using a similarity algorithm:
perform a closeness comparison of each test case in the selected one of the plurality of clusters to all other test cases associated with all other clusters of the plurality of clusters;
when the comparison determines a similarity that is within a predetermined threshold similarity, adding the other test case from the other cluster to a list;
when all comparisons are complete, executing each of the test cases on the list, the executing being one or more of:
automatically causing an automatic execution of the test case on the list, the automatic execution of the test case causing one or more control signals to be sent to a selected electronic device or selected electronic service, the control signal causing actuation of components, members, or features of the selected electronic device or selected electronic service, and automatically monitoring the results of execution of the test case; and
manually causing a manual execution of the test case on the list, wherein a user physically interacts with the device or service to actuate a component or feature of the device or service;
wherein the ML model is subsequently retrained automatically in real-time using a detected defect, the detected defect being a leaked or slipped defect occurring during a next phase or during production and being associated with a new feature being added to the electronic device or electronic service that causes failure of an existing feature, the ML model retraining being effective to automatically change the predetermined threshold similarity or manually create one or more new test cases.
16. The system of claim 15 , wherein the electronic devices comprise one or more of a smart phone, a lap top, a tablet, a cellular phone, or a personal computer.
17. The system of claim 15 , wherein the one or more test cases comprise executable computer instructions.
18. The system of claim 15 , wherein the threshold is dynamic and adjustable.
19. The system of claim 15 , wherein a user utilizes a testing knowledge service to gain further knowledge of the one or more test cases for the corresponding new incoming requirements.
20. The system of claim 15 , wherein a user estimates the regression testing effort needed for any new requirements, defects, or enhancements via knowing an exact count of regression test cases to be executed.
21. The system of claim 15 , wherein suitable manual regression test cases to be automated are identified automatically or by a user based on business criticality and repeatability of the regression test cases.
22. The system of claim 15 , where the electronic services comprise electronic telecommunication services.
23. The system of claim 15 , wherein the system is disposed at a central control center.
24. The system of claim 15 , wherein the system is disposed at a factory.
25. The system of claim 15 , wherein the closeness comparison between selected test cases is determined by using any similarity algorithms.
26. The system of claim 15 , wherein the electronic model is further refined and physically changed according to the new requirement.
27. The system of claim 15 , wherein the vectorized new requirement comprises an angle and a magnitude.
28. The system of claim 15 , wherein the vectorized new requirement is created using NLP.