Method for generating reliability tests based on orthogonal arrays and field data
A method for generating reliability tests for a telephone system is based upon sampling an orthogonal array which covers various combinations of test parameters. Field data is collected of actual telephone activity on a telephone system. The field data is evaluated so as to determine call-mix characteristics. Probabilistic weights for the different call-mix characteristics are obtained, and then the probabilistic weights are used to sample the test case scenarios generated in the orthogonal array which have the same call-mix characteristics. These test case scenarios are used to run tests on the telephone system. These tests are preferably performed using automated test scripts. After the test data is collected, reliability metrics are calculated from the test data.
1. A method for generating reliability tests for telephone systems, comprising:
generating an orthogonal array which includes test cases having combinations of test parameters;
obtaining call characteristics from field data based upon actual telephone usage;
obtaining probabilistic weights for different call characteristics based upon a frequency of occurrence in the field data;
using the probabilistic weights to select desirable test cases from the orthogonal array;
running tests on the telephone system using the selected desirable test cases;
collecting test data from the tests that have been run; and
calculating reliability metrics for the telephone system based on the test data.
2. The method according to claim 1 wherein the tests are run using automated test scripts.
3. The method according to claim 2 wherein the orthogonal array is generated using the Taguchi method.
4. The method according to claim 3 wherein the orthogonal array includes tests cases which include at least call types, scenarios, and durations.
5. The method according to claim 4 wherein the call types are selected from the group including a transfer, point-to-point, hold, conference, and others.
6. The method according to claim 5 wherein the test scripts are run for the different call types.
7. The method according to claim 1 wherein the reliability metrics are selected from the group including at least a mean time to failure, a mean time to failure with a predetermined confidence factor, the probability that a telephone will fail after a certain time period, and the probability that a telephone will survive for a certain time period.
8. The method according to claim 4 wherein the test cases of the orthogonal array are divided into groups by common characteristics and wherein a test case is randomly selected from one of the groups once the group is selected based upon its probabilistic weight as determined from the field data.
9. A computer program on a computer readable medium that generates reliability tests for telephone systems, comprising:
generating an orthogonal array which includes test cases having combinations of test parameters;
obtaining call characteristics from field data based upon actual telephone usage;
obtaining probabilistic weights for different call characteristics based upon a frequency of occurrence in the field data;
using the probabilistic weights to select desirable test cases from the orthogonal array;
running tests on the telephone system using the selected desirable test cases;
collecting test data from the tests that have been run; and
calculating reliability metrics for the telephone system based on the test data.
10. The computer program according to claim 9 wherein the tests are run using automated test scripts.
11. The computer program according to claim 10 wherein the orthogonal array is generated using the Taguchi method.
12. The computer program according to claim 11 wherein the orthogonal array includes tests cases which include at least call types, scenarios, and durations.
13. The computer program according to claim 12 wherein the call types are selected from the group including a transfer, p2p, hold, and conference.
14. The computer program according to claim 13 wherein the test scripts are run for the different call types.
15. The computer program according to claim 9 wherein the reliability metrics are selected from the group including at least a mean time to failure, a mean time to failure with a predetermined confidence factor, the probability that a telephone will fail after a certain time period, and the probability that a telephone will survive for a certain time period.
16. The computer program according to claim 12 wherein the test cases of the orthogonal array are divided into groups by common characteristics and wherein a test case is randomly selected from one of the groups once the group is selected based upon its probabilistic weight as determined from the field data.
17. A method for generating reliability tests for communication systems, comprising:
generating an orthogonal array which includes test cases having combinations of test parameters;
obtaining operational characteristics from field data based upon actual communication usage;
obtaining probabilistic weights for different operational characteristics based upon a frequency of occurrence in the field data;
using the probabilistic weights to select desirable test cases from the orthogonal array;
running tests on the communication system using the selected desirable test cases;
collecting test data from the tests that have been run; and
calculating reliability metrics for the communication system based on the test data.
18. The method according to claim 17 wherein the tests are run using automated test scripts.
19. The method according to claim 18 wherein the orthogonal array is generated using the Taguchi method.
20. The method according to claim 19 wherein the orthogonal array includes tests cases which include at least operational types, scenarios, and durations.
21. The method according to claim 20 wherein the test scripts are run for the different operational types.
22. The method according to claim 17 wherein the reliability metrics are selected from the group including at least a mean time to failure, a mean time to failure with a predetermined confidence factor, the probability that a communication will fail after a certain time period, and the probability that a communication device will survive for a certain time period.
23. The method according to claim 20 wherein the test cases of the orthogonal array are divided into groups by common characteristics and wherein a test case is randomly selected from one of the groups once the group is selected based upon its probabilistic weight as determined from the field data.
24. A computer program on a computer readable medium that generates reliability tests for communication systems, comprising:
generating an orthogonal array which includes test cases having combinations of test parameters;
obtaining operational characteristics from field data based upon actual communication usage;
obtaining probabilistic weights for different operational characteristics based upon a frequency of occurrence in the field data;
using the probabilistic weights to select desirable test cases from the orthogonal array;
running tests on the communication system using the selected desirable test cases;
collecting test data from the tests that have been run; and
calculating reliability metrics for the communication system based on the test data.
25. The computer program according to claim 24 wherein the tests are run using automated test scripts.
26. The computer program according to claim 25 wherein the orthogonal array is generated using the Taguchi method.
27. The computer program according to claim 26 wherein the orthogonal array includes tests cases which include at least operational types, scenarios, and durations.
28. The computer program according to claim 27 wherein the test scripts are run for the different operational types.
29. The computer program according to claim 26 wherein the reliability metrics are selected from the group including at least a mean time to failure, a mean time to failure with a predetermined confidence factor, the probability that a communication device will fail after a certain time period, and the probability that a communication device will survive for a certain time period.
30. The computer program according to claim 27 wherein the test cases of the orthogonal array are divided into groups by common characteristics and wherein a test case is randomly selected from one of the groups once the group is selected based upon its probabilistic weight as determined from the field data.