System and method for precision drive testing of a cellular network
A method and system for Precision Drive Testing (PDT) of cellular wireless networks (e.g., 2G/3G/4G/LTE, 5G, 6G). The PDT system and method are cloud-based, automated, remotely controlled and allow for directed testing of user/device or machine experience of network(s) and service(s) performance by running pre-defined use cases, specific test scripts on pre-defined (pre-calculated) routes or within pre-defined (pre-calculated) areas of interest for the use case. The PDT may be automatically and remotely triggered by AI/ML based network sensing by drive test agents and/or outcomes (results) of network and/or service performance data and/or customer analytics or raw network and/or service and/or customer data, and/or network planning data, and/or new deployments of networks/technologies/services/devices, and benchmarking data.
1 . A method for automatically directing drive testing by field agents, each of whom is provisioned with a drive test agent configured to assess network performance and/or service and/or user experience in a geographical area, the method comprising:
obtaining a use case for the drive test agent in response to an automated request from either an external data source or an internal data source;
automatically retrieving one or more test scripts corresponding to the obtained use case;
obtaining from the use case, information regarding the location of one or more network sites or one or more areas of interest to which the automated request pertains, and determining the geographical area to be drive tested;
calculating one or more drive test routes for each drive test agent, in order to cover the determined geographical area;
sending to one or more drive test agents, the test scripts corresponding to the obtained use case along with definition of done criteria and drive test routes, so that each corresponding field agent can conduct a drive test within said determined geographical area, the definition of done criteria including at least a statistical significance level threshold and an acceptable measurement error;
automatically receiving real time feedback from each drive test agent regarding the status of that drive test;
automatically sending one or more instructions to said each drive test agent in response to the real time feedback provided by that agent, said instructions comprising one or more of route updates, corrected test scripts, modified test scripts and repeated test scripts; and
automatically receiving and uploading final drive test results upon satisfaction of the definition of done criteria by each drive test agent.
2 . The method according to claim 1 , wherein the external data source comprises network data and/or service data and/or user experience data regarding quality degradation and/or failures and/or alarms.
3 . The method according to claim 1 , wherein the internal data source comprises real time network and/or real time service and/or real time user experience quality sensing obtained from one or more predictive machine learning (ML)/artificial intelligence (AI) algorithms embedded in the drive test agent.
4 . The method according to claim 1 , further comprising sending to the one or more drive test agents, one or more of specific set of key performance indicators (KPIs) to be collected, device/network events to be collected, and drive test success metrics.
5 . The method according to claim 4 , further comprising:
for the obtained use case, applying KPI masks to optimize the number of test scripts and the size of the collected KPIs data set.
6 . The method according to claim 1 , wherein the final drive test results include collected key performance indicators (KPIs), device/network events, fault flags report, drive test success metrics of pass/fail tests and driven route(s) with any information about changes to the original route(s).
7 . The method according to claim 1 , further comprising, after automatically receiving and uploading final drive test results: enriching the use case/test scripts pairs with new and/or modified use case/test scripts, based on lessons learned using machine learning (ML)/artificial intelligence (AI) techniques.
8 . The method according to claim 1 , wherein the real time feedback from each drive test agent regarding the status of the drive test is one from the group consisting of:
(i) satisfying the definition of done;
(ii) test device or agent and/or equipment malfunction;
(iii) modified drive routes due to unexpected traffic events;
(iv) failed test script due to lack of network and/or service access; or
(v) failed test script due to lack of KPIs required by the obtained use case.
9 . The method according to claim 1 , wherein:
the use case is obtained in response to a problem automatically identified by said external data source, the problem pertaining to network and/or service and/or customer experience quality degradation and/or failures and/or alarms.
10 . The method according to claim 9 , comprising:
the test scripts sent to each drive test agent contain information about the specific data to be collected; and
the specific data to be collected comprises a set of key performance indicators and/or device/network events sufficient to solve the problem corresponding to the use case.
11 . The method according to claim 1 , wherein:
the use case is obtained in response to a problem identified by one or more predictive machine learning (ML)/artificial intelligence (AI) algorithms embedded on the drive test agent, the problem pertaining to real time network and/or real time service and/or real time user experience quality sensing.
12 . The method according to claim 11 , comprising:
the test scripts sent to each drive test agent contain information about the specific data to be collected; and
the specific data to be collected comprises a set of key performance indicators and/or device/network events sufficient to solve the problem corresponding to the use case.
13 . The method according to claim 1 , wherein determining the geographical area to be drive tested comprises:
(a) calculating a boundary of the geographical area based on the geolocation of the network sites under study; and/or
(b) calculating a boundary of the geographical area based on the geolocation coordinates of the area of interest.
14 . The method according to claim 13 , wherein the step of calculating the one or more drive test routes comprises:
(a) geographical optimization to calculate routes which maximize the covered area within the whole geographical area to be tested; or
(b) optimizing a route to a cover list of sweet spots previously defined by one or more of the following criteria (i) geography, (ii) prediction, and (iii) testing.
15 . The method according to claim 1 , wherein:
the use case is obtained in response to an automated request corresponding to a predetermined monitoring and/or maintenance schedule.
16 . The method according to claim 1 , wherein:
the use case is obtained in response to a request made pursuant to (a) a new deployment, wherein the new deployment is a new site, a new device or a new service, and/or (b) a benchmarking task.
17 . The method according to claim 16 , wherein:
the step of determining one or more drive test routes comprises obtaining a predefined route corresponding to a location of the new deployment and/or a location of a benchmarking task.
18 . The method according to claim 1 , further comprising:
determining the number of drive test agents to whom the test scripts are to be sent, prior to sending the test scripts, wherein:
the number of drive test agents required for a drive test specific to a use case is determined using one or more of the following: (i) number of routes to be driven (ii) minimum required statistical significance of the measurements, and (iii) acceptable measurement standard error.
19 . The method according to claim 1 , comprising:
ordering a halt to a particular drive test agent's drive test, upon automatically receiving real time feedback from that drive test agent indicating a problem with either that drive test agent's equipment or the network.
20 . The method according to claim 1 , comprising:
updating the drive test of a given drive test agent, in response to automatically receiving real time feedback from that drive test agent.
21 . The method according to claim 1 , comprising:
determining the definition of done criteria further including minimum performance requirements for a set of key performance indicators (KPIs) constituting the test scripts or indication of equipment malfunction and/or or network malfunction.
22 . The method according to claim 1 , wherein all steps are performed without human intervention.
23 . The method according to claim 1 , wherein at least one of the following steps is performed by a human: (a) obtaining a use case; and (b) retrieving one or more test scripts.
24 . The method according to claim 1 , comprising:
providing six software entities collectively configured to implement a precision drive testing (PDT) methodology, the six software entities comprising:
(a) a use cases library defined by network/customer problems or network planning or new deployments or benchmarking and which require a drive test (DT);
(b) a test scripts library with minimized test key performance indicators (KPIs) (use cases centric masks on KPIs measured in a KPIs database), real time error/malfunctioning flags and success metric (pass/fail);
(c) a PDT application manager configured to implement control PDT triggering:
period, event triggered, scheduled;
(d) a PDT application manager executed by engineer or automated;
(e) a PDT Orchestrator configured to (i) create work orders, (ii) assign area/route drive test agents to the work orders, and (iii) create definition of done criteria; and
(f) PDT Area/routes categories: geography, predicted, tested, sweet spots.
25 . The method according to claim 1 , wherein the internal data source comprises real time network, real time service, or real time user experience quality sensing obtained from one or more predictive machine learning (ML)/artificial intelligence (AI) algorithms embedded in or accessible by the drive test agent.
26 . A method for automatically directing drive testing by field agents to assess network performance, service, or user experience in a geographical area, the method comprising:
automatically retrieving one or more test scripts corresponding to a use case;
automatically receiving real time feedback from a drive test agent regarding the status of a drive test;
automatically sending one or more instructions to the drive test agent in response to the received real time feedback provided by the drive test agent, the instructions comprising one or more of route updates, corrected test scripts, modified test scripts, or repeated test scripts; and
automatically receiving and uploading final drive test results upon satisfaction of a definition of done criteria by the drive test agent, the definition of done criteria including at least a statistical significance level threshold and an acceptable measurement error.