IP Library › Granted Patent US 12663289

Granted Patent B2

US 12663289 · App. 17/649,698 · Granted Jun 23, 2026

Utilizing models to evaluate geolocation estimate quality without independent test data

Inventors: Christopher Michael Murphy (Bath, GB); Dave Padfield (Marlborough, GB); Oliver Tyce (Wroughton, GB)

Assignee: VIAVI Solutions Inc.

G01C21/3841 G01C21/005 G01C21/3848

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12663289
App. No.: 17/649,698
Granted: Jun 23, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A device may receive a geolocation estimation model and geolocation input data identifying geolocations. The device may process one or more sets of the geolocation input data, with a quality evaluator model, to generate one or more quality evaluation metrics. The device may modify the geolocation estimation model based on the one or more quality evaluation metrics.

Claims (136)

1 . A method, comprising:

receiving, by a device, geolocation input data identifying one or more geolocations and a machine learning model configured to generate one or more geolocation estimates,

wherein the geolocation input data includes one or more measured quantities associated with one or more of a channel quality indicator, a reference signal code power (RSCP), a received power level (RxLev), an E c /N 0 measurement, a frequency, a center frequency, a spectral power distribution, a phase, a phase shift, an angle of arrival, a time of signal arrival, or a time difference of arrival with respect to another signal;

removing, by the device, data from the geolocation input data to generate sub-populated geolocation input data,

wherein removing the data from the geolocation input data comprises one or more of:

removing all attributes of a particular type,

randomly removing attributes,

systematically exploring all sub-populated attribute combinations, or

utilizing combinations of attributes for which performance of the machine learning model has a current weakness relative to other combinations of attributes;

generating, by the device and using the machine learning model, the one or more geolocation estimates based on the sub-populated geolocation input data;

generating, by the device, a plurality of quality evaluation metrics based on the one or more geolocation estimates,

wherein the plurality of quality evaluation metrics includes one or more of:

a metric identifying a degree to which the one or more geolocation estimates are consistent with other geolocation estimates, or

a metric identifying a degree to which the one or more geolocation estimates are consistent with the geolocation input data;

combining, by the device, the plurality of quality evaluation metrics to generate one or more aggregate quality evaluation metrics,

wherein combining the plurality of quality evaluation metrics comprises one or more of:

averaging the plurality of quality evaluation metrics,

performing a summation the plurality of quality evaluation metrics,

performing a weighted summation of the plurality of quality evaluation metrics,

performing a multiplication of the plurality of quality evaluation metrics, or

performing a reduction of the plurality of quality evaluation metrics to an extreme value; and

causing, by the device and to improve an accuracy of geolocation estimates generated by the machine learning model, a modification of the machine learning model to be implemented in a network,

wherein the modification is performed by constructing an objective function based on the one or more aggregate quality evaluation metrics.

2 . The method of claim 1 , wherein the geolocation input data further includes one or more of:

observations associated with the geolocations, or

configuration values associated with the geolocations.

3 . The method of claim 1 , wherein the one or more geolocation estimates include:

first estimates of locations, of a first entity during a first time period, based on a first set of one or more measurements performed by the first entity during the first time period, and

second estimates of locations, of a second entity during a second time period, based on a second set of one or more measurements performed by the second entity during the second time period, and

wherein generating the plurality of quality evaluation metrics comprises:

generating the plurality of quality evaluation metrics based on the first estimates and the second estimates.

4 . The method of claim 1 , wherein the one or more geolocation estimates include:

a sequence of first estimates of locations, of a first entity during a first time period, based on a first sequence of sets of one or more measurements performed by the first entity during the first time period, and

a sequence of second estimates of locations, of a second entity during a second time period, based on a second sequence of sets of one or more measurements performed by the second entity during the second time period, and

wherein generating the plurality of quality evaluation metrics comprises:

generating the plurality of quality evaluation metrics based on the sequence of first estimates and the sequence of the second estimates.

5 . The method of claim 1 , wherein the one or more geolocation estimates include:

estimates of locations, of a first entity during a first time period, based on:

a set of one or more measurements associated with relative positions of the first entity during the first time period and of a second entity during a second time period, or

information identifying the relative positions, and

wherein generating the plurality of quality evaluation metrics comprises:

generating the plurality of quality evaluation metrics based on the estimates.

6 . The method of claim 1 , wherein generating the plurality of quality evaluation metrics comprises:

determining a first time associated with reception of a first signal transmitted by a first transmitting device;

determining a second time associated with reception of a second signal transmitted by a second transmitting device;

calculating a difference between the first time and the second time; and

generating one of the plurality of quality evaluation metrics based on the difference.

7 . A device, comprising:

one or more processors configured to:

receive geolocation input data identifying one or more geolocations and a machine learning model configured to generate one or more geolocation estimates,

remove data from the geolocation input data to generate sub-populated geolocation input data,

wherein the one or more processors, to remove the data from the geolocation input data, are configured to one or more of:

remove all attributes of a particular type,

randomly remove attributes,

systematically explore all sub-populated attribute combinations, or

utilize combinations of attributes for which performance of the machine learning model has a current weakness relative to other combinations of attributes;

generate, using the machine learning model, the one or more geolocation estimates based on the sub-populated geolocation input data;

generate a plurality of quality evaluation metrics based on the one or more geolocation estimates,

wherein the plurality of quality evaluation metrics includes one or more of:

a metric identifying a degree to which the one or more geolocation estimates are consistent with other geolocation estimates, or

a metric identifying a degree to which the one or more geolocation estimates are consistent with the geolocation input data,

combine the plurality of quality evaluation metrics to generate one or more aggregate quality evaluation metrics,

wherein the one or more processors, to combine the plurality of quality evaluation metrics, are configured to one or more of:

average the plurality of quality evaluation metrics,

perform a summation the plurality of quality evaluation metrics,

perform a weighted summation of the plurality of quality evaluation metrics,

perform a multiplication of the plurality of quality evaluation metrics, or

perform a reduction of the plurality of quality evaluation metrics to an extreme value; and

cause, to improve an accuracy of geolocation estimates generated by the machine learning model, a modification of the machine learning model to be implemented in a network,

wherein the modification is performed by constructing an objective function based on the one or more aggregate quality evaluation metrics.

8 . The device of claim 7 , wherein the one or more processors, to generate the plurality of quality evaluation metrics, are configured to:

determine a likelihood that an entity is located at an estimated geolocation of the one or more geolocation estimates; and

generate one of the plurality of quality evaluation metrics based on the likelihood.

9 . The device of claim 7 , wherein the one or more processors, to generate the plurality of quality evaluation metrics, are configured to:

identify a sequence of estimates of a location of an entity as the location changes over time; and

generate one of the plurality of quality evaluation metrics based on the sequence of estimates.

10 . The device of claim 7 , wherein the one or more processors, to generate the plurality of quality evaluation metrics, are configured to:

identify a classification of a state class for an entity; and

generate one of the plurality of quality evaluation metrics based on the classification.

11 . The device of claim 7 , wherein the one or more processors are configured to one or more of:

provide the plurality of quality evaluation metrics for display; or

retrain a different machine learning model, configured to generate the plurality of quality evaluation metrics, based on the plurality of quality evaluation metrics.

12 . The device of claim 7 , wherein the one or more processors are further configured to one or more of:

modify one or more parameters of the machine learning model based on the plurality of quality evaluation metrics; or

correct an error in the geolocation input data based on the plurality of quality evaluation metrics.

13 . The device of claim 7 , wherein the one or more processors are further configured to one or more of:

modify, based on the plurality of quality evaluation metrics, a different machine learning model used to generate the plurality of quality evaluation metrics; or

receive feedback associated with the plurality of quality evaluation metrics and modify the different machine learning model based on the feedback.

14 . A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:

one or more instructions that, when executed by one or more processors of a device, cause the device to:

receive geolocation input data identifying one or more geolocations and a machine learning model configured to generate one or more geolocation estimates,

remove data from the geolocation input data to generate sub-populated geolocation input data,

wherein the one or more instruction, to cause the device to remove the data from the geolocation input data, cause the device to one or more of: