IP Library › Granted Patent US 12694298
Granted Patent B1
US 12694298 · App. 17/850,761 · Granted Jul 28, 2026

Verification of neural network model evolution

Inventors: Ahmed Irfan (Cupertino, CA); Homer Strong (Seattle, WA); Soon Ho Kong (Cupertino, CA); Atiye Alaeddini (Kirkland, WA); Aleksandar Nevenov Chakarov (San Jose, CA)
Assignee: Amazon Technologies, Inc.
G06N3/086
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12694298
App. No.
17/850,761
Granted
Jul 28, 2026
Kind
B1
Abstract

A plurality of result pairs may be compared of at least part of an input region of neural network model input data. Each of the plurality of result pairs may include a first result provided by a first neural network model for a respective subregion of the input region and a second result provided by a second neural network model for the respective subregion. The second neural network model may be an update of the first neural network model. It may be determined that there exists a non-equivalent result pair within the plurality of result pairs, wherein the first result of the non-equivalent result pair provided by the first neural network model differs from the second result of the non-equivalent result pair provided by the second neural network model. The second neural network model may be retrained based at least in part on existence of the non-equivalent result pair.

Claims (44)

1 . A computing system comprising:

one or more processors; and

one or more memories having stored therein instructions that, upon execution by the one or more processors, cause the computing system to perform computing operations comprising:

comparing a plurality of result pairs of at least part of an input region of neural network model input data, each of the plurality of result pairs including a first result provided by a first neural network model for a respective subregion of the input region and a second result provided by a second neural network model for the respective subregion of the input region, wherein the second neural network model is an update of the first neural network model, and wherein the second neural network model is trained subsequent to training of the first neural network model;

determining that there exists a non-equivalent result pair within the plurality of result pairs, wherein the first result of the non-equivalent result pair provided by the first neural network model differs from the second result of the non-equivalent result pair provided by the second neural network model;

retraining, based at least in part on existence of the non-equivalent result pair, the second neural network model;

repeating the comparing, the determining, and the retraining until there are no non-equivalent result pairs within the input region; and

indicating, based on an absence of the non-equivalent result pairs within the input region, that the second neural network model is ready to deploy.

2 . The computing system of claim 1 , wherein the retraining is performed using a retraining dataset that includes synthetic data corresponding to the non-equivalent result pair.

3 . The computing system of claim 1 , wherein the operations further comprise:

determining that the non-equivalent result pair corresponds to a valid use case, wherein the retraining is based in part on the non-equivalent result pair corresponding to the valid use case.

4 . The computing system of claim 3 , wherein other non-equivalent result pairs that do not correspond to valid use cases trigger refining of the input region for the comparing.

5 . A computer-implemented method comprising:

comparing a plurality of result pairs of at least part of an input region of neural network model input data, each of the plurality of result pairs including a first result provided by a first neural network model for a respective subregion of the input region and a second result provided by a second neural network model for the respective subregion of the input region;

determining that there exists a non-equivalent result pair within the plurality of result pairs, wherein the first result of the non-equivalent result pair provided by the first neural network model differs from the second result of the non-equivalent result pair provided by the second neural network model;

retraining, based at least in part on existence of the non-equivalent result pair, the second neural network model;

repeating the comparing, the determining, and the retraining until there are no non-equivalent result pairs within the input region; and

indicating, based on an absence of the non-equivalent result pairs within the input region, that the second neural network model is ready to deploy.

6 . The computer-implemented method of claim 5 , wherein the determining that there exists the non-equivalent result pair within the plurality of result pairs comprises:

determining a non-equivalent parent region within the input region that includes at least one non-equivalent result pair; and

iteratively partitioning the non-equivalent parent region into descendant regions until a threshold partition size is reached.

7 . The computer-implemented method of claim 6 , wherein each of the descendant regions that is a non-equivalent region is partitioned into two or more other of the descendant regions until the threshold partition size is reached.

8 . The computer-implemented method of claim 5 , wherein the retraining is performed using a retraining dataset that includes synthetic data corresponding to the non-equivalent result pair.

9 . The computer-implemented method of claim 5 , further comprising:

determining that the non-equivalent result pair corresponds to a valid use case, wherein the retraining is based in part on the non-equivalent result pair corresponding to the valid use case.

10 . The computer-implemented method of claim 9 , wherein other non-equivalent result pairs that do not correspond to valid use cases trigger refining of the input region for the comparing.

11 . The computer-implemented method of claim 5 , wherein the first neural network model and the second neural network model provide recommendations related to removal of security permissions.

12 . The computer-implemented method of claim 5 , wherein the first neural network model is trained using a first training dataset, wherein the second neural network model is trained using a second training dataset, and wherein the second training dataset is at least partially different from the first training dataset.

13 . The computer-implemented method of claim 5 , wherein the second neural network model is an update of the first neural network model.

14 . One or more non-transitory computer-readable storage media having stored thereon computing instructions that, upon execution by one or more computing devices, cause the one or more computing devices to perform computing operations comprising:

comparing a plurality of result pairs of at least part of an input region of neural network model input data, each of the plurality of result pairs including a first result provided by a first neural network model for a respective subregion of the input region and a second result provided by a second neural network model for the respective subregion of the input region;

determining that there exists a non-equivalent result pair within the plurality of result pairs, wherein the first result of the non-equivalent result pair provided by the first neural network model differs from the second result of the non-equivalent result pair provided by the second neural network model;

retraining, based at least in part on existence of the non-equivalent result pair, the second neural network model;

repeating the comparing, the determining, and the retraining until there are no non-equivalent result pairs within the input region; and

indicating, based on an absence of the non-equivalent result pairs within the input region, that the second neural network model is ready to deploy.

15 . The one or more non-transitory computer-readable storage media of claim 14 , wherein the determining that there exists the non-equivalent result pair within the plurality of result pairs comprises:

determining a non-equivalent parent region within the input region that includes at least one non-equivalent result pair; and

iteratively partitioning the non-equivalent parent region into descendant regions until a threshold partition size is reached.

16 . The one or more non-transitory computer-readable storage media of claim 15 , wherein each of the descendant regions that is a non-equivalent region is partitioned into two or more other of the descendant regions until the threshold partition size is reached.

17 . The one or more non-transitory computer-readable storage media of claim 14 , wherein the retraining is performed using a retraining dataset that includes synthetic data corresponding to the non-equivalent result pair.

18 . The one or more non-transitory computer-readable storage media of claim 14 , wherein the operations further comprise:

determining that the non-equivalent result pair corresponds to a valid use case, wherein the retraining is based in part on the non-equivalent result pair corresponding to the valid use case.

19 . The one or more non-transitory computer-readable storage media of claim 18 , wherein other non-equivalent result pairs that do not correspond to valid use cases trigger refining of the input region for the comparing.

20 . The one or more non-transitory computer-readable storage media of claim 14 , wherein the second neural network model is an update of the first neural network model.