IP Library › Granted Patent US 12681783

Granted Patent B1

US 12681783 · App. 18/538,929 · Granted Jul 14, 2026

Approximate duplicate event filter

Inventors: Karl Eric Anderson (Columbia, MD); Kelly Anne Rooker (Ellicott City, MD); Nebi Mert Aydin (Odenton, MD); Joshua Kupershmidt (Somerville, MA); Menachem Mendel Lerner (Baltimore, MD); Samuel Macaluso (Arlington, VA); Jacob A. Niebloom (Baltimore, MD); Martin C Stonebraker (Sparks, MD); Aditya Tiwari (Woburn, MA)

Assignee: Amazon Technologies, Inc.

G06F9/542 G06F12/0864 G06F40/284

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12681783
App. No.: 18/538,929
Granted: Jul 14, 2026
Kind: B1

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Methods and systems of using an approximate duplicate event filter are disclosed. Event processing systems of a service provider network such as a security service may receive a large volume of events to analyze. Events that have previously occurred can be given a lower priority for analysis than new events. Events that are similar to previous events can likely also be given a lower priority for analysis. These non-exact match events can be detected by comparing token representations of new events to token representations of previous events. Events may be considered according to the context in which the events occur.

Claims (57)

1 . A system, comprising,

one or more processors;

one or more memory storing program instructions for implementing an event-deduplication engine for an event analysis service; and

one or more storage devices configured to implement a cache that stores a state of event history for events processed by the event analysis service, wherein the state of event history comprises token representations of prior events, wherein the token representations are associated with context keys;

wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to:

identify a possible new event based on a context key of the possible new event;

generate a set of token representations of the possible new event;

evaluate the set of the token representations of the possible new event, wherein evaluation of the set of the token representations of the possible new event comprises generating a ratio of a number of the token representations of the possible new event that are included in the state of event history and associated with the context key of the possible new event to a number of the token representations of the possible new event;

determine, based on the evaluation of the set of the token representations of the possible new event, whether the possible new event is an approximate duplicate of one or more of the prior events based on the ratio being above a threshold for identifying approximate duplicates;

output, based on a determination that the possible new event is not an approximate duplicate of a recent prior event, the possible new event; and

update the state of event history to include the set of the token representations of the possible new event.

2 . The system of claim 1 , wherein to generate the set of the token representations of the possible new event, the program instructions further cause the one or more processors to:

identify grammatical components of a string included in the possible new event with a word parser; or

generate a set of fixed length substring patterns.

3 . The system of claim 1 , wherein to generate the set of the token representations of the possible new event, the program instructions further cause the one or more processors to:

generate hash values for the token representations, wherein the stored token representations of the prior event are stored as hash values; and

wherein to update the state of event history to include the set of the token representations of the possible new event the program instructions further cause the one or more processors to:

store the generated hash values for the token representations in the cache storing the state of event history.

4 . The system of claim 1 , wherein the program instructions, when executed on or across the one or more processors, further cause the one or more processors to:

determine whether the context key of the possible new event is associated with a context key stored in a context key history; and

output the possible new event, based on a determination that the context key of the possible new event is not associated with a context key stored in the context key history.

5 . The system of claim 1 , wherein the one or more processors are a plurality of processors and wherein the program instructions, when executed on or across the plurality of processors, cause the plurality of processors to:

determine whether the context key of the possible new event is included in the context keys associated with the token representations of the state of event history; and

perform said evaluation of the set of the token representations of the possible new event on a processor of the plurality of processors designated for the context key of the possible new event, wherein the processor has access to the stored token representations of the state of event history associated with the context key of the possible new event.

6 . The system of claim 1 , wherein to update the state of event history to include the set of the token representations of the possible new event, the program instructions, when executed on or across the one or more processors, further cause the one or more processors to expire individual ones of the token representations of the prior events from the state of event history while the event analysis service continues to process the possible new event or other possible new events.

7 . A method, comprising,

identifying a possible new event based on a context key of the possible new event;

generating a set of token representations of the possible new event;

evaluating the set of the token representations of the possible new event, wherein evaluation of the token representations of the possible new event comprises generating a probabilistic determination of a duplicate event based on a number of token representations of the possible new event that are included in a state of event history for events processed by an event analysis service, wherein the state of event history comprises a set of token representations of prior events;

determining, based on the evaluation of the token representations of the possible new event, that the possible new event is not an approximate duplicate of one or more of the prior events based on the probabilistic determination of a duplicate event and a threshold for approximate new events;

outputting, based on a determination that the possible new event is not an approximate duplicate of a recent prior event, the possible new event; and

updating the state of event history to include the token representations of the possible new event.

8 . The method of claim 7 , further comprising:

determining the context key of the possible new event is not included in the state of event history, wherein the state of event history comprises a set of context keys of the prior events; and

outputting, based on a determination that the context key of the possible new event is not included in the state of event history, the possible new event.

9 . The method of claim 7 , further comprising:

determining a second possible new event is included in the state of event history; and

labeling, based on a determination that the second possible new event is included in the state of event history, the second possible new event as a duplicate event.

10 . The method of claim 7 , wherein updating the state of event history to include the token representations of the possible new event comprises expiring individual ones of the token representations of prior events from the state of event history while the event analysis service continues to process the possible new event or other possible new events.

11 . The method of claim 10 , wherein said expiring individual ones of the token representations of prior events from the state of event history is performed based on an amount of available storage for the state of event history.

12 . The method of claim 8 , wherein updating the state of event history to include the token representations of the possible new event comprises expiring one or more of the token representations of prior events from the state of event history based on a time since the one or more token representations of the prior events were included in the state of event history.

13 . The method of claim 7 , wherein the state of event history comprises the set of token representations of prior events, wherein the set of token representations comprises a combination of a context key of an individual one of a prior event and an individual one of a token representation of the individual one of the prior event, represented as a hash value.

14 . The method of claim 7 , wherein the probabilistic determination is a Jaccard distance.

15 . The method of claim 7 , wherein the probabilistic determination is a score generated based on a weight assigned to individual ones of the token representations of the possible new event.

16 . The method of claim 15 , wherein the weight is based on a frequency of occurrence within the state of event history of individual ones of the token representations of the possible new event.

17 . The method of claim 15 , wherein the weight is based on a count of occurrence within the state of event history of individual ones of the token representations of the possible new event.

18 . The method of claim 7 , further comprising removing or replacing strings in the possible new event that are determined to be random strings before said generating a set of token representations of the possible new event.

19 . A non-transitory computer-readable storage medium storing program instructions that, when executed on or across one or more processors, cause the one or more processors to:

identify a possible new event based on a context key of the possible new event;

generate a set of token representations of the possible new event;

evaluate the set of the token representations of the possible new event, wherein evaluation of the token representations of the possible new event comprises generating a probabilistic determination based on a number of token representations of the possible new event that are included in a state of event history for events processed by an event analysis service, wherein the state of event history comprises a set of token representations of prior events;

determine, based on the evaluation of the token representations of the possible new event, whether the possible new event is an approximate duplicate of one or more of the prior events, based on the probabilistic determination;

output, based on a determination that the possible new event is not an approximate duplicate of a recent prior event, the possible new event; and

update the state of event history to include the token representations of the possible new event.

20 . The computer-readable storage medium of claim 19 , wherein the probabilistic determination is:

a ratio of a number of the token representations of the possible new event that are included in the state of event history and associated with the context key of the possible new event to a number of the token representations of the possible new event; or

a score generated based on a weight assigned to individual ones of the token representations of the possible new event.