Granted Patent B1

US 12,613,930 · App. 17/855,370 · Granted Apr 28, 2026

Inventors: Jean-Philippe E. Martin (Mountain View, CA); Úlfar Erlingsson (Palo Alto, CA); Yijou Chen (Cupertino, CA)

Assignee: Fortinet, Inc.

G06F16/9537G06F9/455G06F9/545G06F16/254G06F16/9024G06F16/9038G06F16/9535G06F21/57H04L43/045H04L43/06H04L63/10H04L67/306H04L67/535G06F16/2456

Abstract

Claims (35)

1 . A method of ensuring exactly once data ingestion, the method comprising:

interposing an event streaming platform and a data ingestion pipeline between a computing environment and a data warehouse platform;

receiving, by the data ingestion pipeline, from an event stream of the event streaming platform, event data comprising a plurality of first records, wherein the event data is received in real-time as it is collected from respective compute assets of a plurality of compute assets operable in the computing environment;

generating, by the data ingestion pipeline, a plurality of second records each corresponding to a first record of the plurality of first records and comprising a log sequence number (LSN) describing a location in the event stream for a corresponding first record;

sending, by the data ingestion pipeline, to a data warehouse, the plurality of second records; and

after detecting occurrence of an error or a restart of the data ingestion pipeline, ensuring a given second record of the plurality of second records is stored exactly once in the data warehouse by:

querying, by the data ingestion pipeline, the data warehouse for an LSN for a last stored second record; and

requesting, by the data ingestion pipeline, from the event streaming platform, next event data based on the LSN for the last stored second record.

2 . The method of claim 1 , wherein each of the plurality of second records includes a machine identifier (MID) describing a source of the corresponding first record.

3 . The method of claim 2 , wherein the MID is based on a hash of one or more data points associated with the source of the corresponding first record.

4 . The method of claim 2 , wherein the MID is one of a plurality of MIDs generated at least partially in parallel for two or more groups of first records.

5 . The method of claim 1 , wherein each of the LSNs comprises a partition identifier for the event stream and an offset in an identified partition.

6 . The method of claim 1 , further comprising:

receiving a first confirmation from the data warehouse indicating a successful storage of one or more of the second records; and

sending, in response to the first confirmation, a second confirmation to the event streaming platform describing one or more of the first records corresponding to the one or more of the second records.

7 . The method of claim 6 , wherein the first confirmation comprises a highest LSN for the one or more of the second records.

8 . The method of claim 7 , wherein the second confirmation comprises the highest LSN for the one or more second records.

9 . The method of claim 1 wherein the next event data comprises one or more first records occurring after the LSN in the event stream.

10 . A computer program product for ensuring exactly once data ingestion, the computer program product embodied on a non-transitory computer readable medium, the computer program product including computer program instructions configurable to carry out the steps of;

receiving, by a data ingestion pipeline interposed between a computing environment and a data warehouse platform, from an event stream of an event streaming platform interposed between the data ingestion pipeline and the computing environment, event data comprising a plurality of first records;

generating, by the data ingestion pipeline, a plurality of second records each corresponding to a first record of the plurality of first records and comprising a log sequence number (LSN) describing a location in the event stream for a corresponding first record; and

sending, to a data warehouse, the plurality of second records; and

after detecting occurrence of an error or a restart of the data ingestion pipeline, ensuring a given second record of the plurality of second records is stored exactly once in the data warehouse by:

querying, by the data ingestion pipeline, the data warehouse for an LSN for a last stored second record; and

requesting, by the data ingestion pipeline, from the event streaming platform, next event data based on the LSN for the last stored second record.

11 . The computer program product of claim 10 , wherein the next event data comprises one or more first records occurring after the LSN in the event stream.

12 . The computer program product of claim 10 , wherein each of the plurality of second records includes a machine identifier (MID) describing a source of the corresponding first record.

13 . The computer program product of claim 12 , wherein the MID is based on a hash of one or more data points associated with the source of the corresponding first record.

14 . The computer program product of claim 12 , wherein the MID is one of a plurality of MIDs generated at least partially in parallel for two or more groups of first records.

15 . The computer program product of claim 10 , wherein each of the LSNs comprises a partition identifier for the event stream and an offset in an identified partition.

16 . The computer program product of claim 10 , wherein the steps further comprise:

receiving a first confirmation from the data warehouse indicating a successful storage of one or more of the second records; and

sending, in response to the first confirmation, a second confirmation to the event streaming platform describing one or more of the first records corresponding to the one or more of the second records.

17 . The computer program product of claim 16 , wherein the first confirmation comprises a highest LSN for the one or more of the second records.

18 . The computer program product of claim 17 , wherein the second confirmation comprises the highest LSN for the one or more second records.

Assignments (1)

Continuity (9)

References Cited (400)