Granted Patent B1

US 12,596,618 · App. 18/651,689 · Granted Apr 7, 2026

Inventors: Sayantan Bhattacharyya (Sydney, AU); Wendi Qiu (Norwest, AU); Christopher Michael Sutton (North Ryde, AU)

Assignee: Cisco Technology, Inc.

G06F11/1464G06F11/1469H04L67/1074

Abstract

Claims (69)

1 . A computer-implemented method, comprising:

receiving, by a cluster manager of a peer cluster comprising a plurality of peer nodes, a notification of a corrupted bucket stored in the peer cluster; and

performing, by the cluster manager, a bucket recovery operation, comprising:

for each peer node of a subset of the plurality of peer nodes, sending a request to the peer node to report a bucket health metric for a bucket copy corresponding to the corrupted bucket stored on the peer node,

obtaining, from each peer node of the subset of the plurality of peer nodes, a corresponding bucket health metric comprising a searchable event count of the respective bucket copy stored on the peer node,

electing, as a canonical bucket copy, a bucket copy having a searchable event count with a highest value, wherein the canonical bucket copy is stored on an elected peer node, and

sending a replicate command to the elected peer node storing the canonical bucket copy to replicate the canonical bucket copy to the plurality of peer nodes of the peer cluster.

2 . The method of claim 1 , wherein the subset of the plurality of peer nodes includes peer nodes that store the bucket copy corresponding to the corrupted bucket.

3 . The method of claim 1 , wherein

the request to report the bucket health metric is received by a first peer node of the subset of the plurality of peer nodes; and

wherein the method further comprises rebuilding, by the first peer node, the bucket copy stored on the first peer node responsive to the request.

4 . The method of claim 1 , further comprising:

reporting, by a first peer node of the subset of the plurality of peer nodes, to the cluster manager, a bucket rebuild process status of the first peer node and an updated bucket health metric of the bucket copy stored on the first peer node,

wherein the updated bucket health metric includes at least the searchable event count.

5 . The method of claim 1 , further comprising:

by the cluster manager, prior to sending the replicate command:

sending an instruction to a first peer node of the subset of the plurality of peer nodes to discard the respective bucket copy stored on the first peer node,

wherein the searchable event count corresponding to the respective bucket copy stored on the first peer node is less than the searchable event count corresponding to the canonical bucket copy.

6 . The method of claim 1 , further comprising:

adding, by the cluster manager, an identifier of the corrupted bucket to a corrupted bucket queue, wherein the corrupted bucket queue comprises a plurality of identifiers corresponding to a plurality of corrupted buckets.

7 . The method of claim 6 , further comprising:

scheduling, by the cluster manager, a plurality of concurrent instances of a bucket recovery process for each corrupted bucket of at least a subset of the plurality of identifiers in the corrupted bucket queue,

wherein the plurality of concurrent instances of the bucket recovery process is less than a concurrent instance threshold.

8 . The method of claim 1 , wherein the peer cluster further comprises the plurality of peer nodes, at least one search head, and the cluster manager, and

wherein a number of plurality of peer nodes in the peer cluster corresponds at least to a replication factor, and

wherein the plurality of peer nodes stores a plurality of machine generated events in a plurality of buckets, the plurality of buckets comprising the corrupted bucket; and

wherein a number of bucket copies corresponding to the corrupted bucket corresponds to the replication factor.

9 . The method of claim 1 , further comprising:

receiving, by the cluster manager, the notification of the corrupted bucket from an administrative application, as at least one of a representational state transfer (REST) call, a command line instruction (CLI), and a Web User Interface (WebUI) call.

10 . The method of claim 1 , wherein receiving the notification of the corrupted bucket comprises:

receiving, by the cluster manager, the notification of the corrupted bucket from a first peer node of the peer cluster.

11 . The method of claim 1 , further comprising:

generating, by a first peer node, a read error notification; and

adding, by a peer engine executing on the first peer node, an identifier of the corrupted bucket to a corrupted bucket queue corresponding to the first peer node, wherein the corrupted bucket queue corresponding to the first peer node comprises a plurality of identifiers of a plurality of corrupted buckets.

12 . The method of claim 11 , further comprising:

periodically sending, by the first peer node, the corrupted bucket queue as the notification to the cluster manager by a REST Hypertext Transfer Protocol (HTTP) request.

13 . The method of claim 1 , further comprising:

determining, by a first peer node, during a restart process,

a mismatch between a bucket journal size corresponding to a bucket health metric of a bucket of the first peer node and a bucket journal size stored in a bucket manifest entry corresponding to the bucket from a bucket manifest stored on the first peer node, and

transmitting, by the first peer node, the notification responsive to the mismatch.

14 . A computing device, comprising:

a processor; and

a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising:

receiving, by a cluster manager of a peer cluster comprising a plurality of peer nodes, a notification of a corrupted bucket stored in the peer cluster; and

performing, by the cluster manager, a bucket recovery operation, comprising:

for each peer node of a subset of the plurality of peer nodes, sending a request to the peer node to report a bucket health metric for a bucket copy corresponding to the corrupted bucket stored on the peer node,

obtaining, from each peer node of the subset of the plurality of peer nodes, a corresponding bucket health metric comprising a searchable event count of the respective bucket copy stored on the peer node,

electing, as a canonical bucket copy, a bucket copy having a searchable event count with a highest value, wherein the canonical bucket copy is stored on an elected peer node, and

sending a replicate command to the elected peer node storing the canonical bucket copy to replicate the canonical bucket copy to the plurality of peer nodes of the peer cluster.

15 . The computing device of claim 14 , wherein

the subset of the plurality of peer nodes includes peer nodes that store the bucket copy corresponding to the corrupted bucket.

16 . The computing device of claim 14 , wherein

the request to report a bucket health metric is received by a first peer node of the subset of the plurality of peer nodes; and

wherein the operations further comprise rebuilding, by the first peer node, the bucket copy stored on the first peer node responsive to the request.

17 . The computing device of claim 14 , wherein the operations further comprise:

reporting, by a first peer node of the subset of the plurality of peer nodes, to the cluster manager, a bucket rebuild process status of the first peer node and an updated bucket health metric of the bucket copy stored on the first peer node,

wherein the updated bucket health metric includes at least the searchable event count.

18 . The computing device of claim 14 , wherein the operations further comprise:

prior to sending the replicate command:

sending an instruction to a first peer node of the subset of the plurality of peer nodes to discard the respective bucket copy stored on the first peer node,

wherein the searchable event count corresponding to the respective bucket copy stored on the first peer node is less than the searchable event count corresponding to the canonical bucket copy.

19 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving, by a cluster manager of a peer cluster comprising a plurality of peer nodes, a notification of a corrupted bucket stored in the peer cluster; and

performing, by the cluster manager, a bucket recovery operation, comprising:

for each peer node of a subset of the plurality of peer nodes, sending a request to the peer node to report a bucket health metric for a bucket copy corresponding to the corrupted bucket stored on the peer node,

obtaining, from each peer node of the subset of the plurality of peer nodes, a corresponding bucket health metric comprising a searchable event count of the respective bucket copy stored on the peer node,

electing, as a canonical bucket copy, a bucket copy having a searchable event count with a highest value, wherein the canonical bucket copy is stored on an elected peer node, and

sending a replicate command to the elected peer node storing the canonical bucket copy to replicate the canonical bucket copy to the plurality of peer nodes of the peer cluster.

20 . The non-transitory computer-readable medium of claim 19 , wherein the subset of the plurality of peer nodes includes peer nodes that store the bucket copy corresponding to the corrupted bucket.

Assignments (3)

References Cited (17)