Method of handling a failure in a task pipeline between a source of alerts and a security incident and event management (SIEM) system
A method of handling a failure in a pipeline between a source of alerts and a Security Incident & Event Management (SIEM) platform involves: deserializing data from at least one error state object in an error log; transferring the deserialized data to an offline environment; reconstituting at least some tasks of the pipeline, including the failed task, using contents of the error log and contents from a task log; re-running the at least some tasks of the pipeline; identifying a cause of the failed task; based upon the cause of the failed task, transferring at least one of a modified task or replacement task for use in the pipeline; instantiating a new version of the pipeline incorporating the at least one of the modified task or the replacement task; and re-executing the new version of the pipeline to process alerts and provide the processed alerts to the SIEM platform.
1. A method of handling a failure in a task pipeline executed in a system comprising a case creation platform having at least one processor, non-transient storage, the pipeline and an error log containing at least one error state object resulting from a non-skippable failed task, wherein the case creation platform is functionally between a source of alerts and a Security Incident & Event Management (SIEM) platform, the method comprising:
deserializing data from the at least one error state object in the error log;
transferring the deserialized data to an offline environment comprising at least one processor and non-transient storage;
reconstituting at least some tasks of the pipeline, including the failed task, in the offline environment using contents of the error log and contents from a task log associated with the pipeline; wherein the reconstituting further comprises using a local output task in place of a sink task of the pipeline;
re-running the at least some tasks of the pipeline in the offline environment;
identifying a cause of the failed task based upon the re-running;
based upon the cause of the failed task, transferring at least one of a modified task or replacement task to the system for use in the pipeline;
instantiating a new version of the pipeline in the system incorporating the at least one of the modified task or the replacement task; and
re-executing the new version of the pipeline to process alerts and provide the processed alerts to the SIEM platform.
2. The method of claim 1 , wherein the reconstituting at least some tasks of the pipeline comprises: re-instantiating all tasks of the pipeline.
3. The method of claim 1 , wherein the reconstituting further comprises: using a local input task in place of a source task of the pipeline.
4. The method of claim 1 , wherein the re-running of the at least some tasks of the pipeline in the offline environment comprises: running an entirety of the pipeline using specific alert data originally sourced by the source task from the error log.
5. The method of claim 1 , wherein the re-running of the at least some tasks of the pipeline in the offline environment comprises: running a portion of the pipeline beginning from an input to the a specific task where the failure occurred.
6. The method of claim 1 , wherein the re-running the at least some tasks of the pipeline in the offline environment further comprises: re-running using a copy of input data received from the task log.
7. The method of claim 1 , wherein the re-running the at least some tasks of the pipeline in the offline environment further comprises: re-running using alternative data as input data.
8. The method of claim 7 wherein the alternative data comprises data newly gathered by a source task of the pipeline.
9. The method of claim 8 wherein the alternative data comprises data obtained using a parameter different from an original parameter.
10. The method of claim 9 wherein the parameter includes different time constraints.