Machine learning for detecting and modifying faulty controls
In some aspects, a computing system may use machine learning to determine whether a control is faulty or generate recommendations to make a modification to a control. may identify a portion of problematic computer code that implements the faulty control through the use of machine learning. A computing system may use machine learning to generate embeddings that map incident data and control data (e.g., computer-readable code of a control) to the same vector space. Further, a computing system may use a weighting mechanism that may be used to weight each sample used to train a machine learning model, which may allow a model to train more efficiently.
1 . A system for improving machine learning model training via training subsets associated with weights related to network control experience of control submitters, the system comprising:
one or more processors and one or more non-transitory media having instructions recorded thereon that, when executed by the one or more processors, cause operations comprising:
receiving, as part of a set of network controls, access restrictions corresponding to security compliance, the access restrictions comprising (i) a first text description describing a first restriction associated with a first user and with a first fault label indicating whether the first restriction is faulty and (ii) a second text description describing a second restriction associated with a second user and with a second fault label indicating whether the second restriction is faulty;
in connection with receiving the access restrictions, obtaining control experience scores associated with users that submitted the access restrictions, the control experience scores comprising (i) a first control experience score for the first user that is derived from first experience data indicating control experience of the first user and (ii) a second control experience score for the second user that is derived from second experience data indicating control experience of the second user;
based on the first control experience score satisfying a threshold score and the second control experience score failing to satisfy the threshold score, generating, as part of a training dataset, training samples comprising a first training sample associated with a first weight derived from the first control experience score, while excluding the second text description of the second restriction from the training dataset, wherein the first training sample comprises the first text description describing the first restriction and the first fault label;
training, based on the training dataset, a machine learning model to detect faulty network controls, wherein the machine learning model applies the first weight derived from the first control experience score to the first training sample to train the machine learning model; and
based on the first control experience score satisfying the threshold score and the second control experience score failing to satisfy the threshold score, causing presentation of a control template, comprising a plurality of prompts for writing a new control, on a second user interface for the second user, without causing presentation of the control template on a first user interface for the first user.
2 . A method comprising:
receiving a plurality of controls corresponding to compliance comprising (i) a first text description describing a first control associated with a first user and with a first label indicating whether the first control is faulty and (ii) a second text description describing a second control associated with a second user and with a second label indicating whether the second control is faulty;
in connection with receiving the plurality of controls corresponding to compliance, obtaining control experience scores associated with users that submitted the plurality of controls, the control experience scores comprising a first control experience score for the first user that is derived from first experience data indicating control experience of the first user and a second control experience score for the second user that is derived from second experience data indicating control experience of the second user;
based on the first control experience score satisfying a threshold score, generating, as part of a training dataset, training samples comprising the first text description, describing the first control, that is associated with a first weight derived from the first control experience score;
training, based on the training dataset, a machine learning model to detect faulty control, wherein the machine learning model applies the first weight derived from the first control experience score to a first training sample comprising the first text description describing the first control to train the machine learning model; and
based on the first control experience score satisfying the threshold score and the second control experience score failing to satisfy the threshold score, causing presentation of a control template, comprising a plurality of prompts for writing a new control, on a second user interface for the second user, without causing presentation of the control template on a first user interface for the first user.
3 . The method of claim 2 , wherein generating the training samples comprises, based on the first control experience score satisfying the threshold score and the second control experience score failing to satisfy the threshold score, generating, as part of the training dataset, the training samples comprising the first text description describing the first control, while avoiding inclusion of the second text description of the second control in the training dataset.
4 . The method of claim 2 , further comprising:
based on the second control experience score failing to satisfy the threshold score, activating, on the second user interface for the second user, an update control template comprising a plurality of prompts for updating the second control.
5 . The method of claim 2 , further comprising:
based on the first control experience score satisfying the threshold score, generating the first label indicating that the first control is not faulty.
6 . The method of claim 2 , further comprising:
generating the first label associated with the first control based on a quantity of incidents indicated for the first control by a data record obtained from a database.
7 . The method of claim 2 , wherein the first control experience score is derived from a number of controls viewed or submitted by the first user.
8 . The method of claim 2 , wherein the first control experience score is derived from a weighted average of a number of controls viewed by the first user and a number of controls submitted by the first user.
9 . The method of claim 2 , further comprising deriving the first weight by:
multiplying the first weight by a number of controls viewed or submitted by the first user; and
dividing the first weight by a number of incidents associated with the first control.
10 . The method of claim 2 , wherein training the machine learning model comprises modifying a plurality of weights of the machine learning model through backpropagation based on determining that a classification of the machine learning model does not match a label of a corresponding training sample.
11 . One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, causes operations comprising:
receiving a plurality of controls corresponding to compliance comprising (i) a first text description describing a first control associated with a first user and with a first label indicating whether the first control is faulty and (ii) a second text description describing a second control associated with a second user and with a second label indicating whether the second control is faulty;
in connection with receiving the plurality of controls corresponding to compliance, obtaining control experience scores associated with users that submitted the plurality of controls, the control experience scores comprising a first control experience score for the first user that is derived from first experience data indicating control experience of the first user and a second control experience score for the second user that is derived from second experience data indicating control experience of the second user;
based on the first control experience score satisfying a threshold score, generating, as part of a training dataset, training samples comprising the first text description, describing the first control, that is associated with a first weight derived from the first control experience score;
training, based on the training dataset, a machine learning model to detect faulty control, wherein the machine learning model applies the first weight derived from the first control experience score to a first training sample comprising the first text description describing the first control to train the machine learning model; and
based on the first control experience score satisfying a threshold score and the second control experience score failing to satisfy the threshold score, causing presentation of a control template, comprising a plurality of prompts for writing a new control, on a second user interface for the second user, without causing presentation of the control template on a first user interface for the first user.
12 . The one or more non-transitory computer-readable media of claim 11 , wherein generating the training samples comprises, based on the first control experience score satisfying the threshold score and the second control experience score failing to satisfy the threshold score, generating, as part of the training dataset, the training samples comprising the first text description describing the first control, while avoiding inclusion of the second text description of the second control in the training dataset.
13 . The one or more non-transitory computer-readable media of claim 11 , the operations further comprising:
based on the second control experience score failing to satisfy the threshold score, activating, on the second user interface for the second user, an update control template comprising a plurality of prompts for updating the second control.
14 . The one or more non-transitory computer-readable media of claim 11 , the operations further comprising:
based on the first control experience score satisfying the threshold score, generating the first label indicating that the first control is not faulty.
15 . The one or more non-transitory computer-readable media of claim 11 , the operations further comprising:
generating the first label associated with the first control based on a quantity of incidents indicated for the first control by a data record obtained from a database.
16 . The one or more non-transitory computer-readable media of claim 11 , wherein the first control experience score is derived from a number of controls viewed or submitted by the first user.
17 . The one or more non-transitory computer-readable media of claim 11 , wherein the first control experience score is derived from a weighted average of a number of controls viewed by the first user and a number of controls submitted by the first user.
18 . The one or more non-transitory computer-readable media of claim 11 , the operations further comprising deriving the first weight by:
multiplying the first weight by a number of controls viewed or submitted by the first user; and
dividing the first weight by a number of incidents associated with the first control.