Automated notebook completion using sequence-to-sequence transformer
Generally discussed herein are devices, systems, and methods for generating an automatic interactive digital notebook completion model. A method can include receiving notebook content of an interactive digital notebook, the notebook content including a markdown cell followed by a code cell. The method can include generating input/output examples by, for each input/output example by masking one of (i) content of the markdown cell or (ii) content of the code cell resulting in a masked cell, identifying the masked cell and content of another cell of the markdown cell or the code that is not masked as an input for an input/output example, and identifying the content of the masked cell as an output for the input/output example. The method can include training, based on the input/output examples, a natural language processing model that generates a prediction of the content of a second masked cell as an output.
1 . A method comprising:
receiving notebook content of an interactive digital notebook as the content is entered into the interactive digital notebook, the notebook content including a sequence of respective cells comprising a markdown cell, immediately followed by a code cell, and a grading cell, the markdown cell indicating programming code that is to be entered in the code cell, the grading cell including content that tests whether the programming code in the code cell or the markdown cell is accurate;
providing the notebook content, with one of the code cell and the markdown cell masked, to a natural language processing (NLP) model, the NLP model trained to predict content of respective masked cells of input/output examples, the input/output examples include one of (i) content of the markdown cell of an input/output example of the input/output examples masked or (ii) content of the code cell of an input/output example of the input/output examples masked, and respective grading cells;
receiving from the NLP model a prediction of multiple hypotheses of content of the masked cell;
evaluating the multiple hypotheses to identify hypotheses that are consistent with content in the grading cell;
presenting only hypotheses of the multiple hypotheses that are consistent with content in the grading cell; and
providing, by automatically inserting the predicted content into the masked cell, the prediction on the interactive digital notebook.
2 . The method of claim 1 , wherein the masked cell includes the contents of the code cell masked.
3 . The method of claim 1 , wherein the masked cell includes the contents of the markdown cell masked.
4 . The method of claim 1 , further comprising providing a control code to the NLP model indicating whether the masked cell is the markdown cell or the code cell.
5 . The method of claim 4 , wherein the control code further indicates whether the markdown cell includes a header, formula, or a combination thereof.
6 . The method of claim 4 , wherein the control code further indicates whether the code cell includes a function definition, a class, an import statement, or a combination thereof.
7 . The method of claim 1 , wherein the NLP model is a sequence-to-sequence transformer model.
8 . The method of claim 1 , further comprising:
providing input of a first type to the NLP model, the input including a masked cell; and
receiving, from the NLP model, output of a second different type as contents of the masked cell, wherein the first type is one of markdown or code and the second type is the other of markdown or code.
9 . A system comprising:
processing circuitry;
a memory including instructions that, when executed by the processing circuitry, cause the processing circuitry to perform operations comprising:
receiving notebook content of an interactive digital notebook as the content is entered into the interactive digital notebook, the notebook content including a sequence of respective cells comprising a markdown cell, immediately followed by a code cell, and a grading cell, the markdown cell indicating programming code that is to be entered in the code cell, the grading cell including content that tests whether the programming code in the code cell or the markdown cell is accurate;
providing the notebook content, with one of the code cell and the markdown cell masked, to a natural language processing (NLP) model, the NLP model trained to predict content of respective masked cells of input/output examples, the input/output examples include one of (i) content of the markdown cell of an input/output example of the input/output examples masked or (ii) content of the code cell of an input/output example of the input/output examples masked, and respective grading cells;
receiving from the NLP model a prediction of multiple hypotheses of content of the masked cell;
evaluating the multiple hypotheses to identify hypotheses that are consistent with content in the grading cell;
presenting only hypotheses of the multiple hypotheses that are consistent with content in the grading cell; and
providing, by automatically inserting the predicted content into the masked cell, the prediction on the interactive digital notebook.
10 . The system of claim 9 , wherein the masked cell includes the contents of the code cell masked.
11 . The system of claim 9 , wherein the masked cell includes the contents of the markdown cell masked.
12 . The system of claim 9 , wherein the operations further comprise providing a control code to the NLP model indicating whether the masked cell is the markdown cell or the code cell.
13 . A non-transitory machine-readable medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving notebook content of an interactive digital notebook as the content is entered into the interactive digital notebook, the notebook content including a sequence of respective cells comprising a markdown cell, immediately followed by a code cell, and a grading cell, the markdown cell indicating programming code that is to be entered in the code cell, the grading cell including content that tests whether the programming code in the code cell or the markdown cell is accurate;
providing the notebook content, with one of the code cell and the markdown cell masked, to a natural language processing (NLP) model, the NLP model trained to predict content of respective masked cells of input/output examples, the input/output examples include one of (i) content of the markdown cell of an input/output example of the input/output examples masked or (ii) content of the code cell of an input/output example of the input/output examples masked, and respective grading cells;
receiving from the NLP model a prediction of multiple hypotheses of content of the masked cell;
evaluating the multiple hypotheses to identify hypotheses that are consistent with content in the grading cell;
presenting only hypotheses of the multiple hypotheses that are consistent with content in the grading cell; and
providing, by automatically inserting the predicted content into the masked cell, the prediction on the interactive digital notebook.
14 . The non-transitory machine-readable medium of claim 13 , wherein the operations further comprise providing a control code to the NLP model indicating whether the masked cell is the markdown cell or the code cell and the control code further indicates whether the markdown cell includes a header, formula, or a combination thereof.
15 . The non-transitory machine-readable medium of claim 13 , wherein the operations further comprise providing a control code to the NLP model indicating whether the masked cell is the markdown cell or the code cell and the control code further indicates whether the code cell includes a function definition, a class, an import statement, or a combination thereof.
16 . The non-transitory machine-readable medium of claim 13 , wherein the operations further comprise:
providing input of a first type to the NLP model, the input including a masked cell; and
receiving, from the NLP model, output of a second different type as contents of the masked cell, wherein the first type is one of markdown or code and the second type is the other of markdown or code.