Multi-modal construction of deep learning networks

1. A computer-implemented method for generating source code for a deep learning network, comprising:

extracting, from user-provided multi-modal inputs, one or more items related to generating a deep learning network model, wherein the one or more extracted items comprise an image of a deep learning network design and a classification task to be performed by the deep learning network model;

processing the image to extract information comprising at least one of: text from the image using an optical character recognition process; and edges and nodes from the image using an edge detection process;

retrieving a similar pre-existing deep learning network model, created using deep learning source code corresponding to a first library language, from a repository based at least in part on a comparison of: the information extracted from the image of the deep learning network design; and multiple deep learning network models stored in the repository;

adapting the retrieved pre-existing deep learning network model to the one or more multi-model inputs to generate the deep learning network model, wherein said adapting comprises: changing a last layer in the pre-existing deep learning network model to have a number of nodes corresponding to a number of classes associated with the classification task to be performed by the deep learning network model; and re-training and updating one or more parameters of the retrieved pre-existing deep learning network model;

creating an intermediate representation of the deep learning network model, wherein the intermediate representation comprises: one or more items of data pertaining to the deep learning network model; and one or more design details attributed to the deep learning network model;

automatically converting the intermediate representation into source code corresponding to a second library language that is different than the first library language;

automatically performing a static validation of the deep learning source code to determine whether one or more specified network layers are present in the generated deep learning network model; and

outputting the deep learning source code corresponding to the second library language to at least one user;

wherein the steps are carried out by at least one computing device.

2. The computer-implemented method of claim 1 , wherein the user-provided multi-modal inputs comprise one or more items of natural language conversation data.

3. The computer-implemented method of claim 2 , wherein said extracting comprises employing a knowledge graph to disambiguate one or more user intentions from the one or more items of natural language conversation data.

4. The computer-implemented method of claim 1 , wherein the user-provided multi-modal inputs comprise input source code written in a deep learning framework.

5. The computer-implemented method of claim 1 , wherein the deep learning network design image is automatically extracted from a research paper.

6. The computer-implemented method of claim 1 , wherein the user-provided multi-modal inputs comprise a user-provided dataset.

7. The computer-implemented method of claim 1 , wherein the user-provided multi-modal inputs comprise one or more inputs provided by the user via a drag-and-drop user interface.

8. The computer-implemented method of claim 1 , wherein the static validation comprises determining whether multiple pre-determined layers are present, in a pre-determined sequence, in the deep learning network model.

9. The computer-implemented method of claim 1 , comprising:

generating a suggestion to add one or more layers in the deep learning network model based on said static validation.

10. The computer-implemented method of claim 1 , comprising:

generating a suggestion to delete one or more layers in the deep learning network model based on said static validation.

11. The computer-implemented method of claim 1 , comprising:

generating a suggestion to swap one or more layers in the deep learning network model based on said static validation.

12. The computer-implemented method of claim 5 , wherein the research paper is provided as input by the at least one user, and wherein automatically extracting the image of the deep learning network design comprises:

extracting a candidate image from the research paper;

extracting edges and lines from the candidate image; and

applying a binary classifier to the extracted edges and lines to determine that the candidate image comprises the deep learning network design.

13. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to:

extract, from user-provided multi-modal inputs, one or more items related to generating a deep learning network model, wherein the one or more extracted items comprise an image of a deep learning network design and a classification task to be performed by the deep learning network model;

process the image to extract information comprising at least one of: text from the image using an optical character recognition process; and edges and nodes from the image using an edge detection process;

retrieve a similar pre-existing deep learning network model, created using deep learning source code corresponding to a first library language, from a repository based at least in part on a comparison of: the information extracted from the image of the deep learning network design; and multiple deep learning network models stored in the repository;

adapt the retrieved pre-existing deep learning network model to the one or more multi-model inputs to generate the deep learning network model, wherein said adapting comprises: changing a last layer in the pre-existing deep learning network model to have a number of nodes corresponding to a number of classes associated with the classification task to be performed by the deep learning network model; and re-training and updating one or more parameters of the retrieved pre-existing deep learning network model;

create an intermediate representation of the deep learning network model, wherein the intermediate representation comprises: one or more items of data pertaining to the deep learning network model; and one or more design details attributed to the deep learning network model;

automatically convert the intermediate representation into source code corresponding to a second library language that is different than the first library language;

automatically perform a static validation of the deep learning source code to determine whether one or more specified network layers are present in the generated deep learning network model; and

output the deep learning source code corresponding to the second library language to at least one user.

14. The computer program product of claim 13 , wherein the user-provided multi-modal inputs comprise one or more items of natural language conversation data.

15. A system comprising:

a memory configured to store program instructions; and

at least one processor operably coupled to the memory to execute the program instructions to:

extract, from user-provided multi-modal inputs, one or more items related to generating a deep learning network model, wherein the one or more extracted items comprise an image of a deep learning network design and a classification task to be performed by the deep learning network model;

process the image to extract information comprising at least one of: text from the image using an optical character recognition process; and edges and nodes from the image using an edge detection process;

retrieve a similar pre-existing deep learning network model, created using deep learning source code corresponding to a first library language, from a repository based at least in part on a comparison of: the information extracted from the image of the deep learning network design; and multiple deep learning network models stored in the repository;

adapt the retrieved pre-existing deep learning network model to the one or more multi-model inputs to generate the deep learning network model, wherein said adapting comprises: changing a last layer in the pre-existing deep learning network model to have a number of nodes corresponding to a number of classes associated with the classification task to be performed by the deep learning network model; and re-training and updating one or more parameters of the retrieved pre-existing deep learning network model;

create an intermediate representation of the deep learning network model, wherein the intermediate representation comprises: one or more items of data pertaining to the deep learning network model; and one or more design details attributed to the deep learning network model;

automatically convert the intermediate representation into source code corresponding to a second library language that is different than the first library language;

automatically perform a static validation of the deep learning source code to determine whether one or more specified network layers are present in the generated deep learning network model; and

output the deep learning source code corresponding to the second library language to at least one user.

16. The system of claim 15 , wherein the static validation comprises determining whether multiple pre-determined layers are present, in a pre-determined sequence, in the deep learning network model.

17. The system of claim 15 , wherein the user-provided multi-modal inputs comprise one or more items of natural language conversation data.

18. The system of claim 15 , wherein said extracting comprises employing a knowledge graph to disambiguate one or more user intentions from the one or more items of natural language conversation data.

19. The system of claim 15 , wherein the user-provided multi-modal inputs comprise input source code written in a deep learning framework.

20. A computer-implemented method for generating source code for a deep learning network, comprising:

creating a repository of multiple deep learning network models;

extracting, from user-provided multi-modal inputs, one or more items related to generating a deep learning network model, wherein the one or more extracted items comprise an image of a deep learning network design and a classification task to be performed by the deep learning network model;

processing the image to extract information comprising at least one of: text from the image using an optical character recognition process; and edges and nodes from the image using an edge detection process;

retrieving a similar pre-existing deep learning network model, created using deep learning source code corresponding to a first library language, from the repository based at least in part on a comparison of: the information extracted from the image of the deep learning network design; and the multiple deep learning network models stored in the repository;

adapting the retrieved pre-existing deep learning network model to the one or more user-provided inputs to generate the deep learning network model, wherein said adapting comprises: changing a last layer in the pre-existing deep learning network model to have a number of nodes corresponding to a number of classes associated with the classification task to be performed by the deep learning network model; and re-training and updating one or more parameters of the retrieved pre-existing deep learning network model;

creating an intermediate representation of the deep learning network model, wherein the intermediate representation comprises: one or more items of data pertaining to the deep learning network model; and one or more design details attributed to the deep learning network model;

automatically converting the intermediate representation into source code corresponding to a second library language that is different than the first library language; and

outputting the deep learning source code corresponding to the second library language to at least one user.