Methods and systems for search and ranking of code snippets using machine learning models
Systems and methods for automatically generating search list rankings of code snippets are provided. An exemplary method includes searching source code repositories to identify code snippets in response to a search query, assigning weight values to ranking parameters, and processing the code snippets using machine learning models to generate rating scores for each of the code snippets, where each rating score applies to a corresponding ranking parameter. The method includes generating a combined score for each of the code snippets by combining the rating scores for the code snippet according to the weight values assigned to the corresponding ranking parameters and generating and presenting a user interface including an ordered list of the code snippets based on the combined scores for the code snippets.
1. A system for automatically generating search list rankings of code snippets using one or more machine learning models, the system comprising:
one or more processors and memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
searching one or more source code repositories to identify a plurality of code snippets in response to a search query;
assigning a plurality of weight values to a plurality of ranking parameters, each of the plurality of ranking parameters corresponding to a different type of rating score for plurality of code snippets;
processing the plurality of code snippets using the one or more machine learning models to generate a plurality of rating scores for each of the plurality of code snippets, wherein each rating score of the plurality of rating scores applies to a corresponding ranking parameter of the plurality of ranking parameters;
generating a combined score for each of the plurality of code snippets, wherein the combined score for a code snippet is generated by combining the plurality of rating scores for the code snippet according to the plurality of weight values assigned to the corresponding ranking parameters; and
generating and presenting a user interface comprising an ordered list of the plurality of code snippets based on the combined score for each of the plurality of code snippets,
wherein the ranking parameters comprise a popularity score, a relevancy score, a recency score, an engagement score, and a document type score, wherein:
the popularity score is a standardized measurement relative to a popularity of a code snippet;
the relevancy score is a standardized measurement relative to a relevance of a code snippet to the search query;
the recency score is a standardized measurement relative to how recently a code snippet was used;
the engagement score is a standardized measurement relative to an amount of engagement a code snippet receives; and
the document type score is a standardized measurement relative to a type of document from which the code snippet was derived.
2. The system of claim 1 , wherein the popularity score, the relevancy score, the recency score, the engagement score, and the document type score are normalized across provider data and third party data by a machine learning model.
3. The system of claim 1 , wherein the popularity score is determined based on a number of views and/or user downloads of a code snippet across different providers of the code snippet, the different providers of the code snippet comprising a plurality of code repositories.
4. The system of claim 1 , wherein the relevancy score is determined by computing a similarity between the code snippet and the search query based on a L2 Euclidian distance.
5. The system of claim 1 , wherein the engagement score is determined based on a number of clicks, comments, likes, and shares of a code snippet across different providers of the code snippet, the different providers of the code snippet comprising a plurality of code repositories.
6. The system of claim 1 , wherein the operations further comprise:
scanning, by a data-crawler, at least one of public repositories, cloud providers, Q&A sites, or review sites to retrieve information on popularity, relevancy, recency, engagement, and document type information regarding a code snippet; and
storing, by the data-crawler, the information to the one or more machine learning models.
7. The system of claim 1 , the operations further comprising:
receiving a large volume of technical documents from a variety of sources, wherein the large volume of technical documents comprises at least 30 million documents;
pre-processing the large volume of technical documents to extract relevant data;
semantically vectorizing the technical documents using a pre-trained machine learning model; and
storing the vectorized technical documents in a vector database.
8. A method for automatically generating search list rankings of code snippets, the method comprising:
searching one or more source code repositories to identify a plurality of code snippets in response to a search query;
assigning a plurality of weight values to a plurality of ranking parameters, each of the plurality of ranking parameters corresponding to a different type of rating score for plurality of code snippets;
processing the plurality of code snippets using one or more machine learning models to generate a plurality of rating scores for each of the plurality of code snippets, wherein each rating score of the plurality of rating scores applies to a corresponding ranking parameter of the plurality of ranking parameters;
generating a combined score for each of the plurality of code snippets, wherein the combined score for a code snippet is generated by combining the plurality of rating scores for the code snippet according to the plurality of weight values assigned to the corresponding ranking parameters; and
generating and presenting a user interface comprising an ordered list of the plurality of code snippets based on the combined score for each of the plurality of code snippets,
wherein the ranking parameters comprise a popularity score, a relevancy score, a recency score, an engagement score, and a document type score, wherein:
the popularity score is a standardized measurement relative to a popularity of a code snippet;
the relevancy score is a standardized measurement relative to a relevance of a code snippet to the search query;
the recency score is a standardized measurement relative to how recently a code snippet was used;
the engagement score is a standardized measurement relative to an amount of engagement a code snippet receives; and
the document type score is a standardized measurement relative to a type of document from which the code snippet was derived.
9. The method of claim 8 , wherein the popularity score is determined based on a number of views and/or user downloads of a code snippet across different providers of the code snippet, the different providers of the code snippet comprising a plurality of code repositories.
10. The method of claim 8 , wherein the relevancy score is determined by computing a similarity between the code snippet and the search query based on a L2 Euclidian distance.
11. The method of claim 8 , wherein the engagement score is determined based on a number of clicks, comments, likes, and shares of a code snippet across different providers of the code snippet, the different providers of the code snippet comprising a plurality of code repositories.
12. The method of claim 8 , wherein the method further comprises:
scanning, by a data-crawler, at least one of public repositories, cloud providers, Q&A sites, or review sites to retrieve information on popularity, relevancy, recency, engagement, and document type information regarding a code snippet; and
storing, by the data-crawler, the information to the one or more machine learning models.
13. The method of claim 8 , further comprising:
receiving a large volume of technical documents from a variety of sources, wherein the large volume of technical documents comprises at least 30 million documents;
pre-processing the large volume of technical documents to extract relevant data;
semantically vectorizing the technical documents using a pre-trained machine learning model; and
storing the vectorized technical documents in a vector database.
14. One or more non-transitory computer-readable media storing instructions thereon, wherein the instructions, when executed by one or more processors, cause the one or more processors to:
search one or more source code repositories to identify a plurality of code snippets in response to a search query;
assign a plurality of weight values to a plurality of ranking parameters, each of the plurality of ranking parameters corresponding to a different type of rating score for plurality of code snippets;
process the plurality of code snippets using one or more machine learning models to generate a plurality of rating scores for each of the plurality of code snippets, wherein each rating score of the plurality of rating scores applies to a corresponding ranking parameter of the plurality of ranking parameters;
generate a combined score for each of the plurality of code snippets, wherein the combined score for a code snippet is generated by combining the plurality of rating scores for the code snippet according to the plurality of weight values assigned to the corresponding ranking parameters; and
generate and present a user interface comprising an ordered list of the plurality of code snippets based on the combined score for each of the plurality of code snippets,
wherein the ranking parameters comprise a popularity score, a relevancy score, a recency score, an engagement score, and a document type score, wherein:
the popularity score is a standardized measurement relative to a popularity of a code snippet;
the relevancy score is a standardized measurement relative to a relevance of a code snippet to the search query;
the recency score is a standardized measurement relative to how recently a code snippet was used;
the engagement score is a standardized measurement relative to an amount of engagement a code snippet receives; and
the document type score is a standardized measurement relative to a type of document from which the code snippet was derived.
15. The one or more non-transitory computer-readable media of claim 14 , wherein the popularity score is determined based on a number of views and/or user downloads of a code snippet across different providers of the code snippet, the different providers of the code snippet comprising a plurality of code repositories.
16. The one or more non-transitory computer-readable media of claim 14 , wherein the relevancy score is determined by computing a similarity between the code snippet and the search query based on a L2 Euclidian distance.
17. The one or more non-transitory computer-readable media of claim 14 , wherein the engagement score is determined based on a number of clicks, comments, likes, and shares of a code snippet across different providers of the code snippet, the different providers of the code snippet comprising a plurality of code repositories.