IP Library › Granted Patent US 12,314,707
Granted Patent B2
US 12,314,707 · App. 17/985,849 · Granted May 27, 2025

Pre-training for automating code review activities

Inventors: Nan Duan (Beijing, CN); Shengyu Fu (Redmond, WA); Shuai Lu (Beijing, CN); Neelakantan Sundaresan (Bellevue, WA); Alexey Svyatkovskiy (Bellevue, WA)
Assignee: Microsoft Technology Licensing, LLC.
G06F8/71G06N3/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,314,707
App. No.
17/985,849
Granted
May 27, 2025
Kind
B2
Abstract

A deep learning model is pre-trained with a large-scale of unsupervised data of code review tasks in order to learn the relationships between code changes and a code review. The pre-trained deep learning model predicts a code review given a code diff hunk in a code diff format. The code diff hunk includes the changed code and its surrounding context. The pre-trained deep learning model may then be fine-tuned with supervised data in order to make predictions for several code review activities, such as, code change quality estimation and code refinement.

Claims (54)

1. A computer-implemented method, comprising:

searching a source repository for a plurality of code change snippets and a plurality of code reviews, one or more of the plurality of code reviews associated with select ones of the code change snippets;

obtaining each of the plurality of code change snippets in a code diff format, wherein the code diff format includes one or more tags, wherein a tag represents an edit made to an original code associated with a select code change snippet of the plurality of code change snippets;

transforming each of the plurality of code change snippets into a code diff hunk, wherein the code diff hunk includes changed code and surrounding context;

randomly denoising each of the code diff hunks of the plurality of code change snippets and each of the plurality of code reviews; and

generating a pre-trained deep learning model from the randomly denoised code diff hunks of the plurality of code change snippets and the randomly denoised code reviews, wherein the pre-trained deep learning model predicts a code review given an input code diff hunk.

2. The computer-implemented method of claim 1 ,

prior to randomly denoising each of the code diff hunks of the plurality of code change snippets and each of the plurality of code reviews, transforming each of the one or more tags into a special token, wherein the special token represents a specific edit.

3. The computer-implemented method of claim 1 ,

wherein randomly denoising each of the code diff hunks of the plurality of code change snippets and each of the plurality of code reviews further comprises:

randomly denoising the one or more tags associated with each of the plurality of code change snippets.

4. The computer-implemented method of claim 1 ,

wherein randomly denoising each of the code diff hunks of the plurality of code change snippets and each of the plurality of code reviews further comprises:

randomly denoising tokens associated with each of the plurality of code reviews.

5. The computer-implemented method of claim 1 ,

wherein randomly denoising each of the code diff hunks of the plurality of code change snippets and each of the plurality of code reviews further comprises:

randomly denoising lines of source code in each of the plurality of code change snippets.

6. The computer-implemented method of claim 1 ,

wherein the pre-trained deep learning model includes a neural transformer model with attention including at least one encoder block and at least one decoder block.

7. The computer-implemented method of claim 6 , further comprising:

training the at least one encoder block of the pre-trained deep learning model with a fine-tuning dataset to generate a code quality estimation model, wherein the fine-tuning dataset includes samples of code diff hunks with a label, wherein the label indicates whether a code review is needed for the code diff hunk, wherein the code quality estimation model predicts a label for a given code diff hunk.

8. The computer-implemented method of claim 1 , further comprising:

training the pre-trained deep learning model with a fine-tuning dataset to generate a code refinement model, wherein the fine-tuning dataset includes samples of original source code and an associated code review, wherein the code refinement model predicts a code review given an input original source code and a corresponding code review.

9. The computer-implemented method of claim 1 , wherein the surrounding context includes one or more lines of source code surrounding changed code that have not been changed.

10. A system comprising:

a processor and a memory;

wherein the memory stores a program configured to be executed by the processor,

wherein the program comprises instructions that when executed by the processor performs actions that:

obtain a plurality of code change snippets in a code diff format and a plurality of code reviews, wherein select ones of the plurality of code change snippets are associated with a particular code review, wherein the code diff format includes at least one tag representing an edit;

construct a first plurality of pre-training samples from the plurality of code change snippets and a second plurality of pre-training samples from the plurality of code reviews, wherein each of the first plurality of pre-training samples includes lines of source code in the code diff format with a respective surrounding context, wherein each of the second plurality of pre-training samples includes tokens of a select code review of the plurality of code reviews;

randomly denoise the lines of source code in each of the code change snippets of the first plurality of pre-training samples, the at least one tag in each code change snippet of the first plurality of pre-training samples and the tokens of each code review of the plurality of code reviews of the second plurality of pre-training samples; and

generate a code review deep learning model through pre-training a deep learning model with the first plurality of pre-training samples and the second plurality of pre-training samples, wherein the code review deep learning model learns to generate a particular code review given an input code change snippet with an associated surrounding context in the code diff format.

11. The system of claim 10 , wherein the program comprises instructions that when executed by the processor performs actions that:

obtain a fine-tuning dataset including a plurality of triplets, wherein a triplet includes an original source code snippet, an associated code review, and a corresponding code change snippet, wherein the corresponding code change snippet is derived from application of the associated code review to the original source code snippet; and

generate a code refinement model by fine-tuning the pre-trained deep learning model with the fine-tuning dataset, wherein the code refinement model generates refined code given an input original source code snippet and a corresponding code review.

12. The system of claim 11 , wherein the code refinement model is a neural transformer model having at least one encoder block and at least one decoder block.

13. The system of claim 10 , wherein the deep learning model includes a neural transformer model with attention having at least one encoder block and at least one decoder block.

14. The system of claim 13 , wherein the program comprises instructions that when executed by the processor performs actions that:

construct a fine-tuning dataset including a plurality of fine-tuning samples, each fine-tuning sample including a code diff hunk and an associated label; and

generate a code diff estimation model using the at least one encoder block of the pre-trained deep learning model trained on the fine-tuning dataset, wherein the code diff estimation model predicts whether or not an input code diff hunk needs an associated code review.

15. The system of claim 10 , wherein the program comprises instructions that when executed by the processor performs actions that:

deploy the pre-trained deep learning model in a version-controlled source code repository to automate code review.

16. A hardware storage device having stored thereon computer executable instructions that are structured to be executable by a processor of a computing device to thereby cause the computing device to perform actions that:

obtain a plurality of code change snippets in a code diff format and a plurality of code reviews, wherein select ones of the plurality of code change snippets are associated with a particular code review, wherein the code diff format includes at least one tag representing an edit;

construct a first plurality of pre-training samples from the plurality of code change snippets and a second plurality of pre-training samples from the plurality of code reviews, wherein the first plurality of pre-training samples includes lines of source code in the code diff format with a respective surrounding context, wherein the second plurality of pre-training samples includes tokens of each code review of the plurality of code reviews;

randomly denoise the lines of source code in each of the code change snippets of the first plurality of pre-training samples, the at least one tag in each code change snippet of the first plurality of pre-training samples and the tokens of each code review of the plurality of code reviews of the second plurality of pre-training samples; and

train a deep learning model to learn to generate a code review, when given an input code change snippet with an associated surrounding context in the code diff format, with the first plurality of pre-training samples and the second plurality of pre-training samples.

17. The hardware storage device of claim 16 having stored thereon computer executable instructions that are structured to be executable by the processor of the computing device to thereby cause the computing device to perform actions that:

obtain a fine-tuning dataset including a plurality of triplets, wherein a triplet includes an original source code snippet, an associated code review, and a corresponding code change snippet, wherein the corresponding code change snippet is derived from application of the associated code review to the original source code snippet; and

fine-tune the deep learning model with the fine-tuning dataset for the deep learning model to learn to generate refined source code for a given original source code snippet and a given related code review.

18. The hardware storage device of claim 16 having stored thereon computer executable instructions that are structured to be executable by the processor of the computing device to thereby cause the computing device to perform actions that:

generate a fine-tuning dataset comprising a plurality of fine-tuning samples, wherein a fine-tuning sample comprises a code diff hunk and an associated label; and

train at least one encoder block of the deep learning model with the fine-tuning dataset to learn to determine whether or not an input code diff hunk needs a code review.

19. The hardware storage device of claim 16 , wherein the deep learning model comprises a neural transformer model with attention.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 17, 2022
From: DUAN, NAN; FU, SHENGYU; LU, SHUAI; SUNDARESAN, NEELAKANTAN; SVYATKOVSKIY, ALEXEY
To: MICROSOFT TECHNOLOGY LICENSING, LLC.
Reel/Frame 061811/0858 →
Continuity (1)
Related Publication 20240160435A1 · May 16, 2024
References Cited (109)
US 10503631B1 · Talluri · 2019 [cited by examiner]
US 11150877B2 · Ivankovic · 2021 [cited by applicant]
US 11604626B1 · Sawant · 2023 [cited by applicant]
US 20140196010A1 · Balachandran · 2014 [cited by examiner]
US 20190228319A1 · Gupta et al. · 2019 [cited by applicant]
US 20200249918A1 · Svyatkovskiy et al. · 2020 [cited by applicant]
US 20200293617A1 · Luo · 2020 [cited by applicant]
US 20200341755A1 · Woulfe · 2020 [cited by examiner]
US 20210019249A1 · Gnaneswaran · 2021 [cited by examiner]
US 20210357307A1 · Deng et al. · 2021 [cited by applicant]
US 20220164626A1 · Bird · 2022 [cited by applicant]
US 20240184570A1 · Fu · 2024 [cited by applicant]
US 20250068419A1 · Fu · 2025 [cited by applicant]
WO 2019143539A1 · 2019 [cited by applicant]
International Search Report and Written Opinion received for PCT Application No. PCT/US23/036827, mailed on Feb. 16, 2024, 17 pages. [cited by applicant]
Non-Final Office Action mailed on Apr. 11, 2024, in U.S. Appl. No. 18/074,994, 20 pages. [cited by applicant]
Ackerman, et al., “Software Inspections and the Industrial Production of Software”, In Proceedings of a Symposium on Software Validation: Inspection-Testing-Verification-Alternatives, Oct. 1, 1984. [cited by applicant]
Beller, et al., “Modern code reviews in open-source projects: which problems do they fix?”, In Proceedings of Working Conference on Mining Software Repositories, May 31, 2014, 10 Pages. [cited by applicant]
Brown, et al., “Language models are few-shot learners”, In Proceedings of the 34th International Conference on Neural Information Processing Systems, Dec. 6, 2020, 25 Pages. [cited by applicant]
Chiang, et al., “Breaking Down Multilingual Machine Translation”, In Journal of Computing Research Repository, Oct. 15, 2021, 15 Pages. [cited by applicant]
Chouchen, et al., “WhoReview: A multi-objective search-based approach for code reviewers recommendation in modern code review”, In Journal of Applied Soft Computing, vol. 100, Mar. 2021. [cited by applicant]
Fagan, Michaele. , “A History of Software Inspections”, In Journal of Software Pioneers, Oct. 21, 2011. [cited by applicant]
Fagan, Michaele. , “Design and Code Inspections to Reduce Errors in Program Development”, In Journal of IBM Systems Journal, vol. 15, Issue 3, 1976. [cited by applicant]
Guo, et al., “GraphCodeBert: Pre-Training Code Representations with Data Flow”, In Proceedings of International Conference on Learning Representations, May 3, 2021, 18 Pages. [cited by applicant]
Wang, et al., “CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation”, In Repository of arXiv: 2109.00859, Sep. 2, 2021, 13 Pages. [cited by applicant]
Shuai Lu, “CodeBERT”, Retrieved From: https://github.com/microsoft/CodeBERT/tree/master/CodeReviewer, Aug. 8, 2022, 4 Pages. [cited by applicant]
“From Research to Production”, Retrieved From: https://pytorch.org/, Retrieved On: Nov. 25, 2022, 4 Pages. [cited by applicant]
“Gerrit Code Review”, Retrieved From: https://www.gerritcodereview.com/, Retrieved On: Nov. 25, 2022, 3 Pages. [cited by applicant]
“Let's build from here”, Retrieved From: https://github.com/, Retrieved On: Nov. 25, 2022, 14 Pages. [cited by applicant]
“The AI community building the future”, Retrieved From: https://huggingface.co/, Retrieved On: Nov. 25, 2022, 8 Pages. [cited by applicant]
Li, et al., “Automating code review activities by large-scale pre-training”, In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Nov. 9… [cited by applicant]
Notice of Allowance mailed on Aug. 7, 2024, in U.S. Appl. No. 18/074,994, 12 pages. [cited by applicant]
Allamanis, et al., “Self-Supervised Bug Detection and Repair”, In Proceedings of 35th Conference on Neural Information Processing Systems, Dec. 6, 2021, 12 Pages. [cited by applicant]
Chen, et al., “Sequencer: Sequence-to-Sequence Learning for End-to-End Program Repair”, In Journal of IEEE Transactions on Software Engineering, Sep. 10, 2019, 17 Pages. [cited by applicant]
Dinella, et al., “DeepMerge: Learning to Merge Programs”, In Repository of arXiv:2105.07569v3, Sep. 6, 2021, 12 Pages. [cited by applicant]
Hoang, et al., “CC2Vec Distributed Representations of Code Changes”, In Proceedings of the 2023 CHI conference on human factors in computing systems, Jun. 27, 2020, pp. 518-529. [cited by applicant]
Hong, et al., “CommentFinder: A Simpler, Faster, More Accurate Code Review Comments Recommendation”, In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Soft… [cited by applicant]
International Search Report and Written Opinion received for PCT Application No. PCT/US2024/042595, Nov. 25, 2024, 14 pages. [cited by applicant]
Jin, et al., “InferFix: End-to-End Program Repair with LLMs”, Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Mar. 13, 2023, pp. 1646-… [cited by applicant]
Lu, et al., “LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning”, In Repository of arXiv:2308.11148v2, Sep. 5, 2023, 12 Pages. [cited by applicant]
Svyatkovskiy, et al, “MergeBERT: Program Merge Conflict Resolution via Neural Transformers,” Sep. 8, 2021, 16 pages. [cited by applicant]
Zhang, et al., “BERTScore: Evaluating text generation with BERT”, In Proceedings of Eighth International Conference on Learning Representations, Apr. 26, 2020, pp. 1-43. [cited by applicant]
“GitHub Copilot”, Retrieved from: https://web.archive.org/web/20210927023930/https://copilot.github.com/, Sep. 27, 2021, 16 Pages. [cited by applicant]
Ackerman, et al., “Software Inspections and the Industrial Production of Software”, In Proceedings of a Symposium on Software Validation: Inspection-Testing-Verification-Alternatives, Oct. 1, 1984, pp. 13-40. [cited by applicant]
Ahmad, et al., “Unified Pre-Training for Program Understanding and Generation”, In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, J… [cited by applicant]
Bacchelli, et al., “Expectations, Outcomes, and Challenges of Modern Code Review”, In Proceedings of the 35th International Conference on Software Engineering, May 18, 2013, pp. 712-721. [cited by applicant]
Beller, et al., “Modern code reviews in open-source projects: which problems do they fix?”, In Proceedings of Working Conference on Mining Software Repositories, May 21, 2014, 10 Pages. [cited by applicant]
Borgeaud, et al., “Improving language models by retrieving from trillions of tokens”, In Proceedings of the 39th International Conference on Machine Learning, vol. 162, Jun. 28, 2022, 35 Pages. [cited by applicant]
Bosu, et al., “Impact of Peer Code Review on Peer Impression Formation: A Survey”, In Proceedings ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, Oct. 10, 2013, pp. 133-142. [cited by applicant]
Brown, et al., “Language models are few-shot learners”, In Proceedings of the 34th International Conference on Neural Information Processing Systems, Dec. 6, 2020, pp. 1-75. [cited by applicant]
Chen, et al., “Evaluating Large Language Models Trained on Code”, In Repository of arXiv:2107.03374v2, Jul. 14, 2021, 35 Pages. [cited by applicant]
Chiang, et al., “Breaking Down Multilingual Machine Translation”, In Findings of the Association for Computational Linguistics: ACL 2022, May 22, 2022, pp. 2766-2780. [cited by applicant]
Chouchen, et al., “WhoReview: A multi-objective search-based approach for code reviewers recommendation in modern code review”, In Journal of Applied Soft Computing, vol. 100, Nov. 30, 2020, pp. 1-13. [cited by applicant]
Devanbu, et al., “Deep learning & software engineering: State of research and future directions”, In Repository of arXiv:2009.08525v1, Sep. 17, 2020, pp. 1-37. [cited by applicant]
Devlin, et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, In Repository of arXiv:1810.04805v1, Oct. 11, 2018, 14 Pages. [cited by applicant]
Fagan, Michaele. , “A History of Software Inspections”, In Journal of Software Pioneers, Oct. 21, 2011, pp. 562-573. [cited by applicant]
Fagan, Michaele. , “Design and Code Inspections to Reduce Errors in Program Development”, In Journal of IBM Systems Journal, vol. 15, Issue 3, 1976, pp. 575-607. [cited by applicant]
Feng, et al., “Codebert: A Pre-Trained Model for Programming and Natural Languages”, In Proceedings of the Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP, Nov. 16, 2020, pp. 1536-1547. [cited by applicant]
Guo, et al., “GraphCodeBert: Pre-Training Code Representations with Data Flow”, In Proceedings of International Conference on Learning Representations, 2021, 18 Pages. [cited by applicant]
Guo, et al., “UniXcoder: Unified Cross-Modal Pre-training for Code Representation”, In Repository of arXiv:2203.03850v1, Mar. 8, 2022, 14 Pages. [cited by applicant]
Gupta, et al., “Intelligent code reviews using deep learning”, In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Aug. 20, 2018, 9 Pages. [cited by applicant]
Hellendoorn, et al., “Towards Automating Code Review at Scale”, In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Aug. 23,… [cited by applicant]
Heumüller, et al., “Exploit Those Code Reviews! Bigger Data for Deeper Learning”, In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Eng… [cited by applicant]
Hindle, et al., “On the Naturalness of Software”, In Journal of Communications of the ACM, vol. 59, Issue 5, May 2016, pp. 122-131. [cited by applicant]
Husain, et al., “CodeSearchNet Challenge: Evaluating the State of Semantic Code Search”, In Journal of Computing Research Repository, Sep. 20, 2019, 6 Pages. [cited by applicant]
Kanade, et al., “Learning and Evaluating Contextual Embedding of Source Code”, In Proceedings of 37th International Conference on Machine Learning, Jul. 13, 2020, 12 Pages. [cited by applicant]
Lewis, et al., “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension”, In Repository of arXiv:1910.13461v1, Oct. 29, 2019, 10 Pages. [cited by applicant]
Li, et al., “CodeReviewer: Pre-Training for Automating Code Review Activities”, In Repository of arXiv:2203.09095v2, Oct. 11, 2022, 13 Pages. [cited by applicant]
Li, et al., “Deepreview: automatic code review using deep multi-instance learning”, In Proceedings of Pacific-Asia Conference on Knowledge Discovery and Data Mining, Apr. 14, 2019, pp. 318-330. [cited by applicant]
Liu, et al., “ROBERTa: A Robustly Optimized BERT Pretraining Approach”, In Repository of arXiv:1907.11692v1, Jul. 26, 2019, 13 Pages. [cited by applicant]
Liu, et al., “CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation”, In Proceedings of 35th Conference on Neural Information Processing Systems Track on Datasets and Benchmarks, Jun. 3, … [cited by applicant]
Mukadam, et al., “Gerrit software code review data from android”, In Proceedings of 10th Working Conference on Mining Software Repositories, May 18, 2013, pp. 45-48. [cited by applicant]
Paixao, et al., “CROP: Linking Code Reviews to Source Code Changes”, In Proceedings of IEEE/ACM 15th International Conference on Mining Software Repositories, May 27, 2018, pp. 46-49. [cited by applicant]
Papineni, et al., “BLEU: a method for automatic evaluation of machine translation”, In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Jul. 7, 2002, pp. 311-318. [cited by applicant]
Raffel, et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, In Journal of Machine Learning Research, vol. 21, Issue 140, Jun. 21, 2020, pp. 1-67. [cited by applicant]
Rigby, et al., “Convergent contemporary software peer review practices”, In Proceedings of the 2013 9th joint meeting on foundations of software engineering, Aug. 18, 2013, pp. 202-212. [cited by applicant]
Sadowski, et al., “Modern code review: a case study at google”, In Proceedings of IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track, May 30, 2018, pp. 181-190. [cited by applicant]
Shi, et al., “Automatic code review by learning the revision of source code”, In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, Issue 1, Jul. 17, 2019, pp. 4910-4917. [cited by applicant]
Siow, et al., “CORE: Automating Review Recommendation for Code Changes”, In Proceedings of IEEE 27th International Conference on Software Analysis, Evolution and Reengineering, Feb. 18, 2020, pp. 284-295. [cited by applicant]
Svyatkovskiy, et al., “IntelliCode Compose: Code Generation using Transformer”, In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engin… [cited by applicant]
Thongtanunam, et al., “Who should review my code? A file location-based code-reviewer recommendation approach for Modern Code Review”, In Proceedings of IEEE 22nd International Conference on Software Analysis, Evolution… [cited by applicant]
Tufano, et al., “An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation”, In Journal of ACM Transactions on Software Engineering and Methodology, vol. 24, Issue 4, Sep. 2019, 29 Pag… [cited by applicant]
Tufano, et al., “Towards Automating Code Review Activities”, In Proceedings of IEEE/ACM 43rd International Conference on Software Engineering, May 22, 2021, pp. 163-174. [cited by applicant]
Tufano, et al., “Using Pre-Trained Models to Boost Code Review Automation”, In Repository of arXiv:2201.06850v1, Jan. 18, 2022, pp. 1-12. [cited by applicant]
Tufano, et al., “Using pre-trained models to boost code review automation”, In Proceedings IEEE/ACM 44th International Conference on Software Engineering, May 25, 2022, pp. 2291-2302. [cited by applicant]
Vaswani, et al., “Attention is All You Need”, In Proceedings of Advances in Neural Information Processing Systems, vol. 30, Dec. 4, 2017, 11 Pages. [cited by applicant]
Wang, et al., “CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation”, In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Nov. … [cited by applicant]
Watson, et al., “A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research”, In Journal of ACM Transactions on Software Engineering and Methodology, vol. 31, Issue 2, Mar. 4, 2022, 58 P… [cited by applicant]
Yang, et al., “Mining the modern code review repositories: a dataset of people, process and product”, In Proceedings of IEEE/ACM 13th Working Conference on Mining Software Repositories, May 14, 2022, pp. 460-463. [cited by applicant]
Zanjani, et al., “Automatically recommending peer reviewers in modern code review”, In Journal of IEEE Transactions on Software Engineering, vol. 42, Issue 6, Nov. 12, 2015, pp. 530-543. [cited by applicant]
Zhu, et al., “Multilingual Code Snippets Training for Program Translation”, In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, Issue 10, Jun. 28, 2022, pp. 11783-11790. [cited by applicant]
“CodeBERT”, Retrieved From: https://github.com/microsoft/CodeBERT/tree/master/CodeReviewer, Retrieved On: Nov. 25, 2022, 4 Pages. [cited by applicant]
“Code Generation on Humaneval—State-of-the-art,” Retrieved from the Link: https://paperswithcode.com/sota/code-generation-on-humaneval, 2024, Accessed on Feb. 27, 2024, 25 pages. [cited by applicant]
Wang, et al., “Glue: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding”, In Proceedings of the EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Nov. 1, 2018, … [cited by applicant]
Tufano, Michele., “AutoDev: Automated AI-Driven Development”, Microsoft, Feb. 2, 2024, 3 pages. [cited by applicant]
“Open AI. GPT-4 Technical Report”, arXiv:2303.08774, Mar. 15, 2023, 99 pages. [cited by applicant]
Achiam, et al., OPENAI, Gpt-4 Technical Report, Mar. 4, 2024, 100 pages. [cited by applicant]
Agarwal, et al., “Copilot Evaluation Harness: Evaluating Ilm-Guided Software programming,” Feb. 22, 2024, 14 pages. [cited by applicant]
Smith, et al., “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model,” 2022, 44 pages. [cited by applicant]
Chen, et al., “Vscuda: Llm Based Cuda Extension for Visual Studio Code,” In Proceedings of the SC'23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, NewYork, NY, … [cited by applicant]
Ding, et al., “CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion,” Advances in Neural Information Processing Systems, 36, 2023, 23 pages. [cited by applicant]
Floridi, et al., “Gpt-3: Its Nature, Scope, Limits, and Consequences. Minds and Machines,” vol. 30, 2020, pp. 681-694. [cited by applicant]
Gravitas, et al., “Autogpt,” Retrieved from Link: https://github.com/Significant-Gravitas/AutoGPT, GitHub Repository, 2024, 4 Pages. [cited by applicant]
Srivastava, et al., “Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models,” 2023, pp. 1-95. [cited by applicant]
Nam, et al., “In-IDE Generation-based Information Support with a Large Language Model,” 2023, pp. 1-15. [cited by applicant]
Ouyang, et al., “Training Language Models to Follow Instructions with Human Feedback,” Advances in Neural Information Processing Systems, vol. 35, Mar. 4, 2022, pp. 27730-2774. [cited by applicant]
Rae, et al., “Scaling Language Models: Methods, Analysis Insights from Training Gopher,” 2022, 120 pages. [cited by applicant]
Roziere, et al., “Code Llama: Open Foundation Models for Code,” arXiv preprint arXiv:2308.12950, 2023, 48 pages. [cited by applicant]
Shinn, et al., “Reflexion: Language Agents with Verbal Reinforcement Learning,” 2023, 19 pages. [cited by applicant]
Cited By (1)
US 12,602,226