IP Library › Granted Patent US 12682247
Granted Patent B2
US 12682247 · App. 18/181,044 · Granted Jul 14, 2026

Exploiting domain-specific language characteristics for language model pretraining

Inventors: Inderjeet Jayakumar Nair (Bengaluru, IN); Natwar Modani (Bangalore, IN)
Assignee: ADOBE INC.
G06N3/0895G06F40/279
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12682247
App. No.
18/181,044
Granted
Jul 14, 2026
Kind
B2
Abstract

A method, apparatus, non-transitory computer readable medium, and system of training a domain-specific language model are described. One or more aspects of the method, apparatus, non-transitory computer readable medium, and system include obtaining domain-specific training data including a plurality of domain-specific documents having a document structure corresponding to a domain, and obtaining domain-agnostic training data including a plurality of documents outside of the domain. The domain-specific training data and the domain-agnostic training data are used to train a language model to perform a domain-specific task based on the domain-specific training data and to perform a domain agnostic task based on the domain-agnostic training data.

Claims (46)

1 . A method comprising:

obtaining domain-specific training data including a plurality of domain-specific documents having a document structure corresponding to a domain;

obtaining domain-agnostic training data including a plurality of documents outside of the domain; and

training a language model to perform a domain-specific task and a domain-agnostic task by randomly selecting a training objective from a set of candidate training objectives, wherein the set of candidate training objectives comprises at least one of a Positive-Unlabeled (PU) Learning objective, a soft label classification objective, a masked language modeling (MLM), and an auto-regression objective, wherein the language model is trained to perform the domain-specific task using the Positive-Unlabeled (PU) Learning or the soft label classification objective based on the domain-specific training data, and wherein the language model is trained to perform the domain-agnostic task using the masked language modeling (MLM) or the auto-regression objective based on the domain-agnostic training data.

2 . The method of claim 1 , further comprising:

identifying similar text in the plurality of domain-specific documents; and

labeling dynamic text within the similar text to obtain the domain-specific training data.

3 . The method of claim 2 , further comprising:

identifying a header of the similar text; and

performing soft label classification based on the header, wherein the domain-specific task is based on the soft label classification.

4 . The method of claim 1 , wherein:

the domain-agnostic task comprises an MLM task.

5 . The method of claim 1 , further comprising:

computing a contrastive learning loss, wherein the language model is trained to perform the domain-specific task based on the contrastive learning loss.

6 . The method of claim 1 , wherein:

the domain-specific training data and the domain-specific task relate to the domain in which different documents include similar text based on a same template.

7 . A method comprising:

receiving a query relating to a document in a domain;

generating a domain-specific output based on the query using a language model, wherein the language model is trained to perform a domain-specific task and a domain-agnostic task by randomly selecting a training objective from a set of candidate training objectives, wherein the set of candidate training objectives comprises at least one of a Positive-Unlabeled (PU) Learning objective, a soft label classification objective, a masked language modeling (MLM), and an auto-regression objective, wherein the language model is trained to perform the domain-specific task using the Positive-Unlabeled (PU) Learning or the soft label classification objective based on domain-specific training data, wherein the language model is trained to perform the domain-agnostic task using the masked language modeling (MLM) or the auto-regression objective based on domain-agnostic training data, and wherein the domain-specific training data includes a plurality of domain-specific documents having a document structure corresponding to the domain; and

transmitting a response to the query based on the domain-specific output.

8 . The method of claim 7 , wherein:

the domain-agnostic training data includes domain-agnostic documents.

9 . The method of claim 7 , wherein:

the domain comprises legal documents.

10 . The method of claim 9 , wherein:

the domain-specific output comprises a legal judgement prediction, a legal entity recognition, a legal question answer, an automated legal review, a legal text classification, an unusual clause identification, or a contract type classification.

11 . The method of claim 7 , wherein:

the domain comprises scientific or medical documents and the domain-specific output comprises a medical diagnosis prediction, a scientific or medical entity recognition, a scientific or medical question answer, a scientific or medical text classification, an unusual clause identification, or a document type classification.

12 . A computer system comprising:

one or more processors;

a memory coupled to and in communication with the one or more processors wherein the one or more processors in communication with the memory are configured to perform operations comprising:

obtaining domain-specific training data including a plurality of domain-specific documents having a document structure corresponding to a domain;

obtaining domain-agnostic training data including a plurality of documents outside of the domain; and

training a language model to perform a domain-specific task and a domain-agnostic task by randomly selecting a training objective from a set of candidate training objectives, wherein the set of candidate training objectives comprises at least one of a Positive-Unlabeled (PU) Learning objective, a soft label classification objective, a masked language modeling (MLM), and an auto-regression objective, wherein the language model is trained to perform the domain-specific task using the Positive-Unlabeled (PU) Learning or the soft label classification objective based on the domain-specific training data, and wherein the language model is trained to perform the domain-agnostic task using the masked language modeling (MLM) or the auto-regression objective based on the domain-agnostic training data.

13 . The computer system of claim 12 , wherein the one or more processors in communication with the memory are configured to perform operations further comprising:

identifying similar text in the plurality of domain-specific documents and label dynamic text within the similar text to obtain the domain-specific training data.

14 . The computer system of claim 12 , wherein:

the domain-agnostic training data includes domain agnostic documents.

15 . The computer system of claim 14 , wherein:

the domain-agnostic task comprises an MLM task.

16 . The computer system of claim 12 , wherein:

the language model comprises a transformer model.

17 . The computer system of claim 12 , wherein the one or more processors in communication with the memory are configured to perform operations further comprising:

generating a response to a received query based on an output of the language model, and transmit the response to the received query based on a domain-specific output.

18 . The computer system of claim 12 , wherein:

the domain-specific task relates to a legal document domain.