Quality and efficacy analysis of statement of purpose in higher education admissions using machine learning
Systems and methods for the multifaceted analysis of a written work, such as an educational program admission essay, using machine learning, deep learning and natural language processing. Language, relevance, structure, and flows are evaluated for an overall impactful essay. Essay content is checked to evaluate whether the author has covered an essay's essential aspects. The essay is also analyzed for an effective structure for presenting details as per the essay type. The disclosure includes data preparation for the task, process of data tagging, feature engineering from the essay text, method for transfer learning and fine-tuning language model to adapt to the context of an essay. Finally, a process for building machine learning and deep learning models and technique for ensembling to use both models in combination is disclosed. The system may provide user-adapted feedback based on a persona created from the user profile.
1. A system, comprising:
one or more computing devices configured to:
receive a block of text comprising an essay organized as a series of sentences within a series of one or more paragraphs;
inventory content within the essay by:
dividing the essay into a plurality of text portions;
classifying each text portion of the plurality of text portions as one of a plurality of content classes by applying the essay and plurality of text portions to a language model and applying the output of the language model to a multi-label classifier machine learning model; and
determining, for each content class in the plurality of content classes, a value associated with that content class;
assess a quality of discourse structure within the essay by:
classifying each sentence within the series of sentences as one of a plurality of discourse elements by applying the essay to one or more sequence labeling machine learning models; and
identifying a type of transition associated with a transition from one discourse element to a different discourse element within the series of sentences;
assess a quality of language usage within the essay; and
provide normative feedback relating to the plurality of values associated with the plurality of content classes, the quality of discourse structure, and the quality of language usage.
2. The system of claim 1 wherein the one or more computing devices is or are further configured to:
receive a profile containing information describing one or more characteristics of a subject of the essay;
determine a plurality of topics relating to the one or more characteristics of the subject of the essay;
detect the presence or absence of each one of the plurality of topics within the essay; and
provide feedback regarding one or more of the plurality of topics that are absent from essay.
3. The system of claim 1 wherein the multi-label classifier machine learning model comprises a deep learning multi-class, multi-label classifier model to the essay.
4. The system of claim 3 wherein the deep learning multi-class, multi-label classifier model has been constructed from a one-versus-rest classifier and multi-label classifier.
5. The system of claim 4 wherein each of the one-versus-rest classifier and multi-label classifier have been trained using a fine-tuned language model.
6. The system of claim 5 wherein the fine-tuned language model has been trained using a transfer learning approach, wherein a language model is pre-trained on a general purpose training and said language model is then fine-tuned on a special purpose training set.
7. The system of claim 1 wherein normative feedback relating to the plurality of values associated with the plurality of content classes comprises information regarding a content class that is either over-represented or under-represented within the essay.
8. The system of claim 1 wherein the normative feedback regarding a quality of discourse structure within the essay comprises information relating to the type of transition.
9. The system of claim 1 wherein the one or more computing devices is further configured to assess a quality of language usage within the essay by employing a language model and logic algorithms to detect grammatical and syntactic errors in the output of said language model.
10. A computer-implemented method comprising the steps of:
receiving a block of text comprising an essay organized as a series of sentences within a series of one or more paragraphs;
inventorying content within the essay by:
dividing the essay into a plurality of text portions;
classifying each text portion of the plurality of text portions as one of a plurality of content classes by applying the essay and plurality of text portions to a language model and applying the output of the language model to a multi-label classifier machine learning model; and
determining, for each content class in the plurality of content classes, a value associated with that content class;
assessing a quality of discourse structure within the essay by:
classifying each sentence within the series of sentences as one of a plurality of discourse elements by applying the essay to one or more sequence labeling machine learning models; and
identifying a type of transition associated with a transition from one discourse element to a different discourse element within the series of sentences;
assessing a quality of language usage within the essay; and
providing normative feedback relating to the plurality of values associated with the plurality of content classes, the quality of discourse structure, and the quality of language usage.
11. The method of claim 10 further comprising the steps of:
receiving a profile containing information describing one or more characteristics of a subject of the essay;
determining a plurality of topics relating to the one or more characteristics of the subject of the essay;
detecting the presence or absence of each one of the plurality of topics within the essay; and
providing feedback regarding one or more of the plurality of topics that are absent from the essay.
12. The method of claim 10 wherein the step of providing normative feedback relating to the plurality of values associated with the plurality of content classes further comprises providing information regarding a content class that is either over-represented or under-represented within the essay.
13. The method of claim 10 wherein the step of providing normative feedback regarding a quality of discourse structure within the essay further comprises providing information relating to the type of transition.
14. The method of claim 10 wherein step of assessing a quality of language usage within the essay further comprises employing a language model and logic algorithms to detect grammatical and syntactic errors in the output of said language model.
15. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors of a machine learning system, cause the machine learning system to:
receive a block of text comprising an essay organized as a series of sentences within a series of one or more paragraphs;
inventory content within the essay by:
dividing the essay into a plurality of text portions;
classifying each text portion of the plurality of text portions as one of a plurality of content classes by applying the essay and plurality of text portions to a language model and applying the output of the language model to a multi-label classifier machine learning model; and
determining, for each content class in the plurality of content classes, a value associated with that content class;
assess a quality of discourse structure within the essay by:
classifying each sentence within the series of sentences as one of a plurality of discourse elements by applying the essay to one or more sequence labeling machine learning models; and
identifying a type of transition associated with a transition from one discourse element to a different discourse element within the series of sentences;
assess a quality of language usage within the essay; and
provide normative feedback relating to the plurality of values associated with the plurality of content classes, the quality of discourse structure, and the quality of language usage.
16. The non-transitory computer-accessible storage medium of claim 15 , wherein program instructions that when executed on one or more processors of a machine learning system further cause the machine learning system to:
receive a profile containing information describing one or more characteristics of a subject of the essay;
determine a plurality of topics relating to the one or more characteristics of the subject of the essay;
detect the presence or absence of each one of the plurality of topics within the essay; and
provide feedback regarding one or more of the plurality of topics that are absent from the essay.
17. The system of claim 15 wherein the multi-label classifier machine learning model comprises a deep learning multi-class, multi-label classifier model to the essay.
18. The system of claim 17 wherein the deep learning multi-class, multi-label classifier model has been constructed from a one-versus-rest classifier and multi-label classifier.
19. The system of claim 18 wherein each of the one-versus-rest classifier and multi-label classifier have been trained using a fine-tuned language model.
20. The system of claim 19 wherein the fine-tuned language model has been trained using a transfer learning approach, wherein a language model is pre-trained on a general purpose training and said language model is then fine-tuned on a special purpose training set.