IP Library › Granted Patent US 12,039,622
Granted Patent B2
US 12,039,622 · App. 17/314,600 · Granted Jul 16, 2024

Course assignment by a multi-learning management system

Inventors: Erhan Onal (Sunnyvale, CA); Bryan Lee Baker (Palo Alto, CA); Justin Michael Emge (Santa Clara, CA)
Assignee: Google LLC
G06Q50/2057G06F16/285
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,039,622
App. No.
17/314,600
Granted
Jul 16, 2024
Kind
B2
Abstract

Methods, systems, and apparatus, including computer-readable storage media, for course assignment by a multi-learning management system. The system can receive data from a variety of individual learning management systems offering different courses. The system can use feedback data of a user base for the system to cluster courses by predicted difficulty, and generate, from the clusters, a sequence of courses for a target user. The sequence of courses can include at least one course from each cluster, with courses from a first cluster with a lower overall difficulty measure preceding courses in a second cluster with a higher overall difficulty measure in the sequence, wherein the starting cluster can be calculated according to the estimated level of the target user.

Claims (105)

1. A method, comprising:

receiving, by one or more processors, first data comprising one or more respective difficulty scores for each of a plurality of courses and feedback for one or more courses identified as having been completed by a first user of a plurality of users,

wherein each course corresponds to a topic and is from one or more multiple learning management systems, and

wherein receiving the feedback for the one or more courses comprises:

identifying one or more topics categorizing source code authored by the first user;

prompting one or more user computing devices for expertise scores of the first user with regard to the one or more topics; and

receiving, as part of the feedback for the one or more courses, expertise scores from the one or more user computing devices, including one or more expertise scores of the first user;

standardizing, by the one or more processors, the one or more respective difficulty scores into a common format such that the one or more respective difficulty scores can be accurately compared;

receiving, by the one or more processors, second data comprising respective expertise scores measuring a level of expertise in the topic for each of the plurality of users;

standardizing, by the one or more processors, the respective expertise scores into the common format such that the respective expertise scores can be accurately compared;

generating for each course, by the one or more processors based at least on the common format of the respective difficulty scores for the course in the first data, and common format of the respective expertise scores for at least the first user of the plurality of users, a respective overall difficulty measure for each course;

clustering the plurality of courses into a plurality of clusters,

wherein each cluster represents one or more courses,

wherein each course represented in a cluster corresponds to a respective overall difficulty measure within a respective range associated with the cluster, and

wherein courses in a first cluster with a lower overall difficulty measure precede courses in a second cluster with a higher overall difficulty measure in a sequence of courses;

identifying a starting difficulty threshold based on the respective expertise scores for the first user and the feedback for the one or more courses;

generating, by the one or more processors based on the starting difficulty threshold, the sequence of courses for the first user, comprising at least one course from each of the plurality of clusters; and

providing for output, by the one or more processors, the sequence of courses.

2. The method of claim 1 , wherein each cluster corresponds to a respective difficulty label, and wherein each course in each cluster is labeled with the respective same difficulty label.

3. The method of claim 1 ,

wherein generating the sequence of courses comprises generating the sequence to include only courses with a respective overall difficulty threshold meeting or exceeding the starting difficulty threshold for the first user.

4. The method of claim 3 , wherein generating the sequence of courses further comprises:

identifying courses that have been marked as completed by the first user; and

excluding the identified courses from the sequence of courses.

5. The method of claim 1 , wherein the method further comprises:

generating, for presentation on a user interface of a user computing device, data representing:

a visual representation of the sequence of courses; and

receiving, by the one or more processors, input to modify one or more of the courses in the sequence of courses, and in response to the input:

updating the sequence of courses in accordance with the modification, and

updating the data representing the visual representation of the modified sequence of courses.

6. The method of claim 1 , wherein the method further comprises:

receiving state data for each course in the sequence of courses, wherein the state data characterizes a respective completion state of each course by the first user;

determining, from the state data, that a course in the sequence of courses has a completion state indicating course failure by the first user; and

in response to the determining, generating an updated sequence of courses.

7. The method of claim 6 , wherein generating the updated sequence of courses comprises updating the sequence of courses to include only courses with corresponding overall difficulty measures at or below an overall difficulty measure for the course in the sequence of courses with the course failure completion state.

8. The method of claim 1 , wherein generating the sequence of courses further comprises:

receiving a total sequence length value specifying a total number of courses to include in the sequence and a minimum per-cluster course value specifying the minimum number of courses to include in the sequence of courses for each cluster; and

generating the sequence of courses in accordance with the total sequence length value and the minimum course value.

9. The method of claim 1 , wherein generating the sequence of courses further comprises:

receiving one or more intra-cluster criteria for arranging courses in the sequence from the same cluster; and

generating the sequence of courses in accordance with the one or more intra-cluster criteria.

10. The method of claim 9 ,

wherein the one or more intra-cluster criteria comprise an intra-cluster difficulty criterion, and

wherein generating the sequence of courses in accordance with the one or more intra-cluster criteria comprises generating the sequence such that courses with lower overall difficulty measures precede courses from the same cluster with higher overall difficulty measures.

11. The method of claim 9 , wherein:

the intra-cluster criteria comprises one or more conditions for sequencing courses selected from a same cluster, and

the one or more conditions includes at least one of increasing overall difficult, a popularity of a course in the same cluster, newness of the course in the same cluster, or age of the course in the same cluster.

12. The method of claim 1 , further comprising:

receiving, by the one or more processors, an input to modify the sequence of courses; and

providing for output, by the one or more processors, the modified sequence of courses.

13. The method of claim 1 , further comprising:

comparing, by the one or more processors, the respective overall difficulty measures to add or augment at least one of a label or characteristic of at least one of the plurality of courses to provide a contextual relationship of the at least one of the plurality of courses to a target user base; and

updating, by the one or more processors, labels or characteristics associated with the respective difficulty scores for each of the plurality of courses, maintained in a catalog engine, with the at least one of the added or augmented label or characteristic.

14. A system comprising:

one or more processors configured to:

receive first data comprising one or more respective difficulty scores for each of a plurality of courses and feedback for one or more courses identified as having been completed by a first user of a plurality of users,

wherein each course corresponds to a topic and is from one or more of multiple learning management systems, and

wherein receiving the feedback for the one or more courses comprises:

identifying one or more topics categorizing source code authored by the first user;

prompting one or more user computing devices for expertise scores of the first user with regard to the one or more topics; and

receiving, as part of the feedback for the one or more courses, expertise scores from the one or more user computing devices, including one or more expertise scores of the first user;

standardize the one or more respective difficulty scores into a common format such that the one or more respective difficulty scores can be accurately compared;

receive second data comprising respective expertise scores measuring a level of expertise in the topic for each of the plurality of users;

standardize the respective expertise scores into the common format such that the respective expertise scores can be accurately compared;

generate, for each course, a respective overall difficulty measure based at least on the common format of the respective difficulty scores for the course in the first data, and the common format of the respective expertise scores for at least the first user of the plurality of users;

cluster the plurality of courses into a plurality of clusters,

wherein each cluster represents one or more courses,

wherein each course represented in a cluster corresponds to a respective overall difficulty measure within a respective range associated with the cluster, and

wherein courses in a first cluster with a lower overall difficulty measure precede courses in a second cluster with a higher overall difficulty measure in a sequence of courses;

identify a starting difficulty threshold based on the respective expertise scores for the first user and the feedback for the one or more courses;

generate, based on the starting difficulty threshold, the sequence of courses for the first user, comprising at least one course from each of the plurality of clusters; and

provide for output the generated sequence of courses.

15. The system of claim 14 , wherein each cluster corresponds to a respective difficulty label, and wherein each course in each cluster is labeled with the respective same difficulty label.

16. The system of claim 14 ,

wherein generating the sequence of courses comprises generating the sequence to include only courses with a respective overall difficulty threshold meeting or exceeding the starting difficulty threshold for the first user.

17. The system of claim 16 , wherein in generating the sequence of courses, the one or more processors are further configured to:

identify courses that have been marked as completed by the first user; and

exclude the identified courses from the sequence of courses.

18. The system of claim 14 , wherein the one or more processors are further configured to:

generate, for presentation on a user interface of a user computing device, data representing:

a visual representation of the sequence of courses; and

receive, by the one or more processors, input to modify one or more of the courses in the sequence of courses, and in response to the input:

update the sequence of courses in accordance with the modification, and

update the data representing the visual representation of the modified sequence of courses.

19. The system of claim 14 , wherein the one or more processors are further configured to:

compare the respective overall difficulty measures to add or augment at least one of a label or characteristic of at least one of the plurality of courses to provide a contextual relationship of the at least one of the plurality of courses to a target user base; and

update labels or characteristics associated with the respective difficulty scores for each of the plurality of courses, maintained in a catalog engine, with the at least one of the added or augmented label or characteristic.

20. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving first data comprising one or more respective difficulty scores for each of a plurality of courses and feedback for one or more courses identified as having been completed by a first user of a plurality of users,

wherein each course corresponds to a topic and is from one or more of multiple learning management systems, and

wherein receiving the feedback for the one or more courses comprises:

identifying one or more topics categorizing source code authored by the first user;

prompting one or more user computing devices for expertise scores of the first user with regard to the one or more topics; and

receiving, as part of the feedback for the one or more courses, expertise scores from the one or more user computing devices, including one or more expertise scores of the first user;

standardizing, by the one or more processors, the one or more respective difficulty scores into a common format such that the one or more respective difficulty scores can be accurately compared;

receiving second data comprising respective expertise scores measuring a level of expertise in the topic for each of the plurality of users;

standardizing, by the one or more processors, the respective expertise scores into the common format such that the respective expertise scores can be accurately compared;

generating, for each course, a respective overall difficulty measure based at least on the common format of the respective difficulty scores for the course in the first data, and the common format of the respective expertise scores for at least the first user of the plurality of users;

clustering the plurality of courses into a plurality of clusters,

wherein each cluster represents one or more courses,

wherein each course represented in a cluster corresponds to a respective overall difficulty measure within a respective range associated with the cluster, and

wherein courses in a first cluster with a lower overall difficulty measure precede courses in a second cluster with a higher overall difficulty measure in a sequence of courses;

identifying a starting difficulty threshold based on the respective expertise scores for the first user and the feedback for the one or more courses;

generating, based on the starting difficulty threshold, the sequence of courses for the first user, comprising at least one course from each of the plurality of clusters; and

providing for output the generated sequence of courses.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 11, 2021
From: ONAL, ERHAN; BAKER, BRYAN LEE; EMGE, JUSTIN MICHAEL
To: GOOGLE LLC
Reel/Frame 056200/0576 →
Continuity (1)
Related Publication 20220358611A1 · Nov 10, 2022
Cited By (1)
US 12,488,701