IP Library Granted Patent US 9,754,503
Granted Patent B2
US 9,754,503 · App. 14/666,686 · Granted Sep 5, 2017

Systems and methods for automated scoring of a user's performance

Inventor: Gary Feng (Princeton, NJ)
Assignee: Educational Testing Service
G09B7/00G10L15/04G10L17/26G10L25/00H04N7/002H04N7/147H04W4/023G10L15/26H04N7/14H04N17/00H04N2007/145H04N2017/006H04W4/02
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,754,503
App. No.
14/666,686
Granted
Sep 5, 2017
Kind
B2
Abstract

Systems and methods are provided for evaluating a user's performance. A first mobile communications device is configured to capture first audio and visual data of a user's performance. The first audio and video data are analyzed by the first device during the user's performance to determine if audio and video quality requirements are met. A second mobile communications device is configured to capture second audio and visual data of the user's performance, the second visual data capturing a different field of view than the first visual data. The second audio and visual data are transmitted by the second device to a remote computing system. The transmitted data is analyzed at the remote computing system to determine if the user is receiving utilizing unauthorized material in the performance. A computer-based scoring system is configured to determine a score for the user's performance based on the first audio and visual data.

Claims (80)

1. A system for evaluating a user's performance, the system comprising:

a first mobile communications device configured to:

capture first audio data and first visual data associated with a user's performance,

analyze the first audio data during the user's performance to determine if the first audio data meets an audio quality requirement, and generate a first signal based on a determination that the first audio data does not meet the audio quality requirement,

analyze the first visual data during the user's performance to determine if the first visual data meets a video quality requirement, and generate a second signal based on a determination that the first visual data does not meet the video quality requirement;

a second mobile communications device configured to:

capture second audio data and second visual data associated with the user's performance, the second visual data capturing a different field of view than the first visual data,

transmit the second audio data and the second visual data to a remote computing system, the remote computing system (i) analyzing the second audio data and the second visual data, (ii) determining if the user is receiving unauthorized assistance or utilizing unauthorized material in the performance, and (iii) transmitting a third signal to the second mobile communications device based on a determination that the user is receiving the unauthorized assistance or utilizing the unauthorized material, and

terminate the capturing of the audio and visual data by the first and second mobile communications devices based on a receipt of the first, second, or third signals, the second mobile communications device being configured to (i) receive the first and second signals from the first mobile communications device, and (ii) transmit a signal to the first mobile communications device to terminate the capturing of the first audio and visual data; and

a computer-based scoring system configured to determine a score for the user's performance based on a plurality of numerical measures that are determined by processing the first audio data and the first visual data with a computer processing system, the score being determined automatically and without human intervention.

2. The system of claim 1 , wherein the user's performance includes a spoken utterance, and wherein the computer-based scoring system is configured to:

process the first audio data with the computer processing system to generate a text that is representative of the spoken utterance;

process the text with the computer processing system to determine a first numerical measure indicative of a content of the spoken utterance;

process the first audio data with the computer processing system to determine a second numerical measure indicative of the user's delivery of the spoken utterance;

process the first visual data with the computer processing system to determine a third numerical measure indicative of a behavior of the user during the performance; and

apply a numerical, computer-based scoring model to the first numerical measure, the second numerical measure, and the third numerical measure to automatically determine the score for the user's performance, the numerical, computer-based scoring model including

a first variable and an associated first weighting factor, the first variable receiving a value of the first numerical measure,

a second variable and an associated second weighting factor, the second variable receiving a value of the second numerical measure, and

a third variable and an associated third weighting factor, the third variable receiving a value of the third numerical measure.

3. The system of claim 1 ,

wherein the first mobile communications device is configured to (i) analyze the first visual data during the user's performance to determine if the user's face is included in a field of view captured in the first visual data, (ii) generate a fourth signal based on a determination that the user's face is not included in the field of view captured in the first visual data, and (iii) transmit the fourth signal to the second mobile communications device; and

wherein the second mobile communications device is configured to terminate the capturing of the audio and visual data by the first and second mobile communications devices based on a receipt of the fourth signal.

4. The system of claim 3 , wherein the first mobile communications device is configured to generate an audible sound based on the determination that the user's face is not included in the field of view captured in the first visual data.

5. The system of claim 1 ,

wherein the first mobile communications device is configured to (i) receive a first GPS signal from a satellite, (ii) determine a location of the first mobile communications device based on the first GPS signal, and (iii) determine if the location of the first mobile communications device matches a designated location at which the user's performance is required to occur; and

wherein the second mobile communications device is configured to (i) receive a second GPS signal from a satellite, (ii) determine a location of the second mobile communications device based on the second GPS signal, and (iii) determine if the location of the second mobile communications device matches the designated location at which the user's performance is required to occur.

6. The system of claim 1 ,

wherein the first mobile communications device is configured to (i) analyze the first audio data and the first visual data during the user's performance to determine if the user is receiving the unauthorized assistance or utilizing the unauthorized material in the performance, (ii) generate a fourth signal based on a determination that the user is receiving the assistance or utilizing the material, and (iii) transmit the fourth signal to the second mobile communications device; and

wherein the second mobile communications device is configured to terminate the capturing of the audio and visual data by the first and second mobile communications devices based on a receipt of the fourth signal.

7. The system of claim 1 , wherein the determination of whether the first audio data meets the audio quality requirement comprises:

comparing a noise level in the first audio data to a threshold noise level; and

comparing a volume of a spoken utterance included in the first audio data to a threshold volume level.

8. The system of claim 1 , wherein the determination of whether the first visual data meets the video quality requirement comprises:

determining if a lighting condition in the first visual data meets a lighting requirement; and

determining if the user's face, as captured in the first visual data, has a contrast that meets a facial-contrast requirement.

9. The system of claim 1 , wherein the second mobile communications device is configured to transmit the second audio data and the second visual data to the remote computing system in real-time and during the user's performance, the field of view of the second visual data being larger than that of the first visual data and capturing a larger amount of an environment in which the user's performance occurs as compared to the first visual data.

10. The system of claim 1 ,

wherein the first mobile communications device is configured to (i) monitor a network connectivity of the first mobile communications device during the user's performance to determine an ability of the first mobile communications device to send or receive data over a network, (ii) generate a fourth signal based on a determination that the first mobile communications device is unable to send or receive data over the network, and (iii) transmit the fourth signal to the second mobile communications device; and

wherein the second mobile communications device is configured to terminate the capturing of the audio and visual data by the first and second mobile communications devices based on a receipt of the fourth signal.

11. A method of evaluating a user's performance, the method comprising:

capturing, using a first mobile communications device, first audio data and first visual data associated with a user's performance;

analyzing, using the first mobile communications device, the first audio data during the user's performance to determine if the first audio data meets an audio quality requirement;

generating a first signal based on a determination that the first audio data does not meet the audio quality requirement;

analyzing, using the first mobile communications device, the first visual data during the user's performance to determine if the first visual data meets a video quality requirement;

generating a second signal based on a determination that the first visual data does not meet the video quality requirement;

capturing, using a second mobile communications device, second audio data and second visual data associated with the user's performance, the second visual data capturing a different field of view than the first visual data;

transmitting, using the second mobile communications device, the second audio data and the second visual data to a remote computing system, the remote computing system (i) analyzing the second audio data and the second visual data, (ii) determining if the user is receiving unauthorized assistance or utilizing unauthorized material in the performance, and (iii) transmitting a third signal to the second mobile communications device based on a determination that the user is receiving the unauthorized assistance or utilizing the unauthorized material;

terminating, using the second mobile communications device, the capturing of the audio and visual data by the first and second mobile communications devices based on a receipt of the first, second, or third signals, the second mobile communications device being configured to (i) receive the first and second signals from the first mobile communications device, and (ii) transmit a signal to the first mobile communications device to terminate the capturing of the first audio and visual data; and

determining, using a computer-based scoring system, a score for the user's performance based on a plurality of numerical measures that are determined by processing the first audio data and the first visual data with a computer processing system, the score being determined automatically and without human intervention.

12. The method of claim 11 , wherein the user's performance includes a spoken utterance, and wherein the determining of the score using the computer-based scoring system comprises:

processing the first audio data with the computer processing system to generate a text that is representative of the spoken utterance;

processing the text with the computer processing system to determine a first numerical measure indicative of a content of the spoken utterance;

processing the first audio data with the computer processing system to determine a second numerical measure indicative of the user's delivery of the spoken utterance;

processing the first visual data with the computer processing system to determine a third numerical measure indicative of a behavior of the user during the performance; and

applying a numerical, computer-based scoring model to the first numerical measure, the second numerical measure, and the third numerical measure to automatically determine the score for the user's performance, the numerical, computer-based scoring model including

a first variable and an associated first weighting factor, the first variable receiving a value of the first numerical measure,

a second variable and an associated second weighting factor, the second variable receiving a value of the second numerical measure, and

a third variable and an associated third weighting factor, the third variable receiving a value of the third numerical measure.

13. The method of claim 11 comprising:

analyzing, using the first mobile communications device, the first visual data during the user's performance to determine if the user's face is included in a field of view captured in the first visual data.

14. The method of claim 13 comprising:

generating, using the first mobile communications device, an audible sound based on a determination that the user's face is not included in the field of view captured in the first visual data.

15. The method of claim 11 comprising:

receiving, using the first mobile communications device, a first GPS signal from a satellite;

determining, using the first mobile communications device, a location of the first mobile communications device based on the first GPS signal;

determining, using the first mobile communications device, if the location of the first mobile communications device matches a designated location at which the user's performance is required to occur;

receiving, using the second mobile communications device, a second GPS signal from a satellite;

determining, using the second mobile communications device, a location of the second mobile communications device based on the second GPS signal; and

determining, using the second mobile communications device, if the location of the second mobile communications device matches the designated location at which the user's performance is required to occur.

16. The method of claim 11 comprising:

analyzing, using the first mobile communications device, the first audio data and the first visual data during the user's performance to determine if the user is receiving the unauthorized assistance or utilizing the unauthorized material in the performance.

17. The method of claim 11 , wherein the determination of whether the first audio data meets the audio quality requirement comprises:

comparing a noise level in the first audio data to a threshold noise level; and

comparing a volume of a spoken utterance included in the first audio data to a threshold volume level.

18. The method of claim 11 , wherein the determination of whether the first visual data meets the video quality requirement comprises:

determining if a lighting condition in the first visual data meets a lighting requirement; and

determining if the user's face, as captured in the first visual data, has a contrast that meets a facial-contrast requirement.

19. The method of claim 11 , wherein the transmitting of the second audio data and the second visual data to the remote computing system is in real-time and during the user's performance, the field of view of the second visual data being larger than that of the first visual data and capturing a larger amount of an environment in which the user's performance occurs as compared to the first visual data.

20. The method of claim 11 comprising:

analyzing, using the first mobile communications device, a network connectivity of the first mobile communications device during the user's performance to determine an ability of the first mobile communications device to send or receive data over a network.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 9, 2015
From: FENG, GARY
To: EDUCATIONAL TESTING SERVICE
Reel/Frame 035365/0058 →
Continuity (2)
Provisional Application 61969475 · Mar 24, 2014
Related Publication 20150269857A1 · Sep 24, 2015