Construction method and system of descriptive model of classroom teaching behavior events

1. A construction method of a descriptive model of classroom teaching behavior events, comprising steps as the followings:

(1) acquiring classroom teaching video data to be trained;

(2) dividing the classroom teaching video data to be trained into multiple events according to utterances of a teacher by using a voice activity detection technology; and

(3) performing multi-modal recognition on all events by using multiple artificial intelligence technologies to divide the events into sub-events in multiple dimensions, establishing an event descriptive model according to the sub-events, and describing various teaching behavior events of the teacher in a classroom;

wherein step (3) further comprises:

extracting commonality between events, establishing an event descriptive model that uniformly describes all events according to the commonality and the sub-events, and uniformly describing all teaching behavior events of the teacher that occur in the classroom;

wherein in the event descriptive model, an entire classroom teaching event sequence (E) is defined, E={e 1 , e 2 , . . . , e n }, n indicates that n events occur, e i indicates an event, and e i is expressed by a 6-tuple <id, t, dt, w, a w , R>, wherein id is a unique identifier of an event;

t is a start time of the event;

dt is a duration corresponding to the event whose start time is t;

w is a dimension of the event, w∈W, W={w 0 , w 1 , w 2 , . . . , w m }, and the dimension comprises the teacher's facial expression, speech emotion, gaze, hand gesture, and body posture;

a w is an attribute of an event w, a w ∈{a 1 w , a 2 w , . . . , a l w }R indicates events correlated with a current event and correlations therebetween, and is a 2-tuple sequence defined as R={<e 1 , r 1 >, <e 2 , r 2 >, . . . , <e n , r n >}, where e in a relational 2-tuple <e, r> indicates an event associated with the current event, and r indicates a specific value of the correlation between the two events.

2. The construction method of a descriptive model of classroom teaching behavior events according to claim 1 , wherein the correlation between the two events comprises a dimensional correlation and a temporal correlation.

3. The construction method of a descriptive model of classroom teaching behavior events according to claim 2 , wherein the dimensional correlation is determined by a fuzzy matrix R=(r ij ) mxm , and the fuzzy matrix R is defined as:

R

=

[

r

11

r

12

r

1

⁢

3

⋯

⋯

r

1

⁢

m

r

2

⁢

1

r

2

⁢

2

r

2

⁢

3

⋯

⋯

r

2

⁢

m

⋯

⋯

r

m

⁢

1

r

m

⁢

2

r

m

⁢

3

⋯

⋯

r

m

⁢

m

]

where r ij ∈[0,1], i=1,2, . . . , m, j=1,2, . . . , m , r ij is a degree of correlation between an i-th dimension and a j-th dimension; if r ij =1, then i=j and it indicates a degree of correlation between a same dimension, i.e., a highest correlation; if r ij =0, it indicates that the i-th dimension and the j-th dimension are not correlated at all; and if r ij ∈(0, 1), the closer to 1, the higher the correlation.

4. The construction method of a descriptive model of classroom teaching behavior events according to claim 2 , wherein when durations corresponding to two events are the same, the temporal correlation is calculated according to the Pearson coefficient, and when the durations corresponding to the two events are different, the temporal correlation is calculated by using dynamic time warping.

5. A method for describing classroom teaching behavior events, comprising steps as the followings:

acquiring target classroom teaching video data;

dividing the target classroom teaching video data into multiple teaching events including various teaching behaviors of a teacher according to utterances of the teacher by using a voice activity detection technology; and

inputting all teaching events to an event descriptive model constructed in advance by the construction method according to claim 1 to obtain description results of various teaching behavior events of the teacher in the classroom.

6. A system for describing classroom teaching behavior events, comprising:

a target data acquisition module configured to acquire target classroom teaching video data;

a teaching event division module configured to divide the target classroom teaching video data into multiple teaching events including various teaching behaviors of a teacher according to utterances of the teacher by using a voice activity detection technology; and

a processing module configured to input all teaching events to an event descriptive model constructed in advance by the construction method according to claim 1 to obtain description results of various teaching behavior events of the teacher in the classroom.

7. A computer device comprising a memory and a processor, the memory stores a computer program, and when executing the computer program, the processor is configured to implement the construction method according to claim 1 .

8. A construction system of a descriptive model of classroom teaching behavior events, comprising:

a training data acquisition module configured to acquire classroom teaching video data to be trained;

an event division module configured to divide the classroom teaching video data to be trained into multiple events according to utterances of a teacher by using a voice activity detection technology; and

an event description module configured to perform multi-modal recognition on all events by using multiple artificial intelligence-related technologies to divide the events into sub-events in multiple dimensions, establish an event descriptive model according to the sub-events, and describe various teaching behavior events of the teacher in a classroom;

wherein the event description module is further configured to extract commonality between events, establishing an event descriptive model that uniformly describes all events according to the commonality and the sub-events, and uniformly describing all teaching behavior events of the teacher that occur in the classroom;

wherein in the event descriptive model, an entire classroom teaching event sequence (E) is defined, E={e 1 , e 2 , . . . , e n }, n indicates that n events occur, e i indicates an event, and e i is expressed by a 6-tuple <id, t, dt, w, a w , R>, wherein

id is a unique identifier of an event;

t is a start time of the event;

d t is a duration corresponding to the event whose start time is t;

w is a dimension of the event, w∈W, W={w 0 , w 1 , w 2 , . . . , w m }, and the dimension comprises the teacher's facial expression, speech emotion, gaze, hand gesture, and body posture;

a w is an attribute of an event w, a w ∈{a 1 w , a 2 w , . . . , a l w }

R indicates events correlated with a current event and correlations therebetween, and is a 2-tuple sequence defined as R={<e 1 , r 1 >, <e 2 , r 2 >, . . . , <e n , r n >}, where e in a relational 2-tuple <e, r > indicates an event associated with the current event, and r indicates a specific value of the correlation between the two events.