Visual liveness detection

1. A method for visual liveness detection, the method comprising:

obtaining, by one or more computer processors, a video-audio signal related to a speaker speaking a text;

splitting, by one or more computer processors, the video-audio signal into a video signal which records images of the speaker and an audio signal which records a speech spoken by the speaker;

obtaining, by one or more computer processors, a first sequence indicating visual mouth openness from the video signal;

obtaining, by one or more computer processors, a second sequence indicating acoustic mouth openness based on the text and the audio signal;

measuring, by one or more computer processors, synchrony between the first sequence and the second sequence; and

determining, by one or more computer processors, liveness of the speaker based on the synchrony.

2. The method of claim 1 , wherein obtaining a first sequence comprises:

sampling, by one or more computer processors, the video signal to obtain video frames;

determining, by one or more computer processors, a mouth-open-status of the speaker for a video frame of the video signal; and

sequencing, by one or more computer processors, the mouth-open-statuses of the video frames of the video signal in time order to generate the first sequence.

3. The method of claim 1 , wherein obtaining a second sequence comprises:

segmenting, by one or more computer processors, the text into speech units;

determining, by one or more computer processors, a mouth-open-status of the respective speech units;

detecting, by one or more computer processors, a time point at which the speech starts in the audio signal;

sampling, by one or more computer processors, the audio signal starting from the time point;

determining, by one or more computer processors, audio samples of the audio signal corresponding to the respective speech units;

providing, by one or more computer processors, the audio samples with the respective mouth-open-statuses of the corresponding speech units; and

sequencing, by one or more computer processors, the mouth-open-statuses of the samples of the audio signal in time order to generate the second sequence.

4. The method of claim 1 , wherein measuring synchrony between the first sequence and the second sequence comprises:

aligning, by one or more computer processors, the first sequence with the second sequence; and

measuring, by one or more computer processors, a similarity between the aligned first and second sequences, the similarity representing the synchrony.

5. The method of claim 4 , wherein aligning the first sequence with the second sequence comprises:

removing, by one or more computer processors, a part of the first sequence from the beginning of the first sequence such that the remaining of the first sequence has the same time length as the second sequence; and

down-sampling, by one or more computer processors, one of the first and second sequences which has a larger amount of the mouth-open-statuses, such that the amount of the mouth-open-statuses in the down-sampled sequence equals to the amount of the mouth-open-statuses in the other of the first and second sequences.

6. The method of claim 4 , wherein measuring a similarity between the aligned first and second sequences comprises:

determining, by one or more computer processors, the number of the same mouth-open-statuses in the aligned first and second sequences; and

calculating, by one or more computer processors, a value of the similarity as a ratio of the number to the amount of the mouth-open-statuses of one of the aligned first and second sequences.

7. A computer system for visual liveness detection, the computer system comprising:

one or more computer readable storage media;

program instructions stored on at least one of the one or more computer readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising:

program instructions to obtain a video-audio signal related to a speaker speaking a text;

program instructions split the video-audio signal into a video signal which records images of the speaker and an audio signal which records a speech spoken by the speaker;

program instructions obtain a first sequence indicating visual mouth openness from the video signal;

program instructions obtain a second sequence indicating acoustic mouth openness based on the text and the audio signal;

program instructions measure synchrony between the first sequence and the second sequence; and

program instructions determine liveness of the speaker based on the synchrony.

8. The computer system of claim 7 , wherein the program instructions to obtain a first sequence, further comprises:

program instructions to sample the video signal to obtain video frames;

program instructions to determine a mouth-open-status of the speaker for a video frame of the video signal; and

program instructions to sequence the mouth-open-statuses for the video frames of the video signal in time order to generate the first sequence.

9. The computer system of claim 7 , wherein the program instructions to obtain a second sequence, further comprises:

program instructions to segment the text into speech units;

program instructions to determine a mouth-open-status of the respective speech units;

program instructions to detect a time point at which the speech starts in the audio signal;

program instructions to sample the audio signal starting from the time point;

program instructions to determine audio samples of the audio signal corresponding to the respective speech units;

program instructions to provide the audio samples with the respective mouth-open-statuses of the corresponding speech units; and

program instructions to sequence the mouth-open-statuses of the samples of the audio signal in time order to generate the second sequence.

10. The computer system of claim 7 , wherein the program instructions to measure synchrony between the first sequence and the second sequence, further comprises:

program instructions to align the first sequence with the second sequence; and

program instructions to measure a similarity between the aligned first and second sequences, the similarity representing the synchrony.

11. The computer system of claim 10 , wherein the program instructions to align the first sequence with the second sequence, further comprises:

program instructions to remove a part of the first sequence from the beginning of the first sequence such that the remaining of the first sequence has the same time length as the second sequence; and

program instructions to down-sample one of the first and second sequences which has a larger amount of the mouth-open-statuses, such that the amount of the mouth-open-statuses in the down-sampled sequence equals to the amount of the mouth-open-statuses in the other of the first and second sequences.

12. The computer system of claim 10 , wherein program instructions to measure a similarity between the aligned first and second sequences, the similarity representing the synchrony, further comprises:

program instructions to determine the number of the same mouth-open-statuses in the aligned first and second sequences; and

program instructions to calculate a value of the similarity as a ratio of the number to the amount of the mouth-open-statuses of one of the aligned first and second sequences.

13. A computer program product for visual liveness detection, the computer program product comprising:

one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions comprising:

program instructions to obtain a video-audio signal related to a speaker speaking a text;

program instructions to split the video-audio signal into a video signal which records images of the speaker and an audio signal which records a speech spoken by the speaker;

program instructions to obtain a first sequence indicating visual mouth openness from the video signal;

program instructions to obtain a second sequence indicating acoustic mouth openness based on the text and the audio signal;

program instructions to measure synchrony between the first sequence and the second sequence; and

program instructions to determine liveness of the speaker based on the synchrony.

14. The computer program product of claim 13 , wherein the program instructions to obtain a first sequence, further comprises:

program instructions to sample the video signal to obtain video frames;

program instructions to determine a mouth-open-status of the speaker for a video frame of the video signal; and

program instructions to sequence the mouth-open-statuses for the video frames of the video signal in time order to generate the first sequence.

15. The computer program product of claim 13 , wherein the program instructions to obtain a second sequence, further comprises:

program instructions to segment the text into speech units;

program instructions to determine a mouth-open-status of the respective speech units;

program instructions to detect a time point at which the speech starts in the audio signal;

program instructions to sample the audio signal starting from the time point;

program instructions to determine audio samples of the audio signal corresponding to the respective speech units;

program instructions to provide the audio samples with the respective mouth-open-statuses of the corresponding speech units; and

program instructions to sequence the mouth-open-statuses of the samples of the audio signal in time order to generate the second sequence.

16. The computer program product of claim 13 , wherein the program instructions to measure synchrony between the first sequence and the second sequence, further comprises:

program instructions to align the first sequence with the second sequence;

program instructions to measure a similarity between the aligned first and second sequences, the similarity representing the synchrony.

17. The computer program product of claim 16 , wherein the program instructions to align the first sequence with the second sequence, further comprises:

program instructions to remove a part of the first sequence from the beginning of the first sequence such that the remaining of the first sequence has the same time length as the second sequence; and

program instructions to down-sample one of the first and second sequences which has a larger amount of the mouth-open-statuses, such that the amount of the mouth-open-statuses in the down-sampled sequence equals to the amount of the mouth-open-statuses in the other of the first and second sequences.

18. The computer program product of claim 16 , wherein program instructions to measure a similarity between the aligned first and second sequences, the similarity representing the synchrony, further comprises:

program instructions to determine the number of the same mouth-open-statuses in the aligned first and second sequences; and

program instructions to calculate a value of the similarity as a ratio of the number to the amount of the mouth-open-statuses of one of the aligned first and second sequences.