IP Library Granted Patent US 10,341,694
Granted Patent B2
US 10,341,694 · App. 15/669,127 · Granted Jul 2, 2019

Data processing method and live broadcasting method and device

Inventor: Gang Xu (Beijing, CN)
Assignee: ALIBABA GROUP HOLDING LIMITED
H04N21/2335G10L15/26G10L15/265H04N21/2187H04N21/233H04N21/23406H04N21/23614H04N21/234309H04N21/4884H04N21/8133
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,341,694
App. No.
15/669,127
Granted
Jul 2, 2019
Kind
B2
Abstract

Data processing methods, live broadcasting methods and devices are disclosed. An example data processing method may comprise converting audio and video data into broadcast data in a predetermined format, and performing speech recognition on audio data in the audio and video data, and adding the text information obtained from speech recognition into the broadcast data. In real time, text information obtained from speech recognition according to the audio data can be inserted.

Claims (86)

1. A data processing method, comprising:

converting audio and video data into broadcast data in a predetermined format, and performing speech recognition on audio data in the audio and video data;

in response to text information obtained from the speech recognition comprising a preset keyword, converting the text information to a corresponding operation instruction according to a preset corresponding relationship between the keyword and the operation instruction, and sending the operation instruction to a network device; and

in response to the text information obtained from the speech recognition not comprising the preset keyword, adding text information obtained from speech recognition into the broadcast data.

2. The method according to claim 1 , wherein the performing speech recognition on the audio data in the audio and video data comprises:

segmenting the audio data in the audio and video data; and

performing speech recognition on the segmented audio data segment by segment.

3. The method according to claim 2 , wherein the segmenting the audio data in the audio and video data comprises:

segmenting the audio data in the audio and video data at one or more intervals with energy continuously lower than a predetermined threshold for over a predetermined time length.

4. The method according to claim 1 , wherein the adding the text information obtained from speech recognition into the broadcast data comprises:

generating subtitles according to the text information obtained from speech recognition; and

incorporating the generated subtitles, according to time axis information, into the broadcast data.

5. The method according to claim 1 , wherein the converting audio and video data into broadcast data in a predetermined format comprises:

encoding the audio and video data; and

packaging the encoded audio and video data into a code stream of streaming media.

6. The method according to claim 1 , wherein the converting audio and video data into broadcast data in a predetermined format, and performing speech recognition on the audio data in the audio and video data comprises:

converting the audio and video data, frame by frame, into the broadcast data in a predetermined format, and placing the broadcast data into a first FIFO (first-in-first-out) queue; and

performing speech recognition on the audio data in the audio and video data, and placing the text information obtained from speech recognition into a second FIFO queue.

7. The method according to claim 6 , wherein the adding the text information obtained from speech recognition into the broadcast data comprises:

extracting the broadcast data from the first FIFO queue, and extracting corresponding text information from the second FIFO queue;

adding the extracted text information into the extracted broadcast data; and

placing the broadcast data with the added text information into a third FIFO queue.

8. A live broadcasting method, comprising:

collecting audio and video data;

converting the collected audio and video data into a code stream of streaming media, and performing speech recognition on the audio data in the audio and video data;

in response to text information obtained from the speech recognition comprising a preset keyword, converting the text information to a corresponding operation instruction according to a preset corresponding relationship between the keyword and the operation instruction, and sending the operation instruction to a network device; and

in response to the text information obtained from the speech recognition not comprising the preset keyword, adding text information obtained from speech recognition into the code stream of streaming media, and sending the code stream of streaming media with the added text information to a source site of a content delivery network.

9. A data processing method, comprising:

receiving broadcast audio and video data from a network device for broadcasting;

collecting speech data from the audio and video data during the broadcasting process;

performing speech recognition on the collected speech data;

in response to text information obtained from the speech recognition comprising a preset keyword, converting the text information to a corresponding operation instruction according to a preset corresponding relationship between the keyword and the operation instruction, and sending the operation instruction to the network device; and

in response to the text information obtained from the speech recognition not comprising the preset keyword, sending the text information obtained from the speech recognition to the network device.

10. The method according to claim 9 , wherein the performing speech recognition on the collected speech data comprises:

segmenting the collected speech data; and

performing speech recognition on the segmented speech data segment by segment.

11. The method according to claim 10 , wherein the segmenting the collected speech data comprises:

segmenting the collected speech data at one or more intervals with energy continuously lower than a predetermined threshold for over a predetermined time length.

12. The method according to claim 9 , wherein the method further comprises:

displaying the text information obtained from the speech recognition.

13. A data processing device, comprising a non-transitory computer-readable memory and a processor, wherein:

the memory stores a program for data processing; and

the program for data processing, when executed by the processor, causes the device to:

convert audio and video data into broadcast data in a predetermined format, and perform speech recognition on audio data in the audio and video data;

in response to text information obtained from the speech recognition comprising a preset keyword, convert the text information to a corresponding operation instruction according to a preset corresponding relationship between the keyword and the operation instruction, and send the operation instruction to a network device; and

in response to the text information obtained from the speech recognition not comprising the preset keyword, add the text information obtained from speech recognition into the broadcast data.

14. The data processing device according to claim 13 , wherein to perform speech recognition on the audio data in the audio and video data, the program, when executed by the processor, further causes the device to:

segment the audio data in the audio and video data; and

perform speech recognition on the segmented audio data segment by segment.

15. The data processing device according to claim 14 , wherein to segment the audio data in the audio and video data, the program, when executed by the processor, further causes the device to:

segment the audio data in the audio and video data at one or more intervals with energy continuously lower than a predetermined threshold for over a predetermined time length.

16. The data processing device according to claim 13 , wherein to add the text information obtained from speech recognition into the broadcast data, the program, when executed by the processor, further causes the device to:

generate subtitles according to the text information obtained from speech recognition; and

incorporate the generated subtitles, according to time axis information, into the broadcast data.

17. The data processing device according to claim 13 , wherein to convert audio and video data into broadcast data in a predetermined format, the program, when executed by the processor, further causes the device to:

encode the audio and video data; and

package the encoded audio and video data into a code stream of streaming media.

18. The data processing device according to claim 13 , wherein to convert audio and video data into broadcast data in a predetermined format, and perform speech recognition on the audio data in the audio and video data, the program, when executed by the processor, further causes the device to:

convert the audio and video data, frame by frame, into the broadcast data in a predetermined format, and place the broadcast data into a first FIFO (first-in-first-out) queue; and

perform speech recognition on the audio data in the audio and video data, and place the text information obtained from speech recognition into a second FIFO queue.

19. The data processing device according to claim 18 , wherein to add the text information obtained from speech recognition into the broadcast data, the program, when executed by the processor, further causes the device to:

extract the broadcast data from the first FIFO queue, and extract corresponding text information from the second FIFO queue;

add the extracted text information into the extracted broadcast data; and

place the broadcast data with the added text information into a third FIFO queue.

20. A data processing device, comprising a non-transitory computer-readable memory and a processor, wherein:

the memory stores a program for data processing; and

the program for data processing, when executed by the processor, causes the device to:

collect audio and video data;

convert the collected audio and video data into a code stream of streaming media, and concurrently, perform speech recognition on audio data in the audio and video data;

in response to text information obtained from the speech recognition comprising a preset keyword, convert the text information to a corresponding operation instruction according to a preset corresponding relationship between the keyword and the operation instruction, and send the operation instruction to a network device; and

in response to the text information obtained from the speech recognition not comprising the preset keyword, add text information obtained from speech recognition into the code stream of streaming media, and send the code stream of streaming media with the added text information to a source site of a content delivery network.

21. A data processing device, comprising a non-transitory computer-readable memory and a processor, wherein:

the memory stores a program for data processing; and

the program for data processing, when read and executed by the processor, causes the device to:

receive broadcast audio and video data from a network device for broadcasting;

collect speech data from the audio and video data during the broadcasting process;

perform speech recognition on the collected speech data;

in response to text information obtained from the speech recognition comprising a preset keyword, convert the text information to a corresponding operation instruction according to a preset corresponding relationship between the keyword and the operation instruction, and send the operation instruction to a network device; and

in response to the text information obtained from the speech recognition not comprising the preset keyword, send the text information obtained from the speech recognition to the network device.

22. The data processing device according to claim 21 , wherein to perform speech recognition on the collected speech data, the program, when executed by the processor, further causes the data processing device to:

segment the collected speech data; and

perform speech recognition on the segmented speech data segment by segment.

23. The data processing device according to claim 22 , wherein to segment the collected speech data, the program, when executed by the processor, further causes the data processing device to:

segment the collected speech data at one or more intervals with energy continuously lower than a predetermined threshold for over a predetermined time length.

24. The data processing device according to claim 21 , wherein the program, when executed by the processor, further causes the data processing device to:

display the text information obtained from the speech recognition.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 21, 2026
From: ALIBABA GROUP HOLDING LIMITED
To: CLOUD INTELLIGENCE ASSETS HOLDING (SINGAPORE) PRIVATE LIMITED
Reel/Frame 075478/0225 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 19, 2019
From: XU, GANG
To: ALIBABA GROUP HOLDING LIMITED
Reel/Frame 048373/0993 →
Priority Claims (1)
CN 2016 1 0639385 · Aug 5, 2016 · national
Continuity (1)
Related Publication 20180041783A1 · Feb 8, 2018