Data deduplication method and apparatus, and storage medium
View Patent ↗A data deduplication method includes: acquiring a live streaming data stream to be processed, wherein the data stream includes at least one audio/video packet; and executing following operations on one audio/video packet, each audio/video packet carrying an identifier, and the identifier being determined according to the order of the audio/video packet in the live streaming data stream; determining, according to the identifier of the audio/video packet, whether the audio/video packet is a repeatedly acquired audio/video packet, and if so, discarding the audio/video packet. Accordingly, it is ensured that no unduplicated data is sent to a user, and the problem that in a live streaming process, due to the fact that an intermediate link is reconnected, a user acquired part of duplicated data, and the audio/video rollback phenomenon caused by the duplicated data being played is solved.
1 . A data deduplication method, comprising:
acquiring a live streaming data stream to be processed, the live streaming data stream to be processed comprising at least one audio/video packet;
performing following operations on one audio/video packet of the at least one audio/video packet, wherein an audio/video packet of the at least one audio/video packet carries an audio/video identifier, and the audio/video identifier is determined according to an order of audio/video packets of the at least one audio/video packet in the live streaming data stream to be processed:
determining, according to the audio/video identifier of the one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet;
when it is determined that the one audio/video packet of the at least one audio/video packet is not a repeatedly acquired audio/video packet, recording the audio/video identifier of the one audio/video packet of the at least one audio/video packet, and removing the audio/video identifier of the one audio/video packet of the at least one audio/video packet after recording the audio/video identifier of the one audio/video packet of the at least one audio/video packet, so that a terminal performs audio/video playing based on the one audio/video packet of the at least one audio/video packet with the audio/video identifier removed; and
when it is determined that the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet, discarding the one audio/video packet of the at least one audio/video packet.
2 . The method according to claim 1 , wherein the determining whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet according to the audio/video identifier of the one of the at least one audio/video packet, comprises:
determining whether audio/video packets in the acquired live streaming data stream to be processed comprise an audio/video packet having the same audio/video identifier as the audio/video identifier of the one audio/video packet of the at least one audio/video packet; and
if the audio/video packets in the acquired live streaming data stream to be processed comprise an audio/video packet having the same audio/video identifier as the audio/video identifier of the one audio/video packet of the at least one audio/video packet, determining that the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet.
3 . The method according to claim 2 , wherein before the determining that the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet, the method further comprises:
determining whether the audio/video identifier of the one audio/video packet of the at least one audio/video packet is an audio/video identifier that is reset after audio/video identifiers of the audio/video packets in the live streaming data stream to be processed reach a preset end identifier; and
the determining that the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet comprises:
if the audio/video identifier of the one audio/video packet of the at least one audio/video packet is not an audio/video identifier that is reset after the audio/video identifiers of audio/video packets in the live streaming data stream to be processed reach the preset end identifier, determining that the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet.
4 . The method according to claim 2 , wherein the audio/video identifiers of the audio/video packets of the at least one audio/video packet monotonically increase according to the order of the audio/video packets of the at least one audio/video packet in the live streaming data stream to be processed;
the determining whether the audio/video packets in the acquired live streaming data stream to be processed comprise an audio/video packet having the same audio/video identifier as the audio/video identifier of the one audio/video packet of the at least one audio/video packet comprises:
determining a maximum audio/video identifier of the acquired audio/video packets according to the audio/video packets of the acquired live streaming data stream to be processed;
determining whether the audio/video identifier of the one audio/video packet of the at least one audio/video packet is less than or equal to the maximum audio/video identifier of the acquired audio/video packets; and
if the audio/video identifier of the one audio/video packet of the at least one audio/video packet is less than or equal to the maximum audio/video identifier of the acquired audio/video packets, determining that the acquired audio/video packets in the live streaming data stream to be processed comprise an audio/video packet having the same audio/video identifier as the audio/video identifier of the one audio/video packet of the at least one audio/video packet.
5 . The method according to claim 3 , wherein the audio/video identifiers of the audio/video packets of the at least one audio/video packet monotonically increase according to the order of the audio/video packets of the at least one audio/video packet in the live streaming data stream to be processed;
the determining whether the audio/video identifier of one audio/video packet of the at least one audio/video packet is an audio/video identifier that is reset after the audio/video identifiers of the audio/video packets in the live streaming data stream to be processed reach a preset end identifier comprises:
acquiring the audio/video identifier of a first audio/video packet at an interval of a preset number of audio/video packets in a first preset direction from the one audio/video packet of the at least one audio/video packet in the live streaming data stream to be processed, and the audio/video identifier of a second audio/video packet at an interval of the preset number of audio/video packets in a second preset direction from one audio/video packet of the at least one audio/video packet in the live streaming data stream to be processed, wherein the second preset direction and the first preset direction are opposite; and
if the audio/video identifier of the first audio/video packet is greater than or equal to the audio/video identifier of the second audio/video packet, determining that the audio/video identifier of the one audio/video packet of the at least one audio/video packet is an audio/video identifier that is reset after the audio/video identifiers of the audio/video packets in the live streaming data stream to be processed reach the preset end identifier.
6 . The method according to claim 1 , wherein before the determining, according to the audio/video identifier of the one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet, the method further comprises:
determining whether the live streaming data stream to be processed is a new pushed stream; and
the determining, according to the audio/video identifier of the one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet comprises:
if the live streaming data stream to be processed is not a new pushed stream, determining, according to the audio/video identifier of one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet.
7 . The method according to claim 6 , wherein the live streaming data stream to be processed carries a stream pushing timestamp; and
the determining whether the live streaming data stream to be processed is a new pushed stream comprises:
determining, according to the stream pushing timestamp, whether the stream pushing timestamp changes; and
when it is determined that the stream pushing timestamp changes, determining that the live streaming data stream to be processed is a new pushed stream.
8 . The method according to claim 1 , wherein before the determining, according to the audio/video identifier of the one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet, the method further comprises:
determining whether the audio/video identifier of the one audio/video packet of the at least one audio/video packet is empty; and
the determining, according to the one audio/video identifier of one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet comprises:
if the audio/video identifier of the one audio/video packet of the at least one audio/video packet is not empty, determining, according to the audio/video identifier of the one audio/video packet of the at least one audio/video packet, whether the audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet.
9 . A stream pulling device, comprising:
a processor;
a memory; and
a computer program,
wherein the computer program is stored in the memory and is configured to be executed by the processor, and the computer program comprises instructions used for causing the stream pulling device to:
acquire a live streaming data stream to be processed, the live streaming data stream to be processed comprising at least one audio/video packet;
perform following operations on one audio/video packet of the at least one audio/video packet, wherein an audio/video packet of the at least one audio/video packet carries an audio/video identifier, and the audio/video identifier is determined according to an order of audio/video packets of the at least one audio/video packet in the live streaming data stream to be processed:
determine, according to the audio/video identifier of the one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet;
when it is determined that the one audio/video packet of the at least one audio/video packet is not a repeatedly acquired audio/video packet, record the audio/video identifier of the one audio/video packet of the at least one audio/video packet, and remove the audio/video identifier of the one audio/video packet of the at least one audio/video packet after recording the audio/video identifier of the one audio/video packet of the at least one audio/video packet, so that a terminal performs audio/video playing based on the one audio/video packet of the at least one audio/video packet with the audio/video identifier removed; and
when it is determined that the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet, discard the one audio/video packet of the at least one audio/video packet.
10 . The stream pulling device according to claim 9 , wherein instructions used for causing the stream pulling device to determine whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet according to the audio/video identifier of the one of the at least one audio/video packet, further comprise instructions used for causing the stream pulling device to:
determine whether audio/video packets in the acquired live streaming data stream to be processed comprise an audio/video packet having the same audio/video identifier as the audio/video identifier of the one audio/video packet of the at least one audio/video packet; and
if the audio/video packets in the acquired live streaming data stream to be processed comprise an audio/video packet having the same audio/video identifier as the audio/video identifier of the one audio/video packet of the at least one audio/video packet, determine that the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet.
11 . The stream pulling device according to claim 10 , wherein before determining that the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet, the instructions further comprise instructions used for causing the stream pulling device to:
determine whether the audio/video identifier of the one audio/video packet of the at least one audio/video packet is an audio/video identifier that is reset after audio/video identifiers of the audio/video packets in the live streaming data stream to be processed reach a preset end identifier; and
the instructions used for causing the stream pulling device to determine that the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet further comprises instructions used for causing the stream pulling device to:
if the audio/video identifier of the one audio/video packet of the at least one audio/video packet is not an audio/video identifier that is reset after the audio/video identifiers of audio/video packets in the live streaming data stream to be processed reach the preset end identifier, determine that the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet.
12 . The stream pulling device according to claim 10 , wherein the audio/video identifiers of the audio/video packets of the at least one audio/video packet monotonically increase according to the order of the audio/video packets of the at least one audio/video packet in the live streaming data stream to be processed;
the instructions used for causing the stream pulling device to determine whether the audio/video packets in the acquired live streaming data stream to be processed comprise an audio/video packet having the same audio/video identifier as the audio/video identifier of the one audio/video packet of the at least one audio/video packet further comprise instructions used for causing the stream pulling device to:
determine a maximum audio/video identifier of the acquired audio/video packets according to the audio/video packets of the acquired live streaming data stream to be processed;
determine whether the audio/video identifier of the one audio/video packet of the at least one audio/video packet is less than or equal to the maximum audio/video identifier of the acquired audio/video packets; and
if the audio/video identifier of the one audio/video packet of the at least one audio/video packet is less than or equal to the maximum audio/video identifier of the acquired audio/video packets, determine that the acquired audio/video packets in the live streaming data stream to be processed comprise an audio/video packet having the same audio/video identifier as the audio/video identifier of the one audio/video packet of the at least one audio/video packet.
13 . The stream pulling device according to claim 11 , wherein the audio/video identifiers of the audio/video packets of the at least one audio/video packet monotonically increase according to the order of the audio/video packets of the at least one audio/video packet in the live streaming data stream to be processed;
the instructions used for causing the stream pulling device to determine whether the audio/video identifier of one audio/video packet of the at least one audio/video packet is an audio/video identifier that is reset after the audio/video identifiers of the audio/video packets in the live streaming data stream to be processed reach a preset end identifier comprise the instructions used for causing the stream pulling device to:
acquire the audio/video identifier of a first audio/video packet at an interval of a preset number of audio/video packets in a first preset direction from the one audio/video packet of the at least one audio/video packet in the live streaming data stream to be processed, and the audio/video identifier of a second audio/video packet at an interval of the preset number of audio/video packets in a second preset direction from one audio/video packet of the at least one audio/video packet in the live streaming data stream to be processed, wherein the second preset direction and the first preset direction are opposite; and
if the audio/video identifier of the first audio/video packet is greater than or equal to the audio/video identifier of the second audio/video packet, determine that the audio/video identifier of the one audio/video packet of the at least one audio/video packet is an audio/video identifier that is reset after the audio/video identifiers of the audio/video packets in the live streaming data stream to be processed reach the preset end identifier.
14 . The stream pulling device according to claim 9 , wherein before the stream pulling device is cause to determine, according to the audio/video identifier of the one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet, the instructions further comprise instructions used for causing the stream pulling device to:
determine whether the live streaming data stream to be processed is a new pushed stream; and
the instructions used for causing the stream pulling device to determine, according to the audio/video identifier of the one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet comprises instructions used for causing the stream pulling device to:
if the live streaming data stream to be processed is not a new pushed stream, determine, according to the audio/video identifier of one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet.
15 . The stream pulling device according to claim 14 , wherein the live streaming data stream to be processed carries a stream pushing timestamp; and
the instructions used for causing the stream pulling device to determine whether the live streaming data stream to be processed is a new pushed stream comprise instructions used for causing the stream pulling device to:
determine, according to the stream pushing timestamp, whether the stream pushing timestamp changes; and
when it is determined that the stream pushing timestamp changes, determine that the live streaming data stream to be processed is a new pushed stream.
16 . The stream pulling device according to claim 9 , wherein before the stream pulling device is caused to determine, according to the audio/video identifier of the one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet, the instructions further comprise instructions used for causing the stream pulling device to:
determine whether the audio/video identifier of the one audio/video packet of the at least one audio/video packet is empty; and
the instructions used for causing the stream pulling device to determine, according to the one audio/video identifier of one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet comprise instructions used for causing the stream pulling device to:
if the audio/video identifier of the one audio/video packet of the at least one audio/video packet is not empty, determine, according to the audio/video identifier of the one audio/video packet of the at least one audio/video packet, whether the audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet.
17 . A computer program product, comprising computer instructions, wherein the computer instructions, when executed by a processor, to cause the processor to:
acquire a live streaming data stream to be processed, the live streaming data stream to be processed comprising at least one audio/video packet;
perform following operations on one audio/video packet of the at least one audio/video packet, wherein an audio/video packet of the at least one audio/video packet carries an audio/video identifier, and the audio/video identifier is determined according to an order of audio/video packets of the at least one audio/video packet in the live streaming data stream to be processed:
determine, according to the audio/video identifier of the one audio/video packet of the at least one audio/video packet, whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet;
when it is determined that the one audio/video packet of the at least one audio/video packet is not a repeatedly acquired audio/video packet, record the audio/video identifier of the one audio/video packet of the at least one audio/video packet, and remove the audio/video identifier of the one audio/video packet of the at least one audio/video packet after recording the audio/video identifier of the one audio/video packet of the at least one audio/video packet, so that a terminal performs audio/video playing based on the one audio/video packet of the at least one audio/video packet with the audio/video identifier removed; and
when it is determined that the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet, discard the one audio/video packet of the at least one audio/video packet.
18 . The computer program product according to claim 17 , wherein instructions to cause the processor to determine whether the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet according to the audio/video identifier of the one of the at least one audio/video packet, comprise instructions to cause the processor to:
determine whether audio/video packets in the acquired live streaming data stream to be processed comprise an audio/video packet having the same audio/video identifier as the audio/video identifier of the one audio/video packet of the at least one audio/video packet; and
when the audio/video packets in the acquired live streaming data stream to be processed comprise an audio/video packet having the same audio/video identifier as the audio/video identifier of the one audio/video packet of the at least one audio/video packet, determine that the one audio/video packet of the at least one audio/video packet is a repeatedly acquired audio/video packet.