IP Library › Granted Patent US 10,581,782
Granted Patent B2
US 10,581,782 · App. 15/470,004 · Granted Mar 3, 2020

Generating a stitched data stream

Inventor: Kevin Dechau Tang (Los Angeles, CA)
Assignee: Snap Inc.
H04L51/16H04L51/10H04L51/20H04L65/4069H04L65/605H04N21/2743H04N21/41407H04N21/4394H04N21/4788H04L51/32H04L63/107H04N21/4524
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,581,782
App. No.
15/470,004
Granted
Mar 3, 2020
Kind
B2
Abstract

Systems and methods provide for a server computer to receive a plurality of messages from a plurality of user computing devices, each message of the plurality of messages comprising a data stream, determine a subset of messages of the plurality of messages associated with a similar geolocation and time period, determine a set of messages of the subset of messages based on a match score for each pair of messages, and stitch together the set of messages to generate a stitched data stream from the data streams for each message of the set of messages based on a time period for each message, wherein the stitched data stream comprises messages with data streams that overlap in time periods such that there may be more than one data stream for a given time period.

Claims (59)

1. A method comprising:

receiving, at a server computer, a plurality of messages from a plurality of user computing devices, each message of the plurality of messages comprising a data stream;

determining, by the server computer, a subset of messages of the plurality of messages associated with a similar geolocation and time period;

extracting, by the server computer, an audio fingerprint for each message in the subset of messages;

grouping, by the server computer, the subset of messages into a plurality of pairs of messages, each pair of messages comprising a first message and a second message;

comparing, by the server computer, audio fingerprints of the first message and the second message in each pair of messages to determine a match score for each pair of messages;

determining, by the server computer, a set of messages of the subset of messages associated with a common audio timeline based on the match score for each pair of messages;

generating, by the server computer, a stitched data stream from the data streams of each message of the set of messages by performing operations comprising:

determining, for each message, a time period corresponding to the common audio timeline;

determining, a start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline; and

stitching together, by the server computer, the set of messages starting with the start message to generate the stitched data stream from the data streams for each message of the set of messages based on the time period for each message, wherein the stitched data stream comprises messages with data streams that overlap in time periods such that there is more than one data stream for at least one time period in the stitched data stream; and

providing, by the server computer, the stitched data stream to one or more user computing devices, to be displayed on the one or more computing devices as a continuous data stream that transitions from one message data stream to a next message data stream to a continuous common audio stream.

2. The method of claim 1 , wherein determining a subset of messages of the plurality of messages associated with a similar geolocation comprises determining that a geolocation for each of the subset of messages is associated with the same predetermined region of global positioning system (GPS) coordinates.

3. The method of claim 1 , wherein determining the start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline for the subset of messages comprises selecting the start message from a plurality of messages with similar periods of time in the common audio timeline based on a quality score.

4. The method of claim 1 , wherein determining the start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline for the subset of messages comprises randomly selecting the start message from a plurality of messages with similar periods of time in the common audio timeline.

5. The method of claim 1 , wherein the stitched data stream comprises messages that are prioritized based on a quality score for an order of display in the timeline.

6. The method of claim 1 , wherein the stitched data stream comprises messages that are prioritized based on a random selection for an order of display in the timeline.

7. The method of claim 1 , wherein audio associated with the stitched data stream comprises audio that is an average of audio associated with each data stream in the stitched data stream.

8. The method of claim 1 , wherein stitching together the set of messages starting with the start message to generate the stitched data stream from the data streams for each message of the set of messages based on the time period for each message, further comprises:

determining a first data stream from a message, of a set of messages that each have a time period that intersects with the time period of the start message;

selecting the first data stream as the second data stream for the stitched data stream; and

continuing to determine a data stream that has the longest duration of a set of messages that each have a time period that intersects with a previously selected data stream to select a next data stream for the stitched data stream.

9. The method of claim 1 , wherein extracting the audio fingerprint for each message in the subset of messages comprises:

analyzing the audio for each message in the subset of messages to calculate an audio fingerprint for each message in the subset of messages by extracting features from a spectrogram corresponding the audio for each message in the subset of messages.

10. The method of claim 1 , wherein the continuous audio stream comprises audio that is an average of audio associated with each data stream in the stitched data stream.

11. A server computer comprising:

a processor; and

a computer readable medium coupled with the processor, the computer readable medium comprising instructions stored thereon that are executable by the processor to cause a computing device to perform operations comprising:

receiving a plurality of messages from a plurality of user computing devices, each message of the plurality of messages comprising a data stream;

determining a subset of messages of the plurality of messages associated with a similar geolocation and time period;

extracting an audio fingerprint for each message in the subset of messages;

grouping the subset of messages into a plurality of pairs of messages, each pair of messages comprising a first message and a second message;

comparing audio fingerprints of the first message and the second message in each pair of messages to determine a match score for each pair of messages;

determining a set of messages of the subset of messages associated with a common audio timeline based on the match score for each pair of messages;

generating a stitched data stream from the data streams of each message of the set of messages by performing operations comprising:

determining for each message, a time period corresponding to the common audio timeline;

determining, a start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline; and

stitching together the set of messages starting with the start message to generate the stitched data stream from the data streams for each message of the set of messages based on the time period for each message, wherein the stitched data stream comprises messages with data streams that overlap in time periods such that there is more than one data stream for at least one time period in the stitched data stream; and

providing the stitched data stream to one or more user computing devices, to be displayed on the one or more computing devices as a continuous data stream that transitions from one message data stream to a next message data stream to a continuous common audio stream.

12. The server computer of claim 11 , wherein determining the subset of messages of the plurality of messages associated with the similar geolocation comprises determining that a geolocation for each of the subset of messages is associated with the same predetermined region of global positioning system (GPS) coordinates.

13. The server computer of claim 11 , wherein determining the start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline for the subset of messages comprises selecting the start message from a plurality of messages with similar periods of time in the common audio timeline based on a quality score.

14. The server computer of claim 11 , wherein determining the start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline for the subset of messages comprises randomly selecting the start message from a plurality of messages with similar periods of time in the common audio timeline.

15. The server computer of claim 11 , wherein the stitched data stream comprises messages that are prioritized based on a quality score for an order of display in the timeline.

16. The server computer of claim 11 , wherein the stitched data stream comprises messages that are prioritized based on a random selection for an order of display in the timeline.

17. The server computer of claim 11 , wherein audio associated with the stitched data stream comprises audio that is an average of audio associated with each data stream in the stitched data stream.

18. A non-transitory computer readable medium comprising instructions stored thereon that are executable by at least one processor to cause a computing device to perform operations comprising:

receiving a plurality of messages from a plurality of user computing devices, each message of the plurality of messages comprising a data stream;

determining a subset of messages of the plurality of messages associated with a similar geolocation and time period;

extracting an audio fingerprint for each message in the subset of messages;

grouping the subset of messages into a plurality of pairs of messages, each pair of messages comprising a first message and a second message;

comparing audio fingerprints of the first message and the second message in each pair of messages to determine a match score for each pair of messages;

determining a set of messages of the subset of messages associated with a con on audio timeline based on the match score for each pair of messages;

generating a stitched data stream from the data streams of each message of the set of messages by performing operations comprising:

determining, for each message, a time period corresponding to the common timeline;

determining, a start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline; and

stitching together the set of messages starting with the start message to generate the stitched data stream from the data streams for each message of the set of messages based on the time period for each message, wherein the stitched data stream comprises messages with data streams that overlap in time periods such that there is more than one data stream for at least one time period in the stitched data stream; and

providing the stitched data stream to one or more user computing devices, to be displayed on the one or more computing devices as a continuous data stream that transitions from one message data stream to a next message data stream to a continuous common audio stream.

19. The non-transitory computer readable medium of claim 18 , wherein determining the start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline for the subset of messages comprises selecting the start message from a plurality of messages with similar periods of time in the common audio timeline based on a quality score.

20. The non-transitory computer readable medium of claim 18 , wherein determining the start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline for the subset of messages comprises randomly selecting the start message from a plurality of messages with similar periods of time in the common audio timeline.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 20, 2020
From: TANG, KEVIN DECHAU
To: SNAP INC.
Reel/Frame 051553/0327 →
Continuity (1)
Related Publication 20180278562A1 · Sep 27, 2018
Cited By (6)
US 12,204,807 US 12,231,437 US 12,236,148 US 12,342,026 US 12,387,403 US 12,726,454