IP Library Granted Patent US 11,392,347
Granted Patent B2
US 11,392,347 · App. 16/904,526 · Granted Jul 19, 2022

Audio messaging interface on messaging platform

Inventors: Richard Plom (San Francisco, CA); Reed Martin (San Francisco, CA); Max Rose (San Francisco, CA)
Assignee: Twitter, Inc.
G06F3/167G06F3/0484G10L15/26H04M3/5307
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,392,347
App. No.
16/904,526
Granted
Jul 19, 2022
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for audio messaging interface for messaging platform. One of the methods includes receiving, by a first client on a first user device, a request to record an audio message, wherein the first client is configured to provide a user interface for the platform for a user using the first user device who is logged in to a user account on the platform; recording audio through a microphone of the first user device; generating a platform message by (i) generating a video file that includes the recorded audio as an audio portion of the video file and programmatically generated minimal video content as a video portion of the video file, and (ii) including the video file in the platform message; and posting, by the first client, the platform message to the platform, in response to a post request.

Claims (85)

1. A method, comprising:

receiving, by a first client on a first user device, a request to record an audio message for a social messaging platform, wherein the first client is software running on the first user device, wherein the first client is configured to provide a user interface for the social messaging platform for a user using the first user device who is logged in to a user account on the social messaging platform, and wherein the request is received through the user interface;

recording, by the first client, in response to the request, audio with a first file size received through a microphone of the first user device;

programmatically generating, by the first client, minimal artificial video content with a second file size that is less than the first file size of the recorded audio;

generating a platform message by (i) encoding the recorded audio with frames of the minimal artificial video content and generating a video file that includes the recorded audio as an audio portion of the video file and the minimal artificial video content as a video portion of the video file, and (ii) including the video file in the platform message; and

posting, by the first client, the platform message to the social messaging platform and causing a second client to decode, using a codec for an audiovisual file format, the video file included in the platform message and use the minimal artificial video content as a placeholder in the video file for second video content that will be rendered in place of the minimal artificial video content, in response to a post request received by the first client.

2. The method of claim 1 , further comprising:

receiving, by the first client on the first user device, one or more edit requests to edit the recorded audio;

and in response to the one or more edit requests and before generating the platform message, editing the recorded audio.

3. The method of claim 1 , wherein the minimal artificial video content presents an appearance of a moving waveform within a duration of the audio recording, the method comprising:

receiving, by the first client on the first user device, a request to insert a sound effect at a particular point in the audio recording;

receiving, by the first client on the first user device, a selection of a visual marker corresponding to the sound effect and a time input indicating when in the recorded audio to include the sound effect, wherein the visual marker is selected from among one or more visual markers displayed on the user interface; and

in response to the selection of the visual marker and the time input:

modifying the recorded audio to play the sound effect at the received time input in the recorded audio, and

modifying the minimal artificial video content to display the visual marker on the moving waveform at a position corresponding to the time input.

4. The method of claim 3 , wherein modifying the recorded audio comprises inserting the sound effect at the received time input in the recorded audio.

5. The method of claim 3 , wherein modifying the audio recording comprises overlaying the sound effect over the recorded audio at the received time input, wherein the audio recording includes both the received time input and the sound effect when played back at the received time input.

6. The method of claim 1 , wherein the minimal artificial video content is an unchanging image, a monochrome image, or a loop of a waveform against a static background.

7. The method of claim 6 , wherein:

the minimal artificial video content is the loop of the waveform against the static background; and

the waveform is a visual representation of at least a portion of the recorded audio to the first user device.

8. The method of claim 1 , wherein the video file is an MPEG-4 file.

9. The method of claim 1 , wherein programmatically generating the minimal artificial video content comprises: generating video frames including information describing the recorded audio.

10. The method of claim 1 , wherein the platform message includes a signature comprising a hash value of the minimal artificial video content.

11. The method of claim 1 , wherein the platform message includes a flag that a) indicates that the minimal artificial video content has been programmatically generated, and b) causes the second client to use the minimal artificial video content as the placeholder.

12. A system comprising:

one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving, by a first client on a first user device, a request to record an audio message for a social messaging platform, wherein the first client is software running on the first user device, wherein the first client is configured to provide a user interface for the social messaging platform for a user using the first user device who is logged in to a user account on the social messaging platform, and wherein the request is received through the user interface;

recording, by the first client, in response to the request, audio with a first file size received through a microphone of the first user device;

programmatically generating, by the first client, minimal artificial video content with a second file size that is less than the first file size of the recorded audio;

generating a platform message by (i) encoding the recorded audio with frames of the minimal artificial video content and generating a video file that includes the recorded audio as an audio portion of the video file and the minimal artificial video content as a video portion of the video file, and (ii) including the video file in the platform message; and

posting, by the first client, the platform message to the social messaging platform and causing a second client to decode, using a codec for an audiovisual file format, the video file included in the platform message and use the minimal artificial video content as a placeholder in the video file for second video content that will be rendered in place of the minimal artificial video content, in response to a post request received by the first client.

13. The system of claim 12 , wherein the operations further comprise:

receiving, by the first client on the first user device, one or more edit requests to edit the recorded audio; and

in response to the one or more edit requests and before generating the platform message, editing the recorded audio.

14. The system of claim 12 , wherein the minimal artificial video content presents an appearance of a moving waveform within a duration of the audio recording, the operations further comprising:

receiving, by the first client on the first user device, a request to insert a sound effect at a particular point in the audio recording;

receiving, by the first client on the first user device, a selection of a visual marker corresponding to the sound effect and a time input indicating when in the recorded audio to include the sound effect, wherein the visual marker is selected from among one or more visual markers displayed on the user interface; and

in response to the selection of the visual marker and the time input:

modifying the recorded audio to play the sound effect at the received time input in the recorded audio, and

modifying the minimal artificial video content to display the visual marker on the moving waveform at a position corresponding to the time input.

15. The system of claim 14 , wherein modifying the recorded audio comprises inserting the sound effect at the received time input in the recorded audio.

16. The system of claim 14 , wherein modifying the audio recording comprises overlaying the sound effect over the recorded audio at the received time input, wherein the audio recording includes both the received time input and the sound effect when played back at the received time input.

17. The system of claim 12 , wherein the minimal artificial video content is an unchanging image, a monochrome image, or a loop of a waveform against a static background.

18. The system of claim 17 , wherein:

the minimal artificial video content is the loop of the waveform against the static background; and

the waveform is a visual representation of at least a portion of the recorded audio to the first user device.

19. The system of claim 12 , wherein the video file is an MPEG-4 file.

20. The system of claim 12 , wherein the operations further comprise:

receiving, by the first client, a speech-to-text request; and

in response to the speech-to-text request:

detecting speech in the recorded audio,

generating text corresponding to the detected speech, and

displaying the generated text corresponding to the detected speech;

wherein generating the platform message comprises adding the generated text to the platform message; and

using, by the social messaging platform, one or more extracted terms from the generated text to index the platform message.

21. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving, by a first client on a first user device, a request to record an audio message for a social messaging platform, wherein the first client is software running on the first user device, wherein the first client is configured to provide a user interface for the social messaging platform for a user using the first user device who is logged in to a user account on the platform, and wherein the request is received through the user interface;

recording, by the first client, in response to the request, audio with a first file size received through a microphone of the first user device;

programmatically generating, by the first client, minimal artificial video content with a second file size that is less than the first file size of the recorded audio;

generating a platform message by (i) encoding the recorded audio with frames of the minimal artificial video content and generating a video file that includes the recorded audio as an audio portion of the video file and the minimal artificial video content as a video portion of the video file, and (ii) including the video file in the platform message; and

posting, by the first client, the platform message to the social messaging platform and causing a second client to decode, using a codec for an audiovisual file format, the video file included in the platform message and use the minimal artificial video content as a placeholder in the video file for second video content that will be rendered in place of the minimal artificial video content, in response to a post request received by the first client.

22. The computer-readable storage media of claim 21 , wherein the operations further comprise:

receiving, by the first client on the first user device, one or more edit requests to edit the recorded audio; and

in response to the one or more edit requests and before generating the platform message, editing the recorded audio.

23. The computer-readable storage media of claim 21 , wherein the minimal artificial video content presents an appearance of a moving waveform within a duration of the audio recording, the operations further comprising:

receiving, by the first client on the first user device, a request to insert a sound effect at a particular point in the audio recording;

receiving, by the first client on the first user device, a selection of a visual marker corresponding to the sound effect and a time input indicating when in the recorded audio to include the sound effect, wherein the visual marker is selected from among one or more visual markers displayed on the user interface; and

in response to the selection of the visual marker and the time input:

modifying the recorded audio to play the sound effect at the received time input in the recorded audio, and

modifying the minimal artificial video content to display the visual marker on the moving waveform at a position corresponding to the time input.

24. The computer-readable storage media of claim 23 , wherein modifying the recorded audio comprises inserting the sound effect at the received time input in the recorded audio.

25. The computer-readable storage media of claim 23 , wherein modifying the audio recording comprises overlaying the sound effect over the recorded audio at the received time input, wherein the audio recording includes both the received time input and the sound effect when played back at the received time input.

26. The computer-readable storage media of claim 21 , wherein the minimal artificial video content is an unchanging image, a monochrome image, or a loop of a waveform against a static background.

27. The computer-readable storage media of claim 26 , wherein:

the minimal artificial video content is the loop of the waveform against the static background; and

the waveform is a visual representation of at least a portion of the recorded audio to the first user device.

28. The computer-readable storage media of claim 21 , wherein the video file is an MPEG-4 file.

29. The computer-readable storage media of claim 21 , wherein the operations further comprise:

receiving, by the first client, a speech-to-text request; and

in response to the speech-to-text request:

detecting speech in the recorded audio,

generating text corresponding to the detected speech, and

displaying the generated text corresponding to the detected speech; and

wherein generating the platform message comprises adding the generated text to the platform message.

Assignments (7)
TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENT RIGHTS (REEL 062079, FRAME 0677) Recorded Mar 3, 2026
From: MORGAN STANLEY SENIOR FUNDING, INC., AS COLLATERAL AGENT
To: X CORP. (F/K/A TWITTER, INC.)
Reel/Frame 075015/0574 →
RELEASE OF SECURITY INTEREST Recorded Apr 30, 2025
From: MORGAN STANLEY SENIOR FUNDING, INC., AS COLLATERAL AGENT
To: X CORP. (F/K/A TWITTER, INC.)
Reel/Frame 071127/0240 →
RELEASE OF SECURITY INTEREST Recorded Mar 27, 2025
From: MORGAN STANLEY SENIOR FUNDING, INC.
To: X CORP. (F/K/A TWITTER, INC.)
Reel/Frame 070670/0857 →
SECURITY INTEREST Recorded Oct 28, 2022
From: TWITTER, INC.
To: MORGAN STANLEY SENIOR FUNDING, INC.
Reel/Frame 062079/0677 →
SECURITY INTEREST Recorded Oct 28, 2022
From: TWITTER, INC.
To: MORGAN STANLEY SENIOR FUNDING, INC.
Reel/Frame 061804/0001 →
SECURITY INTEREST Recorded Oct 28, 2022
From: TWITTER, INC.
To: MORGAN STANLEY SENIOR FUNDING, INC.
Reel/Frame 061804/0086 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 15, 2021
From: PLOM, RICHARD; MARTIN, REED; ROSE, MAX
To: TWITTER, INC.
Reel/Frame 057180/0827 →
Continuity (1)
Related Publication 20210397411A1 · Dec 23, 2021