IP Library Granted Patent US 11,594,214
Granted Patent B2
US 11,594,214 · App. 17/866,301 · Granted Feb 28, 2023

System and/or method for semantic parsing of air traffic control audio

Inventors: Michael Pust (Boston, MA); Joseph Bondaryk (Boston, MA); Matthew George (Boston, MA)
Assignee: MERLIN LABS, INC.
G10L15/1815B64C19/00G08G5/0013G10L15/16G10L15/26
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,594,214
App. No.
17/866,301
Granted
Feb 28, 2023
Kind
B2
Abstract

The method S 200 can include: at an aircraft, receiving an audio utterance from air traffic control S 210 , converting the audio utterance to text, determining commands from the text using a question-and-answer model S 240 , and optionally controlling the aircraft based on the commands S 250 . The method functions to automatically interpret flight commands from the air traffic control (ATC) stream.

Claims (68)

1. A system for an aircraft for semantic parsing of air traffic control (ATC) utterances, the system comprising:

an autonomous computing system configured to receive an ATC audio signal from a communication system, the autonomous computing system comprising:

a speech-to-text module configured to determine an utterance hypothesis from the ATC audio signal; and

a question-and-answer (Q/A) module configured to autonomously determine an aircraft command based on the utterance hypothesis using a plurality of natural language queries, the aircraft command the comprising at least one command parameter and a set of values corresponding to each command parameter, wherein each command parameter is selected from a predetermined set of command parameters, wherein each of the set of values and the at least one command parameter are determined based on the respective result of a distinct natural language query of the plurality of natural language queries; and

a validation system communicatively coupled to the autonomous computing system and configured to:

provide the aircraft command to a pilot;

determine a pilot validation of the aircraft command, the pilot validation comprising an audit of a respective result of each natural language query of the plurality of natural language queries; and

automatically provide the aircraft command to a flight processing system of the aircraft based on the pilot validation.

2. The system of claim 1 , wherein the ATC utterances are syntactically non-standardized, wherein the aircraft command is provided to the pilot in a human-interpretable format, wherein the aircraft command is automatically provided to the flight processing system in a standardized machine-readable format.

3. The system of claim 2 , wherein the utterance hypothesis comprises a text transcript, wherein the validation system is further configured to provide the utterance hypothesis to the pilot along with the aircraft command.

4. The system of claim 1 , wherein the pilot validation is determined based on a pilot confirmation input associated with the aircraft command.

5. The system of claim 4 , wherein the pilot confirmation input is received from a remote pilot located offboard the aircraft.

6. The system of claim 1 , wherein the pilot validation is determined automatically.

7. The system of claim 1 , wherein the pilot validation comprises an update to a respective result of a natural language query based on the audit.

8. The system of claim 1 , wherein the system further comprises the flight processing system, wherein the flight processing system comprises a flight management system (FMS), a flight control system (FCS), or an autopilot (AP) system.

9. A method for semantic parsing of air traffic control (ATC) utterances for an aircraft, the method comprising:

receiving an ATC audio signal;

determining an utterance hypothesis from the ATC audio signal with automatic speech recognition (ASR);

querying the utterance hypothesis with a pre-trained neural network model according to a sequence of the natural language queries; and

based on the sequence of natural language queries, determining an aircraft commands associated with the utterance hypothesis, the aircraft command comprising at least one command parameter and a set of corresponding values for each command parameter, wherein each command parameter is selected from a predetermined set of command parameters, wherein each of the set of corresponding values and the at least one command parameter are determined based on the respective result of a distinct natural language query of the sequence;

providing the aircraft command to a pilot;

determining a pilot validation comprising an audit of a respective result of each natural language query of the sequence; and

in response to determining the pilot validation, automatically facilitating operation of the aircraft based on the aircraft command.

10. The method of claim 9 , wherein the ATC utterances are syntactically non-standardized, wherein the aircraft command is provided to the pilot in a human-interpretable format, wherein automatically facilitating operation of the aircraft based on the aircraft command comprises providing the aircraft command to a flight processing system in a standardized machine-readable format.

11. The method of claim 9 , wherein the sequence comprises a tree-based sequence with a plurality of dependencies linking one or more natural language queries to a determination that the aircraft is an intended recipient of an utterance corresponding to the utterance hypothesis.

12. The method of claim 11 , wherein the aircraft command is provided to the pilot is based on the determination that the aircraft is an intended recipient of the utterance.

13. The method of claim 12 , wherein the utterance hypothesis is determined using an integrated automatic speech recognition and sentence boundary detection (ASR/SBD) model, wherein the utterance hypothesis comprises a speaker identification, wherein the determination that the aircraft is the intended recipient is based on the speaker identification.

14. The method of claim 9 , further comprising:

based on the audit, determining an errant result associated with a natural language query of the sequence; and

based on the errant result, updating the aircraft command with a corrected command parameter or value corresponding to the errant result.

15. The method of claim 9 , wherein the pilot validation is determined based on receipt of a pilot confirmation input associated with the aircraft command.

16. The method of claim 9 , wherein the pilot validation is determined automatically.

17. The method of claim 9 , further comprising:

determining a second and a third utterance hypothesis from the ATC audio signal;

querying the second utterance hypothesis with the pre-trained neural network model according to a second set of natural language queries;

based on the second set of natural language queries, determining that the aircraft is not an intended recipient of the utterance hypothesis;

querying the third utterance hypothesis with the pre-trained neural network model according to a third set of natural language queries;

based on the third set of natural language queries, determining a second aircraft command; and

providing the second aircraft command to a pilot,

wherein the first and second aircraft commands are provided along with the utterance hypothesis and the third utterance hypothesis, respectively,

wherein the second utterance hypothesis is associated with a time period between a first time period associated with the utterance hypothesis and a second time period associated with the third utterance hypothesis,

wherein the second utterance hypothesis is not provided to the pilot.

18. A method for semantic parsing of air traffic control (ATC) utterances for an aircraft, the method comprising:

receiving an ATC audio signal;

determining an utterance hypothesis from the ATC audio signal with automatic speech recognition (ASR);

querying the utterance hypothesis with a pre-trained neural network model according to a sequence of the natural language queries;

based on the sequence of natural language queries, determining an aircraft commands associated with the utterance hypothesis;

providing the aircraft command to a pilot;

determining a pilot validation comprising an audit of a respective result of each natural language query of the plurality;

based on the audit, determining an errant result associated with a natural language query of the sequence;

based on the errant result, updating the aircraft command with a corrected command parameter or value corresponding to the errant result; and

in response to determining the pilot validation, automatically facilitating operation of the aircraft based on the aircraft command.

19. A system for an aircraft for semantic parsing of air traffic control (ATC) utterances, the system comprising:

an autonomous computing system configured to receive an ATC audio signal from a communication system, the autonomous computing system comprising:

a speech-to-text module configured to determine an utterance hypothesis from the ATC audio signal; and

a question-and-answer (Q/A) module configured to autonomously determine an aircraft command based on the utterance hypothesis using a plurality of natural language queries; and

a validation system communicatively coupled to the autonomous computing system and configured to:

provide the aircraft command to a pilot;

determine a pilot validation of the aircraft command, wherein the pilot validation is determined based on a pilot confirmation input, received from a remote pilot located offboard the aircraft, associated with the aircraft command; and

automatically provide the aircraft command to a flight processing system of the aircraft based on the pilot validation.

20. A method for semantic parsing of air traffic control (ATC) utterances for an aircraft, the method comprising:

receiving an ATC audio signal;

determining an utterance hypothesis from the ATC audio signal with automatic speech recognition (ASR);

querying the utterance hypothesis with a pre-trained neural network model according to a sequence of the natural language queries; and

based on the sequence of natural language queries, determining an aircraft command associated with the utterance hypothesis, wherein the sequence comprises a tree-based sequence with a plurality of dependencies linking one or more natural language queries to a determination that the aircraft is an intended recipient of an utterance corresponding to the utterance hypothesis;

based on the determination that the aircraft is an intended recipient of the utterance, providing the aircraft command to a pilot;

determining a pilot validation; and

in response to determining the pilot validation, automatically facilitating operation of the aircraft based on the aircraft command.

Assignments (4)
RELEASE OF SECURITY INTEREST Recorded Mar 17, 2026
From: WTI FUND X, INC.
To: MERLIN LABS, INC.
Reel/Frame 074104/0389 →
SECURITY INTEREST Recorded Feb 23, 2024
From: MERLIN LABS, INC.
To: WTI FUND X, INC.
Reel/Frame 066550/0209 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 15, 2022
From: PUST, MICHAEL; BONDARYK, JOSEPH; GEORGE, MATTHEW
To: APOLLO FLIGHT RESEARCH INC.
Reel/Frame 060526/0017 →
CHANGE OF NAME Recorded Jul 15, 2022
From: APOLLO FLIGHT RESEARCH INC.
To: MERLIN LABS, INC.
Reel/Frame 060678/0692 →
Continuity (3)
Continuation 17500358 · Oct 13, 2021
Provisional Application 63090898 · Oct 13, 2020
Related Publication 20220366900A1 · Nov 17, 2022