Aircrew automation system and method with integrated imaging and force sensing modalities

1. An aircrew automation system for use in an aircraft, the aircrew automation system comprising:

a computer system having a processor and one or more interfaces to connect a plurality of systems or subsystems, wherein the computer system is communicatively coupled with a flight control system of the aircraft and is configured to generate commands based at least in part on flight situation data received from (1) one or more of the plurality of systems or subsystems or (2) the flight control system; and

an actuation system operatively coupled with the computer system via at least one of the one or more interfaces, wherein the actuation system comprises:

an optical sensor configured to visually identify one or more cockpit instruments corresponding to the flight control system;

an actuator configured to engage the one or more cockpit instruments;

a force sensor operably coupled to the actuator and configured to measure a force when the actuator makes contact with the one or more cockpit instruments; and

a controller operably coupled with the optical sensor, the actuator, and the force sensor, wherein the controller is configured (1) to map a location, type, and position of the one or more cockpit instruments as a function of data collected by the optical sensor and the force sensor, (2) to execute the commands received from the computer system via the actuator, and (3) to communicate data collected by the optical sensor or the force sensor to the computer system.

2. The aircrew automation system of claim 1 , wherein the optical sensor is an infrared camera or a visual camera.

3. The aircrew automation system of claim 1 , wherein the actuator comprises two or more digits that are individually manipulable.

4. The aircrew automation system of claim 3 , wherein each of the two or more digits comprises a respective force sensor.

5. The aircrew automation system of claim 1 , wherein the actuator is a robotic arm configured to maneuver in six degrees of freedom within a cockpit environment.

6. The aircrew automation system of claim 5 , wherein the optical sensor and the force sensor are collocated at a distal end of the robotic arm, the distal end being configured to make contact with the one or more cockpit instruments.

7. The aircrew automation system of claim 1 , further comprising a knowledge acquisition system operatively coupled with the computer system via at least one of the one or more interfaces to determine information specific to the aircraft.

8. The aircrew automation system of claim 7 , wherein the knowledge acquisition system is configured to access a matrix of force values corresponding to a force profile of the one or more cockpit instruments.

9. The aircrew automation system of claim 8 , wherein the controller is configured to compare measured data from the force sensor to the matrix of force values to identify a cockpit instrument of the one or more cockpit instruments based on a respective force profile.

10. The aircrew automation system of claim 7 , wherein the knowledge acquisition system is configured to access a database containing a matrix of image data corresponding to a layout of the one or more cockpit instruments.

11. The aircrew automation system of claim 10 , wherein the controller is configured to compare measured data from the optical sensor to the matrix of image data to identify a cockpit instrument of the one or more cockpit instruments based on a location of the cockpit instrument according to the layout.

12. The aircrew automation system of claim 1 , further comprising a perception system operatively coupled with the computer system via at least one of the one or more interfaces to monitor the one or more cockpit instruments of the aircraft to generate flight situation data.

13. The aircrew automation system of claim 1 , further comprising a human machine interface operatively coupled with the computer system via at least one of the one or more interfaces to provide an interface between a pilot and the aircrew automation system to display data to and accept commands from the pilot.

14. The aircrew automation system of claim 13 , wherein the human machine interface includes a touch screen display.

15. The aircrew automation system of claim 1 , wherein the actuator is a multi-tool configured to interact with a knob-style cockpit instrument and a switch-style cockpit instrument.

16. A method of implementing an aircrew automation system for use in an aircraft, the method comprising:

receiving a commanded heading from a flight control system of the aircraft to alter aircraft heading;

identifying, via an optical sensor, a knob amongst one or more cockpit instruments that is configured to alter aircraft heading;

activating a robotic arm configured to manipulate the one or more cockpit instruments;

controlling the robotic arm to engage the knob;

determining that a measured force profile for the knob matches a stored force profile value; and

turning the knob to reflect the commanded heading.

17. The method of claim 16 , further comprising the step of reading a first heading before contacting the knob.

18. The method of claim 17 , further comprising the step of calculating an amount of change required to achieve the commanded heading based on the reading of the first heading and the commanded heading.

19. The method of claim 16 , further comprising the steps of:

reading a second heading after turning the knob; and

comparing the second heading to the commanded heading.

20. The method of claim 19 , further comprising the steps of:

determining the second heading does not match the commanded heading; and

calculating a second amount of change required to achieve the commanded heading based on the reading of the second heading and the commanded heading.