Counter Optimization-Based Testing of Flight Envelope Protections in a Fly-By-Wire Control Law Using Deep Q-Learning

David Braun; Rasmus Steffensen; Agnes Steinert; Florian Holzapfel

← Back to list of papers of the 2022 EuroGNC conference

CEAS EuroGNC 2022

Counter Optimization-Based Testing of Flight Envelope Protections in a Fly-By-Wire Control Law Using Deep Q-Learning

David Braun	Research Associate, Institute of Flight System Dynamics, Technical University of Munich, 85748, Garching, Germany.
Rasmus Steffensen	Research Associate, Institute of Flight System Dynamics, Technical University of Munich, 85748, Garching, Germany.
Agnes Steinert	Research Associate, Institute of Flight System Dynamics, Technical University of Munich, 85748, Garching, Germany.
Florian Holzapfel	Professor and Head of the Institute of Flight System Dynamics, Technical University of Munich, 85748, Garching, Germany.

Abstract:
This paper investigates the use of Deep Q-learning for counter optimization-based flight control laws testing. Deep Q-learning is a model-free Reinforcement Learning (RL) method that leverages repeated simulation of a black-box function to identify the optimal control policy. In the scope of flight control laws testing, we employ Deep Q-learning to perform worst-case analyses of highly complex nonlinear flight control systems. In particular, we show how this model-free RL method can be used to identify the worst-case combination of pilot stick inputs and wind disturbances with respect to a specified clearance criterion. Furthermore, we elaborate how the decision making of the RL agent can be constrained in order to prevent combinations of inputs that would result in maneuvers that are unfeasible or, with regard to real world application, operationally irrelevant. Ultimately, this novel approach to flight control law analysis and testing allows to assess the worst-case performance of complex closed-loop systems without the need for model simplifications. By identifying worst-case scenarios that might not be easily predicted by the control engineers, the proposed RL-based counter optimization framework can serve as a valuable tool in the design process of modern flight control laws. We demonstrate the capability of the RL-based worst-case analysis on a high-fidelity closed-loop simulation model of an all attitude turboprop demonstrator aircraft. The response of the load factor protection of the fly-by-wire control law resulting from worst-case inputs and wind gust disturbances is investigated.

Keywords: Worst-Case Analysis; Reinforcement Learning; Testing of Flight Control Laws

View PDF CEAS-GNC-2022-076

David Braun, Rasmus Steffensen, Agnes Steinert, Florian Holzapfel: Counter Optimization-Based Testing of Flight Envelope Protections in a Fly-By-Wire Control Law Using Deep Q-Learning. Proceedings of the 2022 CEAS EuroGNC conference. Berlin, Germany. May 2022. CEAS-GNC-2022-076.

BibTeX entry:

@Incollection{CEAS-GNC-2022-076,

    authors = {Braun, David and Steffensen, Rasmus and Steinert, Agnes and Holzapfel, Florian},

    title = {Counter Optimization-Based Testing of Flight Envelope Protections in a Fly-By-Wire Control Law Using Deep Q-Learning},

    booktitle = {Proceedings of the 2022 {CEAS EuroGNC} conference},

    address = {Berlin, Germany},

    month = may,

    year = {2022},

    note = {CEAS-GNC-2022-076}

}