Master Thesis: Building an Uncertainty-Robust Reinforcement Learning-based model for UAV self-separation under Uncertainty

The autonomous operation of unmanned aerial vehicles (UAVs) plays an increasingly important role in research and commercial applications. These vehicles can assist with crucial applications, such as emergency response, infrastructure monitoring, and parcel delivery, but are expected to lead to traffic densities too great for human air traffic controllers to handle. Work its ongoing to develop autonomous separation management systems, from planning and trajectory generation to conflict detection and resolution. For conflict detection and resolution (CD&R), Reinforcement Learning (RL) shows great promise, outperforming state-of-the art geometric methods in safety and efficiency under certain conditions. These methods can be shown to be robust to position noise, and especially perform better at high traffic densities. However, most work considers a homogeneous policy: that is, all vehicles employ the same self-separation strategy, which is also the basis for the strong performance shown by the RL models. In realistic operations, low-level airspace is heterogeneous, and will include vehicles such as trauma response helicopters. These trauma helicopters showcase different dynamics as they travel through the airspace faster than a typical drone, and are given priority over drone operations, meaning that they themselves may not take any conflict resolution manoeuvres. As this is a largely unexplored topic, several research questions can be derived from this, namely:

 • How do Learning-based autonomous CD&R methods perform in heterogeneous environments, with unresponsive priority vehicles such as trauma helicopters?

• How can the training regimes of the models take priority vehicles into account while guaranteeing safety?

The thesis will be expected to answer these questions.

The internship is in collaboration with the TU Delft

The assignment will include the following tasks:

• Investigation of existing approaches for (RL-based) CD&R, including under uncertainty (Literature Study);

• Design of representative heterogeneous scenarios for evaluation and training;

• Model selection, tuning or development, based on simulation results (with algorithms such as SAC from stable-baselines3 or other);

• A design benchmark for the analysis of system safety and robustness under heterogeneous and homogeneous scenarios.

The final outcome of this assignment will be:

• Research into a priority-aware RL-based UAV conflict resolution model;

• A technical thesis report describing the approach, results and conclusions of the work;

• Optional: a conference paper.

6 months.

•         Master student aerospace engineering, mechanical engineering, control engineering or computer science;

•         Experience with programming (Python, Matlab);

•         Experience with practical application of ML/RL (PyTorch, Keras, Tensorflow or other);

•         Preferably good understanding of (aircraft) dynamics, simulation & control.

For more than 100 years, Royal NLR has been the ambitious knowledge organization with the will to keep innovating. From that motivation, we make the world of transportation safer, more sustainable, more efficient and more effective. We are on the threshold of ground-breaking innovations. Plans and ideas get moving when they are well fed with the right energy. Over 1000 passionate professionals work on research and innovation. From aircraft engineers to psychologists and from mathematicians to application experts.

Our colleagues would love to tell you what it’s like to work at NLR.

You will be working within the Air Traffic Management & Airport department. Your colleagues are focused on solving real-world problems within air traffic management, airspace design, U-Space and other exciting domains.

For more information about the assignment contact Sasha Vlaskin sasha.vlaskin@nlr.nl. In addition to our website, visit our NLRmedia channel on YouTube where you can get a good idea of the organization. from you!

Great! We are looking forward to hearing from you, you can apply via the “apply” button.