Subject

In recent years, the increase of available training data and computing resources provided an outstanding improvement in performances of Deep Learning (DL) models. Nearly every computer vision task is now based on Convolutional Neural Networks (CNNs), able to solve complex problems such as image classification and human pose estimation (HPE).

The integration of DL in Augmented Reality (AR) systems, which overlay 3D virtual objects (holograms) onto the real world, would be beneficial for a wide range of applications. AR head mounted displays, or AR glasses, are currently being explored in the medical field, as they leave the hands of the clinician free while providing a 3D visualization of anatomical structures. CNNs can be used to elaborate and interpret what the clinicians see to provide visual feedback.

This integration is usually performed by running the CNNs on an external server, which receives the input data from the AR glasses (e.g. HoloLens 2) and sends the result back via Wi-Fi. This approach has several drawbacks, as the Wi-Fi introduces latency and is unstable. Unfortunately, running the CNNs directly on the device is not always possible due to the limited computing resources.

Thus far, inference of machine learning models directly on HoloLens has been implemented leveraging Windows Machine Learning (winML), but a recent work demonstrated that using a different approach (Barracuda in Unity) can drastically reduce the inference time. Both the approaches require as input the ONNX format of the model, which size (MB) and floating-point operations per second (FLOPS) can be optimized to achieve real time performances.

Kind of work

The goal of this project is to investigate which DL models can run on HoloLens 2 and, more importantly, the most efficient way to export and run them. A set of evaluation criteria, such as ease of implementation or performance, will be used to compare the different approaches. In order to quantitatively and qualitatively evaluate them, a Proof of Concept (PoC) application will be developed.

Framework of the Thesis

The project will consist of:
• Literature study.
• Implementation and extension of literature-based DL optimization techniques (post-training quantization, transfer learning, knowledge distillation etc…).
• Investigation of different approaches to inference DL model directly on HoloLens 2 (winML, Barracuda…).
• Implementation of a PoC application to test the different approaches.
• Development of a workflow allowing for quantitative and qualitative evaluation of performances.

The CNNs architectures will be optimized and converted in ONNX using common DL frameworks such as PyTorch and Tensorflow. Then, a Universal Windows Platform (UWP) application will be developed (C#/C++) and deployed on HoloLens 2. Finally, the analysis of the different approaches’ performances will be performed in Python.

The work will be done at VUB-ETRO, Pleinlaan 9.

Number of Students

1