Subject

Deep unfolding unrolls an optimization algorithm and maps the (sub)steps to corresponding neural network layers, to obtain a machine learning model that incorporates the domain knowledge from the original algorithm into its architecture. This approach results in very compact and efficient models, with many use cases in signal and image processing.

Kind of work

The objective of this project is to test the performance of our deep unfolding Transformer architecture on the task of video super-resolution, and study the benefit of using an extra high-resolution modality to aid in the super-resolution. The specific research activities will be the following:
• Get acquainted with the relevant deep unfolding concepts and architectures.
• Review the state-of-the-art in (multimodal) video super-resolution.
• Select a suitable multimodal video dataset to be used to training and evaluation.
• Adapt our deep unfolding Transformer implementation for video compressed sensing to video super-resolution, and incorporate the use of side information from another modality into the architecture.
• Evaluate the quality of the model, with or without using side information, leveraging metrics like peak signal to noise ratio and structural similarity index.
• Compare the deep unfolding Transformer model to the state-of-the-art (multimodal) video super-resolution models and discuss the results.

Framework of the Thesis

References and further reading
B. De Weerdt, Y. C. Eldar, and N. Deligiannis, “Designing Transformer networks for sparse recovery of sequential data using deep unfolding,” in 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
I. Marivani, E. Tsiligianni, B. Cornelis, and N. Deligiannis, “Multimodal Image Super-resolution via Deep Unfolding with Side Information,” in 2019 27th European Signal Processing Conference (EUSIPCO).
V. Monga, Y. Li, and Y. C. Eldar, “Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing,” IEEE Signal Processing Magazine, vol. 38, no. 2, pp. 18–44, Mar. 2021.

Number of Students

1

Expected Student Profile

Good knowledge of Machine Learning, AI and data processing. Good programming skills in Python (particularly PyTorch)