Super Resolution Video Enhancing with AMD GPU
By Timo Jyrinki
I’ve had a somewhat recent AMD Radeon RX 560 graphics card in my Mini-ITX PC for over a year already, and one long term interest I have would be to be able to enhance old videos. Thanks to Hackweek I could look at this as one of the things that I’ve waited to have time for. In recent years there have been approaches to use eg neural networks to do super resolution handling of photos and also videos, so that there would be more actual details and shapes than what would be possible via normal image manipulation. The only feasible way of doing those is by using GPUs, but unfortunately the way those are utilized is a bit of a mess, most of all because proprietary one vendor only CUDA is the most used one.
On the open source side, there is OpenCL in Mesa by default and Vulkan Compute, and there’s AMD’s separate open source ROCm that offers a very big platform for computing but also among else OpenCL 2.x support and a source level CUDA to portable code (including AMD support) translator called HIP. I won’t go into HIP, but I’m happy there’s at least the idea of portable code being thrown around. OpenCL standardization has been failing a bit, with the newest 3.0 trying to fix why the industry didn’t adopt 2.x properly and OpenCL 1.2 being the actual (but a bit lacking) baseline. I think the same happened a bit with OpenGL 1.x -> 2.x -> 3.x a long time ago by the way… Regardless if the portable code is OpenCL or Vulkan Compute, the open standards are now making good progress forward.
I first looked at ROCm’s SLE15SP2 installation guide - interestingly, it also installed on Tumbleweed if ignoring one dependency problem of openmp-extras. However, on my Tumbleweed machine I do not have Radeon so this was just install test. I was however surprised that even the kernel module compiled against TW’s 5.11 kernel - and if it would have not, there’s the possibility of using upstream kernel’s module instead.
Over at the Radeon machine side, I installed ROCm without problems. Regardless of that SDK, many real world applications for video enhancing are still CUDA specific so would at minimum require porting to portable code with HIP. These include projects like DAIN, DeOldify etc. However, there are also projects using OpenCL and Vulkan Compute - I’ll take video2x as an example here that supports multiple backends, some of which support OpenCL (like waifu2x-converter-cpp) or Vulkan (like realsr-ncnn-vulkan). I gave the latter a try on my Radeon as it won the CVPR NTIRE 2020 challenge on super-resolution.
I searched my video collection far enough back in time so that I could find some 720p videos, and selected one which has a crowd and details in distance. I selected 4x scaling even though that was probably a time costly mistake as even downscaling that 5120x2880 video probably wouldn’t be any better than 2x scaling to 2560x1440 and eg downscaling that to 1920x1080. I set up the realsr-ncnn-vulkan and ffmpeg paths to my compiled versions, gave the 720p video as input and fired way!
And waited. For 3.5 hours :) Noting that I should probably buy an actual highend AMD GPU, but OTOH my PC case is smallish and on the other hand the market is unbearable due to crypto mining. The end result was fairly good, even if not mind blowing. There are certainly features it handles very well, and it seems to clean up noise a bit as well while improving contrast a bit. It was clearly something that would not be possible by traditional image manipulation alone. Especially the moving video seems like higher resolution in general. Maybe the samples I’ve seen from some other projects out there have been better, then OTOH my video was a bit noisy with movement blur included so not as high quality as would be eg a static photo or a video clip tailored to show off an algorithm.
For showing the results, I upscaled a frame from the original 1280x720 clip to the 4x 5120x2880 resolution in GIMP, and then cropped a few pieces of both it and the matching frame in the enhanced clip. I downscaled them actually then to 1440p equivalent crop. Finally, I also upscaled the original frame to FullHD and downscaled the enhanced one to FullHD as well, to be more in line what could be one actual desired realistic end result - 720p to 1080p conversion while increasing detail.
The “1440p equivalent” comparison shots standard upscale vs realsr-ncnn-vulkan.
The FullHD scaling comparison - open both as full screen in new window and compare.
