VMAF in FFmpeg – Installation and Usage Guide for Ubuntu

In this tutorial, we will see how to compile and install FFmpeg with VMAF support using libvmaf and then we shall see how to compute or calculate the VMAF score between two videos using FFmpeg with VMAF support enabled.


FFmpeg has always been at the top of everyone’s mind when it came to open source video compression software and is best described by Wikipedia as

FFmpeg is a free and open-source project consisting of a vast software suite of libraries and programs for handling video, audio, and other multimedia files and streams.

VMAF from Netflix stands for Video Multi-method Assessment Fusion, and it is a video quality metric that combines human vision modeling with machine learning. It’s become very popular as it succeeds (not fully) at automating subjective quality testing that usually requires humans to watch and score videos.

FFmpeg and Netflix’s VMAF are now part of every video processing and compression engineer’s toolbox. But, installing and using them appears to be difficult for a large number of users.

I see these questions often on Stackoverflow and similar websites –

  • how to compute VMAF?
  • how do I compile VMAF on Ubuntu?
  • how do I calcuate VMAF using FFmpeg?
  • no such filter libvmaf when trying to use VMAF with FFmpeg.
  • VMAF example commandline

and the list simply goes on. So, I decided to write down the steps of compiling and installing VMAF, FFmpeg, and computing VMAF with FFmpeg. Along the way, I’ll show you a few gotchas and common errors in the installation and usage procedure.

Step 0: Machine Setup

For the purpose of this document, I will set up a new $5 droplet on DigitalOcean with Ubuntu 18.04 as its operating system.By the way, if you click here, you’ll receive $100 in DigitalOcean credits over 60 days! That’s 60 days and $100 to experiment and play around with DigitalOcean 🙂

Step 1: Install VMAF on Ubuntu 18.04

You can use VMAF in several different ways and you can read about it here. In order to use VMAF with FFmpeg, we need the help of libvmaf – a C library that can be used with C/C++ code. In other words, we can use it with FFmpeg.

So, here is how to install VMAF on an Ubuntu 18.04 machine –

Step 1: Download the latest release of VMAF from here.

wget https://github.com/Netflix/vmaf/archive/v1.5.2.tar.gz

Step 2: Untar it

untar -xvf https://github.com/Netflix/vmaf/archive/v1.5.2.tar.gz

Step 3: Install dependencies.

sudo apt update
sudo apt install python3 python3-pip python3-setuptools python3-wheel ninja-build doxygen
pip3 install meson
pip3 install Cython
pip3 install numpy

Step 4: Make and Install VMAF

cd vmaf-1.5.2/
sudo make 
sudo make install

Make a note of where VMAF’s libvmaf got installed. In my machine, it was installed to /usr/local/lib/x86_64-linux-gnu as per the console log.

Installing src/libvmaf.so.0.0.0 to /usr/local/lib/x86_64-linux-gnu
Installing src/libvmaf.a to /usr/local/lib/x86_64-linux-gnu

Step 2: Install FFmpeg with VMAF support on Ubuntu 18.04

The first part is complete and you have successfully installed VMAF onto your machine. Now, let’s compile and install FFmpeg with VMAF support. These instructions are taken from FFmpeg’s Ubuntu Compilation Guide

Step 1: Install the Dependencies (including nasm). And, remember to create the following folders

mkdir -p ~/ffmpeg_sources ~/bin

Step 2: Install the Video and Audio Codecs that you are going to need. I installed libx264libx265-devlibnuma-devlibvpx-devlibfdk-aac-devlibmp3lame-dev, and libopus-dev.

Step 3: Make sure FFmpeg knows where to find libvmaf

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/x86_64-linux-gnu/

Step 4: Install libunistring-dev to prevent the following error : ERROR: gnutls not found using pkg-config

sudo apt-get install libunistring-dev

Step 5: Download and compile FFmpeg from sources. Notice that we are now using --enable libvmaf to enable VMAF with FFmpeg.

cd ~/ffmpeg_sources && \
wget -O ffmpeg-snapshot.tar.bz2 https://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2 && \
tar xjvf ffmpeg-snapshot.tar.bz2 && \
cd ffmpeg && \
PATH="$HOME/bin:$PATH" PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure \
  --prefix="$HOME/ffmpeg_build" \
  --pkg-config-flags="--static" \
  --extra-cflags="-I$HOME/ffmpeg_build/include" \
  --extra-ldflags="-L$HOME/ffmpeg_build/lib" \
  --extra-libs="-lpthread -lm" \
  --bindir="$HOME/bin" \
  --enable-gpl \
  --enable-gnutls \
  --enable-libass \
  --enable-libfdk-aac \
  --enable-libfreetype \
  --enable-libmp3lame \
  --enable-libopus \
  --enable-libvorbis \
  --enable-libvpx \
  --enable-libx264 \
  --enable-libx265 \
  --enable-libvmaf \
  --enable-nonfree && \
PATH="$HOME/bin:$PATH" make && \
make install && \
hash -r

That’s it! You should now have an FFmpeg build with VMAF enabled.

Related:  How to Reverse a Video using FFmpeg

Step 3: Computing FFmpeg with VMAF

Now let’s see how we can calculate the VMAF score between two videos using FFmpeg and VMAF.

Step 1: Let’s create two test videos – a 1920x1080p30 video and a 1280x720p30 video using FFmpeg’s built-in libraries that generates test data. The commandlines are shown below and as you can see, it will generate two 5-second duration clips, 30 fps, of 1920x1080p and 1280x720p resolution.

bin/ffmpeg -f lavfi -i testsrc=duration=5:size=1920x1080:rate=30 test_1080p30.mpg
bin/ffmpeg -f lavfi -i testsrc=duration=5:size=1280x720:rate=30 test_720p30.mpg

Step 2: Compute the VMAF score between the two videos.
From the commandline below, you can see that we are providing the two input sequences, and then we are scaling both to 1920×1080 using a bicubic filter. This is important because VMAF works only on sequences having the same resolution (and the frame rate should match, else you’ll have serious problems). We are also asking FFmpeg to compute the PSNR score for these two sequences.

bin/ffmpeg -i test_720p30.mpg -i test_1080p30.mpg -filter_complex "[0:v]scale=1920x1080:flags=bicubic[main]; [1:v]scale=1920x1080:flags=bicubic,format=pix_fmts=yuv420p,fps=fps=30/1[ref]; [main][ref]libvmaf=psnr=true:log_path=vmaflog.json:log_fmt=json" -f null -

and it should print the VMAF score as follows on the console

[libvmaf @ 0x55803f80c7c0] VMAF score: 86.375268

Note:

  1. The detailed per-frame details are contained in the JSON file. It contains the per-frame VMAF and PSNR scores as requested.
  2. In FFmpeg’s filter, you can use either bicubic or lancsoz filters for the upsampling process. Netflix suggests that bicubic is a better choice.

Options for VMAF using FFmpeg

The following commandline options are available in FFmpeg for computing VMAF.

  • model_path Set the model path which is to be used for SVM. Default value: /usr/local/share/model/vmaf_v0.6.1.pkl
  • log_path Set the file path to be used to store logs.
  • log_fmt Set the format of the log file (xml or json).
  • enable_transform This option can enable/disable the score_transform applied to the final predicted VMAF score, if you have specified score_transform option in the input parameter file passed to run_vmaf_training.py Default value: false
  • phone_model Invokes the phone model which will generate VMAF scores higher than in the regular model, which is more suitable for laptop, TV, etc. viewing conditions. Default value: false
  • psnr Enables computing psnr along with vmaf. Default value: false
  • ssim Enables computing ssim along with vmaf. Default value: false
  • ms_ssim Enables computing ms_ssim along with vmaf. Default value: false
  • pool Set the pool method to be used for computing vmaf. Options are minharmonic_mean or mean (default).
  • n_threads Set number of threads to be used when computing vmaf. Default value: , which makes use of all available logical processors.
  • n_subsample Set interval for frame subsampling used when computing vmaf. Default value: 1
  • enable_conf_interval Enables confidence interval. Default value: false

Common problems while using VMAF and FFmpeg.

P1. While installing FFmpeg from the official compilation guide on Ubuntu 18.04, you will face this error ERROR: gnutls not found using pkg-config. To prevent this, you need to install sudo apt-get install libunistring-dev.

P2. Another common problem when running FFmpeg with VMAF is this error libvmaf xxx not found using pkg config. This means that while compiling FFmpeg, you didn’t tell it where the libvmaf files are. Simply run the export command and it should fix the issue.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/x86_64-linux-gnu/

That’s It

I hope you were able to follow this guide and could successfully install VMAF, FFmpeg and compute the VMAF score using FFmpeg. If you faced any issues, let me know via email (check the contact page).

Take care and see you next time!

Check out EasyVMAF – a tool for making VMAF calculations simple.

krishna rao vijayanagar
Krishna Rao Vijayanagar

I’m Dr. Krishna Rao Vijayanagar, and I have worked on Video Compression (AVC, HEVC, MultiView Plus Depth), ABR streaming, and Video Analytics (QoE, Content & Audience, and Ad) for several years.

I hope to use my experience and love for video streaming to bring you information and insights into the OTT universe.

5 Comments

  1. Hi, Can I calculate VMAF score of a video which is not in local of my pc? For example a video which is streaming through HDMI cable or USB cable,? Assume I have the reference video for the same. And also it is mentioned in the post that vmaf works for same resolution videos, for my problem statement reference video should be 1080 pixel resolution and testing video of 720 pixel for this case can I use vmaf ?

    • I have not tried computing the VMAF of a streaming video. A couple of points though – the two videos have to be of the same resolution which is a condition for VMAF. Actually if you look at the example above, one of the videos is 720p and the other is 1080p. However, both of them are beign scaled to 1080p by FFmpeg before comparing them and computing the VMAF.

      VMAF is a slow operation because of its complexity. I am not sure it will actually work for live streaming. You can try capturing a few mins of a live stream and computing the VMAF on it though.

  2. This tutorial is pretty good but has a few errors.

    Problems in Step 1, building VMAF:

    1. You said one should download the latest copy of vmaf, but you pointed to a static location. To truly get the latest version, in Step 1-1 one should use “wget -O vmaf-master.zip https://github.com/Netflix/vmaf/archive/refs/heads/master.zip“. In step 1-2 use “unzip -q vmaf-master.zip” to unpack it, then in step 1-4 “cd vmaf-master” to move into the directory and build it.

    2. The latest version of “vmaf” needs an x86 assembler in order to build, so one needs to add “yasm” to the dependency list in step 1-3.

    3. The three “pip install” commands *must* be prefixed with “sudo.” If you don’t do that, your either won’t be able to build VMAF or you won’t be able to install it.

    In step 1-4, it is not necessary to do the initial “make” command using “sudo,” and doing so is considered bad practice. Simply doing “make” will work. However, it’s necessary to do “sudo make install” to get a successful install.

    I was able to get ffmpeg to build using VMAF 2.1.1, which is the latest version as of July 2021.

    Problems in Step 2, building ffmpeg:

    1. You mentioned your build instructions are “based on FFmpeg’s Ubuntu Compilation Guide” in one paragraph, then in step 2-1 said simply “Install the dependencies.” But you didn’t make it clear that meant the dependencies listed in the Compilation Guide, as opposed to the dependencies listed in step 1-3 (which are only for VMAF.)

    2. Actually, step 2-1 is *two* steps. I glossed over the “install dependencies” part and went straight to the “mkdir” part, because by including that text in step 2-1 I concluded that making the directories was the sole intent of that step.

    2. In the list of dependencies in step 2-4, you’re missing the “-dev” on the very first entry; it should be “libx264-dev” and not just “libx264.”

    3. (Minor) You don’t need the “-p” parameter on “mkdir -p ~/ffmpeg_sources ~/bin”. “-p” is used to make any necessary paths leading up to the directory being created. The only path matching that description is “~” (the user’s home directory), so it’s already present.

  3. I followed the instructions precisely, but at step 5, after entering “sudo make” I seem to be getting an error which says that meson is not found:

    cd third_party/libsvm && make lib
    make[1]: Entering directory ‘/home/ubuntu/vmaf-2.2.0/third_party/libsvm’
    if [ “Linux” = “Darwin” ]; then \
    SHARED_LIB_FLAG=”-dynamiclib -Wl,-install_name,libsvm.so.2″; \
    else \
    SHARED_LIB_FLAG=”-shared -Wl,-soname,libsvm.so.2″; \
    fi; \
    g++ ${SHARED_LIB_FLAG} svm.o -o libsvm.so.2
    make[1]: Leaving directory ‘/home/ubuntu/vmaf-2.2.0/third_party/libsvm’
    meson setup libvmaf/build libvmaf –buildtype release -Denable_float=true && \
    ninja -vC libvmaf/build
    /bin/sh: 1: meson: not found
    Makefile:2: recipe for target ‘all’ failed

    Any insights about how I can try fixing this problem?

    • Hi – thanks for the heads-up. This article needs to be refreshed for the latest version of VMAF. However, there is a very helpful comment right above this. Do give those a shot and let me know if they work. Thanks!

2 Trackbacks / Pingbacks

  1. Calculate PSNR, VMAF, SSIM using FFmpeg - OTTVerse
  2. VMAF, PSNR, SSIM calculation using FFmpeg - GistTree

Leave a Reply

Your email address will not be published.


*