VLC3 compilation error: linking issue with the libavutil library

Question

I tried to compile VLC3 in ubuntu15.10 a long time, but still no luck. I configured FFmpeg as follow:

~/ffmpeg-3.3.3$ ./configure --cpu=native --enable-pthreads --enable-bzlib --enable-libdc1394 --enable-libfreetype --enable-frei0r --enable-gnutls --enable-libgsm --enable-libmp3lame --enable-librtmp --enable-libopencv --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-vaapi --enable-vdpau --enable-libvorbis --enable-libvpx --enable-zlib --enable-libfdk-aac --enable-nonfree --enable-gpl --enable-swscale --enable-libcdio --enable-libxcb --enable-libx264 --enable-libxvid --enable-libopencore-amrnb --enable-version3 --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-nvenc --enable-openssl --enable-decklink --extra-cflags=- I/home/ivan/Downloads/Blackmagic_DeckLink_SDK_10.9.5/Linux/include --extra-ldflags=-ldl --extra-libs=-ldl --enable-pic

And configured VLC3 wth the following parameters:

~/vlc-3.0.0-git$ ./configure --with-decklink-sdk= I/home/ivan/Downloads/Blackmagic_DeckLink_SDK_10.9.5/Linux/include --disable-chromecas

The compilation looks fine. Then I tried:

$ sudo make install

And got the following error:

  CCLD     libavio_plugin.la
/usr/local/lib/libavutil.a(hwcontext_vaapi.o): In function `vaapi_device_free':
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vaapi.c:899: undefined reference to `XCloseDisplay'
/usr/local/lib/libavutil.a(hwcontext_vaapi.o): In function `vaapi_device_create':
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vaapi.c:929: undefined reference to `XOpenDisplay'
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vaapi.c:934: undefined reference to `vaGetDisplay'
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vaapi.c:941: undefined reference to `XDisplayName'
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vaapi.c:931: undefined reference to `XDisplayName'
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vaapi.c:958: undefined reference to `vaGetDisplayDRM'
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vaapi.c:929: undefined reference to `XOpenDisplay'
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vaapi.c:931: undefined reference to `XDisplayName'
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vaapi.c:936: undefined reference to `XDisplayName'
/usr/local/lib/libavutil.a(hwcontext_vdpau.o): In function `vdpau_device_create':
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vdpau.c:431: undefined reference to `XOpenDisplay'
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vdpau.c:437: undefined reference to `XDisplayString'
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vdpau.c:439: undefined reference to `XDefaultScreen'
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vdpau.c:439: undefined reference to `vdp_device_create_x11'
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vdpau.c:433: undefined reference to `XDisplayName'
/usr/local/lib/libavutil.a(hwcontext_vdpau.o): In function `vdpau_device_free':
/home/ivan/ffmpeg-3.3.3/libavutil/hwcontext_vdpau.c:410: undefined reference to `XCloseDisplay'
collect2: error: ld returned 1 exit status
Makefile:12577: recipe for target 'libavio_plugin.la' failed
make[4]: *** [libavio_plugin.la] Error 1
make[4]: Leaving directory '/home/ivan/vlc-3.0.0-git/modules'
Makefile:25280: recipe for target 'install-recursive' failed
make[3]: *** [install-recursive] Error 1
make[3]: Leaving directory '/home/ivan/vlc-3.0.0-git/modules'
Makefile:25679: recipe for target 'install' failed
make[2]: *** [install] Error 2
make[2]: Leaving directory '/home/ivan/vlc-3.0.0-git/modules'
Makefile:1519: recipe for target 'install-recursive' failed
make[1]: *** [install-recursive] Error 1
make[1]: Leaving directory '/home/ivan/vlc-3.0.0-git'
Makefile:1985: recipe for target 'install' failed

I already saw the following questions related this situation: question1 , question2 , question3

As I understand from these questions, I should move to the end the following libraries in the compile command:

-lvdpau -lX11 -lva*

But I still didn't understand where is the place to moving these libraries.
Please see the output results of the pkg-config command:

$ pkg-config --libs libavcodec libavformat libswscale libavutil

-L/usr/local/lib -lavformat -lXv -lX11 -lXext -lstdc++ -lvdpau -lva -lva-x11 -lX11 -lva -lva-drm -lva -lxcb -lxcb-shm -lxcb -lxcb-xfixes -lxcb-render -lxcb-shape -lxcb -lxcb-shape -lxcb -lcdio_paranoia -lcdio_cdda -lcdio -ljack -lasound -ldc1394 -lssl -lcrypto -lxvidcore -lx264 -lvpx -lm -lvpx -lm -lvpx -lm -lvpx -lm -lvorbisenc -lvorbis -logg -lvo-amrwbenc -ltheoraenc -ltheoradec -logg -lspeex -lschroedinger-1.0 -lrtmp -lz -lgmp -lpulse -lopus -lopenjpeg -DOPJ_STATIC /usr/lib/x86_64-linux-gnu/libopencv_calib3d.so -lopencv_calib3d /usr/lib/x86_64-linux-gnu/libopencv_contrib.so -lopencv_contrib /usr/lib/x86_64-linux-gnu/libopencv_core.so -lopencv_core /usr/lib/x86_64-linux-gnu/libopencv_features2d.so -lopencv_features2d /usr/lib/x86_64-linux-gnu/libopencv_flann.so -lopencv_flann /usr/lib/x86_64-linux-gnu/libopencv_gpu.so -lopencv_gpu /usr/lib/x86_64-linux-gnu/libopencv_highgui.so -lopencv_highgui /usr/lib/x86_64-linux-gnu/libopencv_imgproc.so -lopencv_imgproc /usr/lib/x86_64-linux-gnu/l ibopencv_legacy.so -lopencv_legacy /usr/lib/x86_64-linux-gnu/libopencv_ml.so -lopencv_ml /usr/lib/x86_64-linux-gnu/libopencv_objdetect.so -lopencv_objdetect /usr/lib/x86_64-linux-gnu/libopencv_ocl.so -lopencv_ocl /usr/lib/x86_64-linux-gnu/libopencv_photo.so -lopencv_photo /usr/lib/x86_64-linux-gnu/libopencv_stitching.so -lopencv_stitching /usr/lib/x86_64-linux-gnu/libopencv_superres.so -lopencv_superres /usr/lib/x86_64-linux-gnu/libopencv_ts.so -lopencv_ts /usr/lib/x86_64-linux-gnu/libopencv_video.so -lopencv_video /usr/lib/x86_64-linux-gnu/libopencv_videostab.so -lopencv_videostab -lopencore-amrwb -lopencore-amrnb -lmp3lame -lgsm -lfreetype -lfdk-aac -lgnutls -lcrystalhd -lm -llzma -lbz2 -lz -pthread -ldl -lavcodec -lXv -lX11 -lXext -lstdc++ -lvdpau -lva -lva-x11 -lX11 -lva -lva-drm -lva -lxcb -lxcb-shm -lxcb -lxcb-xfixes -lxcb-render -lxcb-shape -lxcb -lxcb-shape -lxcb -lcdio_paranoia -lcdio_cdda -lcdio -ljack -lasound -ldc1394 -lssl -lcrypto -lxvidcore -lx264 -lvpx -lm -lvpx -lm -lv px -lm -lvpx -lm -lvorbisenc -lvorbis -logg -lvo-amrwbenc -ltheoraenc -ltheoradec -logg -lspeex -lschroedinger-1.0 -lrtmp -lz -lgmp -lpulse -lopus -lopenjpeg -DOPJ_STATIC /usr/lib/x86_64-linux-gnu/libopencv_calib3d.so -lopencv_calib3d /usr/lib/x86_64-linux-gnu/libopencv_contrib.so -lopencv_contrib /usr/lib/x86_64-linux-gnu/libopencv_core.so -lopencv_core /usr/lib/x86_64-linux-gnu/libopencv_features2d.so -lopencv_features2d /usr/lib/x86_64-linux-gnu/libopencv_flann.so -lopencv_flann /usr/lib/x86_64-linux-gnu/libopencv_gpu.so -lopencv_gpu /usr/lib/x86_64-linux-gnu/libopencv_highgui.so -lopencv_highgui /usr/lib/x86_64-linux-gnu/libopencv_imgproc.so -lopencv_imgproc /usr/lib/x86_64-linux-gnu/libopencv_legacy.so -lopencv_legacy /usr/lib/x86_64-linux-gnu/libopencv_ml.so -lopencv_ml /usr/lib/x86_64-linux-gnu/libopencv_objdetect.so -lopencv_objdetect /usr/lib/x86_64-linux-gnu/libopencv_ocl.so -lopencv_ocl /usr/lib/x86_64-linux-gnu/libopencv_photo.so -lopencv_photo /usr/lib/x86_64-linux-gnu/lib opencv_stitching.so -lopencv_stitching /usr/lib/x86_64-linux-gnu/libopencv_superres.so -lopencv_superres /usr/lib/x86_64-linux-gnu/libopencv_ts.so -lopencv_ts /usr/lib/x86_64-linux-gnu/libopencv_video.so -lopencv_video /usr/lib/x86_64-linux-gnu/libopencv_videostab.so -lopencv_videostab -lopencore-amrwb -lopencore-amrnb -lmp3lame -lgsm -lfreetype -lfdk-aac -lgnutls -lcrystalhd -lm -llzma -lbz2 -lz -pthread -ldl -lswresample -lm -lswscale -lm -lavutil -lm

And the results only for libavutil library:

$ pkg-config --libs libavutil

-L/usr/local/lib -lavutil -lm

Please also see my VLC Makefile

Can you please to point at the place where these libraries should be reordered? Also my pkg-config output results are different from the results from the questions above. Any advice or links to solve this question will be helpful.

Answer 1

So I tested it on Ubuntu 16.04 LTS using Vagrant because 15.10 default packages repository was not available

vagrant init bento/ubuntu-16.04
vagrant up && vagrant ssh

After extracting the sources I installed the below required packages

sudo apt install -y yasm frei0r-plugins-dev libghc-gnutls-dev libfdk-aac-dev libfreetype6-dev libgsm1-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libopencv-dev libopenjpeg-dev autoconf automake build-essential libass-dev libfreetype6-dev libsdl2-dev libtheora-dev libtool libva-dev libvdpau-dev libvorbis-dev libxcb1-dev libxcb-shm0-dev libxcb-xfixes0-dev pkg-config texinfo wget zlib1g-dev libx264-dev libfdk-aac-dev libopus-dev libvpx-dev librtmp-dev libschroedinger-dev speex libspeex-dev libvo-amrwbenc-dev libxvidcore-dev libcdio-paranoia-dev libbz2-dev lua5.2 liblua5.2-dev liba52-0.7.4-dev libxcb-composite0-dev libxcb-keysyms1 libxcb-xv0-dev libxcb-xvmc0-dev

Then configure ffmpeg using below

./configure --cpu=native --enable-pthreads --enable-bzlib --enable-libdc1394 --enable-libfreetype --enable-frei0r --enable-gnutls --enable-libgsm --enable-libmp3lame --enable-librtmp --enable-libopencv --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-vaapi --enable-vdpau --enable-libvorbis --enable-libvpx --enable-zlib --enable-libfdk-aac --enable-nonfree --enable-gpl --enable-swscale --enable-libcdio --enable-libxcb --enable-libx264 --enable-libxvid --enable-libopencore-amrnb --enable-version3 --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-nvenc --enable-openssl --enable-decklink --extra-cflags=-I/vagrant/decklink/blackmagic/Linux/include --extra-ldflags=-ldl --extra-libs=-ldl --enable-pic --enable-shared

compiled using make -j4 && make install

The configure VLC using below

./configure --with-decklink-sdk=I/vagrant/decklink/blackmagic/Linux/include --disable-chromecast --enable-shared --disable-wayland

Then build using make -j4 && make install