Issue when compiling liquidsoap 2.3.0

[etherwvlf]

I am trying to obtain liquidsoap 2.3.0 using opam.
I do so successfully but there is something I cannot seem to fix.

I am using this string to create my binary:
opam install liquidsoap posix-time2 xmlplaylist tls ssl logs tls-liquidsoap shine speex theora fdkaac alsa pulseaudio inotify mad lame cry taglib samplerate faad opus vorbis flac ffmpeg

It gets created no problem but whenever I verify it with:
./liquidsoap --list-plugins

I am met with only this under "protocols":

# protocols

Methods to get a file. They are the first part of URIs: 'protocol:args'.

- annotate: Add metadata to a request
- mpd: Finds all files with a tag equal to a given value using mpd.

When I query my old version I get:

# protocols

Methods to get a file. They are the first part of URIs: 'protocol:args'.

- file: File protocol. Only local files are supported
- synth: Syntesize audio. Parameters are optional.
- polly: Generate speech synthesis using AWS polly service. Result might be mono, needs aws binary in the path.
- s3: Fetch files from s3 using the AWS CLI
- say: Generate speech synthesis using text2wave. Result is always stereo.
- gtts: Generate speech synthesis using Google translate's text-to-speech API. This requires the `gtts-cli` binary.  Result may be mono.
- pico2wave: Generate speech synthesis using pico2wave. Result may be mono.
- text2wave: Generate speech synthesis using text2wave. Result may be mono.
- stereo: Convert a file to stereo (currently decodes to wav).
- ffmpeg: Decode any file to wave using ffmpeg
- tmp: Mark the given uri as temporary. Useful when chaining protocols
- youtube-pl: Resolve a request as a youtube playlist using youtube-dl. You typically want to use this as `playlist("youtube-pl:...")`.
- youtube-dl: Resolve a request using youtube-dl.
- https: Download http URLs using curl
- http: Download http URLs using curl
- process: Resolve a request using an arbitrary process. `<cmd>` is interpolated with: `[("input",<input>),("output",<output>),("colon",":")]`. `uri` is an optional child request, `<output>` is the name of a fresh temporary file and has extension `.<extname>`. `<input>` is an optional input file name as returned while resolving `uri`.
- replaygain: Compute ReplayGain value. Adds returned value as `"replaygain_track_gain"` metadata
- autocue: Adding automatically computed cues/crossfade metadata
- annotate: Add metadata to a request
- mpd: Finds all files with a tag equal to a given value using mpd.

I am missing:

- file: File protocol. Only local files are supported
- synth: Syntesize audio. Parameters are optional.
- say: Generate speech synthesis using text2wave. Result is always stereo.
- gtts: Generate speech synthesis using Google translate's text-to-speech API. This requires the `gtts-cli` binary.  Result may be mono.
- pico2wave: Generate speech synthesis using pico2wave. Result may be mono.
- text2wave: Generate speech synthesis using text2wave. Result may be mono. 
- ffmpeg: Decode any file to wave using ffmpeg

The others I don't care about. So my binary does not support local files and playlists in it's current condition.
My question is what shall I add to my opam install string to obtain this missing functionality ?
Especially the file protocol.
Thank you.

[vitoyucepi]

Hi @etherwvlf,
Can you show the output of the liquidsoap --build-config command?
By the way, all of the protocols are from the liquidsoap-libs opam package. They are added at the runtime from the standard library. https://github.com/savonet/liquidsoap/blob/3e6205ed220dc3a78db8033a63f8f86674b0479d/src/libs/protocols.liq

[vitoyucepi]

The overall script looks good, but I'd tweak it a bit.

sudo apt-get update;
sudo apt-get install --no-install-recommends -y \
  gcc \
  g++ \
  make \
  patch \
  unzip \
  bzip2 \
  opam \
  git \
;

export OPAMJOBS="$(nproc)";
opam init --disable-sandboxing --bare --auto-setup;
opam switch create 4.14.2;
eval $(opam env);

#export IS_SNAPSHOT=false;
#export LIQUIDSOAP_BUILD_TARGET=standalone; # default,standalone,posix
opam install --confirm-level=unsafe-yes \
  liquidsoap \
  tls-liquidsoap \
  ssl \
  posix-time2 \
  xmlplaylist \
  shine \
  speex \
  theora \
  fdkaac \
  alsa \
  pulseaudio \
  inotify \
  mad \
  lame \
  taglib \
  samplerate \
  faad \
  opus \
  vorbis \
  flac \
  ffmpeg \
;

[etherwvlf]

I tried your script, the resulting file is even a few MB smaller :)
Thank you for your suggestions.

I wanted to avoid hardcoded versions in case something changes in the future.
Shall I change this: opam switch create 4.14.2;
with: opam switch create ocaml-base-compiler;
in order to avoid using hardcoded version ?

#export LIQUIDSOAP_BUILD_TARGET=standalone; # default,standalone,posix

Could you clarify that part further ? What is the difference between default, standalone and posix ?

[etherwvlf]

I do have some further observations.
When I do ./liquidsoap --list-plugins on my build machine it correctly lists protocols:

# protocols

Methods to get a file. They are the first part of URIs: 'protocol:args'.

- file: File protocol. Only local files are supported
- synth: Syntesize audio. Parameters are optional.
- polly: Generate speech synthesis using AWS polly service. Result might be mono, needs aws binary in the path.
- s3: Fetch files from s3 using the AWS CLI
- say: Generate speech synthesis using text2wave. Result is always stereo.
- gtts: Generate speech synthesis using Google translate's text-to-speech API. This requires the `gtts-cli` binary.  Result may be mono.
- pico2wave: Generate speech synthesis using pico2wave. Result may be mono.
- text2wave: Generate speech synthesis using text2wave. Result may be mono.
- stereo: Convert a file to stereo (currently decodes to wav).
- ffmpeg: Decode any file to wave using ffmpeg
- tmp: Mark the given uri as temporary. Useful when chaining protocols
- youtube-pl: Resolve a request as a youtube playlist using youtube-dl. You typically want to use this as `playlist("youtube-pl:...")`.
- youtube-dl: Resolve a request using youtube-dl.
- https: Download http URLs using curl
- http: Download http URLs using curl
- process: Resolve a request using an arbitrary process. `<cmd>` is interpolated with: `[("input",<input>),("output",<output>),("colon",":")]`. `uri` is an optional child request, `<output>` is the name of a fresh temporary file and has extension `.<extname>`. `<input>` is an optional input file name as returned while resolving `uri`.
- replaygain: Compute ReplayGain value. Adds returned value as `"replaygain_track_gain"` metadata
- autocue: Adding automatically computed cues/crossfade metadata
- annotate: Add metadata to a request
- mpd: Finds all files with a tag equal to a given value using mpd.

But when I transfer the binary to a separate debian testing machine and I install some prerequisites in order for it to run correctly, eg:
libssl-dev, libpulse-dev, libvorbis-dev, libopus-dev, libmad0-dev, libmp3lame-dev, libfaad-dev, libflac-dev, libfdk-aac-dev
when I do ./liquidsoap --list-plugins I get:

# Available decoders

Available decoders.

- mad: Use libmad to decode any file if its MIME type or file extension is appropriate.
- ogg: Decode a file as OGG provided that libogg accepts it.
- flac: Use libflac to decode any file or stream if its MIME type or file extension is appropriate.
- ffmpeg: Use FFmpeg to decode any file or stream if its MIME type or file extension is appropriate.
- mp4: Use libfaad to decode MP4 if MIME type or file extension is appropriate.
- aac: Use libfaad to decode AAC if MIME type or file extension is appropriate.
- image: Decoder for static images.
- midi: Decode midi files.
- raw audio: Decode audio/x-raw.
- srt: Decode srt files.
- pcm/basic: Decode audio/basic as headerless stereo U8 PCM at 8kHz.
- aiff: Decode as AIFF any file with a correct header.
- wav: Decode file or streams as WAV.

# audio file formats (duration)

Methods to extract duration from a file.

- vorbis: 
- mad: Compute duration of mp3 files using MAD library.
- ogg/flac: 
- flac: Compute duration of flac files.
- ffmpeg: 
- wav/aiff: Native computation of wav and aiff files duration.

# channel layout converters

Methods for converting channel layouts.

- native: Native channel layout converter.

# image file decoding

Image file decoding methods.

- ffmpeg: Decode images using Ffmpeg.
- ppm: Native decoding of PPM images.

# metadata formats

Methods to extract metadata from a file.

- ogg: 
- flac: 
- ffmpeg: 
- mp4: MP4 tag decoder.
- flac_native: Native FLAC metadata resolver.
- ID3v2: Native decode for ID3v2 tags.
- ID3v1: Native decoder for ID3v1 tags.
- ID3: Native decoder for ID3 tags.
- image: Native decoder for image metadata.
- ogg_native: Native ogg metadata resolver.
- video-metadata: Native metadata decoder for videos.

# playlist formats

Method to parse playlist.


# protocols

Methods to get a file. They are the first part of URIs: 'protocol:args'.

- annotate: Add metadata to a request
- mpd: Finds all files with a tag equal to a given value using mpd.

# samplerate converters

Methods for converting samplerate.

- libsamplerate: High-quality samplerate conversion using libsamblerate.
- ffmpeg: 
- native: Native samplerate converter. This is fast but bad quality: you should avoid using it for now if you are serious about sound.

# stream encoding formats

Methods to encode streams.

- shine: SHINE fixed-point mp3 encoder.
- lame: LAME mp3 encoder.
- ogg: ogg encoder.
- flac: Flac encoder.
- ffmpeg: 
- fdkaac: 
- external: Encode using external programs.
- avi: Native avi encoder.
- ndi: NDI encoder. Only used with `output.ndi`.
- wav: Native wav encoder.

# video converters

Methods for converting video frames.

- ffmpeg: FFmpeg image format converter.
- native: Native image format converter.

It runs correctly and only the protocols list comes lacking, so I am beginning to think that I have to install something more than libssl-dev, libpulse-dev, libvorbis-dev, libopus-dev, libmad0-dev, libmp3lame-dev, libfaad-dev, libflac-dev, libfdk-aac-dev to make it work.

[vitoyucepi]

What is the difference between default, standalone and posix ?

Here's the section of the manual about build time environment variables.
https://www.liquidsoap.info/doc-2.3.0/build.html#environment-variables

I wanted to avoid hardcoded versions in case something changes in the future.
opam switch create 4.14.2;

Currently, the recommended version of ocaml is 4.14.

But when I transfer the binary to a separate debian testing machine

You should transfer not only the binary, but also the standard library and some opam dependencies.

libssl-dev, libpulse-dev, libvorbis-dev, libopus-dev, libmad0-dev, libmp3lame-dev, libfaad-dev, libflac-dev, libfdk-aac-dev

For the production environment, you don't need the -dev packages, normal packages that provide .so files would be sufficient.

By the way, here's a guide on how to build liquidsoap #3609.

[etherwvlf]

Oh, thank you. That should help a lot.
I'll make some tries and see if we could get this closed :D

[etherwvlf]

Thanks mister !
Your advice worked beautifully.

[gAlleb]

shine speex theora fdkaac alsa pulseaudio mad lame faad opus vorbis flac ffmpeg

you really need all of them?

[etherwvlf]

shine speex theora fdkaac alsa pulseaudio mad lame faad opus vorbis flac ffmpeg

you really need all of them?

Not exactly, i am trying to get the feel of the process. Definitely could get rid of shine speex theora at least.

[toots]

./liquidsoap --list-plugins

You should try to disable paging:

% PAGER=none  ./liquidsoap --list-plugins

[vitoyucepi]

Hi @toots,
Which protocols are implemented in the .ml files?

[toots]

Only annotate, everything else is scripted.

[vitoyucepi]

https://github.com/savonet/liquidsoap/tree/c0cd13a1f58164d2cab8cb322b90a24064beda17/src/core/protocols
mpd and annotate, so I'm trying to determine if the liquidsoap-libs package is present.

[etherwvlf]

./liquidsoap --list-plugins

You should try to disable paging:

% PAGER=none  ./liquidsoap --list-plugins

Thanks, certainly eases things a lot :)

[etherwvlf]

https://github.com/savonet/liquidsoap/tree/c0cd13a1f58164d2cab8cb322b90a24064beda17/src/core/protocols mpd and annotate, so I'm trying to determine if the liquidsoap-libs package is present.

You may be on to something here. I transfer the resulting binary to the test machine, install libssl-dev, libpulse-dev, libvorbis-dev, libopus-dev, libmad0-dev, libmp3lame-dev, libfaad-dev, libflac-dev, libfdk-aac-dev and it has everything needed to run but lacks the protocols. The list shows only mpd and annotate which as I understand it, come from the core functionality. The build machine obviously has this liquidsoap-libs package but the test rig doesn't hence the difference in the protocols list.
How can I obtain this liquidsoap-libs preferably as a deb-package to install on the test machine and see if these missing protocols magically show up ?
Or even better yet, could I build it into the end binary to simplify the process even more ?