Beam Languages (Erlang, Elixir & LFE)Introduction
In this document and related Nix expressions we use the term
Beam to describe the environment. BEAM is
the name of the Erlang Virtial Machine and, as far as we know,
from a packaging perspective all languages that run on BEAM are
interchangable. The things that do change, like the build
system, are transperant to the users of the package. So we make
no distinction.
Structure
All Beam-related things are available via top-level
beam attribute, which includes:
interpreters: contains a set of compilers running
on Beam, including multiple Erlang/OTP versions
(beam.interpreters.erlangR19, etc), Elixir
(beam.interpreters.elixir) and LFE
(beam.interpreters.lfe).
packages: contains a set of package sets, each
compiled with a specific Erlang/OTP version, e.g.
beam.packages.erlangR19.
Default Erlang compiler is defined by
beam.interpreters.erlang and aliased as
erlang. Default package set with Beam packages is
defined by beam.packages.erlang and aliased at
top-level as beamPackages.
If you want to create a package set built with a custom Erlang version,
use lambda beam.packagesWith, which accepts an
Erlang/OTP derivative and produces a package set similar to
beam.packages.erlang.
Many Erlang/OTP distributions available in
beam.interpreters have their versions with ODBC and/or
Java enabled. For example there's
beam.interpreters.erlangR19_odbc_javac which
corresponds to beam.interpreters.erlangR19.
We also provide beam.packages.erlang.callPackage, which
simplifies writing Beam package definitions, by injecting all packages from
beam.packages.erlang into top-level context.
Build ToolsRebar3
By default Rebar3 wants to manage it's own dependencies. This is perfectly
acceptable in the normal non-Nix setup. In the Nix world it is not. To
support this we have created two versions of rebar3,
rebar3 and rebar3-open. The
rebar3 version has been patched to remove the ability
to download anything from it. If you are not running it a nix-shell or a
nix-build then its probably not going to work for you.
rebar3-open is the normal, un-modified rebar3. It
should work exactly as would any other version of rebar3. Any Erlang
package should rely on rebar3 and thats really what you
should be using too. See buildRebar3 below.
Mix & Erlang.mk
Both Mix and Erlang.mk work exactly as you would expect. There
is a bootstrap process that needs to be run for both of
them. However, that is supported by the
buildMix and buildErlangMk derivations.
How to install Beam packages
Beam packages are not registered in the top level simply because they are
not relevant to the vast majority of Nix users. They are installable using
the beam.packages.erlang attribute set and aliased as
beamPackages. This attribute points at packages built by
the default Erlang/OTP version in Nixpkgs as defined by
beam.interpreters.erlang.
You can list the avialable packages in the
beamPackages with the following command:
$ nix-env -f "<nixpkgs>" -qaP -A beamPackages
beamPackages.esqlite esqlite-0.2.1
beamPackages.goldrush goldrush-0.1.7
beamPackages.ibrowse ibrowse-4.2.2
beamPackages.jiffy jiffy-0.14.5
beamPackages.lager lager-3.0.2
beamPackages.meck meck-0.8.3
beamPackages.rebar3-pc pc-1.1.0
To install any of those packages into your profile, refer to them by
their attribute path (first column):
$ nix-env -f "<nixpkgs>" -iA beamPackages.ibrowse
The attribute path of any Beam packages corresponds to the name
of that particular package in Hex or its OTP Application/Release name.
Packaging Beam ApplicationsErlang ApplicationsRebar3 Packages
There is a Nix function called buildRebar3 (defined
in beam.packages.erlang.buildRebar3 and aliased at
top-level). We use this function to make a derivation that understands
how to build the rebar3 project. For example, the expression we use to
build the hex2nix
project follows.
{stdenv, fetchFromGitHub, buildRebar3, ibrowse, jsx, erlware_commons }:
buildRebar3 rec {
name = "hex2nix";
version = "0.0.1";
src = fetchFromGitHub {
owner = "ericbmerritt";
repo = "hex2nix";
rev = "${version}";
sha256 = "1w7xjidz1l5yjmhlplfx7kphmnpvqm67w99hd2m7kdixwdxq0zqg";
};
beamDeps = [ ibrowse jsx erlware_commons ];
}
This derivation is callable with
beam.packages.erlang.callPackage (see above). If you
want to call this package using normal callPackage,
you should refer to dependency packages via
beamPackages, e.g.
beamPackages.ibrowse.
The only visible difference between this derivation and
something like stdenv.mkDerivation is that we
have added beamDeps to the derivation. If
you add your Beam dependencies here they will be correctly
handled by the system.
If your package needs to compile native code via Rebar's port
compilation mechenism. You should add compilePort =
true; to the derivation.
Erlang.mk Packages
Erlang.mk functions almost identically to Rebar. The only real
difference is that buildErlangMk is called
instead of buildRebar3
{ buildErlangMk, fetchHex, cowlib, ranch }:
buildErlangMk {
name = "cowboy";
version = "1.0.4";
src = fetchHex {
pkg = "cowboy";
version = "1.0.4";
sha256 =
"6a0edee96885fae3a8dd0ac1f333538a42e807db638a9453064ccfdaa6b9fdac";
};
beamDeps = [ cowlib ranch ];
meta = {
description = ''Small, fast, modular HTTP server written in
Erlang.'';
license = stdenv.lib.licenses.isc;
homepage = "https://github.com/ninenines/cowboy";
};
}
Mix Packages
Mix functions almost identically to Rebar. The only real
difference is that buildMix is called
instead of buildRebar3
{ buildMix, fetchHex, plug, absinthe }:
buildMix {
name = "absinthe_plug";
version = "1.0.0";
src = fetchHex {
pkg = "absinthe_plug";
version = "1.0.0";
sha256 =
"08459823fe1fd4f0325a8bf0c937a4520583a5a26d73b193040ab30a1dfc0b33";
};
beamDeps = [ plug absinthe ];
meta = {
description = ''A plug for Absinthe, an experimental GraphQL
toolkit'';
license = stdenv.lib.licenses.bsd3;
homepage = "https://github.com/CargoSense/absinthe_plug";
};
}
Alternatively one can use buildHex as a shortcut for the above:
{ buildHex, buildMix, plug, absinthe }:
buildHex {
name = "absinthe_plug";
version = "1.0.0";
sha256 =
"08459823fe1fd4f0325a8bf0c937a4520583a5a26d73b193040ab30a1dfc0b33";
builder = buildMix;
beamDeps = [ plug absinthe];
meta = {
description = ''A plug for Absinthe, an experimental GraphQL
toolkit'';
license = stdenv.lib.licenses.bsd3;
homepage = "https://github.com/CargoSense/absinthe_plug";
};
}
How to developAccessing an Environment
Often, all you want to do is be able to access a valid
environment that contains a specific package and its
dependencies. we can do that with the env
part of a derivation. For example, lets say we want to access an
erlang repl with ibrowse loaded up. We could do the following.
~/w/nixpkgs ❯❯❯ nix-shell -A beamPackages.ibrowse.env --run "erl"
Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]
Eshell V7.0 (abort with ^G)
1> m(ibrowse).
Module: ibrowse
MD5: 3b3e0137d0cbb28070146978a3392945
Compiled: January 10 2016, 23:34
Object file: /nix/store/g1rlf65rdgjs4abbyj4grp37ry7ywivj-ibrowse-4.2.2/lib/erlang/lib/ibrowse-4.2.2/ebin/ibrowse.beam
Compiler options: [{outdir,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/ebin"},
debug_info,debug_info,nowarn_shadow_vars,
warn_unused_import,warn_unused_vars,warnings_as_errors,
{i,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/include"}]
Exports:
add_config/1 send_req_direct/7
all_trace_off/0 set_dest/3
code_change/3 set_max_attempts/3
get_config_value/1 set_max_pipeline_size/3
get_config_value/2 set_max_sessions/3
get_metrics/0 show_dest_status/0
get_metrics/2 show_dest_status/1
handle_call/3 show_dest_status/2
handle_cast/2 spawn_link_worker_process/1
handle_info/2 spawn_link_worker_process/2
init/1 spawn_worker_process/1
module_info/0 spawn_worker_process/2
module_info/1 start/0
rescan_config/0 start_link/0
rescan_config/1 stop/0
send_req/3 stop_worker_process/1
send_req/4 stream_close/1
send_req/5 stream_next/1
send_req/6 terminate/2
send_req_direct/4 trace_off/0
send_req_direct/5 trace_off/2
send_req_direct/6 trace_on/0
trace_on/2
ok
2>
Notice the -A beamPackages.ibrowse.env.That
is the key to this functionality.
Creating a Shell
Getting access to an environment often isn't enough to do real
development. Many times we need to create a
shell.nix file and do our development inside
of the environment specified by that file. This file looks a lot
like the packaging described above. The main difference is that
src points to project root and we call the
package directly.
{ pkgs ? import "<nixpkgs"> {} }:
with pkgs;
let
f = { buildRebar3, ibrowse, jsx, erlware_commons }:
buildRebar3 {
name = "hex2nix";
version = "0.1.0";
src = ./.;
beamDeps = [ ibrowse jsx erlware_commons ];
};
drv = beamPackages.callPackage f {};
in
drv
Building in a shell (for mix projects)
We can leverage the support of the Derivation, regardless of
which build Derivation is called by calling the commands themselves.
# =============================================================================
# Variables
# =============================================================================
NIX_TEMPLATES := "$(CURDIR)/nix-templates"
TARGET := "$(PREFIX)"
PROJECT_NAME := thorndyke
NIXPKGS=../nixpkgs
NIX_PATH=nixpkgs=$(NIXPKGS)
NIX_SHELL=nix-shell -I "$(NIX_PATH)" --pure
# =============================================================================
# Rules
# =============================================================================
.PHONY= all test clean repl shell build test analyze configure install \
test-nix-install publish plt analyze
all: build
guard-%:
@ if [ "${${*}}" == "" ]; then \
echo "Environment variable $* not set"; \
exit 1; \
fi
clean:
rm -rf _build
rm -rf .cache
repl:
$(NIX_SHELL) --run "iex -pa './_build/prod/lib/*/ebin'"
shell:
$(NIX_SHELL)
configure:
$(NIX_SHELL) --command 'eval "$$configurePhase"'
build: configure
$(NIX_SHELL) --command 'eval "$$buildPhase"'
install:
$(NIX_SHELL) --command 'eval "$$installPhase"'
test:
$(NIX_SHELL) --command 'mix test --no-start --no-deps-check'
plt:
$(NIX_SHELL) --run "mix dialyzer.plt --no-deps-check"
analyze: build plt
$(NIX_SHELL) --run "mix dialyzer --no-compile"
If you add the shell.nix as described and
user rebar as follows things should simply work. Aside from the
test, plt, and
analyze the talks work just fine for all of
the build Derivations.
Generating Packages from Hex with Hex2Nix
Updating the Hex packages requires the use of the
hex2nix tool. Given the path to the Erlang
modules (usually
pkgs/development/erlang-modules). It will
dump a file called
hex-packages.nix. That file will contain all
the packages that use a recognized build system in Hex. However,
it can't know whether or not all those packages are buildable.
To make life easier for our users, it makes good sense to go
ahead and attempt to build all those packages and remove the
ones that don't build. To do that, simply run the command (in
the root of your nixpkgs repository). that follows.
$ nix-build -A beamPackages
That will build every package in
beamPackages. Then you can go through and
manually remove the ones that fail. Hopefully, someone will
improve hex2nix in the future to automate
that.