Metaprogramming and testing in the D Programming Language

I have recently decided to play around with a few niche languages in the “system programming” realm. Having been programming for many years in higher level programming languages such as Java, Dart and Kotlin (and tried many, many others at the same level, or higher), I feel like I need to expand my horizons as for certain kinds of applications, these languages are just not the best tool for the job.

In this blog post, I want to focus on the D Programming Language because that’s the one that called my attention a bit more than the others, after some initial experimentation.

I also mocked around a bit with Zig and Nim but didn’t feel like they were the right language for me, at least for now (they’re really cool anyway, do check them out!).

Of course, I had already looked into the king of this space, Rust… but Rust, while being a genius language in many ways, doesn’t really make me excited about writing code - to the contrary, the thought of spending weekends battling with the borrow checker fills me with dread. I would absolutely use Rust in a working context (and have done so) due to its safety guarantees (not just memory safety but specially resource- and thread-safety it provides) and outstanding performance (both in terms of low memory consumption and raw speed), but for hobby projects, thanks but… I just don’t see that happening (I already abandoned a few Rust projects halfway through, I’m afraid).

Nim goes a bit too far towards the other end, in my opinion, as it’s a very playful language where I felt that safety is considered less important than speed and, well, joy. So if you like that (it’s also very fast, creates tiny binaries and uses very little memory) it may be just the language for you.

Zig has a lot of promise, but it just doesn’t feel ready at the moment. It’s also turning out to be quite verbose and difficult to use correctly despite its focus on simplicity.

D seems like a good balance. It looks familiar while having some very interesting features. It’s been around long enough that it’s no longer trying to find itself and changing every 6 months.

In this post, I’d like to share what I learned, with a focus on an unusual aspect of the language, its metaprogramming capabilities, and on another feature that very few languages include, despite its importance and obiquity in modern programming practices: unit testing.

A quick introduction to D

D is not a new language in 2023. It’s been around since 2001, but has evolved quite a bit since then, specially since the stabilization of D version 2 around 2010.

It has 3 different and well maintained compilers (see the Downloads Page):

• DMD the reference compiler, written in D itself.
• GDC a GCC frontend for D.
• LDC based on the LLVM.

DMD is normally used for its faster compilation (in fact, it’s probably one of the fastest compilers of any production-grade language), but the other two are normally better at optimizing for runtime speed.

The D Language Tour does an excellent job in introducing D’s features, and the D's Gems section is specially interesting as it shows things D has that most other languages don’t, like Uniform Function Call Syntax (UFCS), Scope Guards, Compile-time Function Evaluation (CTFE), Attributes (e.g. @safe, @nogc, @mustuse) and more.

Also check the Multi-threading section, which includes Message Passing and Thread-Local Storage, which together enable writing concurrent code using something akin to the Actor Model.

In any case, I think it’s appropriate to show some D examples before discussing more advanced features.

Here’s an example (from the official tour) that shows D slices in action:

import std.stdio : writeln;

void main()
{
    int[] test = [ 3, 9, 11, 7, 2, 76, 90, 6 ];
    test.writeln;
    writeln("First element: ", test[0]);
    writeln("Last element: ", test[$ - 1]);
    writeln("Exclude the first two elements: ",
        test[2 .. $]);

    writeln("Slices are views on the memory:");
    auto test2 = test;
    auto subView = test[3 .. $];
    test[] += 1; // increment each element by 1
    test.writeln;
    test2.writeln;
    subView.writeln;

    // Create an empty slice
    assert(test[2 .. 2].length == 0);
}

Compiling and running it:

➜ dmd -of=slices slices.d
➜ ./slices 
[3, 9, 11, 7, 2, 76, 90, 6]
First element: 3
Last element: 6
Exclude the first two elements: [11, 7, 2, 76, 90, 6]
Slices are views on the memory:
[4, 10, 12, 8, 3, 77, 91, 7]
[4, 10, 12, 8, 3, 77, 91, 7]
[8, 3, 77, 91, 7]

You can also run a D program directly from source using dmd -run file.d or rdmd, which comes bundled with DMD. That’s even usable as a shell shebang #!/usr/bin/env rdmd for scripting in D.

It shows quite a few interesting features.

• test.writeln is the same as writeln(test). That’s what UFCS does.
• test[$ - 1] shows how the $ symbol can be used within [] as the length of the array/slice.
• test[2 .. $] a typical slice, similar to Go, again using $.
• test[] += 1 shows a vector operation which may be optimized by the compiler.
• assert(test[2 .. 2].length == 0); D assertion, which we’ll use for unit testing later.

Quite nice.

D Metaprogramming

D has many metaprogramming features. Metaprogramming is, for those not initiated in the arts, programming against the program itself.

Lisp was probably the pioneer in metaprogramming with its macros, but macros are not the only way to do metaprogramming.

For example, D has templates which allow inspecting types at compile-time to specialize the implementation of a function, as this example shows:

@safe:

auto concat(T)(T lhs, T rhs) {
  static if (is(T: double)) {
    // T is convertable to double
    return lhs + rhs;
  } else {
    // `~` is normally the concat operator in D
    return lhs ~ rhs;
  }
}

unittest {
  assert(2.concat(3) == 5);
  assert(4.2.concat(0.8) == 5.0);
  assert("Hello".concat(" D") == "Hello D");
}

Running the unittest:

➜ dmd -w -main -unittest -run tests.d 
1 modules passed unittests

This example is a bit silly because D supports operator overloading, so you would just use that instead.

If you’re familiar with Java, concat would be similar to a generic static method, but unlike in Java, D allows you to inspect the type at compile-time, so it’s possible to specialize the function implementation for certain types.

static if is an if statement that’s executed at compile-time… it doesn’t exist at runtime, only the selected branch does.

Notice how a template has two arguments lists: one with compile-time arguments, and one with the runtime arguments. The compile-time arguments can be omitted if the D compiler can infer them.

It is possible to explicitly provide the compile-time argument to a template function with the ! operator.

For example, the to template in the std.conv stdlib module takes a type as argument, but because it normally can’t be inferred, it’s almost always passed explicitly:

unittest {
  import std.conv: to;

  assert(42.to!string == "42");
}

And this is just the most basic kind of D template.

You can also use the template keyword to do more advanced things, like generating multiple functions:

template BiDirectionalConverter(T1, T2) {
  import std.conv: to;
  T2 convert(T1 t) {
    return t.to!T2();
  }
  T1 convert(T2 t) {
    return t.to!T1();
  }
}
unittest {
  alias StringIntConv = BiDirectionalConverter!(string, int);

  assert(StringIntConv.convert("20") == 20);
  assert(StringIntConv.convert(20) == "20");
}

The octal template from std.conv is used to declare compile-time octal constans in D:

void main() {
  import std.stdio: writeln;
  import std.conv;
  writeln(octal!"750");
}

Running:

➜ dmd -run tests.d
488

I highly recommend skimmming through the D Templates Tutorial to learn more about what is possible.

Another kind of template in D is the mixin template. It’s kind of a copy-and-paste template, it lets you just paste some code directly where you invoke it as if you had written it in the surrounding scope.

mixin template Abcd(T) {
  T a, b, c, d;
}

unittest {
  mixin Abcd!int;
  a = 10;
  assert(a == 10);
  assert(b == 0);
  assert(c == 0);
  assert(d == 0);
}

Just like that, if you ever find yourself having to type the same thing over and over, well just write a mixin template and stop doing that (I wouldn’t advise introducing variables into another scope like that, but the example should at least give some ideas of what you could do).

Finally, if none of the above options were enough to achieve what you wanted, you can actually generate code with Strings using a String mixin:

/// Build a struct with fields a, b and c of type T.
string abcStruct(T)(string name) {
  return "struct " ~ name
    ~ " { "
    ~ T.stringof ~ " a; "
    ~ T.stringof ~ " b; "
    ~ T.stringof ~ " c; "
    ~ " }\n";
}

unittest {
  mixin(abcStruct!string("StringStruct"));
  mixin(abcStruct!int("IntStruct"));

  auto abcstr = StringStruct("hey", "ho", "let's go");
  assert(abcstr.a == "hey");
  assert(abcstr.b == "ho");
  assert(abcstr.c == "let's go");

  auto abcint = IntStruct(42);
  assert(abcint.a == 42);
  assert(abcint.b == 0);
  assert(abcint.c == 0);
}

D can create a file with all mixins it generated during compilation with the -mixin flag:

➜ dmd -w -main -unittest -mixin=mixins.d -run tests.d
1 modules passed unittests

Now, looking at the mixins.d file, we’ll find the structs the D compiler generated:

// expansion at tests.d(67)
struct StringStruct { string a; string b; string c;  }

// expansion at tests.d(68)
struct IntStruct { int a; int b; int c;  }

Alternatively, use a pragma so that the D compiler just prints the generated code during compilation:

pragma(msg, abcStruct!double("DoubleStruct"));

Result:

➜ dmd -w -main -of=tests tests.d
struct DoubleStruct { double a; double b; double c;  }

A lot more mixin tricks can be found at the Programming in D website.

The Code generation (Parser) example in the official D documentation shows how easy it is to generate constant configuration data by parsing a String at compile-time.

Unit testing

In the previous examples, I used unittest blocks to demonstrate some of D’s features. I hope it is self-evident that the code within these blocks is not normally included in the compilation unit, which is why when invoking the compiler with the intent of running tests, you must pass the -unittest option to the compiler (and to actually run the tests, either execute the produced binary, or include the -run option while compiling).

To recap, a unittest looks like this:

unittest {
  assert(2 + 2 == 4);
}

Change 4 to 5 above and run the code:

➜ dmd -w -main -of=tests -run tests.d

The -main option is used so that the compiler generates an empty main function in case there’s none, avoiding this error: undefined reference to 'main'. The -w flag makes the compiler treat warnings as errors, which is generally a good idea. -of is for naming the output file. Use --help to see all options.

If nothing is printed, all tests are ok. This shows that no test actually ran.

Now try again with -unittest:

➜ dmd -w -main -unittest -run tests.d
tests.d(18): [unittest] unittest failure
1/1 modules FAILED unittests

The output is very simple. It just tells you how many modules’ tests failed and the files and lines where an assertion failed.

This may be fine for quick testing, but it would be nice to know exactly why the test failed to save time when debugging the problem.

This is a common complaint in the D Forum.

The D Forum is great, by the way… and it’s where most D enthusiasts seem to hang out. People are helpful and the website is really fast, something uncommon these days.

After having learnt about how D metaprogramming allows so many cool things, I was just thinking as I saw that: it should be almost trivial to get these tests to produce very good error messages! And that’s absolutely true.

But D strives to keep the compiler simple and the language relatively small (I suppose it’s debatable if that’s the case). So features like this are left to libraries.

However, while there are multiple libraries to help with testing, from what I’ve read so far most people are not using them… either because they’re happy with the very basic testing support provided by DMD itself, or because it’s so easy to write your own framework in D that people do just that!

It’s a familiar feeling for anyone who has done Lisp (or other languages with similarly powerful metaprogramming) at some point!

For example, here’s a little template I came up with to make assertions more powerful by displaying the expected and actual results on a failed assertion:

auto assertThat(string desc, string op, T)(T lhs, T rhs) {
  import std.conv: to;
  const str = "assert(lhs " ~ op ~ " rhs, \"" ~
    desc ~ ": \" ~ lhs.to!string() ~ \" " ~ op ~ " \" ~ rhs.to!string())";
  return mixin(str);
}

Now, the assertion looks like this:

unittest {
  assertThat!("adding two and two", "==")(2 + 2, 5);
}

Running it:

➜ dmd -w -main -unittest -run tests.d
tests.d-mixin-20(20): [unittest] adding two and two: 4 == 5
1/1 modules FAILED unittests

Really cool!

EDIT: the above can be accomplished more easily by passing the -checkaction=context option to the compiler when running the tests. With that, the error message would look like this: main.d(6): [unittest] 4 != 5. Thanks to Steven Schveighoffer for pointing it out.

As a side note, D unittests are typically used to verify the attributes of a function are as expected (because the D compiler infers them, usually, as it’s very tedious to manually annotate every function with lots of attributes).

As an example, as I was trying to implement an atree in D, I tried this test:

@safe @nogc nothrow pure unittest {
    auto tree = Tree([0,0,0,2], [10,11,12,13]);
    assertThat!("children(2) basic case", "==")(tree.children(2), [3, -1]);
}

This only works if the functions used in the unittest are all inferred to be annotated with @safe @nogc nothrow pure (the compiler checks these transitively).

Here’s the result:

➜  myd dmd -unittest -run source/app.d 
source/app.d(38): Error: array literal in `@nogc` function `app.__unittest_L37_C26` may cause a GC allocation
source/app.d(38): Error: array literal in `@nogc` function `app.__unittest_L37_C26` may cause a GC allocation
source/app.d(39): Error: `pure` function `app.__unittest_L37_C26` cannot call impure function `app.Tree.children`
source/app.d(39): Error: `@nogc` function `app.__unittest_L37_C26` cannot call non-@nogc function `app.Tree.children`
source/app.d(39): Error: `@nogc` function `app.__unittest_L37_C26` cannot call non-@nogc function `app.assertThat!("children(2) basic case", "==", int[]).assertThat`
source/app.d(31):        which calls `std.conv.to!string.to!(int[]).to`
/usr/include/dmd/phobos/std/conv.d(207):        which calls `std.conv.toImpl!(string, int[]).toImpl`
/usr/include/dmd/phobos/std/conv.d(997):        which calls `std.conv.toStr!(string, int[]).toStr`
/usr/include/dmd/phobos/std/conv.d(122):        which calls `std.array.appender!string.appender`
/usr/include/dmd/phobos/std/array.d(4146):        which calls `std.array.Appender!string.Appender.this`
/usr/include/dmd/phobos/std/array.d(3509):        which wasn't inferred `@nogc` because of:
/usr/include/dmd/phobos/std/array.d(3509):        cannot use `new` in `@nogc` constructor `std.array.Appender!string.Appender.this`
source/app.d(39): Error: array literal in `@nogc` function `app.__unittest_L37_C26` may cause a GC allocation

Very interesting!

Another common use case is to run a single test only… the compiler does not support that, but you can do it yourself, as @jfondren has shown on the D Forum:

module tester1;

unittest { assert(true); }
unittest { assert(!!true); }
unittest { assert(1 != 1); }
unittest { assert(1 > 0); }

version (unittest) {
    bool tester() {
        import std.meta : AliasSeq;
        import std.stdio : writef, writeln;

        alias tests = AliasSeq!(__traits(getUnitTests, tester1));
        static foreach (i; 0 .. tests.length) {
            writef!"Test %d/%d ..."(i + 1, tests.length);
            try {
                tests[i]();
                writeln("ok");
            } catch (Throwable t) {
                writeln("failed");
            }
        }
        return false;
    }

    shared static this() {
        import core.runtime : Runtime;

        Runtime.moduleUnitTester = &tester;
    }
}

void main() {
    assert(false); // this doesn't get run
}

Running it:

➜ dmd -w -main -unittest -run tests.d
Test 1/4 ...ok
Test 2/4 ...ok
Test 3/4 ...failed
Test 4/4 ...ok

Very neat, but probably not something you want to write yourself.

This uses some fairly advanced stuff like the getUnitTests trait (D traits are about metaprogramming, not what you normally think of when you hear the word if you come from Rust or Scala) and UDAs (compile-time annotations).

Anyway, I would go against the flow here and definitely reach out for a testing framework. In a professional context, you’ve got to have test reports, the ability to easily run tests by category or tags, extremely good error messages to avoid wasting time figuring out why a test even failed, a rich assertion library, a mocking library if possible (though I rarely use mocks these days, sometimes they’re hard to avoid)… and probably more.

The D Wiki links to some frameworks… most are abandoned or still in beta. I haven’t had the time to try it yet, but unit-threaded seems to be the most promising alternative (still maintained and appears to have most functionality I would want).

I might get back to it later, which would require me to also learn how to use D’s package manager and build tool, dub which seems to be quite nice.

Final thoughts

I had fun learning D and discovering how it solved problems that every language has to grapple with in fairly unique ways.

You can feel immediately as you get started that D is not a language with a lot of corporate backing (which can be a good thing!). As I downloaded DMD on my Mac, it failed to compile (or link, rather) even a hello world program due to Mac Sonoma apparently changing something that broke D’s linker. Trying it on my other machine which runs Linux Kubuntu, I had a few other issues, mostly related to the fact that I used the snap installer which was outdated by over 3 years.

If you’re on Linux, apparently the best way to obtain a D compiler is to run the installer script listed in the Downloads Page.

If you’re on Mac, try the LDC compiler until DMD gets fixed (please check it though, because as of writing the bug has already been fixed, but not released yet).

IDE support seems ok-ish, but far from what you get with more mainstream languages like Java, Kotlin, Typescript or even Rust.

I tried using emacs (you need to get d-mode and then install serve-d, the LSP server which also powers VS Code’s D support) first and was a bit underwhelmed.

Then I noticed that the D IntelliJ Plugin is quite capable, and being a big user of Jebrains products, that was a very good surprise (normally, niche languages don’t have great support in IntelliJ at all)!

Hats off to the developers of the IntelliJ plugin! It offers a very good experience out-of-the-box, which nice templates for generating snippets, code navigation (including going into the D stdlib, which is great for learning), inline documentation is nicely styled, DScanner linting so warnings are shown right in the code, auto-formatting via dfmt, support for dub is built-in… it can even run tests if you use d-unit as a dependency (I should’ve chosen that I guess!).

Anyway, there’s a few more nice surprises with D as well, like its good support for CPU and memory profiling and its very good Documentation Tool, ddoc (D documentation can be executed at compile time, like in Rust, ensuring docs examples are always working!).

It feels like with just a little bit more attention, the UX a developer gets with D could be really excellent.

I hope that this post draws a few more people to D. I will certainly keep an eye on it and maybe write more and more code using it as I get more comfortable with the language and tooling. I am definitely getting tired of VM-based languages like Java, and not really enjoying writing Rust, I feel like D might become my next favourite language.