As I'm closing on 1 year of working on Mass I was quite curious to see if my approach to compile-time execution via JIT makes sense and how it compares with other languages that are capable of compile time execution, namely C++ and Zig. I do not have access to Jai, so can not measure that.

For now I just did two test program. The first one runs a counter 1 000 000 times at compile time and stores a result into a compile-time constant. The counter is then printed at runtime to verify the output value.

The goal of the second test program is to constant fold 1 000 definitions computing the sum of integer `1`. We then sum all the definitions at compile time as well to make sure that the compilers do not skip the computation for unreferenced constants. Because of the large amount of source code (2mb), this test not only measures the speed of constant folding itself but also parsing.

To measure just the compiled time execution and not the rest of the compiler, the code is first compiled with the constant hard-coded, then with the compile time execution and I get the difference.

All code is compiled without any optimization to minimize the non-relevant time spent in the compiler.

I have also inspected the generated assembly (x86_64) to verify that the value is indeed computed at compile time. All test are performed on Windows 10 (WSL2 for Clang). All times are provided in milliseconds.

Here are the results:

Counter 

1
2
3
4
5
6
Language     | Hardcoded | Compile Eval | Delta (ms) | X Times Slower
------------ | ----------|--------------|------------|----------------
Mass         | 12        | 16           | 4          | baseline
C++ (MSVC)   | 330       | 2270         | 1940       | 485x
C++ (CLang)  | 1065      | 1874         | 809        | 202x
Zig          | 1220      | 11714        | 10494      | 2623x


The results are pretty much what you would expect considering that both C++ and Zig do interpretation while Mass does a single-pass JIT. Clang seems to do reasonably well for an interpreter although doing anything computationally expensive would still slow down your compilation time dramatically.

Doing complex things with Zig comptime does not seem feasible at this time.

Constant Folding 

1
2
3
4
5
6
Language     | Hardcoded | Constant Folding | Delta (ms) | X Times Slower
------------ | ----------|------------------|------------|----------------
Mass         | 12        | 4362             | 4350       | 11.54x
C++ (MSVC)   | 330       | 1190             | 860        | 2.28x
C++ (CLang)  | 1065      | 1442             | 377        | baseline
Zig          | 1220      | 3818             | 2598       | 6.89x


Clang unsurprisingly is the fastest here as constant folding is its bread and butter. MSVC is slightly behind and Zig is almost 7x slower.

Mass is the more than an order of magnitude slower than Clang. After poking a bit under the hood I can see that the majority of time is actually spent in parsing as it is currently O(n^2) in complexity. The actual JIT part takes around 500ms. There is definitely lots of improvement to be done.

-------------------

You can see the full source code and more detailed description in the repository