Erlang for .NET

Why Compile Erlang for the CLR?

Kappuccino Kid — Sat, 22 Sep 2007 21:46:00 +0000

Erlang for the CLR? Why do such a crazy thing?

The .NET ecosystem lacks a concurrent programming language. C#, Visual Basic and so on have limited language support for concurrency (basically providing the equivalent of Java's synchronized.) Erlang would ideally fill this gap.
Windows programmers expect multi-language interoperation, to be able to readily consume common Windows APIs and frameworks, and to be able to integrate with Windows applications. The CLR is the substrate that enables all of these scenarios. Incidentally, a lot of the .NET and Win32 libraries and frameworks complement the common Erlang libraries. (UI libraries, for example.)
The CLR has a security model for dynamically-loaded code called Code Access Security (CAS.) CAS provides metadata for dynamically-loaded code to declare the permissions it requires, APIs for loading code with specific sets of permissions and for demanding permissions dynamically, and security policy management tools. Erlang's security system is much simpler: permissions are associated with entire Erlang virtual machines ('nodes') which can either be completely trusted, or not trusted at all. Bringing CAS to Erlang would make secure, distributed programming in Erlang first class.
Erlang occupies an interesting spot: It is dynamic, functional, and concurrent. That is new territory on the CLR.
The CLR runs on servers, the Windows client, and in browsers. That is new territory for Erlang.

Inside BEAM, the Erlang Virtual Machine

Kappuccino Kid — Thu, 20 Sep 2007 05:35:00 +0000

OTP R11B-5 includes more than one million lines of Erlang. The kernel and standard library are about a third of these lines. The emulator, BEAM, is around 200,000 lines of C.

BEAM's C main functions are platform-specific and live in erts/emulator/sys/unix, win32, and so on. (Windows actually has two mains: a simple one similar to the Unix entrypoints in erl_main.c, and a more complex one in erl_main_sae.c which is used in bootstrapping the toolchain.) The conventional mains immediately call erl_start which processes command line arguments, copies argv into a list of strings, creates a process and runs otp_ring0 with the argument list. otp_ring0 looks up and runs init:boot/1. There is more initialization after this, but from otp_ring0 on, apart from some built-in functions, Erlang is implemented in Erlang.

In practice, the Erlang compiler (another 30,000 lines of Erlang) compiles Erlang to BEAM bytecodes. When erl_start runs otp_ring0 and so on, it is loading BEAM binaries generated by the Erlang compiler earlier. The BEAM instruction set is register-based and has 130 instructions. These include instructions for arithmetic, comparsions and boolean logic; manipulating strings, tuples and lists; stack and heap allocation and freeing; type tests for Erlang primitives (numbers, lists, process IDs, references, and so on); jumps, structured exception handling, and calls and returns; sending messages and reading from the process' mailbox; waiting and timeouts; and so on. The instructions are listed in lib/compiler/src/beam_opcodes.erl. In BEAM, erts/emulator/beam/ops.tab contains rewrite rules that map typed BEAM instructions into emulator internals. A Perl script (beam_makeops) translates this file into C code (mostly macros.)

Some Erlang built-in function (BIFs) calls are translated into an extended instruction set not generated by the compiler. For example, the BIF erlang:self/3 is rewritten as a single 'self' instruction. Other BIF linkage is via a set of instructions for calling BIFs of varying arities. erts/emulator/beam/bif.tab maps BIF labels into C externs. For example, calls to the BIF erlang:spawn/3 boil down to a call to spawn_3 (found in erts/emulator/bif.c), which in turn thunks to erl_create_process.

Incidentally, processes are not related to threads or operating system processes. Instead, BEAM maintains queues of processes that are scheduled to run. There is one queue per priority level. Schedulers pick processes from the queues and run them. Processes maintain their registers and a pointer to the next instruction to execute.

The garbage collector, in erts/emulator/beam/erl_gc.c, is a mark-and-sweep copying collector. The most interesting thing about the BEAM garbage collector is that the garbage collected heaps are per-process. BEAM can do this because Erlang processes communicate explicitly via messages, and not via mutating shared memory. The CLR, in contrast, divides heaps into generations protected by write barriers and does incremental collection to avoid having to garbage collect all of a program's heap at the same time. This extra complexity is part of the price of using a shared heap to communicate between threads in the CLR.

Stranger Bedfellows: Erlang and the Dynamic Language Runtime

Kappuccino Kid — Wed, 19 Sep 2007 08:00:00 +0000

The Dynamic Language Runtime (DLR) sprung from IronPython internals and will provide the integration point between the CLR and dynamic languages (Python, Ruby, JavaScript and others) and define interoperation between components in different dynamic languages. The former includes abstract syntax for dynamic languages, a member resolution cache, and call-site thunks that cache method resolution. The member resolution cache is purged when a type is introduced or a type's representation changes. The call-site thunks can be strongly typed (at least, for up to six arguments); 'object' acts as a type wildcard the binding of which determines cache validity.

Is the DLR relevant to Erlang? Erlang is untyped, but crucially symbol resolution is straightforward and static, unlike Ruby/Python/etc. member resolution. The DLR is probably useful around the edges, for example in CLR interoperability, where DLR dynamic sites could cache the result of Reflection look-ups on plain old CLR objects, which must presumably appear to Erlang programs as processes (or Erlang must be heavily bastardized.) Erlang .NET's internals will probably more closely resemble F#--pattern matching and tuples--than any DLR language.

Erlang and .NET: Strange bedfellows?

Kappuccino Kid — Tue, 18 Sep 2007 07:56:00 +0000

There are a few challenges in implementing Erlang for .NET:

Erlang is dynamic, functional, concurrent and distributed. The original languages on the CLR were static, object-oriented, not especially concurrent (there are limited language primitives for per-thread locks) and not distributed (there is a deprecated RMI remote-stub mechanism, Remoting; and a few web services libraries.) However there have been some bold attempts at dynamic languages (IronPython, Gardens Point Ruby) and functional languages (F#.)
Erlang processes are cheap. In the CLR, processes are expensive (a CLR thread reserves 1 MB of stack space!) In fact, reliable, fast concurrent programming on Windows with the CLR is downright difficult. Objects are about the cheapest interesting thing in the CLR, so I guess Erlang processes map to objects.
Although the core Erlang language is small, it isn't set down formally. The implementation is the definition, which is one of the problems that derailed a Perl implementation for .NET. For a working definition, Erlang's bytecode format might be a good starting point, however there have been many of those, and like Erlang itself, the meaning of the bytecodes aren't written down either. The current incarnation ('BEAM') is embodied in about 100,000 lines of C code.
Because there is no shared memory between processes, Erlang's garbage collector can collect per-process. This means even naive stop-the-world garbage collection is highly concurrent in a global sense because of Erlang programs' heavy use of processes. For short-lived processes, the default heap is small enough for per-process memory management to work effectively as an area-based allocator. This limits the benefit of using the CLR's excellent generational garbage collector, because there is no way to hint the CLR that memory will be scoped to a particular process (although arguably short-lived processes should fit into gen-0 heap.)
Erlang has C and Java interoperation, but C and Java aren't .NET (the former supported reasonably well with P/Invoke, and the latter with an object model like .NET's, yet not quite.) More difficult than just calling .NET libraries from Erlang, .NET developers typically need a language to produce static constructs to have something to call, which could be difficult given Erlang's baroque syntax and dynamic nature. .NET has little-known terms for various aspects of interoperation (CTS and CLS producer/consumer) and even these don't cover many common scenarios (such as .NET generics.) Nonetheless, the CLR is leveraging Metcalfe's law for heterogenous components, and Erlang should be another exponent in that equation.
Erlang for the CLR should have syntax highlighting, IntelliSense, and debugger integration with Visual Studio.
The Erlang community is relatively small, and disjoint to the Microsoft developer community, and likely not interested in Erlang for the CLR.

There are, however, some consoling things:

The CLR has an excellent JIT, and Erlang has not been exposed to an excellent JIT. Because Erlang is essentially a dynamic language, I would have dismissed this--surely Erlang has more to gain from a better garbage collector than a JIT? But anecdotally (from a CLR architect) IronPython's solid performance is due to both the JIT and the GC.
Erlang libraries are mostly implemented in Erlang, and not C. This means porting the relatively small Erlang runtime system should give immediate access to OTP, Mnesia, and other big Erlang libraries.
The CLR supports asynchronous IO with thread pools and IAsyncResult. The lack of asynchronous IO was one of the factors making Erlang implementations on the JVM impractical--Erlang's thread scheduler is built around asynchronous IO, and the JVM's wasn't. (Since Java 1.5, java.util.concurrent has been addressing this issue though.)
Erlang's dynamism is relatively constrained. For example, Erlang's eval takes a list of bindings for the evaluator, as opposed to JavaScript's eval which can access variables in the callee's scope.