Higgs FFI: Calling C From JS

CMS_Higgs-eventMy friend Maxime Chevalier-Boisvert has been working on a JavaScript VM called Higgs as part of her research into dynamic language optimization. In my free time I’ve been playing with it and using it for various toy projects such as a window manager, games, etc. In this and some upcoming posts I’m going to talk about the work I’ve been doing on and with Higgs.

An implementation of JavaScript on its own is not terribly useful for someone like me. The language doesn’t provide anything in the way of file I/O, graphics, access to the system, etc; the sorts of things the programs I want to write generally require. That’s something Maxime and I had been talking about and we agreed Higgs needed a FFI to interface with C code. This way we could just wrap existing functions/libraries instead of implementing the functionality from scratch.

Recipe For Iron Chef

Interfacing with C code is pretty simple in terms of the basic principles. C programs follow a specific calling convention defined by the platform’s ABI. The fiddly parts are dealing with different platforms and figuring out how to match the semantics and implementation of your language with those of C code.

Higgs fortunately makes things pretty easy. For one, it currently focuses on x86_64 POSIX systems only; so we don’t have to spend time duplicating code to work with Windows DLLs and calling conventions, or x86, etc. Higgs also has a JIT compiler, which means it comes with an assembler. Another nice thing is quite a bit of the Higgs runtime is written in JavaScript and we can use the same facilities to write much of the FFI.

The FFI for Higgs is underpinned by a few low level ops that are implemented in D as part of the interpreter. These allow the dynamic loading of a library, looking up symbols in the library, calling functions in the library, etc.

The following code demonstrates using these low level ops to call malloc:

// Load library
var l = $ir_load_lib("");

// Lookup malloc
var malloc = $ir_get_sym("malloc");

// Call malloc
var ptr = $ir_call_ffi(null, malloc, "*,i32", 128);

Behind the scenes, $ir_load_lib uses dlopen; passing an empty string allows access to the global symbol object of the current program (the Higgs interpreter). Next, we need to find the location for the malloc function. This is accomplished with the $ir_get_sym op; which uses dlsym. Finally, once we have an address for the function we want to call we use $ir_call_ffi to actually call it.

This last step is the most involved. The first argument is a placeholder, the second argument is the address of the function to call, the third argument is a string representing the return and argument types of the function; any following arguments are passed to the function being called. $ir_call_ffi uses this information to generate a bit of assembly (a CodeBlock) which sets up shuttling arguments out of the JS interpreter, calling the C function, and getting the return value into the JS interpreter. After it’s generated, the CodeBlock is placed in the placeholder slot and will be used for subsequent calls.

One Step Beyond

That’s a lot of code just to call one function. Fortunately we have a library to generate a little bit of wrapping code, so we can make this process easier:

// import ffi library
var ffi = require("lib/ffi");

// Load library
var c = ffi.load("");

// Lookup malloc, generate wrapper function
c.fun("malloc", "*,i32");

// Call malloc
var ptr = c.malloc(128);

Steal This API

That’s better, but still a bit more work than we’d like to do. LuaJIT has a really nice FFI library and the Higgs FFI borrows heavily from its API. The Higgs FFI library has a simple parser for C declarations and can generate wrapper code for them.

Now all we need to do is:

var ffi = require("lib/ffi");

ffi.c.cdef(["void* malloc (size_t size);"]);

var ptr = ffi.c.malloc(128);

In addition to wrapping functions, the Higgs FFI can wrap data. Recently I was experimenting with pipes for IPC using the pipe function. The pipe function expects an empty array of 2 ints, which it will populate with the file descriptors for your pipes. We could allocate the memory ourselves, but the Higgs FFI knows how to create and wrap C arrays for us. One way would be to use the functions provided by the FFI library, but we can take advantage of some automatic wrapping with a typedef:

var ffi = require("lib/ffi");

    "typedef int PipePair[2];",
    "int pipe(int pipefd[2]);"

// Create an array of 2 ints
var pipes = ffi.c.PipePair();

// Call the pipe function and pass a ptr to our array

// Get the file descriptors for our pipes
var in_pipe = pipes.get(0);
var out_pipe = pipes.get(1);

The Higgs FFI can also wrap other types of data like structs. This makes it fairly easy to quickly work with and write wrappers for C libraries:

var ffi = require("lib/ffi");
var console = require("lib/console");

var foo = ffi.load("/path/to/mylib.so");

    "struct account { int num; double balance; char name[10]; };",
    "typedef struct account customer;",
    "customer *bar();"

var bob = foo.account(foo.bar());
console.log("Account:", "#" + bob.get_num());
console.log("Balance:", "$" + bob.get_balance().toFixed(2));
console.log("Name:\t", bob.get_name());

What’s Next

The Higgs FFI is still under development and in a bit of flux. I’ve been implementing it as needed, so currently it can only handle the use cases I have come across in my experiments. Going forward I want to increase its ability to do automatic wrapping, improve the API, clean up the code, and so on.

I also plan on using the FFI to write a lot of bindings to allow writing all sorts of programs with Higgs. We already have a library for file I/O and some standard C functions like popen. I’m working on even more bindings for things like the file system, SDL, X11, etc.

If this area interests you, contributions are more than welcome. If you’d like to try writing bindings for your favorite C library, contribute to the FFI, or anything else; feel free to contact me or Maxime with any questions, comments, or anything else.