The Amazing Cosmic Awareness of Open Source: or How I Learned to Stop Worrying and Love Contributing

cosmicawareness“I should contribute to an open source project.” – the thought tumbled around the back of my mind for a long time. I think a lot of coders experience this; they want to contribute but a vague sense of anxiety holds them back. I know I was, and still am, rather shy about it.

Just open-sourcing your own projects can be quite intimidating. It’s putting yourself (or your code at least) out there. It’s also feels like taking on a bit of responsibilty – you now have (potential) users to think of. Joining someone elses project can compound all this by adding even more concerns; what if I mess something up, what if they hate my code, etc. It’s no wonder people are sometimes reluctant to do it. I recall having just such feelings, but I’m glad I’ve overcome them.

These days my OSS efforts are focused on my friend Maxime’s JavaScript VM – Higgs. In my spare time I manage to make some small contributions, and it’s probably one of the most enriching things I’ve done lately. I’ve learned a lot from looking at Higgs’ code, and discussing it’s workings with Max; as well as from discussions about possible designs and her feedback on my pull requests.

The ability to read and understand someone else’s code is an important skill to cultivate, and contributing to an open source project is great practice. It’s also helped me develop my skills in collaborating with others. The more I work with other coders the more I learn to appreciate a diverse team. Everyone has different ways of approaching and conceptualizing problems, different areas of expertise – working with others on open source is a great way to develop the ability to consider and deal with such differences. Working on Higgs has also allowed me to step into areas that are fairly outside of what I normally do – I’m generally a web developer rather than a systems programmer. All of these things can be a little uncomfortable, but it’s good to get out of your comfort zone sometimes; it’s one of the best ways to grow.

Since I’ve started working on Higgs, I’ve had the chance to encourage others to help us out on the project and I’ve come to experience the other side of things. A lot of people are interested in contributing to the project but have some hesitation. “I don’t know much about compilers.” or “I’ve never used D.” (the language the core of Higgs is implemented in) are common responses I’ve heard. To those I would say: “Good, This is your chance to learn!”

Beyond that, it’s important to remember that a lot of open source projects are large (Higgs has code in D, Javascript, and Python) and have a lot of different things they need. In the case of Higgs there is quite a lot of Javascript powering the runtime and the standard libraries that doesn’t require much special knowledge to work on. For example, the ability to construct functions with new Function(…) was implemented in pure Javascript (in a PR graciously submitted by Sorella).

Testing and documentation are another area that’s sometimes overlooked by potential contributors. Most open source projects are in constant need of more/updated documentation. Additionally the more people testing the software, the more bugs are found. Maxime and I both exclusively develop on linux, so we were totally unaware that Higgs was failing to build on OSX and BSD until Paul Fryzel was kind enough to let us know and help us fix it.

If you’re reading this and can relate, I hope I’ve inspired you a bit to get out there and push some code. Find a project you’re interested in with a welcoming team and get to it. If you’re interested in helping with Higgs, come talk to us in #higgsjs (webchat) on freenode or check out the open issues.

Some possible ways you could help:

  • Making sure Higgs builds/passes tests on your system.
  • Testing Higgs features/libraries and checking for bugs.
  • Adding more tests.
  • Adding more documentation.
  • Requesting/suggesting new features.
  • Adding new features to the libraries.
  • Writing new libraries.
  • Porting existing libraries to work on Higgs.
  • Anything else you can think of 🙂

Remember, you can do it Duffy Moon.

Advertisements

Higgs FFI: Calling C From JS

CMS_Higgs-eventMy friend Maxime Chevalier-Boisvert has been working on a JavaScript VM called Higgs as part of her research into dynamic language optimization. In my free time I’ve been playing with it and using it for various toy projects such as a window manager, games, etc. In this and some upcoming posts I’m going to talk about the work I’ve been doing on and with Higgs.

An implementation of JavaScript on its own is not terribly useful for someone like me. The language doesn’t provide anything in the way of file I/O, graphics, access to the system, etc; the sorts of things the programs I want to write generally require. That’s something Maxime and I had been talking about and we agreed Higgs needed a FFI to interface with C code. This way we could just wrap existing functions/libraries instead of implementing the functionality from scratch.

Recipe For Iron Chef

Interfacing with C code is pretty simple in terms of the basic principles. C programs follow a specific calling convention defined by the platform’s ABI. The fiddly parts are dealing with different platforms and figuring out how to match the semantics and implementation of your language with those of C code.

Higgs fortunately makes things pretty easy. For one, it currently focuses on x86_64 POSIX systems only; so we don’t have to spend time duplicating code to work with Windows DLLs and calling conventions, or x86, etc. Higgs also has a JIT compiler, which means it comes with an assembler. Another nice thing is quite a bit of the Higgs runtime is written in JavaScript and we can use the same facilities to write much of the FFI.

The FFI for Higgs is underpinned by a few low level ops that are implemented in D as part of the interpreter. These allow the dynamic loading of a library, looking up symbols in the library, calling functions in the library, etc.

The following code demonstrates using these low level ops to call malloc:

// Load library
var l = $ir_load_lib("");

// Lookup malloc
var malloc = $ir_get_sym("malloc");

// Call malloc
var ptr = $ir_call_ffi(null, malloc, "*,i32", 128);

Behind the scenes, $ir_load_lib uses dlopen; passing an empty string allows access to the global symbol object of the current program (the Higgs interpreter). Next, we need to find the location for the malloc function. This is accomplished with the $ir_get_sym op; which uses dlsym. Finally, once we have an address for the function we want to call we use $ir_call_ffi to actually call it.

This last step is the most involved. The first argument is a placeholder, the second argument is the address of the function to call, the third argument is a string representing the return and argument types of the function; any following arguments are passed to the function being called. $ir_call_ffi uses this information to generate a bit of assembly (a CodeBlock) which sets up shuttling arguments out of the JS interpreter, calling the C function, and getting the return value into the JS interpreter. After it’s generated, the CodeBlock is placed in the placeholder slot and will be used for subsequent calls.

One Step Beyond

That’s a lot of code just to call one function. Fortunately we have a library to generate a little bit of wrapping code, so we can make this process easier:

// import ffi library
var ffi = require("lib/ffi");

// Load library
var c = ffi.load("");

// Lookup malloc, generate wrapper function
c.fun("malloc", "*,i32");

// Call malloc
var ptr = c.malloc(128);

Steal This API

That’s better, but still a bit more work than we’d like to do. LuaJIT has a really nice FFI library and the Higgs FFI borrows heavily from its API. The Higgs FFI library has a simple parser for C declarations and can generate wrapper code for them.

Now all we need to do is:

var ffi = require("lib/ffi");

ffi.c.cdef(["void* malloc (size_t size);"]);

var ptr = ffi.c.malloc(128);

In addition to wrapping functions, the Higgs FFI can wrap data. Recently I was experimenting with pipes for IPC using the pipe function. The pipe function expects an empty array of 2 ints, which it will populate with the file descriptors for your pipes. We could allocate the memory ourselves, but the Higgs FFI knows how to create and wrap C arrays for us. One way would be to use the functions provided by the FFI library, but we can take advantage of some automatic wrapping with a typedef:

var ffi = require("lib/ffi");

ffi.c.cdef([
    "typedef int PipePair[2];",
    "int pipe(int pipefd[2]);"
]);

// Create an array of 2 ints
var pipes = ffi.c.PipePair();

// Call the pipe function and pass a ptr to our array
ffi.c.pipe(pipes.handle);

// Get the file descriptors for our pipes
var in_pipe = pipes.get(0);
var out_pipe = pipes.get(1);

The Higgs FFI can also wrap other types of data like structs. This makes it fairly easy to quickly work with and write wrappers for C libraries:

var ffi = require("lib/ffi");
var console = require("lib/console");

var foo = ffi.load("/path/to/mylib.so");

foo.cdef([
    "struct account { int num; double balance; char name[10]; };",
    "typedef struct account customer;",
    "customer *bar();"
]);

var bob = foo.account(foo.bar());
console.log("Account:", "#" + bob.get_num());
console.log("Balance:", "$" + bob.get_balance().toFixed(2));
console.log("Name:\t", bob.get_name());

What’s Next

The Higgs FFI is still under development and in a bit of flux. I’ve been implementing it as needed, so currently it can only handle the use cases I have come across in my experiments. Going forward I want to increase its ability to do automatic wrapping, improve the API, clean up the code, and so on.

I also plan on using the FFI to write a lot of bindings to allow writing all sorts of programs with Higgs. We already have a library for file I/O and some standard C functions like popen. I’m working on even more bindings for things like the file system, SDL, X11, etc.

If this area interests you, contributions are more than welcome. If you’d like to try writing bindings for your favorite C library, contribute to the FFI, or anything else; feel free to contact me or Maxime with any questions, comments, or anything else.