The Amazing Cosmic Awareness of Open Source: or How I Learned to Stop Worrying and Love Contributing

cosmicawareness“I should contribute to an open source project.” – the thought tumbled around the back of my mind for a long time. I think a lot of coders experience this; they want to contribute but a vague sense of anxiety holds them back. I know I was, and still am, rather shy about it.

Just open-sourcing your own projects can be quite intimidating. It’s putting yourself (or your code at least) out there. It’s also feels like taking on a bit of responsibilty – you now have (potential) users to think of. Joining someone elses project can compound all this by adding even more concerns; what if I mess something up, what if they hate my code, etc. It’s no wonder people are sometimes reluctant to do it. I recall having just such feelings, but I’m glad I’ve overcome them.

These days my OSS efforts are focused on my friend Maxime’s JavaScript VM – Higgs. In my spare time I manage to make some small contributions, and it’s probably one of the most enriching things I’ve done lately. I’ve learned a lot from looking at Higgs’ code, and discussing it’s workings with Max; as well as from discussions about possible designs and her feedback on my pull requests.

The ability to read and understand someone else’s code is an important skill to cultivate, and contributing to an open source project is great practice. It’s also helped me develop my skills in collaborating with others. The more I work with other coders the more I learn to appreciate a diverse team. Everyone has different ways of approaching and conceptualizing problems, different areas of expertise – working with others on open source is a great way to develop the ability to consider and deal with such differences. Working on Higgs has also allowed me to step into areas that are fairly outside of what I normally do – I’m generally a web developer rather than a systems programmer. All of these things can be a little uncomfortable, but it’s good to get out of your comfort zone sometimes; it’s one of the best ways to grow.

Since I’ve started working on Higgs, I’ve had the chance to encourage others to help us out on the project and I’ve come to experience the other side of things. A lot of people are interested in contributing to the project but have some hesitation. “I don’t know much about compilers.” or “I’ve never used D.” (the language the core of Higgs is implemented in) are common responses I’ve heard. To those I would say: “Good, This is your chance to learn!”

Beyond that, it’s important to remember that a lot of open source projects are large (Higgs has code in D, Javascript, and Python) and have a lot of different things they need. In the case of Higgs there is quite a lot of Javascript powering the runtime and the standard libraries that doesn’t require much special knowledge to work on. For example, the ability to construct functions with new Function(…) was implemented in pure Javascript (in a PR graciously submitted by Sorella).

Testing and documentation are another area that’s sometimes overlooked by potential contributors. Most open source projects are in constant need of more/updated documentation. Additionally the more people testing the software, the more bugs are found. Maxime and I both exclusively develop on linux, so we were totally unaware that Higgs was failing to build on OSX and BSD until Paul Fryzel was kind enough to let us know and help us fix it.

If you’re reading this and can relate, I hope I’ve inspired you a bit to get out there and push some code. Find a project you’re interested in with a welcoming team and get to it. If you’re interested in helping with Higgs, come talk to us in #higgsjs (webchat) on freenode or check out the open issues.

Some possible ways you could help:

  • Making sure Higgs builds/passes tests on your system.
  • Testing Higgs features/libraries and checking for bugs.
  • Adding more tests.
  • Adding more documentation.
  • Requesting/suggesting new features.
  • Adding new features to the libraries.
  • Writing new libraries.
  • Porting existing libraries to work on Higgs.
  • Anything else you can think of 🙂

Remember, you can do it Duffy Moon.

Higgs FFI: Calling C From JS

CMS_Higgs-eventMy friend Maxime Chevalier-Boisvert has been working on a JavaScript VM called Higgs as part of her research into dynamic language optimization. In my free time I’ve been playing with it and using it for various toy projects such as a window manager, games, etc. In this and some upcoming posts I’m going to talk about the work I’ve been doing on and with Higgs.

An implementation of JavaScript on its own is not terribly useful for someone like me. The language doesn’t provide anything in the way of file I/O, graphics, access to the system, etc; the sorts of things the programs I want to write generally require. That’s something Maxime and I had been talking about and we agreed Higgs needed a FFI to interface with C code. This way we could just wrap existing functions/libraries instead of implementing the functionality from scratch.

Recipe For Iron Chef

Interfacing with C code is pretty simple in terms of the basic principles. C programs follow a specific calling convention defined by the platform’s ABI. The fiddly parts are dealing with different platforms and figuring out how to match the semantics and implementation of your language with those of C code.

Higgs fortunately makes things pretty easy. For one, it currently focuses on x86_64 POSIX systems only; so we don’t have to spend time duplicating code to work with Windows DLLs and calling conventions, or x86, etc. Higgs also has a JIT compiler, which means it comes with an assembler. Another nice thing is quite a bit of the Higgs runtime is written in JavaScript and we can use the same facilities to write much of the FFI.

The FFI for Higgs is underpinned by a few low level ops that are implemented in D as part of the interpreter. These allow the dynamic loading of a library, looking up symbols in the library, calling functions in the library, etc.

The following code demonstrates using these low level ops to call malloc:

// Load library
var l = $ir_load_lib("");

// Lookup malloc
var malloc = $ir_get_sym("malloc");

// Call malloc
var ptr = $ir_call_ffi(null, malloc, "*,i32", 128);

Behind the scenes, $ir_load_lib uses dlopen; passing an empty string allows access to the global symbol object of the current program (the Higgs interpreter). Next, we need to find the location for the malloc function. This is accomplished with the $ir_get_sym op; which uses dlsym. Finally, once we have an address for the function we want to call we use $ir_call_ffi to actually call it.

This last step is the most involved. The first argument is a placeholder, the second argument is the address of the function to call, the third argument is a string representing the return and argument types of the function; any following arguments are passed to the function being called. $ir_call_ffi uses this information to generate a bit of assembly (a CodeBlock) which sets up shuttling arguments out of the JS interpreter, calling the C function, and getting the return value into the JS interpreter. After it’s generated, the CodeBlock is placed in the placeholder slot and will be used for subsequent calls.

One Step Beyond

That’s a lot of code just to call one function. Fortunately we have a library to generate a little bit of wrapping code, so we can make this process easier:

// import ffi library
var ffi = require("lib/ffi");

// Load library
var c = ffi.load("");

// Lookup malloc, generate wrapper function"malloc", "*,i32");

// Call malloc
var ptr = c.malloc(128);

Steal This API

That’s better, but still a bit more work than we’d like to do. LuaJIT has a really nice FFI library and the Higgs FFI borrows heavily from its API. The Higgs FFI library has a simple parser for C declarations and can generate wrapper code for them.

Now all we need to do is:

var ffi = require("lib/ffi");

ffi.c.cdef(["void* malloc (size_t size);"]);

var ptr = ffi.c.malloc(128);

In addition to wrapping functions, the Higgs FFI can wrap data. Recently I was experimenting with pipes for IPC using the pipe function. The pipe function expects an empty array of 2 ints, which it will populate with the file descriptors for your pipes. We could allocate the memory ourselves, but the Higgs FFI knows how to create and wrap C arrays for us. One way would be to use the functions provided by the FFI library, but we can take advantage of some automatic wrapping with a typedef:

var ffi = require("lib/ffi");

    "typedef int PipePair[2];",
    "int pipe(int pipefd[2]);"

// Create an array of 2 ints
var pipes = ffi.c.PipePair();

// Call the pipe function and pass a ptr to our array

// Get the file descriptors for our pipes
var in_pipe = pipes.get(0);
var out_pipe = pipes.get(1);

The Higgs FFI can also wrap other types of data like structs. This makes it fairly easy to quickly work with and write wrappers for C libraries:

var ffi = require("lib/ffi");
var console = require("lib/console");

var foo = ffi.load("/path/to/");

    "struct account { int num; double balance; char name[10]; };",
    "typedef struct account customer;",
    "customer *bar();"

var bob = foo.account(;
console.log("Account:", "#" + bob.get_num());
console.log("Balance:", "$" + bob.get_balance().toFixed(2));
console.log("Name:\t", bob.get_name());

What’s Next

The Higgs FFI is still under development and in a bit of flux. I’ve been implementing it as needed, so currently it can only handle the use cases I have come across in my experiments. Going forward I want to increase its ability to do automatic wrapping, improve the API, clean up the code, and so on.

I also plan on using the FFI to write a lot of bindings to allow writing all sorts of programs with Higgs. We already have a library for file I/O and some standard C functions like popen. I’m working on even more bindings for things like the file system, SDL, X11, etc.

If this area interests you, contributions are more than welcome. If you’d like to try writing bindings for your favorite C library, contribute to the FFI, or anything else; feel free to contact me or Maxime with any questions, comments, or anything else.

On holistic programming languages.

m0003 540My friend Maxime recently blogged about “Programming Without Text Files”. I’ve always wanted to be one of those bloggy people, and being fairly lazy and uncreative I’ve decided to shamelessly purloin the topic for my inaugural post. I figured I’d share some of my own thoughts in this area of “programming without text files” and beyond.

So, what is “programming without text files” anyway? What does it mean for a programmer to “[edit] the underlying data structures directly?” I think that this is probably a subtle distinction, but a potentially important one.

All programming is ultimately done by manipulating an AST – what difference is it if we do it “directly”? After all, IDEs can give us a lot of nice features (autocompletion, autorefactoring, etc) so that we are not always laboriously typing out a textual representation of our program. Isn’t writing code which essentially “generates” an AST and then using all these fancy IDE features conceptually the same thing?

I think both conceptually and practically there are differences. Unless you’re a lisper there is generally a few layers between you and the AST. IDEs have to work hard for these features. They scan, munge, parse, rescan, wash, rinse, and repeat…and then all this work is lost when we send our program off to the compiler/VM/linter/whatever where the cycle repeats.

I also think on a certain level the way many programmers conceptualize programs and the way our tools accept them are very similar – they are a series of letters, numbers, and “punctuation”. For us, it’s translated to an AST almost as an afterthought; it’s just something the compiler does as part of its magic. We catch glimpses of it here and there when we use fancy IDE features like code folding, syntax checking, etc …but it remains obscured by the maya of text files. We still tend to think of programs as some English (or what-have-you) text.

Was the programming work-flow really perfected decades ago? What if we change the way we think about and treat programs? Can we make a language where programming is a first class citizen?

It reminds me of the debate that ensued when Zed Shaw chose to use sqlite databases to store the configuration for Mongrel2 instead of text files. It makes sense – a db is much easier to query and manipulate programmatically. Yet many objected that they were not “human readable”, or worried that their text-focused tools would have trouble dealing with a db.

I could see similar objections here – “If my program is some opaque binary blob of an AST or whatever, how am I supposed to check it into git? How am I supposed to git diff etc?”. I would answer: “Maybe you don’t need to!”

When you start thinking of your programs holistically all this stuff that was “metadata” becomes just data. It’s all a part of your program. You don’t need to worry about cluttering up your pretty little text file anymore; you have a “living, breathing” data structure – you can store revision information for nodes in the nodes themselves if you want to. You can store all sorts of information, and have your program displayed with as much or little detail as you please.

Now that we’re not operating on mere lines of text, your revision control can be aware of the semantics of your language. I can get a revision history of any/all code that touches the I/O lib without a bunch of grep/awk/sh massaging? Yes please!

*.pyc files? F that! Let program and byte code live harmoniously. Let me see the AST and byte code side by side. Let programs easily hack their own binaries if they want. Cache generated code and data in the program file itself, why not?

Tired of slow coworkers who need everything spelled out? Or coders who write comments like Ayn Rand writes novels? We can have granularity in our comments…and no need to come up with some elaborate markup, and then a tool to parse it, and then integrate it with the editor. We already have a model of programming that is very explicitly about this sort of tree-walking/manipulation. Give each comment a “owner”, “level”, etc and flip a switch for “ELI5” or “pithy” comment display. Have a conversation in the comments, heck upvote comments or attach rage faces to bad code.

y u no face

I’m editing on Linux, Y U NO just hide the Windows code!

No need to scroll through hundreds of lines of code you don’t care about, dealing with nonsense like:

    #ifdef WINDOWS_BLAH

Give me an editor that recognizes my code is fractal – none of this “jump to definition” nonsense: I want infinite zoom. Give me a real modal editor – not these piddling “insert” and “command” modes. I want trying-to-understand-code-someone-else-wrote-mode; I want bang-out-code-as-quickly-as-possible-mode; I want debugging-mode; I want writing-documentation-mode; I want security-audit-mode; I want i18n-mode.

In my opinion, this is all just the beginning of really interesting things you could do. I’ve mostly focused on the programmer/user experience, but such a change could make the lives of compiler/tooling authors easier as well.

I don’t pretend to fully understand the trade-offs involved in such a shift – to be sure there are drawbacks.  I’m also not about to step up and volunteer to implement it (though I’d certainly offer to help) – it’s quite a task. However, I am suspicious of the idea that we really can’t do better than plain text files.

These ideas have been explored before (smalltalk, lisp, MPS, etc), indeed versions of most of this stuff can be found around …but perhaps the computing world of today is the time and place to start taking them more seriously and revisit them. We live in a world of powerful computers and improved interfaces which could make this more practical.

To be sure, this stuff is not prohibited by our text files – it just becomes easier and more obvious when we discard them. Perhaps it is time we throw down our text shackles, and code free.