New Blog

Hello, my new blog can be found here: http://atomiccodebin.com

The Amazing Cosmic Awareness of Open Source: or How I Learned to Stop Worrying and Love Contributing

“I should contribute to an open source project.” – the thought tumbled around the back of my mind for a long time. I think a lot of coders experience this; they want to contribute but a vague sense of anxiety holds them back. I know I was, and still am, rather shy about it.

Just open-sourcing your own projects can be quite intimidating. It’s putting yourself (or your code at least) out there. It’s also feels like taking on a bit of responsibilty – you now have (potential) users to think of. Joining someone elses project can compound all this by adding even more concerns; what if I mess something up, what if they hate my code, etc. It’s no wonder people are sometimes reluctant to do it. I recall having just such feelings, but I’m glad I’ve overcome them.

These days my OSS efforts are focused on my friend Maxime’s JavaScript VM – Higgs. In my spare time I manage to make some small contributions, and it’s probably one of the most enriching things I’ve done lately. I’ve learned a lot from looking at Higgs’ code, and discussing it’s workings with Max; as well as from discussions about possible designs and her feedback on my pull requests.

The ability to read and understand someone else’s code is an important skill to cultivate, and contributing to an open source project is great practice. It’s also helped me develop my skills in collaborating with others. The more I work with other coders the more I learn to appreciate a diverse team. Everyone has different ways of approaching and conceptualizing problems, different areas of expertise – working with others on open source is a great way to develop the ability to consider and deal with such differences. Working on Higgs has also allowed me to step into areas that are fairly outside of what I normally do – I’m generally a web developer rather than a systems programmer. All of these things can be a little uncomfortable, but it’s good to get out of your comfort zone sometimes; it’s one of the best ways to grow.

Since I’ve started working on Higgs, I’ve had the chance to encourage others to help us out on the project and I’ve come to experience the other side of things. A lot of people are interested in contributing to the project but have some hesitation. “I don’t know much about compilers.” or “I’ve never used D.” (the language the core of Higgs is implemented in) are common responses I’ve heard. To those I would say: “Good, This is your chance to learn!”

Beyond that, it’s important to remember that a lot of open source projects are large (Higgs has code in D, Javascript, and Python) and have a lot of different things they need. In the case of Higgs there is quite a lot of Javascript powering the runtime and the standard libraries that doesn’t require much special knowledge to work on. For example, the ability to construct functions with new Function(…) was implemented in pure Javascript (in a PR graciously submitted by Sorella).

Testing and documentation are another area that’s sometimes overlooked by potential contributors. Most open source projects are in constant need of more/updated documentation. Additionally the more people testing the software, the more bugs are found. Maxime and I both exclusively develop on linux, so we were totally unaware that Higgs was failing to build on OSX and BSD until Paul Fryzel was kind enough to let us know and help us fix it.

If you’re reading this and can relate, I hope I’ve inspired you a bit to get out there and push some code. Find a project you’re interested in with a welcoming team and get to it. If you’re interested in helping with Higgs, come talk to us in #higgsjs (webchat) on freenode or check out the open issues.

Some possible ways you could help:

• Making sure Higgs builds/passes tests on your system.
• Testing Higgs features/libraries and checking for bugs.
• Adding more tests.
• Adding more documentation.
• Requesting/suggesting new features.
• Adding new features to the libraries.
• Writing new libraries.
• Porting existing libraries to work on Higgs.
• Anything else you can think of 🙂

Remember, you can do it Duffy Moon.

Contribute to the Higgs JS Compiler

Pointers Gone Wild

Higgs is an open source JIT compiler for JavaScript targeting x86-64 platforms. It’s also the main piece of infrastructure I’m working on as part of my PhD in compiler design at the Université de Montréal. I’ve been working on Higgs for about a year now, and it’s gotten to the point where most of JavaScript ES5 is supported. Along the way, I received help, notably from Tommy Everett, who implemented a user-friendly FFI library to interface with C code, along with proxies for standard C libraries. I’d like to keep making Higgs more complete and useful. This post is an invitation: we’re looking looking for contributors.

The Higgs Roadmap

In the long run, I’d like for Higgs to become useful both inside and outside of the academic community. It’s probably not going to be able to compete on performance with commercial JS engines like V8 and IonMonkey, at least…

View original post 390 more words

Conversations with a time traveler

 

Spring Cleaning (in the Wintertime)

I’ve been going through a lot of my old disks and drives lately trying to get a handle on my digital clutter. It’s been an interesting experience, finding lots of old stuff I forgot about. Years ago I used to frequent a BBS called “The Church of Potent IP” (get it?). It was one of the many proto-4chans of the time, and the content was about what you would expect; a few different sections mostly covering technology and pop culture but also other hobbies/interests. Sometimes the exchanges were quite humorous or thought provoking and I would save them as txt files on floppy disk. I found quite a few of these in my archives, but the following post to “The Pit” (a general discussion area) stood out.

It’s really long (100+ forum “pages”) so I’m reposting it here for posterity in heavily abridged and edited form – with potentially personal information and irrelevent bits trimmed out. I’ve also tried to move the comments around so the answers directly follow the questions and follow a more coherent flow. I don’t seem to have posted in the thread myself; my copy of it ends with a bunch of unanswered posts so I probably arrived after the poster left. As far as I know they never posted again. My editors notes and comments are interspersed.

Post:  So..I’m a time traveler.

TT (op):

Hey guys. It’s friday and I’m bored so I figured I’d tell you.

I’m a young male living in the United States. From my point of view this is the past but I’ve been here so long I’m starting to think of it as my time. I was told the trip would be one way probably so I will probably never go home but its possible I might.

Ask me any questions if you want but you probably won’t be happy with my answers. I can’t tell you the lottery numbers or crap like that.

I’ve removed many predictable “kill yourself”, “yea right”, and “lol” replies here and elsewhere…

HY:

When are you from?

TT(op):

I can’t tell you exactly because I can’t give too much information. It’s pretty far though. More than 200 years.

The poster ignored a lot of questions. It seems anything too open ended like “what’s the future like?” or too specific like “how do I die?” was skipped. They also ignored the many very angry posts calling them out on not answering questions, so I’m just cutting them all out.

IBJ:

Was titor for real?

They’re asking about John Titor, as was clarified in some posts I’m cutting out. He was also brought up a lot in later posts but this is the OP’s only comment on it.

TT(op):

He wasn’t that I know of but he could be I guess. I seriously doubt it though because I never heard of him before now and things he said. The scientist guys explained to me that “time travel” is the wrong thing to say. They told me they were going to project or translate me into the past and that was what it felt like. They drugged me and then I just kind of woke up here. I didn’t come in a truck lol.

This sparked a pretty big and occasionally heated discussion, but it was ignored by OP.

PX:

“scientist guys”?

TT(op):

Ya the guys who sent me back. They are scientists lol. Its hard to explain but they are kind of like a click or maybe a cult. All scientists are like that in my time. They are really annoying. There are a lot of cults in my time but they’re not really religious. It’s hard to explain.

I believe they meant “clique”…

PB:

You live in the US now? Did you live there in the future? What’s your native language?

TT(op):

Yea I live in the US. That’s where I woke up and I’ve stayed here. I’m an illegal alien lol. In my time people move around alot and you wouldn’t understand our maps. We don’t have countries like you do, they’re different. The last question is hard to explain. People mostly speak creole so my native language is mostly english and spanish but also some cantonese and hindustani. I also know some slang from a bunch of other languages from talking to other kids. Really people in my time mostly don’t speak. It’s hard to explain but with technology we don’t need to.

There was some interesting discussion about language that followed, but it’s spread out over a lot of posts so I’ll just include the gist. It seems in the future language is very fragmented. Mandarin Chinese is almost entirely obsolete. Japanese, Korean, and Russian will merge somewhat as well as Spanish, English, and German. OP made a reference to some people speaking a hybrid of Farsi, Arabic, and Portuguese as well as one of Afrikaans, Cantonese, and French. They also mentioned that “only small groups of people can understand eachother”, but did not elaborate on that. The OP didn’t give specific reasons for any of this but implied it was mostly due to gradual cultural/policitcal shifts with a few major global events.

MP:

you keep saying “hard to explain”… seems like a cop out

TT(op):

lol it is! Try talking to someone from long ago. There’s so much you need to explain for one thing then you need to explain a bunch of others. On top of that there’s a bunch of stuff you can’t tell them. I don’t feel like doing it.

PB:

What if you change history or something?

TT(op):

I don’t think I can. I asked the scientist guys about it and they said “why are you planning on it” and laughed at me. I said I was worried about accidents but they didn’t seem worried about it. One said “you can’t fall from the top of a mountain to the bottom” and they laughed at me some more. They always talk all zen like that and it’s really annoying. I’m not gonna take any chances though and just keep my head down. It can’t be totally right because I was sort of sent here to change history.

MP:

Wait, your here to change history? Also keeping your head down? wtf do you call getting online and telling the world your a time traveler?

TT(op):

I’m supposed to do this lol. Technically I was sent back as a test I guess but also this famous guy asked me to do something. He’s pretty powerful. He wanted me to make sure a certain part he needed ended up in a certain spot. It was kind of hard because I had to steal it from man airport but I did it. I don’t know why he wanted that one because we can make anything
in our time so he didn’t really need it. He’s kind of crazy though and was obsessed with having an original one. Most people in my time are pretty crazy I think but he’s out there. He’s like a celebrity for us I guess. You guys actually know him in your time lol. He’s dead but someone in my time reconstructed him and people went nuts for it. I don’t know why tho he just kind of babbles a lot and complains about his ex wives, he is pretty funny sometimes though.

I’m removing a lot of questions such as “what was the part?” and “who is it?” that the poster ignored. They also refused to comment further on their motivations for posting besides that they were “bored”.

CT:

tell us who really killed JFK!

I should note that a lot of these questions/answers drew speculation and heated debate from other posters. I’m cutting it all out along with basically anything that’s not a question. There’s just too much of it, and most of it is fairly predictable. It’s also important that there was a section of the BBS devoted to conspiracy theories so there were a lot of questions like this.
I imagine a lot of the participants were from that section, hence the surprising amount of credulity in the face of a pretty silly claim.

TT(op):

lol that one I actually know. Before I came here I did a little research on this time so I wasn’t totally ignorant. I also learned about it in the school I go to here. It was the Oswald guy. He did the killing pretty much alone but he had some help with other stuff. There was a conspiracy but it’s not like you guys seem to think. It’s just that a lot of stuff was covered up because it was embaressing to some people. Think something kind of like patty hearst not the cia killing him or anything.

NW:

What is neurocam?

As I recall neurocam was some website a lot of conspiracy forums of the time were speculating about. I think it was some kind of RPG or ARG that they were convinced was real.

TT(op):

How should I know. From the webpage I think it’s an art project. lol

I’m removing a lot of speculation about it and I’m gonna cut out any further conspiracy-related questions since the OP didn’t comment on them.

PX:

How does your time machine work?

TT(op):

I don’t know lol. Whenever I asked about it the scientist guys got kind of weird. I kept asking and finally they just kind of flipped and said only an idiot would send someone who actually understands it back. I think they felt bad though because they did tell me a bit about it. They just said a bunch of stuff about fractional dimensions and zeno and something about wei’s work on heisenburg and I didn’t really get it. Then when they saw I didn’t get it they laughed at me some more. Those guys are dicks.

There was some interesting discussion that followed this, but most of the OPs comments were along the lines of “I don’t know” or “I can’t tell you”. I wouldn’t be able to judge whether most of the questions/posts were cogent or intelligent anyway to be honest, so I’m just goint to cut them out. As far as I know they were pretty standard re: discussions about the physics of time travel. The main upshot of the exchange was that the OP was very emphatic about gravity not being involved in the process “per say” (sic); but that tremendous amounts of energy were.

MOH:

This is nuts. YOu go to school?

TT(op):

Yea. I live a pretty normal life now lol. One of the reasons they sent me is because I’m young and it was easy to just slip in somewhere and say I ran away from home because they beat me. Working on getting a ssn but that’s hard lol.

MEL:

I gotta know…do you have sex robots?

TT(op):

lol yea. It’s hard to explain tho because it depends on what you mean by robot. You would probably think alot of the people from my time are robots. We do have robots that are just for sex though.

 

…in which I give up

It’s getting rather tedious to try and edit this big discussion down into something digestible. I’m going to pack it in here and consider just posting the raw logs or maybe posting more selections at another time. It’s just all fascinating to me because it reminds me of all the exchanges on old BBS and such that have probably been lost forever, or are perhaps still sitting gathering dust somewhere on diskettes in a closet. Even in a society with a great capacity for storing massive amounts of information, we lose so much of it. It makes me wonder about all the events and people just totally lost to history throughout the ages. I’ll consider posting some of the other threads I saved as well, but this is probably the most interesting one.

Higgs FFI: Calling C From JS

My friend Maxime Chevalier-Boisvert has been working on a JavaScript VM called Higgs as part of her research into dynamic language optimization. In my free time I’ve been playing with it and using it for various toy projects such as a window manager, games, etc. In this and some upcoming posts I’m going to talk about the work I’ve been doing on and with Higgs.

An implementation of JavaScript on its own is not terribly useful for someone like me. The language doesn’t provide anything in the way of file I/O, graphics, access to the system, etc; the sorts of things the programs I want to write generally require. That’s something Maxime and I had been talking about and we agreed Higgs needed a FFI to interface with C code. This way we could just wrap existing functions/libraries instead of implementing the functionality from scratch.

Recipe For Iron Chef

Interfacing with C code is pretty simple in terms of the basic principles. C programs follow a specific calling convention defined by the platform’s ABI. The fiddly parts are dealing with different platforms and figuring out how to match the semantics and implementation of your language with those of C code.

Higgs fortunately makes things pretty easy. For one, it currently focuses on x86_64 POSIX systems only; so we don’t have to spend time duplicating code to work with Windows DLLs and calling conventions, or x86, etc. Higgs also has a JIT compiler, which means it comes with an assembler. Another nice thing is quite a bit of the Higgs runtime is written in JavaScript and we can use the same facilities to write much of the FFI.

The FFI for Higgs is underpinned by a few low level ops that are implemented in D as part of the interpreter. These allow the dynamic loading of a library, looking up symbols in the library, calling functions in the library, etc.

The following code demonstrates using these low level ops to call malloc:

// Load library
var l = $ir_load_lib("");

// Lookup malloc
var malloc = $ir_get_sym("malloc");

// Call malloc
var ptr = $ir_call_ffi(null, malloc, "*,i32", 128);

Behind the scenes, $ir_load_lib uses dlopen; passing an empty string allows access to the global symbol object of the current program (the Higgs interpreter). Next, we need to find the location for the malloc function. This is accomplished with the $ir_get_sym op; which uses dlsym. Finally, once we have an address for the function we want to call we use $ir_call_ffi to actually call it.

This last step is the most involved. The first argument is a placeholder, the second argument is the address of the function to call, the third argument is a string representing the return and argument types of the function; any following arguments are passed to the function being called. $ir_call_ffi uses this information to generate a bit of assembly (a CodeBlock) which sets up shuttling arguments out of the JS interpreter, calling the C function, and getting the return value into the JS interpreter. After it’s generated, the CodeBlock is placed in the placeholder slot and will be used for subsequent calls.

One Step Beyond

That’s a lot of code just to call one function. Fortunately we have a library to generate a little bit of wrapping code, so we can make this process easier:

// import ffi library
var ffi = require("lib/ffi");

// Load library
var c = ffi.load("");

// Lookup malloc, generate wrapper function
c.fun("malloc", "*,i32");

// Call malloc
var ptr = c.malloc(128);

Steal This API

That’s better, but still a bit more work than we’d like to do. LuaJIT has a really nice FFI library and the Higgs FFI borrows heavily from its API. The Higgs FFI library has a simple parser for C declarations and can generate wrapper code for them.

Now all we need to do is:

var ffi = require("lib/ffi");

ffi.c.cdef(["void* malloc (size_t size);"]);

var ptr = ffi.c.malloc(128);

In addition to wrapping functions, the Higgs FFI can wrap data. Recently I was experimenting with pipes for IPC using the pipe function. The pipe function expects an empty array of 2 ints, which it will populate with the file descriptors for your pipes. We could allocate the memory ourselves, but the Higgs FFI knows how to create and wrap C arrays for us. One way would be to use the functions provided by the FFI library, but we can take advantage of some automatic wrapping with a typedef:

var ffi = require("lib/ffi");

ffi.c.cdef([
    "typedef int PipePair[2];",
    "int pipe(int pipefd[2]);"
]);

// Create an array of 2 ints
var pipes = ffi.c.PipePair();

// Call the pipe function and pass a ptr to our array
ffi.c.pipe(pipes.handle);

// Get the file descriptors for our pipes
var in_pipe = pipes.get(0);
var out_pipe = pipes.get(1);

The Higgs FFI can also wrap other types of data like structs. This makes it fairly easy to quickly work with and write wrappers for C libraries:

var ffi = require("lib/ffi");
var console = require("lib/console");

var foo = ffi.load("/path/to/mylib.so");

foo.cdef([
    "struct account { int num; double balance; char name[10]; };",
    "typedef struct account customer;",
    "customer *bar();"
]);

var bob = foo.account(foo.bar());
console.log("Account:", "#" + bob.get_num());
console.log("Balance:", "$" + bob.get_balance().toFixed(2));
console.log("Name:\t", bob.get_name());

What’s Next

The Higgs FFI is still under development and in a bit of flux. I’ve been implementing it as needed, so currently it can only handle the use cases I have come across in my experiments. Going forward I want to increase its ability to do automatic wrapping, improve the API, clean up the code, and so on.

I also plan on using the FFI to write a lot of bindings to allow writing all sorts of programs with Higgs. We already have a library for file I/O and some standard C functions like popen. I’m working on even more bindings for things like the file system, SDL, X11, etc.

If this area interests you, contributions are more than welcome. If you’d like to try writing bindings for your favorite C library, contribute to the FFI, or anything else; feel free to contact me or Maxime with any questions, comments, or anything else.

On holistic programming languages.

My friend Maxime recently blogged about “Programming Without Text Files”. I’ve always wanted to be one of those bloggy people, and being fairly lazy and uncreative I’ve decided to shamelessly purloin the topic for my inaugural post. I figured I’d share some of my own thoughts in this area of “programming without text files” and beyond.

So, what is “programming without text files” anyway? What does it mean for a programmer to “[edit] the underlying data structures directly?” I think that this is probably a subtle distinction, but a potentially important one.

All programming is ultimately done by manipulating an AST – what difference is it if we do it “directly”? After all, IDEs can give us a lot of nice features (autocompletion, autorefactoring, etc) so that we are not always laboriously typing out a textual representation of our program. Isn’t writing code which essentially “generates” an AST and then using all these fancy IDE features conceptually the same thing?

I think both conceptually and practically there are differences. Unless you’re a lisper there is generally a few layers between you and the AST. IDEs have to work hard for these features. They scan, munge, parse, rescan, wash, rinse, and repeat…and then all this work is lost when we send our program off to the compiler/VM/linter/whatever where the cycle repeats.

I also think on a certain level the way many programmers conceptualize programs and the way our tools accept them are very similar – they are a series of letters, numbers, and “punctuation”. For us, it’s translated to an AST almost as an afterthought; it’s just something the compiler does as part of its magic. We catch glimpses of it here and there when we use fancy IDE features like code folding, syntax checking, etc …but it remains obscured by the maya of text files. We still tend to think of programs as some English (or what-have-you) text.

Was the programming work-flow really perfected decades ago? What if we change the way we think about and treat programs? Can we make a language where programming is a first class citizen?

It reminds me of the debate that ensued when Zed Shaw chose to use sqlite databases to store the configuration for Mongrel2 instead of text files. It makes sense – a db is much easier to query and manipulate programmatically. Yet many objected that they were not “human readable”, or worried that their text-focused tools would have trouble dealing with a db.

I could see similar objections here – “If my program is some opaque binary blob of an AST or whatever, how am I supposed to check it into git? How am I supposed to git diff etc?”. I would answer: “Maybe you don’t need to!”

When you start thinking of your programs holistically all this stuff that was “metadata” becomes just data. It’s all a part of your program. You don’t need to worry about cluttering up your pretty little text file anymore; you have a “living, breathing” data structure – you can store revision information for nodes in the nodes themselves if you want to. You can store all sorts of information, and have your program displayed with as much or little detail as you please.

Now that we’re not operating on mere lines of text, your revision control can be aware of the semantics of your language. I can get a revision history of any/all code that touches the I/O lib without a bunch of grep/awk/sh massaging? Yes please!

*.pyc files? F that! Let program and byte code live harmoniously. Let me see the AST and byte code side by side. Let programs easily hack their own binaries if they want. Cache generated code and data in the program file itself, why not?

Tired of slow coworkers who need everything spelled out? Or coders who write comments like Ayn Rand writes novels? We can have granularity in our comments…and no need to come up with some elaborate markup, and then a tool to parse it, and then integrate it with the editor. We already have a model of programming that is very explicitly about this sort of tree-walking/manipulation. Give each comment a “owner”, “level”, etc and flip a switch for “ELI5” or “pithy” comment display. Have a conversation in the comments, heck upvote comments or attach rage faces to bad code.

I’m editing on Linux, Y U NO just hide the Windows code!

No need to scroll through hundreds of lines of code you don’t care about, dealing with nonsense like:

    #ifdef WINDOWS_BLAH

Give me an editor that recognizes my code is fractal – none of this “jump to definition” nonsense: I want infinite zoom. Give me a real modal editor – not these piddling “insert” and “command” modes. I want trying-to-understand-code-someone-else-wrote-mode; I want bang-out-code-as-quickly-as-possible-mode; I want debugging-mode; I want writing-documentation-mode; I want security-audit-mode; I want i18n-mode.

In my opinion, this is all just the beginning of really interesting things you could do. I’ve mostly focused on the programmer/user experience, but such a change could make the lives of compiler/tooling authors easier as well.

I don’t pretend to fully understand the trade-offs involved in such a shift – to be sure there are drawbacks.  I’m also not about to step up and volunteer to implement it (though I’d certainly offer to help) – it’s quite a task. However, I am suspicious of the idea that we really can’t do better than plain text files.

These ideas have been explored before (smalltalk, lisp, MPS, etc), indeed versions of most of this stuff can be found around …but perhaps the computing world of today is the time and place to start taking them more seriously and revisit them. We live in a world of powerful computers and improved interfaces which could make this more practical.

To be sure, this stuff is not prohibited by our text files – it just becomes easier and more obvious when we discard them. Perhaps it is time we throw down our text shackles, and code free.