On holistic programming languages.

m0003 540My friend Maxime recently blogged about “Programming Without Text Files”. I’ve always wanted to be one of those bloggy people, and being fairly lazy and uncreative I’ve decided to shamelessly purloin the topic for my inaugural post. I figured I’d share some of my own thoughts in this area of “programming without text files” and beyond.

So, what is “programming without text files” anyway? What does it mean for a programmer to “[edit] the underlying data structures directly?” I think that this is probably a subtle distinction, but a potentially important one.

All programming is ultimately done by manipulating an AST – what difference is it if we do it “directly”? After all, IDEs can give us a lot of nice features (autocompletion, autorefactoring, etc) so that we are not always laboriously typing out a textual representation of our program. Isn’t writing code which essentially “generates” an AST and then using all these fancy IDE features conceptually the same thing?

I think both conceptually and practically there are differences. Unless you’re a lisper there is generally a few layers between you and the AST. IDEs have to work hard for these features. They scan, munge, parse, rescan, wash, rinse, and repeat…and then all this work is lost when we send our program off to the compiler/VM/linter/whatever where the cycle repeats.

I also think on a certain level the way many programmers conceptualize programs and the way our tools accept them are very similar – they are a series of letters, numbers, and “punctuation”. For us, it’s translated to an AST almost as an afterthought; it’s just something the compiler does as part of its magic. We catch glimpses of it here and there when we use fancy IDE features like code folding, syntax checking, etc …but it remains obscured by the maya of text files. We still tend to think of programs as some English (or what-have-you) text.

Was the programming work-flow really perfected decades ago? What if we change the way we think about and treat programs? Can we make a language where programming is a first class citizen?

It reminds me of the debate that ensued when Zed Shaw chose to use sqlite databases to store the configuration for Mongrel2 instead of text files. It makes sense – a db is much easier to query and manipulate programmatically. Yet many objected that they were not “human readable”, or worried that their text-focused tools would have trouble dealing with a db.

I could see similar objections here – “If my program is some opaque binary blob of an AST or whatever, how am I supposed to check it into git? How am I supposed to git diff etc?”. I would answer: “Maybe you don’t need to!”

When you start thinking of your programs holistically all this stuff that was “metadata” becomes just data. It’s all a part of your program. You don’t need to worry about cluttering up your pretty little text file anymore; you have a “living, breathing” data structure – you can store revision information for nodes in the nodes themselves if you want to. You can store all sorts of information, and have your program displayed with as much or little detail as you please.

Now that we’re not operating on mere lines of text, your revision control can be aware of the semantics of your language. I can get a revision history of any/all code that touches the I/O lib without a bunch of grep/awk/sh massaging? Yes please!

*.pyc files? F that! Let program and byte code live harmoniously. Let me see the AST and byte code side by side. Let programs easily hack their own binaries if they want. Cache generated code and data in the program file itself, why not?

Tired of slow coworkers who need everything spelled out? Or coders who write comments like Ayn Rand writes novels? We can have granularity in our comments…and no need to come up with some elaborate markup, and then a tool to parse it, and then integrate it with the editor. We already have a model of programming that is very explicitly about this sort of tree-walking/manipulation. Give each comment a “owner”, “level”, etc and flip a switch for “ELI5” or “pithy” comment display. Have a conversation in the comments, heck upvote comments or attach rage faces to bad code.

y u no face

I’m editing on Linux, Y U NO just hide the Windows code!

No need to scroll through hundreds of lines of code you don’t care about, dealing with nonsense like:

    #ifdef WINDOWS_BLAH

Give me an editor that recognizes my code is fractal – none of this “jump to definition” nonsense: I want infinite zoom. Give me a real modal editor – not these piddling “insert” and “command” modes. I want trying-to-understand-code-someone-else-wrote-mode; I want bang-out-code-as-quickly-as-possible-mode; I want debugging-mode; I want writing-documentation-mode; I want security-audit-mode; I want i18n-mode.

In my opinion, this is all just the beginning of really interesting things you could do. I’ve mostly focused on the programmer/user experience, but such a change could make the lives of compiler/tooling authors easier as well.

I don’t pretend to fully understand the trade-offs involved in such a shift – to be sure there are drawbacks.  I’m also not about to step up and volunteer to implement it (though I’d certainly offer to help) – it’s quite a task. However, I am suspicious of the idea that we really can’t do better than plain text files.

These ideas have been explored before (smalltalk, lisp, MPS, etc), indeed versions of most of this stuff can be found around …but perhaps the computing world of today is the time and place to start taking them more seriously and revisit them. We live in a world of powerful computers and improved interfaces which could make this more practical.

To be sure, this stuff is not prohibited by our text files – it just becomes easier and more obvious when we discard them. Perhaps it is time we throw down our text shackles, and code free.



13 thoughts on “On holistic programming languages.

  1. Hello, I’ve found this entry from your friend’s post “Programming without Text Files” (which I found on reddit). I’ve read ideas along the lines of both of your posts for quite a while now, and would like to talk about it with both of you about it (just want to chat, to see if our ideas converge).

    It’s really hard to find like-minded people on this specific subject; most people seem to be quite fine with code-as-text, separation of AST and code among other issues raised by both posts. What they don’t seem to see is that, were those ideas to be shifted toward what we see as “programming without text files”, their workflow would barely (if even) change (let alone change for the worst).

    Either way, firstly, I’m sorry for the bad english. Secondly, I’m looking forward to discuss such ideas with both of you (already contacted Maxime, waiting for reply).

    Keep rockin’ and thanks for the attention, as well for having a different idea of what programming should be in 2013!

    • Thank you for commenting. Your English seems pretty good by the way.

      Yes, I think it’s important to keep in mind that the actual act of inputting code could be very much the same – this would just open up possibilities for alternative ways or new tooling.

      I’ve added a contact form to my about page if you (or any one else) would like to get in touch.

    • Hi there. You say you tried to contact me but I never got your message. Either way, I’d be interested in chatting. Maybe Tom, yourself and I could do a group chat at some point (google hangouts?).

      Tom and I have been discussing various ideas we’d maybe like to integrate into a new programming language, and this is just one of them. We’d be interested in finding people like-minded people to discuss, and perhaps eventually collaborators to help us make these ideas come true.

      • Hi there! I believe I tweeted you (my handle is @knuckvice, but I don’t use Twitter that much), but maybe you don’t receive tweets from people you don’t follow? Either way, Tom has got my email address, and you can get it from him. I emailed him some of my ideas and what I’m working on, but I look forward for a group chat (real-time is better than email in my opinion (yep, I still use IRC)).

  2. Have you had a look at http://mbeddr.com? This is a full-blown C IDE base on Jetbrains MPS which does already most of the things you propose. In addition to that we extended C with e.g. support for physical units, components, statemachines. In addition to that we make use of the projectional nature of MPS. E.g. it is possible (if you want to) to use other notations than text. For example we have used tables to build decision tables. Another example is the way we build product line support. Instead of using IFDEFs from C we have special annotations and different projections. This allows you to look at your source code either in the product line view or only see the code for one given configuration.

    We have published a whole bunch of research papers on that topic:

    A good overview can be found here: http://mbeddr.files.wordpress.com/2010/06/voelteretal-mbeddr-final.pdf

    • No, I haven’t seen it. Jetbrain’s MPS is something I’ve seen and have been meaning to check out in more depth, so I suppose I should get around to that as well.

      It’s interesting that you (obliquely) bring up embedded programming because I was thinking you could do a lot of fun things in this area with a language like Forth (or perhaps Factor).

      I could see an IDE which starts a new “word box” when you type ‘:’. Then with a hotkey bring up a stack display on both the left and the right – the former letting you specify a “stub stack” of values, and the latter showing the stack after the word is evaluated using it; all updated live.

      This would allow a very interactive and test-driven workflow. Specify the starting stack and code until you have a word which results in the stack you want. When you are satisfied another hotkey could turn it into a unit test for the word.

      I can think of a lot of fun ways to interact with and display dictionaries too. It would be well suited to self-modifying programs as well.

      All sorts of ideas you could explore with many different languages. I’m glad other people think so too, and are actually working on it.

      Thanks for the comment and the links; I’ll definitely check them out.

      • Sure 🙂 In case you are interested your contributions are very welcome. mbeddr as well as MPS are both open source under commercial friendly licenses 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s