The other facet of Metalua: making IDEs smarter

Metalua is basically a tool to manipulate Lua programs in Lua. There are two main possible applications of it:

• using it as a self-extensible language, a.k.a. "Lisp without the parenthetical speech impediment";
• using it to analyze, and possibly modify, plain Lua source code.

I find the first application to be the funnier one, by far. It also raises plenty of software engineering open problems, about dynamic parsers, macro composition, hygiene, multiple views of a same object, etc. However, it remains a niche within a niche. Yet the later application, static source analysis and transformation, has a much wider potential audience. Java IDEs have transformed the expectations of many developers; we now expect lot of intelligence and assistance from and IDE, and it requires a deep static understanding of the programs being written. This is very tricky with dynamic languages such as Lua: with Java, you spend tremendous amounts of time making your program intelligible to the type system, with declarations, adapters, interface implementations etc. All this tedious bookkeeping is reused by the IDE to understand your programs. Dynamic languages free you from all this, but that leaves the IDE mostly clueless. So without statically checked types, either your IDE can know and do very little about your programs, or it has to make wild guesses based on heuristics ("heuristics" being an euphemism for "algorithmic-like thingy which sometimes fails, without even realizing how badly it failed"). The incentive to have those heuristics for Lua manipulations written in Lua, and easily modified, is huge: you want Lua hackers to tweak their own heuristics, in the language they all know--Lua. If they need to learn Eclipse, DLTK, XText, your own Lua framework, and possibly Java to describe their peculiar way of declaring a module, they simply won't: they'll keep using the IDE as a Notepad with syntax highlight. Providing a better IDE support Here comes the Metalua-based solution: interface your IDE with Metalua; let the latter tell the former what the code means, and perform any refactoring asked by the user. If Metalua is usable enough for your above-average-but-not-quite-a-wizard Lua hacker, then you can expect interesting things to happen. I believe such interesting things are indeed about to happen:

• my company is about to release a Lua support plugin for Eclipse, based on Metalua, as part of the wider Eclipse project Koneki.
• I'm working on making Metalua more accessible for source-to-source analysis and transformation.

Metalua makes Lua source file manipulations easier, by turning them into AST. unfortunately, visiting and modifying these trees is still a bit tedious. Moreover, such trees are easy to compile into bytecode, but not to convert back into proper source code. Of course, transforming an AST into a syntactically valid source code is trivial, and Metalua does this. But when an AST has been generated from a source file, modified, and redumped, you want the resulting file to keep as much of the original formatting as possible: comments, spaces, line jumps, syntax sugar etc. You'd also like this happen automatically: AST are supposed to be easy to parse; if keeping all the formatting details accurate is a chore, AST lose most of their appeal.

The two needs identified for IDE support are therefore:

• a robust, readable, intuitive tree visitor library;
• a robust "source -> AST -> refactored AST -> back to source" round-trip converter.

I'm currently focusing on the first points, although I think I've also got a rather elegant solution fo the second one. It's called TreeQuery, it's inspired by declarative tree visitors such as XPath or jQuery, and it defines a robust and readable Domain-Specific Language in Lua. It will be the subject of my next post.