Wednesday, October 12, 2011

The other facet of Metalua: making IDEs smarter

Metalua is basically a tool to manipulate Lua programs in Lua. There are two main possible applications of it:
  • using it as a self-extensible language, a.k.a. "Lisp without the parenthetical speech impediment";
  • using it to analyze, and possibly modify, plain Lua source code.
I find the first application to be the funnier one, by far. It also raises plenty of software engineering open problems, about dynamic parsers, macro composition, hygiene, multiple views of a same object, etc. However, it remains a niche within a niche. Yet the later application, static source analysis and transformation, has a much wider potential audience. Java IDEs have transformed the expectations of many developers; we now expect lot of intelligence and assistance from and IDE, and it requires a deep static understanding of the programs being written. This is very tricky with dynamic languages such as Lua: with Java, you spend tremendous amounts of time making your program intelligible to the type system, with declarations, adapters, interface implementations etc. All this tedious bookkeeping is reused by the IDE to understand your programs. Dynamic languages free you from all this, but that leaves the IDE mostly clueless. So without statically checked types, either your IDE can know and do very little about your programs, or it has to make wild guesses based on heuristics ("heuristics" being an euphemism for "algorithmic-like thingy which sometimes fails, without even realizing how badly it failed"). The incentive to have those heuristics for Lua manipulations written in Lua, and easily modified, is huge: you want Lua hackers to tweak their own heuristics, in the language they all know--Lua. If they need to learn Eclipse, DLTK, XText, your own Lua framework, and possibly Java to describe their peculiar way of declaring a module, they simply won't: they'll keep using the IDE as a Notepad with syntax highlight. Providing a better IDE support Here comes the Metalua-based solution: interface your IDE with Metalua; let the latter tell the former what the code means, and perform any refactoring asked by the user. If Metalua is usable enough for your above-average-but-not-quite-a-wizard Lua hacker, then you can expect interesting things to happen. I believe such interesting things are indeed about to happen:
  • my company is about to release a Lua support plugin for Eclipse, based on Metalua, as part of the wider Eclipse project Koneki.
  • I'm working on making Metalua more accessible for source-to-source analysis and transformation.
Metalua makes Lua source file manipulations easier, by turning them into AST. unfortunately, visiting and modifying these trees is still a bit tedious. Moreover, such trees are easy to compile into bytecode, but not to convert back into proper source code. Of course, transforming an AST into a syntactically valid source code is trivial, and Metalua does this. But when an AST has been generated from a source file, modified, and redumped, you want the resulting file to keep as much of the original formatting as possible: comments, spaces, line jumps, syntax sugar etc. You'd also like this happen automatically: AST are supposed to be easy to parse; if keeping all the formatting details accurate is a chore, AST lose most of their appeal.
The two needs identified for IDE support are therefore:
  • a robust, readable, intuitive tree visitor library;
  • a robust "source -> AST -> refactored AST -> back to source" round-trip converter.
I'm currently focusing on the first points, although I think I've also got a rather elegant solution fo the second one. It's called TreeQuery, it's inspired by declarative tree visitors such as XPath or jQuery, and it defines a robust and readable Domain-Specific Language in Lua. It will be the subject of my next post.


Emmanuel said...
This comment has been removed by the author.
Emmanuel said...

"Nice, sounds exciting. I work mostly with dynamic languages inside vim these days, and seldom miss a (working) autocompletion feature."

Hmmm I guess syntactic analysis is only just half what's required to get autocompletion in a dynamic language. For instance for a language like ruby you'd also need to know the semantics of how methods are looked up.

Fabien said...

You always need to know method lookup semantics; the catch is, sometimes you can't. As soon as a language supports introspection, dynamic extension and other variants of monkey-patching, you can't make a perfect job, you can merely handle the most common idioms through heuristics.

(Notice that Java support becomes less than perfect if you start playing with custom class-loaders, too).

Lua is even worse than Ruby, it encourages a build-your-own-dispatch-semantics approach to OO (and it's sometimes almost as bad as it sounds).

That's why making it reasonably doable to extend/modify the heuristics is important: if people make their own dispatch semantics, they'll want their own object hierarchy analysis tool.

Such tools won't be trivial to write, though, but that's a good think IMO: it will encourage people to stick to the couple of well supported OO models, instead of gratuitously rewriting their own variants.

Peter Odding said...

I'm very interested in better text editor / development environment integration for Lua 5.1 and know of a few related projects that may be interesting to people programming in Lua and looking for a better development environment:

LuaInspect (by David Manura) builds upon Metalua to perform static analysis of Lua 5.1 source code. It's integrated with SciTE and Vim (the latter is a Vim plug-in written by me). Note that I'm not all that happy with the Vim plug-in but hey, it's a start :-). I also maintain a Lua file type plug-in for Vim that makes it easier and more fun to work with Lua in Vim.

In my free time I'm currently also working on a type inference engine for Python, with the goal of providing awesome completion support in Vim using only static source code analysis. Based on my experiences with Python I'm considering doing something similar for Lua, and I'm guessing Metalua would be the best foundation for such an undertaking.

By the way thanks Fabien for creating and maintaining Metalua, it provides the basis for all of these tools and without Metalua the Lua ecosystem wouldn't be where it is today!