How does Elixir's EEx work?

Elixir's builtin templating library, EEx, is nifty. The more I understand it, the more interesting it becomes. Unlike most of the standard library, the documentation for EEx is sparse. But fear not, all software can be understood.

How do templating engines usually work?

On the surface, EEx is like other templating languages: the API accepts arbitrary text optionally sprinkled with special placeholders that are evaluated with a runtime-supplied object.

They all look like this:

Template.render("Hello {{ @who }}", { who: "world" })
=> "Hello world"

That's where the similarity ends. Most template libraries take the input text and break it into "instructions":

"Hello {{ @who }}!"
      |
      |
      v
[Text("Hello "), Expr("@who"), Text("!")]

They then go through this list, executing the code that matches the instruction using the runtime-provided object to lookup variables, and building up the output string along the way. More complex constructs like loops can be implemented by emitting "goto" instructions and keeping track of a loop counter.

A good, readable example of this is the TinyTemplate.

These libraries "execute" the template as they go, just like interpreters.

EEx on the other hand is more of a compiler. When you call it, it does not output a string. It outputs Elixir code.

EEx outputs Elixir code!

Note that I'll be only talking about the EEx.compile_string/2 function, as all other functions in the EEx module depend on it.

The docs for EEx do explicitely say that this function compiles a string into an Elixir syntax tree. But I was still surprised when I actually internalized it.

Take the string one <%= "two" %> three <%= "four" %> five for example. If you gave the equivalent of this to another templating library , you would expect to get back one two three four five. But that is not what you get from EEx.

iex(5)> EEx.compile_string(~s(one <%= "two" %> three <%= "four" %> five))
{:__block__, [],
 [
   {:=, [],
    [
      {:arg0, [], EEx.Engine},
      {{:., [], [{:__aliases__, [alias: false], [:String, :Chars]}, :to_string]},
       [], ["two"]}
    ]},
   {:=, [],
    [
      {:arg1, [], EEx.Engine},
      {{:., [], [{:__aliases__, [alias: false], [:String, :Chars]}, :to_string]},
       [], ["four"]}
    ]},
   {:<<>>, [],
    [
      "one ",
      {:"::", [], [{:arg0, [], EEx.Engine}, {:binary, [], EEx.Engine}]},
      " three ",
      {:"::", [], [{:arg1, [], EEx.Engine}, {:binary, [], EEx.Engine}]},
      " five"
    ]}
 ]}

I'll walk through how to read this indented data structure in another post, but for now it's important to understand that this data structure is Elixir code. The Elixir code you write is actually the textual/human representation of this data. When you execute the textual representation of Elixir code, the compiler transforms it to this data structure. It's just Elixir tuples. Just like any other pieace of data, you can save this to a variable, write it to a file, or even send it over the network.

The docs call this an Abstract Syntax Tree" or a quoted expression. I think that distinction blurs the fact that this datastructure is the language. And the language is data ... Hey! that sounds familiar.

You can see the human representation of this data structure:

iex(8)> EEx.compile_string(~s(one <%= "two" %> three <%= "four" %> five))
        |> Macro.to_string()
        |> IO.puts()
arg0 = String.Chars.to_string("two")
arg1 = String.Chars.to_string("four")
<<"one ", arg0::binary, " three ", arg1::binary, " five">>
:ok

Instead of printing out one two three four five, EEx handed us a sequence of expressions that evaluate to a binary/string with the contents one two three four five. It compiled the string template to executable code.

So how does it work?

The EEx.Compiler module converts the input template into a list of what I'll be calling "chunks". For example

~s(one <%= "two" %> three <%= "four" %> five)

... gets converted to ...

[
  {:text, ~c"one ", %{line: 1, column: 1}},
  {:expr, ~c"=", ~c" \"two\" ", %{line: 1, column: 5}},
  {:text, ~c" three ", %{line: 1, column: 17}},
  {:expr, ~c"=", ~c" \"four\" ", %{line: 1, column: 24}},
  {:text, ~c" five", %{line: 1, column: 37}},
  {:eof, %{line: 1, column: 42}}
]

Text outside the special markers is represented as a tuple tagged with :text. Elixir expressions inside the special markers are tagged with :expr.

The docs don't explicitly mention this, but EEx also supports block syntax. Tuples tagged with :start_expr, :middle_expr and :end_expr are used for this case. You can do this:

"""
Listing:
<%= for post <- posts do %>
  Name: <%= post.title %>
  Href: <%= post.href %>
<% end %>
"""

... or even this ...

"""
<%= if true do %>
  "something"
<% else %>
  "something else"
<% end %>
"""

The last example gets converted to the following chunks:

[
  {:start_expr, ~c"=", ~c" if true do ", %{line: 1, column: 1}},
  {:text, ~c"\n  \"something\"\n", %{line: 1, column: 18}},
  {:middle_expr, [], ~c" else ", %{line: 3, column: 1}},
  {:text, ~c"\n  \"something else\"\n", %{line: 3, column: 11}},
  {:end_expr, [], ~c" end ", %{line: 5, column: 1}},
  {:text, ~c"\n", %{line: 5, column: 10}},
  {:eof, %{line: 6, column: 1}}
]

The EEx.Compiler.compile function calls the similarly named function in the EEx.Engine module for each of these chunks, pasing in the chunk as well as accumulated state.

In the default EEx.Engine implementation, the accumulated always has this structure:

%{ binary: [ ... ], dynamic: [ ... ], vars_count: num }

This makes sense when you realize that compiling the template always returns code with the following structure:

# The "dynamic" section
# A variable declaration for each non text "chunk" in the template
arg0 = String.Chars( ... )
arg1 = String.Chars( ... )

# The "binary" section
# A binary as the last expression
<<"verbatim text from template", arg0::binary , "text", arg1::binary>>

The vars_count is used to keep a counter to generate the argX variable names. The binary list contains the :text chunks. The dynamic list contains AST/quoted expressions/Elixir code for the expressions inside the special markers.

The job of EEx.Engine is to build up this accumulated state as it's handed the chunks of the template. EEx.Engine.handle_body is called at the very end with the accumulated state in order to build generate the Elixir code. The "secret weapon" that makes this possible is that Elixir already exposes functions that take the textual representation of Elixir code and turn it into data.