XFormula is a modular language front-end and parser generator for building domain-specific languages (DSLs).
A language rarely stays small. A new literal becomes a new expression. An optional syntax becomes a dialect. An experimental feature needs to be enabled without rewriting unrelated grammar. An AST node needs to participate in the language without being manually threaded through every production.
XFormula is designed for that kind of evolution: build languages as systems, not as monolithic grammars.
Instead of treating the grammar as the language's central source of truth, XFormula lets independent "features" contribute syntax, grammar, AST nodes, and semantic transformations. The language is assembled from those features.
Under the hood, XFormula generates an EBNF grammar for Lark Parser Toolkit. Around that grammar, it provides the type system, object model, transformation logic, and customization layer needed to build a composable language; so one can add language features without turning the grammar into the place where the whole language has to be known.
For flexibility, Lark supports LALR(1), Earley, and CYK parsing algorithms, and XFormula's default features are additionally designed to be compatible with the LALR(1) algorithm, since it is known for its speed and efficiency in both time (CPU) and space (memory).
A feature that should be independent becomes another production. A production needs to know about every possible alternative. AST construction becomes tied to grammar structure. Optional syntax creates conditional branches. Removing a feature means finding and correcting every place where the grammar and transformations know about it.
The language becomes difficult to evolve because its pieces are no longer independent.
XFormula takes the opposite approach. A feature can declare:
- the tokens it introduces,
- the grammar constructs it provides,
- the AST nodes those constructs produce,
- and the transformations that connect parsed input to semantic objects.
Those declarations are composed through a shared syntax context. The central grammar does not need to enumerate every concrete feature. That means a language can evolve by composition rather than by continually rewriting its grammar.
XFormula separates three concerns that are often mixed together in parser implementations:
- Lexical syntax: What source text should be recognized as a token?
- Grammar: How are those tokens composed into language constructs?
- Semantic transformation: How does the parse result become an AST node?
A "feature" can participate in all three.
Source text
│
▼
┌────────────────┐
│ Lexical syntax │ What is this?
└───────┬────────┘
│
▼
┌───────────────┐
│ Grammar │ How does it compose?
└───────┬───────┘
│
▼
┌────────────────────┐
│ Semantic transform │ What does it mean?
└──────────┬─────────┘
│
▼
AST
This separation gives each feature a clear responsibility while allowing the language as a whole to remain compositional.
The key mechanism behind the modularity provided by XFormula is dynamic syntax composition.
A syntax feature does not need to modify a central grammar definition. Instead,
it contributes definitions to a shared
SyntaxContext.
Other definitions discover those contributions through "tags" and "priorities."
For example:
BOOL token ───────────> Bool ─────┐
│
NONE token ───────────> None_ ────┤
│
└──> Literal ───> Start
Literal does not need to know that Bool and None_ exist.
The concrete features declare their relationship to Literal. XFormula
assembles the resulting grammar from those declarations. Adding another literal
therefore extends the same language structure without requiring the existing
Literal implementation to be rewritten.
This is useful when a language has:
- optional syntax,
- experimental features,
- dialects,
- domain-specific extensions,
- independently developed language features.
The main building blocks behind this mechanism are:
Syntax composition is only useful if the resulting semantic model composes as well.
XFormula represents relationships between language constructs directly in the Python type hierarchy.
For example:
Bool
└── Literal
└── Term
└── Primary
└── Operand
└── SimpleExpression
└── Expression
└── Node
A parsed Bool is therefore not merely a concrete parser result. It is also a Literal, a Term, an Operand, an Expression, and a Node.
This allows generic operations to work against abstractions such as Literal,
Expression, or Node without requiring every concrete syntax feature to be
explicitly registered.
For one more example:
parser.parse("none").__class__.__mro__produces an inheritance chain similar to:
None_
Literal
Term
Primary
Operand
SimpleExpression
Expression
HasValue
Node
Configurable
ABC
object
The important consequence is that a syntax feature can specialize the common language model without losing its semantic identity.
XFormula is not merely:
- A collection of regular expressions.
- A static grammar file.
- A wrapper around Lark.
- A parser with a different API.
Rather, it is:
- A modular language front-end. See the list of default operators and precedences.
- A parser generator built around feature composition.
- A system for composing lexical syntax, grammar, AST nodes, and transformations.
- A generator of EBNF grammars for Lark.
- A typed object model for language constructs.
- A customization layer for evolving DSLs.
For demonstration purposes, we will re-build the two literals bool and
none:
true
false
none
and turn them into these AST nodes:
Bool(value=True)
Bool(value=False)
None_(value=None)pip install xformulaA terminal describes how the lexer recognizes source text and how that token
is transformed at runtime.
The none token:
from xformula.runtime.core.context.abc import RuntimeContext
from xformula.syntax.grammar.ebnf import non_terminal
from xformula.syntax.grammar.terminals.abc import Terminal
from xformula.syntax.lexer.tokens.abc import Token
class NONE(Terminal[None]):
class Meta:
priority = 2000
tags = {
non_terminal("None"): 0,
}
def build_grammar(self) -> str:
define = self.ebnf.define
regex = self.ebnf.regex
bound = self.regex.bound
word = self.regex.word
return define(regex(bound(word("none"))))
def transform_token(
self,
runtime_context: RuntimeContext,
token: Token,
) -> None:
return NoneThe boolean token works in exactly the same way:
class BOOL(Terminal[bool]):
class Meta:
priority = 2000
tags = {
non_terminal("Bool"): 1000,
}
def build_grammar(self) -> str:
define = self.ebnf.define
regex = self.ebnf.regex
any_of = self.regex.any_of
bound = self.regex.bound
word = self.regex.word
return define(
regex(
any_of(
bound(word("false")),
bound(word("true")),
),
),
)
def transform_token(
self,
runtime_context: RuntimeContext,
token: Token,
) -> bool:
return token.value.lower() == "true"The regular expressions are the easy part. The important boundary is the one between syntax recognition and typed runtime values.
"true"
↓
BOOL token
↓
bool("true")
↓
Bool(value=True)
For literals, XFormula provides a generic Literal node that can be
specialized for different value types.
import dataclasses
from xformula.syntax.ast.nodes import Literal
@dataclasses.dataclass()
class None_(Literal[None]):
value: None = dataclasses.field(
kw_only=True,
init=False,
default=None,
)@dataclasses.dataclass()
class Bool(Literal[bool]):
value: bool = dataclasses.field(
kw_only=True,
default=False,
)The AST stores the semantic value rather than the original source representation.
A non-terminal describes how a grammar construct is assembled and, when
necessary, how its parse tree should be transformed.
For None:
from xformula.runtime.core.context.abc import RuntimeContext
from xformula.syntax.core.features.literals.ast.nodes import None_ as NoneNode
from xformula.syntax.grammar.ebnf import non_terminal
from xformula.syntax.grammar.non_terminals.abc import NonTerminal
from xformula.syntax.parser.trees.abc import ParseTree
class None_(NonTerminal[NoneNode]):
class Meta:
definition_name = "None"
atomic = True
tags = {
non_terminal("Literal"): -1000,
}
def build_grammar(self) -> str:
return self.ebnf.define_tagged_alternation()
def transform_parse_tree(
self,
runtime_context: RuntimeContext,
tree: ParseTree,
) -> NoneNode:
return NoneNode()The Bool non-terminal can consume the already transformed value produced by
BOOL terminal:
from typing import cast
from xformula.runtime.core.context.abc import RuntimeContext
from xformula.syntax.core.features.literals.ast.nodes import Bool as BoolNode
from xformula.syntax.grammar.ebnf import non_terminal
from xformula.syntax.grammar.non_terminals.abc import NonTerminal
from xformula.syntax.parser.trees.abc import ParseTree
class Bool(NonTerminal[BoolNode]):
class Meta:
atomic = True
tags = {
non_terminal("Literal"): -2000,
}
def build_grammar(self) -> str:
return self.ebnf.define_tagged_alternation()
def transform_parse_tree(
self,
runtime_context: RuntimeContext,
tree: ParseTree[bool],
) -> BoolNode:
value = cast(bool, tree.children[0])
return BoolNode(
value=value,
)Finally, Literal collects whatever definitions have tagged themselves as
Literal:
from typing import TypeVar, cast
from xformula.runtime.core.context.abc import RuntimeContext
from xformula.syntax.ast.nodes.abc import Literal as LiteralNode
from xformula.syntax.grammar.ebnf import non_terminal
from xformula.syntax.grammar.non_terminals.abc import NonTerminal
from xformula.syntax.parser.trees.abc import ParseTree
T = TypeVar("T")
class Literal(NonTerminal[LiteralNode[T]]):
class Meta:
tags = {
non_terminal("Start"): -1,
}
def build_grammar(self) -> str:
return self.ebnf.define_tagged_alternation()
def transform_parse_tree(
self,
runtime_context: RuntimeContext,
tree: ParseTree[T],
) -> LiteralNode[T]:
return cast(LiteralNode, tree.children[0])Notice what is missing. Literal does not enumerate Bool and None_. It
does not need to know which literal features exist. The concrete features
declare their relationships, and XFormula assembles the language from those
declarations. That is the base of the compositional model.
A feature is the unit XFormula uses to compose language functionality.
from xformula.syntax.core.features.abc import Feature
class LiteralFeature(Feature):
def setup(self) -> None:
self.non_terminal_types.extend(
[
None_,
Bool,
Literal,
],
)
self.terminal_types.extend(
[
NONE,
BOOL,
],
)This is the point where the pieces become a language component. A feature can now be enabled, combined with other features, or omitted entirely.
The parser is created from a SyntaxContext.
from xformula.syntax.core.context import SyntaxContext
from xformula.syntax.core.features.polyfill import PolyfillFeature
from xformula.syntax.parser import Parser
syntax_context = SyntaxContext(
feature_types=[
LiteralFeature,
PolyfillFeature,
],
)
parser = Parser(
syntax_context=syntax_context,
)Now the language can be used:
ast = parser.parse("true")
print(ast)
# Bool(value=True)
print(ast.value)
# True
print(parser.parse("none"))
# None_(value=None)The generated grammar is also available:
print(parser.ebnf_document)For this example, the result is approximately:
?start : literal
?literal : bool
| none
bool : BOOL
none : NONE
BOOL.2000 : /\bfalse\b|\btrue\b/
NONE.2000 : /\bnone\b/Note that the grammar is an artifact of the feature composition rather than the primary source of truth.
XFormula intentionally uses non-atomic non-terminals where no custom transformation is required.
Consider:
?start : literal
?literal : bool | noneThe leading ? tells Lark that these rules do not need to create an additional
tree node, so XFormula can use intermediate grammar rules to compose the
language without forcing those rules to become unnecessary AST layers.
This keeps the resulting AST focused on semantic constructs rather than
implementation details of the grammar.
Also, the PolyfillFeature provides missing non-terminals automatically and
non-atomically when they are only needed as structural or tagging points.
XFormula generates EBNF for Lark Parser Toolkit. This keeps the generated grammar separate from the Python implementation of the language itself. The grammar can therefore serve as an interchange point for environments that support Lark-compatible implementations.
The dynamic transformation behavior provided by XFormula is more specific to the XFormula runtime. Reproducing that behavior elsewhere requires equivalent transformation logic, particularly the automatic operator precedence and associativity handling implemented by:
See the extra features section in the Lark documentation for the available implementations.
django-xformula uses XFormula and its default syntax features to transform formulas into SQL queries through Django's ORM.
Simply, the architecture is:
User formula
│
▼
Parser
│
▼
AST
│
▼
Django ORM expression
│
▼
SQL
This project is licensed under the MIT License. See the LICENSE file for details.