From AOSP Makefiles to a Custom JIT VM: The 15-Year Evolution of Smart Build Engine

At ExtBit LLC, our core directive is /"Extend the bit, protect your data."/ While much of our current engineering focuses on Web3 security and zero-trust protocol architecture, my foundation in systems engineering was forged at the very bottom of the software stack: build systems.

For over 15 years, I have been obsessed with a single architectural problem: How do you orchestrate the compilation of massive, multi-component systems software without collapsing under the weight of your own build infrastructure?

This is the origin story of the Smart Build Engine (https://extbit.com/smart). It is a journey that spans nearly two decades—starting with the bottlenecked Makefiles of the Android Open Source Project (AOSP), cycling through half a dozen language prototypes, and culminating in the design of a custom JIT virtual machine and proprietary instruction set written in Go.

If you have ever wondered why designing a custom JIT engine is 50x more complex than writing standard application software, this teardown is for you.

1The Catalyst: The AOSP Namespace Collapse (2010)

The origin of Smart can be traced back over 15 years to my time working deep inside the Android (AOSP) GNU Make-powered build system.

Google engineers had accomplished something incredible: they built a modular build management system entirely out of Makefiles. However, Make is fundamentally constrained by a flat, global namespace. Even when utilizing include directives to pull in discrete modules, all variables and rules eventually smash into a single, global execution context.

Modifying these flat Makefiles was agonizing. Tracing dependencies across hundreds of included files felt like defusing a bomb in the dark. A single misplaced variable assignment in a sub-module could silently corrupt the build graph of an entirely unrelated component.

I knew there had to be a way to achieve true modularity, local scoping, and granular execution control without forcing developers to abandon the pragmatic, macro-driven simplicity that made Make so ubiquitous in the first place.

2The Prototype Graveyard: Searching for the Right Engine

Building a reliable build orchestrator requires extreme precision. Over the next decade, I experimented with multiple architectures to escape the flat-namespace trap, leaving a trail of prototypes across my GitHub repositories:

1. The Perl6/PIR Experiment (18 Years Ago): Long before modern tooling matured, I attempted to implement a new Domain Specific Language (DSL) targeting the Parrot Intermediate Representation (PIR) (http://github.com/duzy/smart.pir). Parrot was designed to host dynamic languages, but bridging a build graph onto its VM proved too abstract for the raw file I/O speed required by build systems.
2. The Make-Wrapper Era (14–15 Years Ago): Believing that I could isolate Makefile execution for general-purpose use, I built http://github.com/duzy/smart.mk and its C++ successor, http://github.com/duzy/smart-make. These projects attempted to corral standard Make into a modular structure using sandboxed execution.
3. The LLVM Build Epiphany: While contributing to various large-scale open-source projects, including wrestling with LLVM's massive compilation requirements, it became clear that wrapping existing tools was a dead end.

I needed absolute control over the execution graph. I needed a custom engine.

3Designing the Smart DSL: Balancing Power and Empathy

When I finally set out to build the definitive version of Smart in Golang, I faced a critical design choice: What should the DSL look like?

Engineers hate learning new syntax just to compile their code. I realized that the lowest barrier to adoption was maintaining a syntax structurally similar to Make. The concepts of Macros and Rules are universally understood and highly pragmatic.

The Smart DSL looks and feels like a modern, highly-evolved Makefile, but under the hood, it possesses strict lexical scoping, explicit module boundaries, and dynamic dependency resolution.

But parsing the DSL was only half the battle. Executing it with the speed required for large C++/Go projects required a total architectural rewrite.

4The 50x Complexity Multiplier: Moving from AST to JIT

Initially, the Go-based Smart engine evaluated the DSL using a standard Recursive-Descent interpreter. It walked the Abstract Syntax Tree (AST) node by node. It worked, but it was slow, memory-heavy, and difficult to optimize dynamically.

To achieve maximum throughput, I had to tear down the recursive-descent implementation to its most granular components and rebuild the engine as a Just-In-Time (JIT) virtual machine.

This is where the complexity multiplier hits. Designing a JIT engine is not just writing code; it is designing custom, virtual hardware.

*1. Inventing the Instruction Set (ISA)* I had to design a bespoke virtual assembly language. Every DSL concept—variable resolution, rule triggering, string interpolation, file stat checks—had to be compiled down into highly optimized, single-byte opcodes.

*2. State and Execution Contexts* Unlike standard application development where the Go runtime handles memory safely, a JIT engine requires manually managing the instruction pointer (IP), operand stacks, and local variable frames. A single off-by-one error in a stack pop instruction does not result in a clean panic; it results in silent, cascading build corruption.

*3. The Execution Loop* By compiling the DSL down to bytecode, the Smart JIT engine executes build logic exponentially faster than AST traversal. Conditional branches, macro expansions, and rule evaluations are executed directly against a highly optimized memory buffer.

5The Result: Smart Build Engine Today

Today, Smart (http://extbit.com/smart) stands as a production-grade, JIT-compiled build orchestration engine. It solves the exact AOSP flat-namespace problem that frustrated me 15 years ago, providing complete namespace isolation, deterministic graph execution, and blazing-fast performance.

The journey from hacked Makefiles to a custom Golang bytecode VM is proof that in systems engineering, you cannot outrun your foundation. Sometimes, to fix a bottleneck, you have to design the virtual CPU from scratch.