Tprrt's Blog

Introduction

The previous two articles on this blog — one on cycle-level optimization, one on the no-heap ECS underneath it — were both about the Game Boy Advance: an FPU-less CPU with no hardware polygon fill at all, where every trick is about avoiding soft-float calls and touching as few VRAM bytes as possible. The Nintendo DS backend in framer-engine is the opposite kind of retro target — it has a real fixed-function 3D GPU, with hardware transform, lighting, and rasterization. That sounds like it should make lighting the easy part. It doesn't. It just moves the constraint from "the CPU can't do float math" to "the GPU can't do the thing you're asking it to do, and it won't tell you that — it'll just render something else instead."

This article is about chasing exactly that down: a real rendering bug, a real hardware surprise straight out of GBATEK, and a fix that gets meaningfully closer to real point-light falloff without ever abandoning the GPU's own lighting hardware for a software fallback.

The hardware constraint that drives everything

Every light in framer-engine — directional, point, or spot — is the same struct light component, and every backend (GBA's software rasterizer, desktop OpenGL/Vulkan/software, and this one) is expected to shade point lights with real distance falloff: the closer an object is, the brighter it gets, fading to nothing at the light's configured range. The DS's own hardware lighting docs rule that out before any code gets written. libnds's glLight():

void glLight(int id, rgb color, v10 x, v10 y, v10 z)
Only parallel light sources are supported on the DS

Four hardware light slots, each one a color and a direction vector — no position, no range, no falloff term anywhere in the API. The DS's GPU computes real per-vertex Lambertian shading (N·L) against those four directions entirely in fixed-function hardware, at zero CPU cost per triangle — genuinely excellent, for a directional light. A point light, on this hardware, can only ever be an approximation: pick a direction, accept that "distance" isn't a thing the lighting unit knows about, and find whatever ways are still available to make that approximation better without leaving hardware lighting behind entirely for a per-vertex CPU-side shading pass (which the DS's ARM946E-S can do, but which throws away the entire reason to use the GPU's fixed-function path in the first place).

Bug 1: an ambient floor that was zero on purpose

Before any of the point-light work, the hardware 3D path had a simpler problem: GL_AMBIENT — the DS's material property for "the color when a face isn't catching any light at all" — was set to black, unconditionally, the instant any Light entity existed in the scene:

/* what shipped, briefly */
glMaterialf(GL_AMBIENT, RGB15(0, 0, 0));

glLight() only lights a face whose normal faces toward one of the four active directions — anything else gets zero contribution from diffuse, and with GL_AMBIENT pinned to black, zero contribution from ambient too. The result: any face pointing away from every active light rendered as pure, flat black, on hardware that was otherwise correctly shading everything facing the right way. The fix looked simple — accumulate every ambient/directional light's color into a running sum each frame, and feed that into GL_AMBIENT instead of a hardcoded zero:

s_ambient_r += gl->r;
s_ambient_g += gl->g;
s_ambient_b += gl->b;
/* ...clamped to 1.0 after the loop, then: */
glMaterialf(GL_AMBIENT,
    RGB15((int)(r * s_ambient_r * 31.0f), /* ...g, b... */));

It rendered correctly with exactly one active light. It did not render correctly with two.

The hardware surprise: ambient is summed once per active light

Adding a second point light to the scene — the natural next step, since the whole point of this work was to support more than one — turned every previously-correctly-shaded face white. Not brighter: blown-out, clipped-to-white white, on geometry that had looked exactly right a commit earlier.

The cause is in GBATEK's own description of the DS's polygon lighting equation, easy to miss because nothing in libnds's glMaterialf() doc comment mentions it: the hardware doesn't compute MaterialAmbient once and add it to the final result. It computes MaterialAmbient × LightColor, separately, for every currently active light, and sums those. Diffuse and specular are supposed to work that way — that's how multiple colored lights are meant to combine — but ambient, conceptually, is "the floor when nothing is lighting this face," and nothing about that concept should scale with how many lights happen to be turned on. The DS hardware doesn't make that distinction. Set GL_AMBIENT to the value you actually want as a floor, and with three active lights, the hardware hands you back three times that, added into every face's final color.

The first attempted fix was the obvious one: divide by the number of active lights before setting GL_AMBIENT, so the hardware's own summation would reconstruct roughly the intended value.

/* looked right, wasn't */
float ambient_scale = 1.0f / (float)s_num_lights;

This fixed the two-point-light overexposure. It also made a sphere lit by one dim ambient light and two much brighter colored point lights render far too dark — because a flat 1/N correction has no way to know that those three active lights aren't equally bright. It divides down a lone dim ambient contribution by exactly as much as it divides down two point lights that have nothing to do with the ambient floor at all. A cube's few flat faces happened to look fine anyway (whichever face caught a point light's direction was still bright from diffuse); a sphere's curved surface — mostly not facing any of the three light directions — was left with almost nothing but that over-divided ambient term, and rendered close to black again, just from a different arithmetic mistake than bug 1.

The actual fix reconstructs the target ambient exactly, not approximately: track the real per-channel sum of every active light's own color (not just the count of lights), and divide by that instead.

/* MaterialAmbient * sum(LightColor) == target once hardware sums it
 * back, regardless of how many lights are active or how bright any
 * one of them is. */
float ambient_scale_r = 1.0f / fmaxf(s_light_color_sum_r, 0.05f);
glMaterialf(GL_AMBIENT,
    RGB15((int)(r * amb_r * ambient_scale_r * 31.0f), /* ...g, b... */));

Since MaterialAmbient × sum(LightColor) is exactly what the hardware computes internally, setting MaterialAmbient = target / sum(LightColor) makes the hardware's own summation land back on target — no matter how many lights are active or how differently bright each one is. The fmaxf(..., 0.05f) floor matters for a real edge case this reasoning otherwise misses entirely: if every active light has an exact 0 in one color channel (a pure-hue light, easy to reach — a pure green point light has no red or blue component at all), the sum for that channel is genuinely zero, and no MaterialAmbient value can make hardware produce a nonzero result by multiplying against a color that has none. Flooring the divisor doesn't fully fix that case — the DS hardware mathematically cannot recover a floor for a channel no active light contributes to — but it keeps the reconstruction from returning inf and cascading into garbage the moment a scene reaches for a light whose color isn't already a happy accident of "has something in every channel."

RGB555 has an opinion about how bright "bright enough" is

With the arithmetic fixed, faces pointing away from every light were no longer wrong — just still visually indistinguishable from unlit black in a real screenshot. RGB555 gives each channel 5 bits, 32 steps; a first ambient floor of 0.18 on a 0.8-albedo material works out to roughly 0.8 * 0.18 * 31 ≈ 4/31 — technically nonzero, comfortably lost to ordinary display gamma and contrast. Raising the floor to 0.45 reaches roughly 11/31, about 35%: the difference between "the math is correct" and "a person looking at the screen can tell the math is correct." Getting the equation right and getting the number right turned out to be two separate bugs, not one.

Real per-object falloff, without leaving hardware lighting

With the ambient floor actually behaving, the remaining problem was the one this article opened with: point lights, as hardware directional lights, weren't just "no falloff" in the abstract — the direction itself was computed once per frame, not per object, from the world origin:

/* once per frame, not once per object */
float lx = gl->x, ly = gl->y, lz = gl->z; /* the light's position */
/* ...normalize (lx, ly, lz) as a direction from the origin... */
glLight(s_num_lights, hw_color, dx, dy, dz);

Every object in the scene, regardless of where it actually sat relative to that light, saw the identical direction and the identical brightness, with no falloff at all. A "point" light behaved exactly like a directional one from every single object's point of view — which is a stronger regression than the hardware's own real limitation demands. The DS genuinely cannot do per-pixel or per-vertex-across-one-object falloff for a point light; it can absolutely do a different, correct direction and brightness for each separate object, if something recomputes those per object instead of once for the whole frame.

That's the fix: move the glLight() upload from once-per-frame to once-per-object, called right before that object's own triangles submit, computing real direction and falloff from that object's actual world position:

static void nds_upload_lights_for_object(const float pos[3])
{
    for (int i = 0; i < s_num_lights; i++) {
        /* ...ambient/directional lights reuse the once-per-frame
         * direction/color computed earlier -- no position to react
         * to, so recomputing per object would be pure waste... */
        float lx = gl->x - pos[0], ly = gl->y - pos[1], lz = gl->z - pos[2];
        float d2 = lx * lx + ly * ly + lz * lz;
        float atten = 1.0f - d2 * gl->inv_range2; /* same falloff every backend uses */
        /* ...clamp atten to [0, 1], normalize (lx, ly, lz), scale color by atten... */
        glLight(i, hw_color, dx, dy, dz);
    }
}

An object near the light now renders bright; an object near the edge of its range renders dim; an object outside the range entirely gets nothing from it — a real approximation of point-light falloff, per object, running through the exact same hardware N·L diffuse calculation the GPU always did. Nothing about the triangle count, vertex count, or rasterization work changes even slightly — this only changes the values written into two hardware registers that were already being written once per object either way.

It isn't free in the sense of costing literally zero CPU cycles — a point light's direction still needs a square root to normalize, and that now happens once per (point light, object) pair instead of once per frame. But it's the same fixed-point sqrtf32/divf32 primitives this code already used for the once-per-frame version, on a scene with a handful of lights and a handful of objects — a few extra fixed-point operations, not a new soft-float cost, and not one more vertex for the GPU to transform or one more pixel for it to fill. Against the cost of submitting an object's actual geometry at all, it doesn't move the needle, which is the only sense of "free" that was ever on the table for a change like this.

Two smaller wins that came out of the same pass

Two more things fell out of looking this closely at the hardware lighting path, neither one changing a single triangle:

GL_ANTIALIAS was already enabled at startup, and doing about half of its job. The DS only blends an edge under antialiasing where the polygon ID changes across it — and every object in the scene defaulted to the same ID (0), so hardware edge smoothing only ever applied where an object's silhouette met the background. Two overlapping objects, sharing the same ID, got a hard edge between them regardless. Assigning each object its own ID (an index mod 63, since glClearPolyID(63) reserves 63 for the backdrop) makes that blending apply between objects too — a different bitmask, OR'd into a glPolyFmt() call that already happens once per object, at the same cost as before.

And the built-in sphere/cone mesh, generated at 12 sectors × 6 stacks (120 triangles), was sized for a target this GPU isn't. That segment count lives in a comment noting the DS's real per-frame budget is "on the order of a couple thousand polygons" — a 360-triangle sphere at 20×10 is still a rounding error against that, and visibly rounder on screen, for a target whose real constraint was never triangle count in the first place.

Verifying a change with nothing to count

The GBA articles on this blog leaned on cycle_probe.py for every claim — a deterministic emulator's own cycle counter, diffed between frames, turning "did this help" into a yes/no number. None of that tooling applies here, and that's not an oversight: this whole article is about work that happens on the GPU's fixed-function pipeline, not the ARM9's own cycles. There's no CPU-side cost to isolate with a breakpoint, because there almost isn't one — the actual verification for this kind of change is the build and test suite (native, GBA, and NDS cross builds, all passing unchanged throughout), plus looking at the actual rendered frame in melonDS to confirm the sphere in examples/lighting is now visibly shaded by two differently-colored, differently-positioned lights instead of one flat approximation. Different kind of hardware, different kind of proof.

Where this goes next

The DS's fixed-function GPU has real capabilities this backend still doesn't touch at all: hardware texture sampling from VRAM, a specular shininess table, per-vertex fog. None of those are CPU-cost questions — they're GPU features sitting unused, the same shape of opportunity this article's point-light work turned out to be. The natural next piece, given how this one went, is real diffuse texturing: the highest-value use of hardware this backend doesn't ask anything of yet, and — if the pattern from the ambient bug repeats — probably has its own GBATEK surprise waiting to be found the same way this one was, by trying the obvious thing first and actually looking at what came out the other end.

Introduction

The previous article on this blog was about squeezing cycles out of framer-engine's GBA renderer. This one is about something that has to be settled before any of that renderer work is possible at all: how do you run an Entity Component System — the architecture pattern the whole engine is built around — on a console with 32KB of fast RAM, 256KB of slow RAM, no operating system, and no malloc() you can lean on?

framer-engine's answer is a second, much smaller ECS implementation living behind the same interface as the desktop one. Every game-logic line — component definitions, systems, queries — is identical between a PC build and a GBA build. Only one file differs: which world.c gets linked in.

Two ECS backends, one interface

framer-engine's default ECS backend is Flecs, a full-featured ECS used for desktop and handheld targets with a real OS underneath (PSP, 3DS, Switch, PC). Flecs is excellent at what it does — archetypes, queries, observers, relationships — but all of that bookkeeping assumes a working heap it can grow and shrink as entities and component types come and go. That's simply not available on a target like the GBA: 32KB of IWRAM, 256KB of EWRAM, both fixed-size and fully accounted for from the moment the ROM boots, with no MMU and no OS to page anything in. There's no heap to assume.

Rolling a second full-featured ECS to fit that budget wasn't the answer either — that's a lot of complexity to maintain for a problem that doesn't need it. What GBA-class scenes actually look like is a handful of objects (examples/simple_cube uses one mesh and a camera; examples/spinning_shapes uses three), not thousands, and the component types are known and fixed at compile time, not user-extensible at runtime. That's a much smaller problem than "general-purpose ECS," and src/ecs/static/world.c — about 550 lines of plain C — is sized to match it instead of to match Flecs:

/* Entities */
static bool s_alive[FRAMER_STATIC_MAX_ENTITIES];
static uint64_t s_comp_mask[FRAMER_STATIC_MAX_ENTITIES];

/* Component data store: row = component slot, column = entity slot.
 * Each entity slot is FRAMER_STATIC_MAX_COMPONENT_SIZE bytes wide. */
static uint8_t s_store[FRAMER_STATIC_MAX_COMPONENTS]
                       [FRAMER_STATIC_MAX_ENTITIES *
                        FRAMER_STATIC_MAX_COMPONENT_SIZE];

Every one of those arrays is a fixed-size global, sized by compile-time constants. There is no framer_entity_create() call anywhere in this file that can fail by running out of memory in some unpredictable way — it can only fail by running out of array slots, a number known before the program even starts.

Both backends sit behind the exact same include/framer/ecs.h — framer_world_t, framer_component_register(), framer_query_create(), framer_system_register(), the FRAMER_GET/FRAMER_SET/FRAMER_FIELD macros. A component defined with FRAMER_COMPONENT_DEFINE(Velocity) and a system registered with framer_system_register() compiles and runs unchanged on either backend; which one a build gets is a single Meson option, -Decs=flecs or -Decs=static, and GBA's cross file pins it to static (Flecs needs an OS, and bare-metal GBA doesn't have one) — the build refuses to configure any other way for that target.

Entities are array slots, nothing more

An entity in this backend isn't an object — it's a 1-based index into those parallel arrays. framer_entity_create() finds the lowest dead slot and claims it, first-fit:

framer_entity_t framer_entity_create(framer_world_t *world)
{
    int i;

    for (i = 0; i < FRAMER_STATIC_MAX_ENTITIES; i++) {
        if (!s_alive[i]) {
            s_alive[i] = true;
            s_comp_mask[i] = 0;
            if (i + 1 > s_entity_high)
                s_entity_high = i + 1;
            return EID(i);
        }
    }
    return 0; /* pool exhausted */
}

That's the entire allocator. No free list to maintain, no fragmentation to worry about — there's nothing to fragment when every slot is the same fixed size and lives at a compile-time-known address. The price for that simplicity is explicit and deliberate: destroying entity 5 and creating a new one immediately afterward hands back the same ID, 5, for a completely different logical entity. There's no generation counter to tell the two apart. For the scene sizes and lifetimes this backend targets — a handful of objects that mostly live for the whole level, not a churn of thousands spawning and despawning every frame — that's an acceptable trade, not an oversight; it's directly covered by a unit test (test_entity_slot_recycled_first_fit) precisely so it stays a known, intentional property instead of a surprise.

Components: a bitmask and a flat array

Each entity slot carries one uint64_t bitmask, one bit per registered component type. framer_component_set() is a memcpy into a fixed-stride row of the flat s_store array, plus a bit set:

void framer_component_set(framer_world_t *world, framer_entity_t e,
                          framer_id_t id, const void *data)
{
    int ei = EIDX(e);
    int ci = cidx(id);
    /* ...bounds and liveness checks elided... */
    memcpy(&s_store[ci][ei * FRAMER_STATIC_MAX_COMPONENT_SIZE], data,
           s_comp[ci].size);
    s_comp_mask[ei] |= ((uint64_t)1u << ci);
    s_any_mask |= ((uint64_t)1u << ci);
}

A query is just a precomputed mask built from the component ids it asks for; matching an entity against it is one AND and one comparison ((s_comp_mask[ei] & mask) == mask). The uint64_t bitmask is also the hard ceiling on how many distinct component types can exist in one world — 64 — which is generous for a retro scene's needs but means the type system itself enforces "don't try to build something Flecs-shaped on top of this."

War story: a silent NULL deref vs. a loud abort()

The single most important property of this backend isn't the data layout — it's what happens when a limit is hit. Early on, framer_component_register() returned 0 when a component was too big or the registry was full, the same "just signal failure" convention used everywhere else in this API. That sounds reasonable until you trace what a 0 component id actually does downstream: framer_query_create() silently skips it when building a query's mask, and the system that registered that query goes on to call FRAMER_FIELD() for a field that was quietly dropped — which dereferences NULL on the system's very next matching entity. That's exactly what happened on a real GBA build: a Text component at 288 bytes against a 64-byte cap, and 17 registered component types against a cap of 12. The crash that surfaced wasn't "your component is too big," it was a NULL-pointer SIGSEGV deep inside a render system, two layers removed from the actual mistake.

The fix removes the silent path entirely:

if (size > FRAMER_STATIC_MAX_COMPONENT_SIZE) {
    fprintf(stderr,
        "framer_component_register: \"%s\" is %lu bytes, "
        "exceeding FRAMER_STATIC_MAX_COMPONENT_SIZE (%d)\n",
        name ? name : "?", (unsigned long)size,
        (int)FRAMER_STATIC_MAX_COMPONENT_SIZE);
    abort();
}

abort() rather than an assert() or a GCC/Clang-specific builtin, because this backend also has to compile under cc65 and SDCC for 8-bit targets — plain C89 abort() is the one failure primitive guaranteed to exist everywhere this code runs. The test suite verifies the contract, not just the arithmetic: test_ecs_static_limits.c forks a child process, registers one component past the limit inside it, and asserts the child died of SIGABRT rather than returning normally — proving the fail-loud path actually fires, not just that the size check's math is correct.

The broader lesson generalizes past this one bug: on a backend built entirely out of fixed-size arrays, every hard limit is a wall, not a suggestion. The only choice that matters is whether you hit that wall with a clear error message at the exact call site that caused it, or with a corrupted query and a crash three function calls away. This backend picked loud, on purpose, everywhere a hard cap exists.

Sizing the pools, and a second silent-failure bug

Each target's meson.build picks FRAMER_STATIC_MAX_COMPONENTS, FRAMER_STATIC_MAX_ENTITIES, and FRAMER_STATIC_MAX_COMPONENT_SIZE to fit what that platform actually needs — there's no universal default that's right for every target, because the engine registers its full set of core components (Transform, Velocity, Sprite, Light, Text, Camera, and so on — 19 today) unconditionally, regardless of whether a given example actually uses all of them:

# Embedded (system == 'none'):
-DFRAMER_STATIC_MAX_COMPONENTS=19
-DFRAMER_STATIC_MAX_ENTITIES=96
-DFRAMER_STATIC_MAX_COMPONENT_SIZE=64

That MAX_ENTITIES=96 number has its own bug story behind it. Each registered component type — not each entity actually created — reserves one sentinel slot out of the same entity pool, so the entities actually available to a scene is MAX_ENTITIES minus however many component types exist. At MAX_ENTITIES=64 and 19 components, that left 45 usable slots — comfortably enough for examples/simple_cube, but one short of examples/input_tester's 48 (47 on-screen panel entities plus one camera). The failure mode was, again, silent: framer_entity_create() returning 0 once the pool filled, and the caller that wanted one more entity for a gamepad-axis label simply never got it — the text just never appeared on screen, with nothing in the logs to say why. Raising the cap to 96 (77 free after the 19 sentinels) fixed it with headroom to spare. On the GBA build that growth costs about 544 bytes of IWRAM (three per-entity arrays scale with the cap), which was checked against the build's actual free IWRAM margin before landing — on a 32KB budget, guessing isn't good enough, you measure.

That measurement habit is the same one from the performance article: arm-none-eabi-size -A on a current GBA build shows exactly where this backend's memory actually goes. The flat component store (19 × 96 × 64 bytes ≈ 114KB) is placed in EWRAM's .sbss section — too big for IWRAM, and zero-initialized for free by the startup code without costing any ROM space:

#ifdef GBA
#define _FRAMER_STORE_ATTR __attribute__((section(".sbss")))
#else
#define _FRAMER_STORE_ATTR
#endif

static uint8_t _FRAMER_STORE_ATTR
    s_store[FRAMER_STATIC_MAX_COMPONENTS]
           [FRAMER_STATIC_MAX_ENTITIES * FRAMER_STATIC_MAX_COMPONENT_SIZE];

Everything else — the alive flags, the masks, the per-frame iterator's entity list — is small enough to live in IWRAM, the GBA's fast 32KB scratch memory, where the CPU actually wants its hot working data. On the current examples/simple_cube build that's roughly 19.5KB of IWRAM used out of 32KB, leaving real headroom for the next component or two — a number worth checking again every time that count grows, the same way the 96-entity fix had to be checked against it.

The scan loop, briefly

framer_world_progress() walks every registered system, in phase order, and for each one scans entity slots up to the high-water mark (s_entity_high, one past the highest slot any entity or component sentinel has ever occupied) looking for bitmask matches. That scan, and the sticky s_any_mask check that skips it entirely for systems whose component type no entity has ever had, was the single biggest win in the previous article's performance work — covered there in full, since it's a perf story more than a design story. The design point that matters here is simpler: this is a linear scan over a flat array, not a sparse-set or archetype-table lookup. That's the right trade at GBA scene sizes (tens of entities), and the wrong one at thousands — which is exactly the line where you'd reach for Flecs instead.

What this design explicitly gives up

None of the above is free, and being upfront about the trade-offs is the point of having two backends instead of pretending one ECS fits every target:

• No entity generations. As covered above, IDs are recycled immediately and look identical to the entity that previously held them.
• No archetypes, no sparse sets. Matching is a linear scan with a bitmask test, not a cache-optimized contiguous iteration over exactly the matching entities. Fine at dozens of entities; the wrong tool past that.
• 64 component types, total, forever, for the whole world. Not per query — for every component type that exists anywhere in the engine, shared across every system. The engine's current 19 leaves room to grow, but a "just add a component" change always has a final cost attached: someone, somewhere, has to recheck that ceiling.
• No relationships, no hierarchies, no observers. Flecs has all of these; this backend has entities, components, and queries, deliberately nothing more.

Every one of these is a real capability Flecs has and this backend doesn't. They're also exactly the features that cost the heap, the dynamic bookkeeping, and the unpredictable-at-compile-time memory use that a bare-metal ROM target can't afford. The two-tier split exists so that trade only has to be made once, explicitly, per target — not silently, by whichever ECS happened to compile.

Where this goes next

The same problem — "no OS, no heap, fixed memory map, known component set" — is true of every retro target on framer-engine's roadmap, not just the GBA. The 32-bit-era consoles mentioned in the previous article's closing section, starting with the PlayStation 1, sit in an interesting middle ground: dramatically more RAM and a real GPU compared to the GBA, but still no OS and still nothing resembling a desktop heap. The expectation going in is that they'll want this same static backend, just with much larger pool constants — not Flecs, and not a third ECS implementation. Whether that expectation survives contact with an actual PS1 build, the way the "obviously correct" tricks in the performance article sometimes didn't survive contact with measurement, is exactly the kind of thing a future article on this blog will have to report honestly either way.

Introduction

framer-engine is a small cross-platform ECS game engine I've been building, with backends ranging from desktop OpenGL/Vulkan down to bare software rendering on 8/16-bit consoles. The Game Boy Advance backend renders actual textured/shaded 3D meshes — cubes, cones, spheres — through a CPU-only software rasterizer, on a 16.78MHz ARM7TDMI with no FPU and no hardware polygon fill. Every float operation is a soft-float library call, every divide is a library call, and every pixel is a CPU read-modify-write into VRAM.

examples/simple_cube, captured straight from mGBA — the demo this article's simple_cube numbers are measured on.

This article is about what it actually takes to make that fast — not in theory, but measured. Every number below comes from scripts/debug/gba/cycle_probe.py, a small script that sets a breakpoint on the engine's vblank-wait function inside headless mGBA and reads the emulator's cycle counter on every hit. mGBA's CPU emulation is deterministic: the same ROM run with the same inputs produces bit-identical cycle counts every time, which means an optimization claim isn't "it looked smoother" — it's "frame N now costs X fewer cycles, every single run." I discard the first ~100 frames as warm-up (caches, branch predictor-equivalent effects, lazy first-frame setup) and average the steady-state window after that.

That discipline matters more than any individual trick below, because twice during this work an optimization that was obviously, mathematically correct measured as a regression. More on that at the end.

The hardware constraint that drives everything

The GBA's display is locked to the LCD's scanout rate. A frame takes exactly 280896 cycles of the system clock to display, whether or not your CPU work fits inside it — if you go over, you just drop to displaying every other frame (or worse), the displayed frame rate quantizing to 59.73 / n for whatever integer multiple of that budget your frame actually costs. There's no "GPU" to defer to and no way to partially miss the deadline gracefully. The entire optimization exercise is: get the CPU-side frame cost under (or as close as possible to) 280896 cycles.

Every technique below exists because of two specific limits:

1. No FPU. Any float/double arithmetic — multiply, divide, sqrtf(), sinf()/cosf()/tanf() — compiles to a call into ARM's soft-float runtime. That's not "slower than native float," it's "a function call plus a software algorithm" for every single operation.
2. No hardware rasterizer. Mode 4's bitmap layers are just VRAM you write to with the CPU. Every triangle the software renderer fills is pixels the ARM7TDMI itself has to compute and store, one at a time.

Technique 1: fixed-point math instead of float

The most foundational change is also the simplest to state: the hot path (per-vertex transform, per-pixel rasterization) uses Q12 fixed-point integers instead of float, via a small fix_t type (src/backends/renderer/common/sw3d_fixed.h):

typedef int32_t fix_t;

#define FIX_SHIFT 12
#define FIX_ONE   (1 << FIX_SHIFT)   /* 4096 == 1.0 */

static inline fix_t fix_mul(fix_t a, fix_t b)
{
    return (fix_t)(((int64_t)a * (int64_t)b) >> FIX_SHIFT);
}

fix_mul's int64_t intermediate looks like it should be expensive, but on ARM it lowers to a single hardware SMULL (signed multiply, 64-bit result) instruction — no library call, no precision tricks, just the right type for the CPU's native multiply. Compare that to a float * float, which on this target is a soft-float call doing mantissa/exponent bookkeeping in software.

Division is the one place fixed-point still hurts, because there's no hardware divider on the ARM7TDMI either way — fixed-point divide still costs a library call (__aeabi_idivmod et al.), just an integer one instead of a float one. The perspective-divide hot path exploits a narrower fact about that specific division to cut its cost further:

/* fix_div()'s general implementation widens to a 64-bit intermediate to
 * stay correct for arbitrary numerators, but the perspective divide's
 * numerator is always FIX_ONE, so FIX_ONE << FIX_SHIFT never exceeds
 * 32 bits. */
static inline fix_t fix_reciprocal(fix_t b)
{
    return (fix_t)(((int32_t)FIX_ONE << FIX_SHIFT) / b);
}

That one change — replacing the general 64-bit fix_div() with a 32-bit-only reciprocal for the one call site where the numerator is known to always be FIX_ONE — measured a ~50,000 cycle/frame saving on examples/spinning_shapes, just from giving the divide routine a narrower, cheaper problem to solve.

Technique 2: a LUT instead of sinf()/cosf()

framer_transform_get_matrix() (the shared, cross-platform transform code) builds rotation matrices via cglm's glm_rotate_{x,y,z}(), which call cosf()/sinf(). On desktop that's a couple of FPU instructions; on the GBA it's a soft-float libm round trip, once per axis, per object, per frame.

examples/spinning_shapes, captured from mGBA — three objects rotating on all three axes every frame, the demo behind every spinning_shapes number in this article.

The GBA backend instead carries its own 256-entry sine table (sw3d_raster.c), reading cosine from the same table at a quarter-turn offset, with linear interpolation between samples:

static const fix_t gba_sin_lut[256] = { /* ... */ };

static void gba_fast_sincosf(fix_t angle_turns_256, fix_t *s, fix_t *c)
{
    int idx = angle_turns_256 & 0xff;
    int cidx = (idx + 64) & 0xff; /* cos(x) == sin(x + tau/4) */

    *s = gba_sin_lut[idx];
    *c = gba_sin_lut[cidx];
}

256 entries means ~1.4° between samples — far finer than visible on a 240x160 screen, so the linear interpolation error never shows up as visible jitter. Swapping this in for the float sin/cos chain, measured A/B (git stash + identical build/measure commands) on spinning_shapes (which rotates 3 objects on all 3 axes every frame): 1,294,320 → 1,234,341 cycles/frame, a ~4.6% reduction, from removing one class of soft-float call entirely.

A follow-up went further: rather than building the rotation matrix the way cglm does — up to three separate generic 4x4 matrix multiplies, one per nonzero Euler axis, each a 64-multiply-add matmul even though most entries of a pure-axis rotation matrix are 0 or 1 — the combined Rz·Ry·Rx product's 9 nonzero 3x3 entries are expanded by hand from the three angles' sin/cos (still sourced from the LUT above) and folded into the output with a single glm_mat4_mul instead of up to three:

/* out = Rz * Ry * Rx, 9 nonzero entries expanded by hand instead of
 * three generic 4x4 matmuls. */
out[0][0] = cy * cz;
out[0][1] = cy * sz;
out[0][2] = -sy;
out[1][0] = sx * sy * cz - cx * sz;
out[1][1] = sx * sy * sz + cx * cz;
out[1][2] = sx * cy;
out[2][0] = cx * sy * cz + sx * sz;
out[2][1] = cx * sy * sz - sx * cz;
out[2][2] = cx * cy;

This was verified against the original three-matmul path via NumPy differential testing across all 8 zero/nonzero axis combinations plus thousands of random angle triples (max absolute error ~1e-16) before it ever touched the renderer. Measured gain: only 1,309,080 → 1,306,224 cycles/frame, ~0.22% — much smaller than the raw operation count suggests, because the compiler's optimizer already folds away most of the original chain's zero/one multiplies once each Rz/Ry/Rx factor starts from an identity-seeded matrix. The lesson here isn't "this technique didn't matter" — it's that hand-expanding math only pays for itself once you've checked what the compiler was already doing for you. A second, branch-free variant that skipped the matrix multiply altogether (translation column copy + per-column scale) was also tried and measured worse in every iteration than this simpler one-matmul version — discarded in favor of what actually measures faster.

Technique 3: Quake III's fast inverse square root

Triangle shading needs each surviving triangle's world-space normal, normalized — once per shaded triangle, per frame, in the single hottest loop of the renderer. cglm's glm_vec3_normalize() calls sqrtf() and then divides by it: two soft-float library calls per triangle.

The fix is the famous bit-hack:

static float sw3d_fast_inv_sqrt(float number)
{
    union { float f; uint32_t i; } conv = { .f = number };

    conv.i = 0x5f3759df - (conv.i >> 1);
    conv.f *= 1.5f - (0.5f * number * conv.f * conv.f); /* one Newton-Raphson step */
    return conv.f;
}

One magic-constant bit-shift gets a rough inverse-square-root estimate straight from the float's IEEE bit pattern (no sqrt call at all), and one Newton-Raphson correction step sharpens it to be visually indistinguishable from the real thing for lighting purposes. Replacing both the sqrt and the divide with this one function, used at every site in the GBA backend that previously called glm_vec3_normalize() (face-normal lighting in the renderer, and the rasterizer's own triangle-normal centroid computation), removes two soft-float calls per triangle for one cheap integer/float hybrid op.

Technique 4: an ECS dispatch early-out

Not every win is renderer-specific. framer_world_progress(), the ECS scheduler's per-frame loop, walked every registered system's full entity range every frame — including systems whose query needs a component type that no entity in the scene has ever had. simple_cube registers collider/velocity/rigidbody systems unconditionally on every platform (component import is unconditional, regardless of whether the scene actually uses them), so most of those systems were scanning entities every frame only to match zero of them, every single time.

The fix tracks a sticky OR of every component bit ever set across the world's lifetime, and skips a system's scan entirely — O(1), no entity walk at all — whenever its query's required mask includes a bit outside that set, which can provably never match:

/* s_any_mask: sticky OR of every component bit ever set across the
 * world's lifetime. A query whose mask requires a bit outside this set
 * can never match any entity — skip the per-entity scan entirely. */
if ((q->mask & world->s_any_mask) != q->mask)
    continue;

This is the single largest win found across the whole project: simple_cube: 308650 → 288629 cycles/frame; spinning_shapes: 757806 → 744701 cycles/frame (both steady-state averages over frames 101-150). A scheduler-level fix, not a renderer trick, but it followed from the exact same discipline: measure where the cycles actually go, don't assume.

Technique 5: making divides Bresenham-shaped

The scanline rasterizer (sw3d_fill_triangle()/sw3d_fill_quad()) originally tested every pixel inside each triangle's bounding box against all three edge functions to decide if it was inside. The replacement computes each row's [lo, hi] x-span directly per edge, incrementally, which is exactly Bresenham's line algorithm applied to "x as a function of y" along a triangle edge:

/* Incrementally tracks bound(y) = floor((b0 + (y - y0) * d) / a) for a
 * fixed positive `a`, one row at a time, with zero divisions after
 * init. The GBA's ARM7TDMI has no hardware divider, so trading one
 * division per edge (at init) for what used to be a same-sign test on
 * every bounding-box pixel is the whole point. */
struct row_bound {
    long val, step, rem, err, a;
};

This turns "one division-equivalent test per candidate pixel" into "one division per triangle edge, plus an integer add per row" — a meaningful shape change on hardware with no hardware divider at all.

It also produced one of the more unusual micro-optimizations in the codebase. The one division this scheme still needs per edge (floordiv_pos()) is built on a / b and a % b in C, which GCC is supposed to fuse into a single __aeabi_idivmod call when both are needed. Disassembly showed that fusion happening on one branch (a > 0) but not the other (a < 0, which negates both operands first) — an extra, redundant __aeabi_idiv call alongside the __aeabi_idivmod for the same division, confirmed to be a GCC codegen quirk specific to that branch (restructuring the C source produced byte-identical codegen either way, so it wasn't fixable from the C side). The actual fix is to call the library function directly and unpack its packed 64-bit r0:r1 quotient/remainder result by hand, removing the compiler's latitude to make the wrong call-fusion choice at all:

extern long long __aeabi_idivmod(long numerator, long denominator);

static long floordiv_pos(long a, long b)
{
    long long qr = __aeabi_idivmod(a, b);
    long q = (long)(uint32_t)qr;
    long r = (long)(qr >> 32);

    if (r != 0 && a < 0)
        q--; /* C truncates toward zero; floor() needs a -1 correction */
    return q;
}

Saved roughly 25,000-30,000 cycles/frame on spinning_shapes — for removing one redundant library call the compiler was inserting on its own, on one branch only, for no reason a compiler flag could fix.

Technique 6: let the hardware scale a smaller image

The GBA has no hardware polygon fill, full stop — every pixel the rasterizer covers is a CPU read-modify-write into VRAM, which is the hard floor under every other optimization in this list: at some point you've removed every avoidable division and float op, and you're still bound by "how many pixels does the CPU have to touch."

The way around that floor isn't a CPU optimization at all: Mode 4's BG2 background layer supports affine transforms even though it's a flat bitmap — the same trick behind GBA titles that faked SNES Mode-7-style scaling. The renderer draws only a 120x80 corner of the framebuffer (a quarter the pixels of the real 240x160 screen) and lets BG2's affine matrix stretch that corner across the full screen at scanout time, for free, in hardware:

#if GBA_RENDER_SCALE == 1
static inline void gba_clear_buffer(vu16 *base) { /* full-res clear */ }
#else
static inline void gba_clear_buffer(vu16 *base)
{
    /* only clear the GBA_RENDER_WIDTH x GBA_RENDER_HEIGHT corner that's
     * actually sampled by BG2's affine matrix — the rest of the page is
     * never displayed, so clearing it is wasted work. */
}
#endif

On spinning_shapes this dropped steady-state cost from ~1.55M to ~1.28M cycles/frame — roughly 10.8fps → 13.1fps, a ~17% reduction — at the cost of visibly blockier 2x-nearest-neighbor-scaled edges. It's opt-in (-Dgba_half_res) rather than default, because unlike every other technique here it's a genuine, visible quality trade-off rather than a free win — worth calling out, since this whole article is otherwise about zero-visual-cost changes.

The measurement discipline that makes any of this credible

None of the numbers above are estimates. scripts/debug/gba/cycle_probe.py drives headless mGBA, sets a breakpoint on the engine's vblank-wait call (the one point every frame reliably passes through exactly once), and reads the emulator's own cycle counter on every hit. Because mGBA's CPU core is a deterministic interpreter/JIT — not a real, jittery piece of silicon — the same ROM, same breakpoint, same number of warm-up frames discarded, produces bit-identical cycle counts on every run. That turns "did this help?" from a vibes question into a yes/no one: rebuild, re-run the probe, diff the number.

That discipline is also what caught the two times this project tried an "obviously correct" optimization that wasn't.

War story 1: caching screen-space half-extents that never change

The camera's screen-space half-width/half-height, once converted to fixed-point, don't change frame to frame unless the camera's projection changes — so hoisting that fixed-point conversion out of the per-vertex projection loop and caching it looked like a pure, free win: same values, computed once instead of once per vertex.

It measured as a regression.

The likely cause, confirmed by inspecting the generated assembly rather than guessing: this project builds with link-time optimization (LTO) and -Doptimization=3 across the board, and LTO's inlining heuristics are sensitive to function and loop size in ways that aren't intuitive from the C source. Adding a cache check (even a cheap one) to an already-hot, already-inlined loop changed the cost/benefit math the inliner used elsewhere in the same translation unit, and the net effect of removing unrelated, more valuable inlining outweighed the arithmetic actually saved. The "obviously correct" loop-invariant hoist was correct about the math and wrong about the measured outcome.

War story 2: skipping integration work for a zero velocity

The same pattern showed up again, independently, in velocity_integration_system(). Most entities in simple_cube have a Velocity component that's exactly zero every frame — adding a zero-vector early-out before the glm_vec3_scale/glm_vec3_add calls is mathematically a no-op (scaling and adding a zero vector changes nothing), so it looked like free cycles for every entity that wasn't actually moving:

/* tempting, and wrong on this build */
if (glm_vec3_isvalid(v->linear) && glm_vec3_norm2(v->linear) == 0.0f &&
    glm_vec3_norm2(v->angular) == 0.0f)
        continue;

Measured: +112 cycles/frame on simple_cube, +312 on spinning_shapes. A regression, on a change with no behavior difference whatsoever. Same root cause as the screen-extent cache: the early-out added code size and a branch to a hot loop, LTO's inlining decisions shifted in response, and whatever inlining was lost elsewhere cost more than the skip saved. It was reverted in the same session it was tried, per the same rule that caught it: measure before keeping, no exceptions for changes that "can't possibly" make things worse.

The takeaway isn't "don't trust loop-invariant hoisting" or "don't trust early-outs" — both are completely standard, usually-correct techniques. It's that once a build is leaning on LTO and aggressive optimization levels to do a lot of the heavy lifting, the compiler's own decisions become part of the system you're optimizing, and they don't always move in the direction your mental model of the code predicts. The only way to know is the same cycle_probe.py round-trip used for every win in this article: change one thing, measure, keep it only if the number actually goes down.

War story 3: quantizing colors in the wrong number system

The GBA backend's shaded sprites and triangles go through gba_palette_index(): a linear scan (up to 256 entries, with a nearest- color distance calculation once full) that maps a computed RGB555 value onto BG_PALETTE, since Mode 4 is 8bpp paletted, not true color. A one-entry cache short-circuits two consecutive calls requesting the exact same value — but continuous per-frame lighting math (examples/ lighting, added since this article's original techniques, orbits two colored point lights around a static sphere and cube) makes each shaded triangle request a slightly different value almost every frame, defeating that cache and driving the palette to saturation within seconds.

The fix looked obvious: round each lit color channel to a coarser step — 16 buckets instead of RGB555's own 32 — before it ever reaches the palette lookup. A gradually-changing light then collapses onto a much smaller, more repeated set of values: more cache hits, and a palette that stays truer to intent for longer before saturating. The first implementation did this in float space, right where the lit color was already a float:

/* looked free, measured otherwise */
static inline float gba_quantize_channel(float v)
{
    const float steps = 16.0f;
    return (float)(int)(v * steps + 0.5f) * (1.0f / steps);
}

Measured with cycle_probe.py on the same orbiting-lights demo, 150 frames, steady-state average over frames 101-150: 646528 cycles/frame with no quantization at all, versus 658869 with it — a ~1.9% regression, not the improvement it was meant to be.

The cause, once measured rather than assumed: this is still the same FPU-less ARM7TDMI every other technique in this article exists to work around. v * steps + 0.5f, the cast, and * (1.0f / steps) are three more soft-float library calls, paid on every one of the three channels, for every shaded triangle, every frame — and that cost was larger than whatever palette-scan time it was saving. An optimization aimed specifically at this hardware's constraint had itself ignored that same constraint.

The fix moves the rounding into integer space instead, using a value the code already computes. f_to_5bit() (the existing float→RGB555-channel helper) produces a clamped 0-31 integer; masking off its low bit gives 16 buckets — the identical bucket count as the float version — for the cost of one AND on a value that has to be computed either way:

static inline u16 f_to_5bit_quantized(float v)
{
    return (u16)(f_to_5bit(v) & ~1u);
}

Same demo, same steady-state measurement: 642455 cycles/frame — a genuine, if modest, ~0.63% improvement over the no-quantization baseline. The idea behind the optimization was sound; it just had to be expressed in a number system this CPU can actually multiply in for free.

Where this leaves things

After all of the above, examples/simple_cube sits at 288074 cycles/frame — 16777216 / 288074, the same ratio cycle_probe.py itself reports for every measurement in this article — works out to ~58.24fps, against a true-60fps budget of 280896 cycles (~59.73fps). That's about 2.5% over budget, down from a starting point of roughly 7-8% over before this round of work. spinning_shapes — three fully shaded objects rotating on all three axes every frame, a heavier scene by design — sits at 741378 cycles/frame, ~22.63fps. Both are ceilings for these specific demo scenes on real, cycle-accurate emulation, not estimates: add more triangles or lights to either scene and the frame cost (and fps) moves accordingly. Closing the rest of that gap on simple_cube would mean moving into riskier territory: caching ECS query results across frames (not just the existence-of-any-entity check from Technique 4), or pre-converting mesh vertex data to fixed-point ahead of time instead of per-vertex at raster time — the latter complicated by the fact that the same mesh struct is also populated through framer-engine's public, float-only custom-mesh API, so caching it would mean either changing that API or building a runtime cache-on-first-use scheme. Both are real options, just bigger ones than "swap a divide for a multiply" — a good place to stop for now and pick back up deliberately, rather than rush into more soft-float removal for diminishing, harder-to-verify returns.

What's next

The GBA backend was the first proof that framer-engine's "real ECS, real 3D, software-rendered, no FPU" approach actually holds up on constrained hardware. The next targets are mainly a step up in capability rather than a step down: 32-bit-era consoles like the PlayStation 1, and handhelds with genuine 3D hardware acceleration — PSP, Nintendo DS, and 3DS. That side of the plan is mostly for fun: getting framer-engine to a point where it's genuinely pleasant to build small demos and little indie games on real retro hardware, GBA included.

But at least one of those targets — most likely the PSP, the one with the most conventional FPU-plus-GPU setup of the group — is also there for a different reason. Every technique in this article exists because the GBA has no FPU and no hardware rasterizer; on a platform that has both, none of those specific tricks apply, and the interesting question flips from "how do I avoid the hardware's weaknesses" to "how far can the engine and the hardware actually go together, pushed deliberately to their limits, with the GPU and FPU doing what they're meant to do." That's a different kind of optimization work — closer to traditional real-time-3D budgeting (draw calls, vertex throughput, fill rate) than to soft-float avoidance — and it needs the same measurement discipline as everything above, just pointed at a different bottleneck. Whether the specific tricks in this article carry over at all won't be clear until that work actually starts; future articles will cover whatever turns out to be that generation's equivalent surprise.

Introduction

Retro console development has experienced a renaissance in recent years, thanks to passionate homebrew communities and modern open-source tooling. What was once the domain of professional game studios with expensive proprietary SDKs is now accessible to anyone with a Linux machine and a passion for classic gaming hardware.

This guide catalogs the best open-source game engines and frameworks available for developing games on classic consoles, from the 8-bit Game Boy Color to sixth generation systems like the PlayStation 2. All tools mentioned are compatible with Linux development environments, making them perfect for a fully free and open-source workflow.

8-bit and 16-bit Consoles

Fifth and Sixth Generation Consoles

Development Tools and Workflow

Recommendations by Experience Level

Community Resources

General Communities:

• NESDev Forums: Multi-platform retro development - Forums
• GBAtemp: DS/3DS homebrew - Website
• devkitPro Discord: Nintendo handheld development

Platform-Specific:

• GBADev: Game Boy Advance - Website
• PSXDev: PlayStation 1 - Website
• PS2Dev Forums: PS2, PSP - Forums
• DCEmulation: Dreamcast - Website
• SegaXtreme: Saturn, Mega Drive - Website
• N64brew: Nintendo 64 - Website
• GC-Forever: GameCube/Wii - Website
• r/vitahacks: PS Vita homebrew

Conclusion

The retro console homebrew scene has never been more vibrant or accessible. With modern open-source toolchains, documentation, and active communities, developing games for classic consoles is now within reach of any motivated developer with a Linux machine.

Whether you want to create a simple Game Boy puzzle game with GB Studio's visual editor, or push the limits of the PlayStation 2's Emotion Engine with assembly-optimized code, the tools are available and the communities are welcoming.

The best part? This entire workflow can be accomplished with 100% free and open-source software, from the development tools to the graphics editors to the music trackers. This guide should give you everything you need to start your retro game development journey.

Happy coding, and may your sprites never flicker!