twigslot

Search

SearchSearch

Firedancer: A Deep Dive into the Fastest Solana Validator

Feb 16, 202630 min read

Firedancer: A Deep Dive into the Fastest Solana Validator

Firedancer is a from-scratch reimplementation of the Solana validator in C, built by Jump Trading’s crypto division. It’s one of the most impressive systems-level C codebases I’ve read in years — not because it does anything conceptually novel, but because it applies decades of low-latency trading infrastructure wisdom to blockchain with ruthless consistency.

This post is a technical deep dive into the Firedancer codebase. I’ll cover the architecture, the C coding patterns, the IPC system, the crypto implementations, and the performance techniques that make it fast. I’ll cite actual file paths and include real code snippets from the repository.

If you want to follow along: github.com/firedancer-io/firedancer.

Table of Contents

  1. Why Firedancer Exists
  2. Architecture: Tiles and Shared Memory
  3. The Memory Model: Workspaces, Not Malloc
  4. Tango: The IPC Backbone
  5. Ballet: Hand-Optimized Crypto
  6. Disco: Tile Orchestration
  7. Funk: The Accounts Database
  8. Waltz: The Network Stack
  9. C Coding Style and Patterns
  10. Performance Techniques
  11. The Sandbox: Security Through Isolation
  12. Why C Over Rust?
  13. Closing Thoughts

Why Firedancer Exists

The existing Solana validator (Agave, formerly known as the Labs client) is written in Rust. It uses async/await, Tokio, crossbeam channels, and the standard Rust concurrency toolkit. It works. But it has performance ceilings that are hard to break through when your concurrency model is “threads sharing memory protected by locks and channels.”

Jump Trading has spent 20 years building low-latency trading systems. Their insight, stated plainly in the Firedancer docs:

Leveraging our experience over the last two decades in building this global network, we believe we are well-positioned to significantly improve the power, reliability, and security of the Solana ecosystem.

The key architectural decision: tiles. Instead of threads sharing an address space with locks, Firedancer uses isolated processes communicating via shared memory with lock-free data structures. This is the architecture of a modern trading system, not a web server.

The secondary benefit is client diversity. A bug in Agave doesn’t take down Firedancer and vice versa. The codebase is written from scratch in C with minimal third-party dependencies — even the crypto, TLS, and QUIC implementations are custom.

Architecture: Tiles and Shared Memory

The fundamental unit of concurrency in Firedancer is the tile. A tile is an OS process pinned to a specific CPU core. Tiles don’t share address space in the traditional sense — they communicate exclusively through shared memory regions mapped into each tile’s address space.

This is conceptually similar to how modern FPGAs or microservices work: each tile is a self-contained processing unit with well-defined inputs and outputs. The difference from microservices is that the “network” between tiles is shared memory with sub-microsecond latency.

Why not threads?

Threads share an address space. This means:

  1. Cache coherence overhead: When thread A writes to a cache line that thread B reads, the hardware must shuttle that line between cores (MESI protocol). Even with careful padding, false sharing is hard to eliminate in a large codebase.

  2. Fault isolation: If a thread corrupts memory, every thread in the process is affected. With separate processes, a crash in one tile doesn’t bring down the validator.

  3. Sandboxing: You can’t apply per-thread seccomp filters. With processes, each tile can have a minimal seccomp policy — the networking tile can do sendto/recvfrom, the crypto tile can do almost nothing.

  4. Predictable performance: No garbage collector, no async runtime, no work-stealing scheduler. Each tile is a tight polling loop on a dedicated core.

Comparison with Agave

Agave (the Rust validator) uses:

  • Tokio for async I/O
  • Crossbeam channels for inter-thread communication
  • Arc<Mutex<T>> and Arc<RwLock<T>> for shared state
  • Rayon for parallel iteration

These are excellent abstractions, but they introduce:

  • Unpredictable latency: Tokio’s work-stealing scheduler means you don’t know which core runs your task, leading to cache thrashing.
  • Lock contention: Even RwLock has overhead — readers must atomically increment a counter, causing cache line bouncing.
  • Hidden allocations: Arc allocates, Vec::push allocates, String::from allocates. In a trading system, every allocation is a potential latency spike.

Firedancer’s model eliminates all of these. Each tile is a single-threaded polling loop. No locks. No allocations after startup. No scheduler.

The Memory Model: Workspaces, Not Malloc

Firedancer’s memory model is built on two layers: shared memory regions (fd_shmem) and workspaces (fd_wksp).

Shared Memory (src/util/shmem/fd_shmem.h)

At the bottom is fd_shmem, which wraps Linux’s hugetlbfs. Firedancer allocates memory backed by huge pages (2MB) and gigantic pages (1GB) for TLB efficiency:

#define FD_SHMEM_NORMAL_LG_PAGE_SZ   (12)  // 4KB
#define FD_SHMEM_HUGE_LG_PAGE_SZ     (21)  // 2MB
#define FD_SHMEM_GIGANTIC_LG_PAGE_SZ (30)  // 1GB

These are NUMA-aware. The startup scripts reserve pages on each NUMA node:

sudo bin/fd_shmem_cfg alloc 8 gigantic 0 \
                      alloc 8 gigantic 1 \
                      alloc 256 huge 0 \
                      alloc 256 huge 1

This eliminates TLB misses for the working set. A 1GB gigantic page means a single TLB entry covers the entire page. On a typical x86 CPU with 4-level page tables, a TLB miss for a 4KB page requires up to 4 memory accesses to walk the page table. With gigantic pages, you get nearly zero TLB misses.

Workspaces (src/util/wksp/fd_wksp.h)

On top of shared memory sits fd_wksp, which is Firedancer’s custom allocator. From the header:

This is not designed to be algorithmically optimal, HPC implementation or efficient at doing lots of tiny allocations. Rather it is designed to be akin to an “mmap” / “sbrk” style allocator of last resort, done rarely and then ideally at application startup.

The key properties:

  1. Global addressing: Every allocation in a workspace has a “global address” (gaddr) that is valid across all processes that have joined the workspace. This is how tiles communicate — they pass gaddr values through shared memory, and each tile translates to a local virtual address.

  2. No malloc after startup: All data structures are allocated from workspaces during initialization. At runtime, there are zero dynamic memory allocations. This is critical for latency — malloc can take hundreds of nanoseconds in the worst case (when it needs to call mmap or brk).

  3. Persistence: A workspace survives process restarts. If a tile crashes, you can restart it and it picks up exactly where the previous process left off. No deserialization needed.

  4. Relocatability: Because everything uses global addresses rather than pointers, a workspace can be checkpointed to disk, copied to a different machine, and resumed.

Usage looks like:

fd_wksp_t * wksp = fd_wksp_attach( "my-wksp-numa-0" );
ulong gaddr = fd_wksp_alloc( wksp, align, sz );
void * laddr = fd_wksp_laddr( wksp, gaddr );
// ... use laddr ...
fd_wksp_free( wksp, gaddr );

This is the foundation everything else is built on.

Tango: The IPC Backbone

src/tango/ is the inter-tile messaging system. It’s the nervous system of Firedancer — every piece of data flowing through the validator (transactions, shreds, votes) passes through tango. The name is fitting: it takes two to tango (producer and consumer).

Tango implements single-producer, multi-consumer (SPMC) lock-free message passing. The three core primitives are:

mcache — Metadata Cache (src/tango/mcache/fd_mcache.h)

The mcache is a ring buffer of fragment metadata. Each entry is an fd_frag_meta_t — a compact descriptor telling consumers where to find the actual data.

#define FD_MCACHE_ALIGN (128UL)  // Double cache line aligned

The alignment to 128 bytes (two cache lines) is deliberate. On x86, a cache line is 64 bytes. Aligning to double cache lines prevents the hardware prefetcher from pulling in adjacent metadata — which would cause false sharing between the producer writing entry N and a consumer reading entry N-1.

The mcache uses sequence numbers rather than head/tail pointers. This is a classic technique from lock-free programming. The producer monotonically increments a sequence number for each published fragment. Consumers track which sequence number they’ve consumed up to. If a consumer falls behind and the ring buffer wraps around, it detects an overrun and recovers.

The publish operation is atomic at the cache-line level:

static inline void
fd_mcache_publish( fd_frag_meta_t * mcache,
                   ulong            depth,
                   ulong            seq,
                   ulong            sig,
                   ulong            chunk,
                   ulong            sz,
                   ulong            ctl,
                   ulong            tsorig,
                   ulong            tspub ) {
  // ... writes metadata atomically to the correct cache line
}

The chunk field is a global address (chunk index into the dcache) — this is how zero-copy works. The producer writes data into the dcache and publishes a metadata entry pointing to it. Consumers read the metadata, translate the chunk to a local address, and read the data directly. No copies.

dcache — Data Cache (src/tango/dcache/fd_dcache.h)

The dcache holds the actual fragment payloads. It’s a data region that the producer writes into and consumers read from.

#define FD_DCACHE_ALIGN      (4096UL)    // Page aligned
#define FD_DCACHE_SLOT_ALIGN (128UL)     // Double cache line slots

The dcache has a “guard region” before the data area:

#define FD_DCACHE_GUARD_FOOTPRINT (3968UL)

This guard region provides flexibility to align payloads so that consumers always see consistent alignment regardless of how the producer writes. It’s a subtle detail that matters when you’re trying to use SIMD instructions on incoming data — misaligned loads on AVX-512 can be significantly slower.

fseq — Flow Control Sequence (src/tango/fseq/fd_fseq.h)

The fseq is the flow control mechanism. It’s a single sequence number that a consumer publishes to tell the producer how far it has consumed:

#define FD_FSEQ_ALIGN     (128UL)
#define FD_FSEQ_FOOTPRINT (128UL)  // Exactly one double cache line

The fseq is exactly 128 bytes — one double cache line. This means the consumer’s flow control update occupies its own cache line and doesn’t interfere with anything else. The producer reads the fseq periodically (not on every publish) to decide whether it has room to write more data.

This is credit-based flow control: the consumer gives the producer credits by advancing the fseq. The producer spends credits by publishing fragments. If the producer runs out of credits, it backs off. No locks, no CAS operations — just loads and stores with careful ordering.

cnc — Command and Control (src/tango/cnc/fd_cnc.h)

The cnc is for out-of-band control signals — “boot”, “run”, “halt”, “fail”. Each tile has a cnc that management threads use to monitor health and control lifecycle. The state machine is well-defined:

     new → BOOT → RUN → (HALT → BOOT | FAIL)
                    ↓
                   USER → RUN

The cnc includes a heartbeat mechanism. A monitoring thread can detect that a tile has hung by observing that the heartbeat hasn’t advanced. This is how Firedancer achieves process-level fault detection without polling or signals.

Putting It Together

The data flow through a pair of tiles looks like:

Producer Tile:
  1. Write payload into dcache slot
  2. Publish fd_frag_meta_t into mcache (sequence number, chunk offset, size)
  3. Periodically check fseq for flow control credits

Consumer Tile:
  1. Poll mcache for new sequence numbers
  2. Read fd_frag_meta_t to get chunk offset
  3. Translate chunk to local address, process payload
  4. Advance fseq to return credits to producer

No locks. No syscalls. No copies. The only synchronization is store ordering (which x86 gives you for free for same-address accesses — stores are never reordered with respect to later loads of the same location).

Ballet: Hand-Optimized Crypto

src/ballet/ contains Firedancer’s cryptographic implementations. These are written from scratch, not pulled from OpenSSL or libsodium, because cryptographic verification is one of the primary bottlenecks in a Solana validator.

Ed25519 (src/ballet/ed25519/)

Solana uses Ed25519 for all transaction signatures. A validator verifying a block might need to verify hundreds of thousands of signatures. Firedancer has three implementations:

  1. Portable (src/ballet/ed25519/) — works everywhere
  2. AVX-512 (src/ballet/ed25519/avx512/) — exploits 512-bit SIMD

The AVX-512 implementation uses a novel representation called fd_r43x6_t — GF(p) elements in radix-2^43 with 6 limbs packed into an AVX-512 vector:

/* A fd_r43x6_t represents a GF(p) element, where p = 2^255-19, in a
   little endian 6 long radix 2^43 limb representation.  The 6 limbs are
   held in lanes 0 through 5 of an AVX-512 vector.  That is:
 
     ( x0 + x1 2^43 + x2 2^86 + x3 2^129 + x4 2^172 + x5 2^215 ) mod p
 
   where xn is the n-th 64-bit vector lane treated as a long. */

This is a field element arithmetic library that operates on 6 limbs simultaneously using a single 512-bit register. One AVX-512 multiply instruction does 8 64-bit multiplications in parallel. Since a field multiplication in GF(2^255-19) decomposes into ~36 multiply-add operations on limbs, you can interleave multiple independent field multiplications across the 8 lanes.

The result: signature verification throughput that saturates memory bandwidth before it saturates compute.

SHA-256 (src/ballet/sha256/fd_sha256.h)

The SHA-256 implementation follows the same pattern — aligned, padded, cache-line-aware:

#define FD_SHA256_ALIGN     (128UL)
#define FD_SHA256_FOOTPRINT (128UL)
 
struct __attribute__((aligned(FD_SHA256_ALIGN))) fd_sha256_private {
  uchar buf[ FD_SHA256_PRIVATE_BUF_MAX ];  // 64 bytes at offset 0
  uint  state[ 8 ];                         // 32 bytes at offset 64
  ulong magic;
  ulong buf_used;
  ulong bit_cnt;
  // Padding to 128 bytes
};

The struct is exactly 128 bytes — one double cache line. The buf (the pending input block) and the state (the hash state) are on separate 64-byte cache lines. This means the CPU can prefetch the state while filling the buffer.

On systems with SHA-NI (Intel SHA extensions), Firedancer uses the hardware acceleration:

#ifndef FD_HAS_SHANI
#define FD_HAS_SHANI 0
#endif

Other Crypto

Ballet also includes:

  • Blake3 — used in Solana’s Proof of History
  • Base58 — with an AVX implementation (src/ballet/base58/fd_base58_avx.h)
  • Reed-Solomon — for erasure coding of shreds (src/ballet/reedsol/)
  • Keccak-256 — for Ethereum compatibility
  • HMAC, SHA-512, SHA-1 — various protocol needs

Each of these follows the same patterns: portable fallback + architecture-specific optimizations, 128-byte aligned state, no dynamic allocation, explicit state management (init/append/fini instead of “hash this buffer”).

Disco: Tile Orchestration

src/disco/ is the “main loop” of the validator. It defines the tiles and how they’re wired together. The name means “distributed components” (I think).

The Stem (src/disco/stem/fd_stem.h)

The fd_stem is the core abstraction for a tile’s event loop. It manages the mcache outputs and flow control:

struct fd_stem_context {
   fd_frag_meta_t ** mcaches;
   ulong *           seqs;
   ulong *           depths;
   ulong *           cr_avail;
   ulong *           min_cr_avail;
   ulong             cr_decrement_amount;
};

Publishing from a tile goes through fd_stem_publish:

static inline void
fd_stem_publish( fd_stem_context_t * stem,
                 ulong               out_idx,
                 ulong               sig,
                 ulong               chunk,
                 ulong               sz,
                 ulong               ctl,
                 ulong               tsorig,
                 ulong               tspub ) {
  ulong * seqp = &stem->seqs[ out_idx ];
  ulong   seq  = *seqp;
  fd_mcache_publish( stem->mcaches[ out_idx ], stem->depths[ out_idx ],
                     seq, sig, chunk, sz, ctl, tsorig, tspub );
  stem->cr_avail[ out_idx ] -= stem->cr_decrement_amount;
  *stem->min_cr_avail = fd_ulong_min( stem->cr_avail[ out_idx ],
                                       *stem->min_cr_avail );
  *seqp = fd_seq_inc( seq, 1UL );
}

This is the hot path. Notice there are no branches (except the implicit branch in fd_ulong_min, which is likely compiled to a cmov), no function calls that could prevent inlining, no indirect jumps. It’s a straight-line sequence of loads, stores, and arithmetic.

The Input Side

The fd_stem_tile_in structure tracks each input:

struct __attribute__((aligned(64))) fd_stem_tile_in {
  fd_frag_meta_t const * mcache;
  uint                   depth;
  uint                   idx;
  ulong                  seq;
  fd_frag_meta_t const * mline;
  ulong *                fseq;
  uint                   accum[6];
};

This is aligned to 64 bytes (one cache line). Each input’s polling state fits in a single cache line, so polling multiple inputs doesn’t cause cache thrashing.

Tile Types

The disco module defines the specific tiles that make up the validator:

  • net (src/disco/net/) — kernel-bypass networking via XDP (AF_XDP sockets)
  • quic — QUIC protocol handling for TPU (transaction processing unit)
  • verify — Ed25519 signature verification
  • dedup — transaction deduplication
  • pack — block packing (ordering transactions into microblocks)
  • bank — transaction execution
  • poh — Proof of History
  • shred — shred creation and distribution (Turbine protocol)
  • store — block storage
  • gui — monitoring dashboard

Each tile is a separate process, pinned to a core, communicating via tango.

The Topology

The base configuration defines constants like:

#define FD_NET_MTU            (2048UL)   // Max packet size through net tile
#define FD_TPU_MTU            (1232UL)   // Max transaction size (IPv6 min MTU - headers)
#define FD_SHRED_STORE_MTU    (41792UL)  // Size of an fd_shred34_t
#define FD_MAX_TXN_PER_SLOT   (98039UL)  // Bounded by CU limits

The data flow is:

Network → Net Tile → QUIC Tile → Verify Tile → Dedup Tile → Pack Tile → Bank Tile → PoH Tile → Shred Tile → Network

Each arrow is a tango link (mcache + dcache + fseq). Each tile polls its inputs, does its computation, and publishes to its outputs. The entire pipeline runs in parallel, with backpressure naturally propagating through flow control credits.

Funk: The Accounts Database

src/funk/fd_funk.h implements Firedancer’s accounts database. It’s a key-value store with transactional semantics, designed specifically for blockchain’s fork-handling requirements.

From the header comment (which is worth reading in full — it’s one of the best-documented files in the codebase):

Funk is a hybrid of a database and version control system designed for ultra high performance blockchain applications.

The Data Model

Records are (xid, key) → value triples, where:

  • xid = transaction ID (16 bytes) — identifies which fork/block this record belongs to
  • key = record key (40 bytes) — the account address
  • value = variable-length binary data — the account data

Transaction Tree

This is where Funk differs from Agave’s AccountsDB. In Agave, accounts are stored in append-only “account files” with periodic snapshots. Fork handling is done by maintaining a bank for each fork, each with its own view of modified accounts.

Funk models forks as a transaction tree:

root → published_1 → published_2 → ... → last_published
                                              ↓
                                        fork_A → child_A1
                                              ↓
                                        fork_B

Transactions (in the database sense, not Solana transactions) can fork from any point. Looking up a record in a fork walks up the ancestor chain until it finds the record. Publishing a fork collapses the linear history and cancels competing forks. This directly models Solana’s consensus mechanism where multiple forks are in flight.

Performance Properties

From the header:

Most are fast O(1) time and all are small O(1) space. There is no use of dynamic allocation to hold temporaries and no use of recursion to bound stack utilization at trivial levels.

Records are stored in workspace memory (fd_wksp), so they:

  • Are NUMA-aware and TLB-optimized
  • Survive process restarts without deserialization
  • Can be accessed zero-copy by multiple tiles
  • Are relocatable (can be checkpointed and moved between hosts)

Each record requires 128 bytes of metadata. For 100 million accounts, that’s ~13 GiB just for metadata. Record values use fd_alloc (Firedancer’s lockfree allocator) from the same workspace.

Why This Beats AccountsDB

Agave’s AccountsDB has several pain points:

  1. Append-only files: Account updates append to growing files, requiring periodic cleaning and compaction.
  2. Snapshot overhead: Creating and loading snapshots involves serializing/deserializing the entire account state.
  3. Fork handling: Each bank maintains its own delta, which is reconciled during slot finalization.

Funk avoids all of this:

  1. In-place updates: Values are updated in-place in shared memory.
  2. Instant restart: The workspace is the database. No serialization needed.
  3. Native fork handling: The transaction tree directly models Solana’s fork structure.

Waltz: The Network Stack

src/waltz/ implements Firedancer’s network stack. Solana requires QUIC for transaction submission (TPU), and Firedancer implements QUIC, TLS 1.3, and the IP stack from scratch.

TLS 1.3 (src/waltz/tls/fd_tls.h)

The TLS implementation is minimal — it only supports what Solana needs:

/* fd_tls is not a general purpose TLS library.  It only provides the
   TLS components required to secure peer-to-peer QUIC connections as
   they appear in Solana network protocol. */

Specifically:

  • Key exchange: X25519 (ECDH on Curve25519)
  • Authentication: Ed25519 raw public keys (RPK) — no certificate authorities
  • Cipher suite: TLS_AES_128_GCM_SHA256 only
  • No: certificate chains, session resumption, pre-shared keys, older TLS versions

This is the right approach. General-purpose TLS libraries like OpenSSL carry enormous complexity for features a Solana validator will never use. By implementing only what’s needed, the attack surface is dramatically smaller and the code is auditable.

IP Stack (src/waltz/ip/)

Firedancer includes its own IPv4 routing table (fd_fib4), FIB (Forwarding Information Base) management, and netlink integration for route discovery. This is because the net tile uses kernel-bypass (XDP), so it can’t rely on the kernel’s IP stack for routing decisions.

HTTP Server (src/waltz/http/)

There’s even a custom HTTP server (fd_http_server.h) used for the validator’s RPC interface and monitoring GUI. Because of course there is.

C Coding Style and Patterns

This is where Firedancer gets really interesting from a software engineering perspective. They’ve developed a consistent, idiomatic C style that achieves many of the guarantees typically associated with higher-level languages.

The Object Lifecycle Pattern

Every data structure in Firedancer follows the same lifecycle:

align() → footprint() → new() → join() → [use] → leave() → delete()

For example, from fd_mcache.h:

ulong            fd_mcache_align    ( void );
ulong            fd_mcache_footprint( ulong depth, ulong app_sz );
void *           fd_mcache_new      ( void * shmem, ulong depth, ulong app_sz, ulong seq0 );
fd_frag_meta_t * fd_mcache_join     ( void * shmcache );
void *           fd_mcache_leave    ( fd_frag_meta_t const * mcache );
void *           fd_mcache_delete   ( void * shmcache );

This is the C equivalent of RAII. The separation of new (format memory) and join (get a local handle) enables the shared-memory model: one process calls new, many processes call join. The separation of leave (release local handle) and delete (unformat memory) means you can restart a process without losing the data.

Every single data structure in the codebase follows this pattern. Consistency matters.

Generic Containers via Macros (src/util/tmpl/)

C doesn’t have generics. Firedancer solves this with a macro-based code generation system. The pattern:

struct mymap {
  ulong key;
  uint  hash;
  // ... other fields ...
};
typedef struct mymap mymap_t;
 
#define MAP_NAME        mymap
#define MAP_T           mymap_t
#define MAP_LG_SLOT_CNT 12
#include "util/tmpl/fd_map.c"

This #include expands to a complete implementation of a hash map specialized for mymap_t. You get:

ulong     mymap_align    ( void );
ulong     mymap_footprint( void );
void *    mymap_new      ( void * shmem );
mymap_t * mymap_join     ( void * shmymap );
void *    mymap_leave    ( mymap_t * mymap );
void *    mymap_delete   ( void * shmymap );
ulong     mymap_key_max  ( void );
mymap_t * mymap_insert   ( mymap_t * mymap, ulong key );
mymap_t * mymap_remove   ( mymap_t * mymap, mymap_t * entry );
mymap_t * mymap_query    ( mymap_t * mymap, ulong key, mymap_t * null );

This is C++ templates implemented with the preprocessor. The advantages over C++ templates:

  1. Predictable codegen: You can see exactly what code is generated.
  2. No name mangling: Symbol names are human-readable.
  3. No header bloat: The template is instantiated exactly once.
  4. Debuggable: GDB can step through the generated code just fine.

The available containers include:

  • fd_map.c — open-addressed hash map with bounded size
  • fd_map_dynamic.c — resizable hash map
  • fd_map_chain.c — chained hash map
  • fd_map_giant.c — hash map for very large datasets
  • fd_map_perfect.c — perfect hash map (compile-time keys)
  • fd_set.c — bitset
  • fd_heap.c — binary heap

Function Attribute Macros (src/util/fd_util_base.h)

Firedancer defines a set of function attribute macros that serve as documentation and optimization hints:

#define FD_FN_PURE    /* No side effects, result depends only on inputs and memory */
#define FD_FN_CONST   /* No side effects, result depends only on inputs (not memory) */
#define FD_FN_UNUSED  __attribute__((unused))
#define FD_FN_SENSITIVE __attribute__((strub)) __attribute__((zero_call_used_regs("all")))
#define FD_WARN_UNUSED __attribute__((warn_unused_result))

FD_FN_SENSITIVE is particularly interesting — it’s used on functions that handle private keys. It tells the compiler to clear all registers and scrub the stack frame on return, reducing the risk of key material lingering in memory. The comment cites a 2023 cryptographic engineering paper (Section 3.2).

Note that FD_FN_PURE and FD_FN_CONST expand to nothing by default. The comment explains why in delightful detail:

Recent compilers seem to take an undocumented and debatable stance that pure functions do no writes to memory. This is a sufficient condition but not a necessary one.

The macros still exist as documentation: when you see FD_FN_PURE, you know the function has no side effects, even if the compiler doesn’t get to optimize on that.

Compiler Tricks

The codebase has a rich set of compiler-interaction macros:

// Tell the compiler to forget what it knows about a variable
#define FD_COMPILER_FORGET(var) \
  __asm__( "# FD_COMPILER_FORGET(" #var ")" : "+r" (var) )
 
// Same but volatile — value is now unpredictable
#define FD_COMPILER_UNPREDICTABLE(var) \
  __asm__ __volatile__( "# FD_COMPILER_UNPREDICTABLE(" #var ")" : "+m,r" (var) )
 
// Compiler memory fence (no hardware fence)
#define FD_COMPILER_MFENCE() \
  __asm__ __volatile__( "# FD_COMPILER_MFENCE()" ::: "memory" )

FD_COMPILER_FORGET is used to prevent the compiler from hoisting operations out of loops or optimizing away “redundant” computations. This is critical in branchless programming — sometimes you need the compiler to actually evaluate both sides of a conditional.

FD_COMPILER_UNPREDICTABLE goes further: it marks a value as volatile, preventing the compiler from treating it as a compile-time constant even if it linguistically is one. This is used to prevent the compiler from optimizing away timing-critical operations.

Type Punning

Firedancer uses a clever type punning helper that keeps strict aliasing enabled:

static inline void *
fd_type_pun( void * p ) {
  __asm__( "# fd_type_pun" : "+r" (p) :: "memory" );
  return p;
}

The asm statement with "memory" clobber tells the compiler “this pointer might now alias anything.” This allows safe type punning (needed for network packet parsing, sockaddr handling, etc.) without disabling strict aliasing optimization globally. It’s a surgical fix instead of -fno-strict-aliasing.

Static Assertions

Used pervasively to catch configuration errors at compile time:

#define FD_STATIC_ASSERT(c,err) _Static_assert(c, #err)
 
// Example usage:
FD_STATIC_ASSERT( FD_MAX_TXN_PER_SLOT<=FD_MAX_TXN_PER_SLOT_CU &&
                  FD_MAX_TXN_PER_SLOT<=FD_MAX_TXN_PER_SLOT_SHRED,
                  max_txn_per_slot );

The FD_IMPORT Pattern

Firedancer has a macro for embedding binary files into object files at compile time:

#define FD_IMPORT_BINARY(name, path) FD_IMPORT(name, path, uchar, 7, "")
#define FD_IMPORT_CSTR(name, path)   FD_IMPORT(name, path, char, 1, ".byte 0")

This uses .incbin in inline assembly to embed files directly into the .rodata section. No xxd, no build system code generation — just a macro. The comment explaining why gcc -M dependency tracking doesn’t work with this is a masterpiece of frustrated systems engineering.

Performance Techniques

Branch Prediction Hints

#define FD_LIKELY(c)   __builtin_expect( !!(c), 1L )
#define FD_UNLIKELY(c) __builtin_expect( !!(c), 0L )

These are used aggressively throughout the codebase. In the hot path of a tile’s polling loop, every error check is wrapped in FD_UNLIKELY. This isn’t just about the branch predictor — it also affects code layout. The compiler will put the unlikely path out of line, keeping the hot path compact and improving instruction cache utilization.

Cache Line Awareness

Everything in Firedancer is aligned to cache line boundaries:

#define FD_MCACHE_ALIGN     (128UL)  // Double cache line
#define FD_DCACHE_ALIGN     (4096UL) // Page aligned
#define FD_DCACHE_SLOT_ALIGN (128UL) // Double cache line
#define FD_FSEQ_ALIGN       (128UL)  // Double cache line
#define FD_SHA256_ALIGN     (128UL)  // Double cache line

The double cache line (128 byte) alignment is a recurring pattern. This is because x86 hardware prefetchers operate on pairs of cache lines. Aligning to 128 bytes ensures that a data structure doesn’t share a prefetch group with unrelated data.

Spin Polling

#if FD_HAS_X86
#define FD_SPIN_PAUSE() __builtin_ia32_pause()
#else
#define FD_SPIN_PAUSE() ((void)0)
#endif

The pause instruction on x86 yields the hyperthread to its sibling without a context switch. Firedancer uses this in spin loops to be a good neighbor when hyperthreading is enabled, but the default deployment pins tiles to physical cores with hyperthreading disabled.

Volatile Access for Shared Memory

#define FD_VOLATILE_CONST(x) (*((volatile const __typeof__((x)) *)&(x)))
#define FD_VOLATILE(x)       (*((volatile       __typeof__((x)) *)&(x)))

These are used instead of C11 atomics for shared memory access. On x86, aligned loads and stores of native-width types are already atomic. The volatile keyword prevents the compiler from caching the value in a register or reordering the access. This is sufficient for the producer-consumer pattern in tango — no need for the overhead of _Atomic or __sync_* builtins for the common case.

For actual atomic read-modify-write operations (used sparingly), there are explicit wrappers:

#define FD_ATOMIC_CAS(p,c,s)         __sync_val_compare_and_swap( (p), (c), (s) )
#define FD_ATOMIC_FETCH_AND_ADD(p,v)  __sync_fetch_and_add( (p), (v) )
#define FD_ATOMIC_XCHG(p,v)          __sync_lock_test_and_set( (p), (v) )

The comment on FD_ATOMIC_XCHG is worth reading — it documents the long history of __sync_lock_test_and_set being misnamed and its behavior varying across compilers and architectures. Firedancer has a compile-time switch to use CAS-loop emulation on platforms where the builtin doesn’t behave as expected.

FD_ONCE: Lock-Free Initialization

#define FD_ONCE_BEGIN do {                                               \
    FD_COMPILER_MFENCE();                                                \
    static volatile int _fd_once_block_state = 0;                        \
    for(;;) {                                                            \
      int _fd_once_block_tmp = _fd_once_block_state;                     \
      if( FD_LIKELY( _fd_once_block_tmp>0 ) ) break;                     \
      if( FD_LIKELY( !_fd_once_block_tmp ) &&                            \
          FD_LIKELY( !FD_ATOMIC_CAS( &_fd_once_block_state, 0, -1 ) ) ) {\
        do
 
#define FD_ONCE_END               \
        while(0);                 \
        FD_COMPILER_MFENCE();     \
        _fd_once_block_state = 1; \
        break;                    \
      }                           \
      FD_YIELD();                 \
    }                             \
  } while(0)

This is a lock-free call_once implementation using CAS. The state variable transitions: 0 (not started) → -1 (in progress) → 1 (done). It’s used for one-time initialization of global state. Note the compiler memory fences to prevent reordering of the initialization code.

Interleaved Metadata Layout

The mcache supports configurable interleaving of fragment metadata:

#define FD_MCACHE_LG_BLOCK      (0)
#define FD_MCACHE_LG_INTERLEAVE (0)

From the comment:

At a LG_INTERLEAVE of i with s byte frag meta data, meta data storage for sequential frags is typically s*2^i bytes apart.

This controls the stride between consecutive fragment metadata entries. With interleaving, sequential entries are spread across multiple cache lines. This prevents false sharing between a producer writing entry N and a fast consumer reading entry N-1, at the cost of worse cache utilization for slow consumers doing sequential scans.

The default (0,0) disables interleaving, optimizing for throughput over latency. The comment suggests (2,7) as a good configuration for latency-sensitive scenarios with 32-byte metadata.

Zero-Copy Message Passing

The entire tango system is zero-copy. When a tile receives a transaction:

  1. The net tile writes the raw packet into its dcache.
  2. It publishes an mcache entry with the chunk index.
  3. The QUIC tile reads the mcache entry, translates the chunk to a local address, and processes the packet in place.
  4. If it needs to forward the payload, it writes into its own dcache and publishes a new mcache entry.

At no point is the payload copied between tiles. The dcache memory is in a shared workspace that all tiles have mapped.

The Sandbox: Security Through Isolation

src/util/sandbox/fd_sandbox.h implements a comprehensive process sandbox. Each tile is sandboxed with:

  1. Environment scrubbing: All environment variables are zeroed and cleared.
  2. File descriptor audit: Only explicitly allowed file descriptors can be open.
  3. Namespace isolation: CLONE_NEWNS, CLONE_NEWNET, CLONE_NEWCGROUP, CLONE_NEWIPC — each tile has its own mount namespace, network namespace, cgroup namespace, and IPC namespace.
  4. UID/GID switching: Tiles drop privileges to an unprivileged user.
  5. Session isolation: No controlling terminal, new process group and session.
  6. Seccomp-BPF: Each tile has a whitelist of allowed syscalls. A networking tile can do sendto/recvfrom. A crypto tile can do almost nothing — not even open or write.
  7. Keyring replacement: Session keyring replaced with anonymous keyring.
  8. Resource limits: Set via rlimits.

From the header:

The security of the sandbox is more important than the security of code which runs before sandboxing. It is strongly preferred to do privileged operations before sandboxing, rather than allowing the privileged operations to occur inside the sandbox.

This is defense in depth. Even if an attacker finds a memory corruption bug in the transaction parser, they’re trapped in a sandbox with no file system access, no network access (the networking tiles are separate processes), and a minimal syscall surface.

This approach was explicitly inspired by browser sandboxing (Chromium’s multi-process architecture). The comment in the Firedancer docs:

Firedancer’s security model is inspired by methods used in the browser world which is under perpetual attack.

Why C Over Rust?

This is the question everyone asks. Firedancer is a greenfield project started in 2022. Rust exists. It has memory safety. Why write a financial system in C?

The Firedancer team’s reasons, as I understand them from the code and docs:

1. Control Over Memory Layout

In Rust, Vec<T> stores (pointer, length, capacity) — 24 bytes on the stack plus a heap allocation. In Firedancer, you’d use a fixed-size array in a workspace with a compile-time known capacity. No heap allocation, no three-word header, exact control over alignment and padding.

Rust’s HashMap uses random state for hash seeding (DoS protection), Robin Hood probing, and allocates buckets on the heap. Firedancer’s fd_map uses a compile-time slot count, is allocated in shared memory, and the hash function is caller-defined.

When you’re counting cache lines and TLB entries, this control matters.

2. No Hidden Allocations or Runtime

Rust has:

  • A global allocator that Box, Vec, String, Arc, etc. use implicitly
  • A panic runtime with unwinding or abort
  • Thread-local storage for the panic handler
  • Format machinery (Display, Debug) that allocates

In Firedancer’s hot path, there are literally zero hidden operations. What you see in the source is what the CPU executes.

3. Predictable Codegen

Rust’s monomorphization can produce surprising amounts of code. A generic function instantiated for 10 types produces 10 copies. Trait objects introduce indirect calls. Iterators with .map().filter().collect() chains can produce excellent code or terrible code depending on optimizer mood.

C macros produce exactly the code you expect. FD_LIKELY/FD_UNLIKELY map directly to compiler intrinsics. Inline assembly is first-class.

4. Auditability

A financial system managing billions of dollars needs to be auditable. C code is straightforward to audit:

  • No hidden destructors running at scope exit
  • No implicit conversions (well, fewer than C++)
  • No trait resolution to figure out which function is actually called
  • No lifetime annotations to decode
  • The call graph is explicit

The security audit of a function like fd_ed25519_verify requires understanding the code, the compiler, and the hardware. In C, the distance between source and assembly is small. In Rust, there are more layers (MIR, LLVM IR, monomorphization, trait resolution) that an auditor must reason about.

5. Shared Memory Is Hard in Rust

Firedancer’s shared-memory model requires casting raw pointers, doing arithmetic on them, and interpreting shared memory regions as typed data structures. In Rust, this requires extensive use of unsafe — so much that you’d lose most of Rust’s safety benefits while paying the ergonomic cost.

Rust’s ownership model assumes a single owner for each piece of data. Shared memory fundamentally violates this — the same memory region is accessed by multiple processes with no language-level synchronization.

The Trade-Off

The honest answer is: Firedancer trades compile-time safety for runtime performance and auditability. The team mitigates the lack of memory safety through:

  • Process isolation (one tile crashing doesn’t affect others)
  • Seccomp sandboxing (limit blast radius)
  • Extensive fuzzing and testing
  • Defensive coding (bounds checks, magic number validation)
  • Simple, auditable code with minimal indirection

Whether this is the right trade-off is debatable. But the performance results speak for themselves.

Closing Thoughts

Reading the Firedancer codebase is a masterclass in systems programming. The coding conventions are consistent. The comments explain why, not just what. The performance techniques are principled, not cargo-culted.

A few things stand out:

The documentation is unusually good. The header comments in fd_funk.h, fd_mcache.h, and fd_wksp.h are better than most READMEs. They explain the design rationale, the constraints, the trade-offs, and the failure modes. When a macro behaves unexpectedly on some compilers, the comment explains exactly why and what was done about it.

The consistency is remarkable. Every data structure has align/footprint/new/join/leave/delete. Every hot-path function is static inline. Every alignment is a power of two and at least double-cache-line. Every error code is well-defined. This consistency reduces cognitive load and makes the codebase navigable even at ~500k lines of C.

The rants are delightful. The doc/rant/ directory contains essays on topics like integer types and frame pointer omission. The comments in fd_util_base.h contain multiple instances of “sigh” regarding compiler behavior. This is a codebase written by people who’ve been fighting compilers for decades and have opinions.

It’s genuinely fast. Firedancer has demonstrated ~1M TPS in test environments. The architecture — tiles, shared memory, zero-copy, lock-free IPC, custom crypto — is the same architecture used in sub-microsecond trading systems. It works because it eliminates every unnecessary abstraction between the application and the hardware.

If you’re interested in high-performance systems programming, studying Firedancer is time well spent. Start with src/util/fd_util_base.h for the coding conventions, then src/tango/mcache/fd_mcache.h for the IPC system, then src/ballet/ed25519/avx512/fd_r43x6.h for the crypto. Read the comments. They’re the best part.

All file paths reference the Firedancer repository at github.com/firedancer-io/firedancer. Code snippets are from the version at time of writing and may have changed.

Graph View

Backlinks