Deep dive into the system architecture: monorepo packages, dependency flow, node isolation, gossip mesh protocol, shard ring, binary wire format, neighbor selection, and Rust node-core internals.
This page covers the full system architecture: how the monorepo is organized, how nodes communicate, how data is replicated and sharded, and what the Rust microservice does.
The codebase is organized into 8 packages and 2 services with strict dependency rules. No package may import from a package that depends on it (no circular dependencies).
Packages:
| Package | Dependencies | Purpose |
|---|---|---|
protocol | None (zero framework deps) | Pure Zod schemas, 46 event types, deterministic validation logic, consensus rules |
db | protocol | Drizzle ORM schemas, PostgreSQL 16 migrations, typed query helpers |
agent | protocol, db | Gossip mesh networking, peer discovery, event propagation, WebSocket management |
server | protocol, db, cache, config | Business logic, tRPC routers, API handlers, settlement orchestration |
cache | None (standalone) | ioredis client, rate limiting, caching strategies, pubsub |
config | None (standalone) | Environment validation, feature flags, node configuration schemas |
cli | protocol, config | Command-line tooling for node operators |
front-door-gateway | cache only | Stateless edge proxy, rate limiting, request routing |
Services:
| Service | Stack | Purpose |
|---|---|---|
node-core | Rust (Axum) | 60+ HTTP endpoints for PSBT operations, Ed25519 signing, AES-GCM encryption, Shamir splitting, scanner loops |
benchmark-dashboard | TypeScript | Development performance dashboard with live JSON Lines streaming |
The dependency graph is strictly acyclic:
protocol
(zero deps)
/ \
v v
db config
/ \ |
v v v
agent server <----+
| |
v v
cache
|
v
front-door-gateway
(cache only)The protocol package sits at the root. It contains all 46 event type definitions, Zod schemas for validation, and deterministic logic (witness selection, fee calculation, threshold checks). Because it has zero dependencies, it can be used by any other package safely. It is also the package that gets version-locked for consensus -- all nodes must agree on the protocol version.
There is no shared database. This is a fundamental design constraint.
Each node runs its own PostgreSQL 16 instance with its own schemas and data. Nodes do not connect to each other's databases. There is no central database, no database replication, no read replicas shared between nodes.
How data consistency works without a shared database:
| Data Type | Replication Strategy |
|---|---|
| Protocol events (listings, orders, attestations, votes) | Fully replicated via gossip -- every node stores every event |
| Chat messages | Sharded via rendezvous hashing -- each thread assigned to specific nodes |
| Media cache | Sharded via rendezvous hashing -- each entry assigned to specific nodes |
| PSBT shards | Shamir 3-of-4 distribution to specific shard-holder nodes |
| Node state (local) | Not replicated -- internal bookkeeping per node |
This isolation means a compromised node cannot tamper with another node's data. It also means the network has no single point of failure for storage.
Node-to-node communication uses a compact binary wire protocol with minimal overhead. Every message starts with a 1-byte type tag followed by the payload.
| Byte | Message Type | Direction | Purpose |
|---|---|---|---|
| 0x01 | HELLO | Bidirectional | Initial handshake with Ed25519 pubkey and protocol version |
| 0x02 | HELLO_ACK | Response | Handshake acknowledgement with challenge nonce |
| 0x03 | AUTH_CHALLENGE | Request | Ed25519 signature challenge |
| 0x04 | AUTH_RESPONSE | Response | Signed challenge response |
| 0x05 | EVENT_NOTIFY | Push | Notify peer of a new event (hash + metadata) |
| 0x06 | EVENT_REQUEST | Pull | Request full event payload by hash |
| 0x07 | EVENT_PAYLOAD | Response | Full event data |
| 0x08 | HEARTBEAT | Bidirectional | Liveness and state vector exchange |
| 0x09 | PEER_LIST | Push | Share known peers for discovery |
| 0x0A | SHARD_STORE | Push | Store an encrypted PSBT shard |
| 0x0B | SHARD_REQUEST | Pull | Request a stored shard for reconstruction |
| 0x0C | SHARD_PAYLOAD | Response | Encrypted shard data |
| 0x0D | CHALLENGE_REQ | Request | PoA challenge for capability verification |
| 0x0E | CHALLENGE_RES | Response | PoA challenge answer |
| 0x0F | COHORT_JOIN | Push | Join a temporary session cohort |
| 0x10 | COHORT_MSG | Push | Message within a session cohort |
| 0x11 | COHORT_LEAVE | Push | Leave a session cohort |
| 0x12 | PING | Bidirectional | Keepalive |
The 18 message types (0x01 through 0x12) cover the full range of node interactions. The binary format avoids JSON parsing overhead for high-frequency messages like heartbeats and event notifications.
Nodes connect to each other over WebSocket at the /gossip endpoint. Every connection begins with Ed25519 authentication -- a node must prove it owns the private key corresponding to its announced public key.
Event Propagation:
The gossip protocol uses a push-pull model:
This is more efficient than broadcasting full events because most neighbors already have most events. The push notification is small (just a hash), and the pull only happens when needed.
Each node maintains connections to exactly 6 neighbors, selected to balance network locality with resilience:
| Slot | Selection Strategy | Purpose |
|---|---|---|
| 3 close neighbors | Lowest latency from recent measurements | Fast propagation to nearby nodes |
| 2 mid-range neighbors | Moderate distance in the ID space | Bridge between clusters |
| 1 random neighbor | Uniformly random from all known nodes | Prevent network partitions |
This K=6 strategy ensures that the gossip mesh does not form disconnected clusters. The 3 close neighbors provide fast local propagation, the 2 mid-range neighbors bridge regions, and the 1 random neighbor provides a probabilistic guarantee against partitions.
Neighbors are periodically re-evaluated based on measured latency and liveness.
Nodes exchange heartbeats to track liveness and synchronize state vectors. The heartbeat protocol uses epidemic (gossip-style) propagation:
Convergence: O(log N) rounds for N nodes
TTL per heartbeat: max(4, ceil(ln(N)))Each heartbeat contains:
With O(log N) convergence, a network of 1,000 nodes converges in roughly 7 rounds. The TTL formula max(4, ceil(ln(N))) ensures heartbeats propagate far enough to reach all nodes without flooding the network -- at 1,000 nodes the TTL is 7, at 10,000 nodes the TTL is 10.
Non-protocol data (chat threads, media cache) is distributed across nodes using rendezvous hashing (highest random weight):
For a given data key and set of N nodes:
For each node:
weight = hash(key + nodeId)
Sort nodes by weight descending
Assign data to the top R nodesThe replica count R is calculated as:
R = ceil(N / 3)For a 9-node network, R = 3. For a 12-node network, R = 4. Each chat thread is cloned to roughly one third of the live mesh, and different thread keys spread those replicas across the full network.
Rendezvous hashing has a key advantage over consistent hashing: when a node joins or leaves, only the data that hashes to that node needs to move. There is no cascading redistribution.
Operators can override the shard ring by enabling full-node mode (FULL_NODE_CHAT=1, FULL_NODE_MEDIA=1), which stores all data locally regardless of shard assignment. This does not affect PSBT shard distribution, which always uses the Shamir 3-of-4 scheme.
For time-sensitive session consensus (like attestation rounds), the protocol creates temporary direct WebSocket connections between participants:
This two-tier approach means the gossip mesh handles bulk event propagation (where O(log N) convergence is fine), while the cohort mesh handles time-critical consensus (where direct connections provide sub-second latency).
The node-core service is a Rust binary built on the Axum web framework. It runs on port 9100 by default and exposes 60+ HTTP endpoints consumed by the TypeScript layer.
Why Rust?
The operations handled by node-core are performance-critical and security-sensitive:
Key Endpoint Categories:
| Category | Examples | Count |
|---|---|---|
| PSBT operations | Create, validate, split, reconstruct, sign | ~15 |
| Cryptography | Ed25519 sign/verify, AES-GCM encrypt/decrypt, Shamir split/combine | ~10 |
| Chain scanning | Mempool watch, block scan, confirmation tracking | ~8 |
| Protocol evaluation | Validate sale, compute fees, check thresholds | ~12 |
| Node management | Health checks, capability probes, metrics | ~8 |
| PoA challenges | Generate challenge, verify response | ~5 |
The TypeScript layer calls node-core via HTTP (RUST_NODE_CORE_URL). This clean separation means the Rust code handles cryptography and chain interaction while TypeScript handles business logic, API routing, and database operations.
The network maintains a threshold wallet where the private key is split across ALL participating nodes using multi-party computation (MPC):
In production, the foundation master node (Node-1) manually activates MPC keygen and the reward token. Full key distribution waits until sufficient decentralization is achieved.
Node-1 is the foundation master node with beta-phase veto and admin powers. However, all foundation actions are subject to full transparency:
A separate rewards MPC wallet at the same 67% BFT threshold handles automated claim payouts for node operator rewards. Claims are signature-based, not transfer-based -- the foundation treasury is auditable at all times.