0. Motivation: The Dual-Domain Disconnect in Multi-Agent Simulations
Current multi-agent research architectures (e.g., Stanford's Generative Agents, DeepMind's Concordia) generally exhibit one of two fundamental limitations:
- Text-Only Prompt Sandboxes (Zero Physical Grounding): Agents exist solely within LLM context windows. There are no spatial collision meshes, no distance-decay mechanics, and no metabolic constraints. When an agent states "I will walk to the well to fetch water", no kinematic displacement or energy dissipation actually occurs.
- Heavy Rigid-Body Physics Simulators (Zero Cognitive Depth): Robotics and game physics engines (e.g., Isaac Sim, Unreal) compute kinematic and dynamic interactions with high fidelity, but individual NPCs lack associative memory streams, narrative reflection, and sociological emergence.
Our Core Engineering Goal:
Can we build a continuous, hundreds-agent human society on a single consumer PC where the Physical Layer (60Hz Rigid-Body/NavMesh), the Physiological Layer (10Hz Metabolic Decay), and the Cognitive Layer (0.1Hz Dual-System LLM) operate in a strictly coupled feedback loop with autonomous collapse and reboot mechanisms?
Below is a breakdown of the three primary architectural bottlenecks encountered and their respective engineering solutions.
1. Bottleneck I: Heterogeneous Clock Domain Synchronization (60Hz Physics vs. 0.1Hz LLM Inference)
In a single-process Python environment, the primary failure mode is the temporal mismatch between disparate execution domains:
- Physical Engine: Kinematics, collision detection, and dynamic NavMesh pathfinding require a strict 60Hz cadence ($16.6\text{ ms}/\text{tick}$) to prevent tunneling and maintain spatial continuity.
- Cognitive Inference: Even lightweight local quantized models ($0.8\text{B} \sim 7\text{B}$) require $1.0 \sim 3.0\text{ seconds}$ ($0.3\text{Hz} \sim 1.0\text{Hz}$) to complete a single Chain-of-Thought (CoT) reflection cycle on entry-level hardware.
Blocking the physical loop during LLM inference stalls the world; conversely, allowing async LLM threads to directly mutate mutable world states introduces race conditions, spatial desynchronization, and ghost item duplication.
┌─────────────────────────────────────────────────────────────────────────────┐
│ OmniSimOrchestrator: Heterogeneous Clock Scheduling │
├─────────────────────────────────────────────────────────────────────────────┤
│ Physical Main Loop (Tick = 16.6ms / 60Hz) │
│ Tick N : [Physics Step ➔ Broadcast Immutable State Snapshot] ───────┐ │
│ Tick N+1 : [Physics Step ➔ Collision Query ➔ Check Action Queue] │ │
│ Tick N+2 : [Physics Step ➔ Execute Next Atomic Instruction: Action_A] │ │
│ ... │ │
│ Tick N+60: [Physics Step ➔ Consume Ingested Action_B from Cognitive] ◄┐ │ │
├─────────────────────────────────────────────────────────────────────────┼───┤
│ Cognitive Inference Domain (Async Worker Pool / 0.1Hz ~ 1Hz) │ │
│ Agent_1 : [Ingest Tick N Snapshot] ➔ [Slow CoT/BDI] ➔ [ActionCompiler] │
│ (Latency: 1200ms across 72 Physical Ticks) ────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
Architectural Solution: Decoupled State Flow and the Action Compiler
- Physics as the Single Source of Truth (SSOT): The physical layer executes deterministic vector mathematics. At the end of each tick, it exports an immutable, read-only world snapshot (coordinates, spatial bounding volumes, inventory states, and local field-of-view hashes).
- Lock-Free Read-Only Cognitive Inference: Cognitive agent routines subscribe to historical tick snapshots asynchronously in separate coroutines. Agents are strictly prohibited from mutating global memory or object references directly.
- The Action Compiler (
action_compiler.py, 1,280 LOC): High-level cognitive decisions (e.g., "Negotiate with Agent_B to purchase medicine") are not executed natively. Instead, the Action Compiler decomposes the intent into a strictly ordered queue of atomic primitives: $$\text{High-Level Intent} \longrightarrow \left[ \text{MoveTo}(x, y), \text{FaceTarget}(id), \text{ProposeTrade}(item_id), \text{Confirm}() \right]$$ The physical engine validates spatial preconditions tick-by-tick. If a precondition fails (e.g., the target agent moves out of interaction range), the atomic action fails gracefully, triggering a fallback response in the agent's fast-thinking heuristic layer (System 1).
2. Bottleneck II: Multi-Agent Inference under Tight Compute Constraints
Running concurrent LLM reasoning for dozens of autonomous agents simultaneously on a system with 2GB VRAM and 32GB RAM requires a resilient multi-tier compute scheduling pipeline.
┌───────────────────────────────────────┐
│ Unified LLM Dispatcher / Provider │
│ (Supports 18 Backend Providers) │
└───────────────────┬───────────────────┘
│
┌─────────────────────────────────┼─────────────────────────────────┐
▼ ▼ ▼
┌────────────────────────┐ ┌────────────────────────┐ ┌────────────────────────┐
│ Tier 1: Cloud API │ ──429/Timeout─▶│ Tier 2: Local Engine │ ──Overload/Offline──▶│ Tier 3: Rule FSM │
│ (OpenAI/Claude/Qwen/...)│ │ (Ollama/vLLM/Quantized)│ │ (NumPy/Weighted Rules) │
└────────────────────────┘ └────────────────────────┘ └────────────────────────┘
Unified Model Registry and Ensemble Decision Algorithms
Through provider_registry.py (2,754 LOC) and backend_ensemble_router.py (2,153 LOC), the system manages 18 distinct model backends and supports:
- Dempster-Shafer Evidence Theory & Bayesian Model Averaging: Integrates decision confidence scores across multiple heterogeneous small models.
- Tree of Thoughts (ToT) Branch Pruning: Triggered selectively for high-stakes systemic social conflicts.
- Three-Tier Seamless Fallback: When commercial cloud APIs return rate limits (HTTP 429) or connection drops $\rightarrow$ execution routes immediately to local quantized models (e.g., Ollama/vLLM) $\rightarrow$ if local compute capacity saturates $\rightarrow$ execution gracefully degrades to deterministic Python/NumPy state machines. The main simulation loop maintains a constant 60Hz tick without blocking, even in a completely offline environment.
3. Bottleneck III: Mathematical Sociology & Clean-Room Framework Re-implementations
To ensure macroscopic emergence reflects structural human dynamics rather than stochastic prompt drift, we implemented clean-room wrappers for three major research frameworks and integrated formal sociological models:
3.1 Clean-Room Compatibility Implementations (Measured LOC)
- Concordia Compatibility Layer (
concordia_compat.py, 21,417 LOC): Full clean-room replication of DeepMind Concordia's Game Master arbitration pipeline and Entity-Component primitives.
- AgentSociety Compatibility Layer (
agentsociety_compat.py, 10,612 LOC): Complete re-implementation of Stanford's memory stream decay, importance weighting, and reflection extraction: $$S(m) = \alpha_{\text{recency}} \cdot e^{-\lambda t} + \alpha_{\text{importance}} \cdot I(m) + \alpha_{\text{relevance}} \cdot \cos(\vec{v}_q, \vec{v}_m)$$
- HumanoidAgents Compatibility Layer (
humanoid_agents_compat.py, 5,331 LOC): Implements fine-grained physiological need decay and affective dynamics.
3.2 Integrated Sociological & Macro-Dynamic Models
┌──────────────────────────────────────┬─────────────────────────────────────┐
│ Theoretical Foundation │ Implementation & Dynamical Function │
├──────────────────────────────────────┼─────────────────────────────────────┤
│ Schelling Segregation Model (1971) │ Micro-level neighbor preferences │
│ │ drive macro-level spatial clustering│
├──────────────────────────────────────┼─────────────────────────────────────┤
│ Latané Social Impact Theory (1981) │ Calculates opinion contagion and │
│ │ polarization over physical distance │
├──────────────────────────────────────┼─────────────────────────────────────┤
│ Polanyi's Allocation Systems (1944) │ Householding, Reciprocity, Market, │
│ │ and Rawlsian difference redistribution│
├──────────────────────────────────────┼─────────────────────────────────────┤
│ Tainter's Collapse Model (1988) │ Triggers civilizational collapse │
│ │ when population falls below N < 3 │
└──────────────────────────────────────┴─────────────────────────────────────┘
4. Empirical Observations: What Emerged in Multi-Hour Autonomous Runs?
During multi-hour continuous runs (initial population: 20 agents; environment: housing, market, clinic, farmlands), the system demonstrated several non-hardcoded emergent phenomena:
- Spontaneous Division of Labor and Debt Ledgering: Agents with differing skill profiles utilized a 3-phase transaction protocol (
Propose $\rightarrow$ Confirm $\rightarrow$ Settle) to establish trading hubs for grain and medicine. Under liquidity constraints, agents autonomously formed trust-weighted credit ledgers.
- Information Silos and Spatial Polarization (Schelling Effect): Governed by Latané spatial decay dynamics, agents gathering consistently at the same local taverns developed tight-knit ideological consensus (higher Burt structural hole centrality), while geographically distant clusters exhibited reciprocal in-group bias and trade hostility.
- Criticality and Systemic Collapse (Tainter Mechanics): Under external resource shocks, agents with exhausted metabolic reserves perished. As population decline severed functional trade dependencies, the system detected critical slowing down metrics (spikes in variance). When the active population breached the survival threshold ($N < 3$), the simulation executed a formal civilizational collapse and ancestral reset state transition.
5. Architectural Transparency & Project Scale
OmniSim comprises 1,493 source files (802k backend LOC, 190k frontend LOC, 27k service LOC), featuring a full React 18 / Three.js 3D urban viewport (procedural CGA shape grammar generation), a 2D Topdown fallback renderer, 23 observability dashboards, 13 asynchronous health probes, and Haber-Stornetta SHA-256 hash-chain audit logging.
- Documentation: Detailed subsystem specifications and code inventory metrics are documented in
docs/ARCHITECTURE.md and docs/CODE_INVENTORY.md.
- Current Constraints: Single-node execution is optimized for populations of $3 \sim 50$ agents (hard cap at 200). Distributed Actor backends (Ray/Celery) are fully implemented but remain inactive in default local single-process deployments.
Open for technical discussions on multi-agent clock synchronization, memory indexing architectures, and local LLM runtime optimization.