To implement a robust, production-grade infrastructure for your 17-Agent cross-border e-commerce swarm, we must treat Agent states and logs exactly like distributed microservices.
Here is a comprehensive engineering design to solve deep JSON nesting (preventing Token Bloat) and build a real-time, queryable "External Log Memory" system for Agent reflection.
Part 1: Lightweight Agent State Serialization Mechanism
To stop JSON payloads from compounding in size as they pass through 17 Agents (e.g., Selection → Pricing → Listing → Ads), we must transition from a Value-Passing (Deeply Nested JSON) model to a Reference-Passing (Flat Session-Graph) model.
1. The Design: The "Pointer-State" Schema
Instead of embedding previous Agents' full history inside the current payload, each Agent operates on a flat, highly structured event packet. It reads and writes data via URI references (Pointers) backed by a shared Redis State Store.
Standard Swarm Payload Envelope (Flat Structure)
{
"$schema": "https://fitvibe.com",
"trace_id": "tr_amz_us_99281a8c",
"event_id": "evt_pricing_adjust_004",
"parent_event_id": "evt_selection_alert_012",
"tenant_id": "tenant_brand_alpha",
"timestamp": 1782715500,
"sender": {
"agent_role": "PricingAgent",
"instance_id": "inst_price_qwen3_4b_02",
"digital_signature": "0x8f2a...c3b1"
},
"state_pointers": {
"product_context": "redis://state/product:sku_hoodie_01",
"market_intelligence": "redis://state/market:sku_hoodie_01",
"ad_performance": "redis://state/ads:sku_hoodie_01"
},
"action_payload": {
"command": "UPDATE_PRICE",
"parameters": {
"sku": "HOODIE-BLK-XL",
"currency": "USD",
"old_price": 29.99,
"new_price": 24.99,
"adjustment_reason_code": "COMPETITOR_UNDER_CUT"
}
}
}
2. Engineering Constraints to Eliminate Token Bloat
- Strict Schema Enforcement (P0): Use fast compiled validation (like ajv in Node or pydantic in Python). If an Agent outputs nested, arbitrary JSON arrays inside action_payload, the Guardrail rejects it immediately at the gateway.
- State Hydration via Dynamic Tool Call: When PricingAgent receives this flat message, it doesn't ingest all historical logs. If it needs to know why the product was selected, it triggers a system tool: fetch_state_by_pointer("redis://state/product:sku_hoodie_01"). This keeps the base LLM prompt minimal and compact.
- Automatic Compaction Cycles: When a trace_id hits 5 hops (hops are tracked in Redis), the Swarm Orchestrator forces a Compaction Event. A specialized "Summarizer Agent" reads the full Redis state, compresses the history into a 200-word markdown summary, rewrites the root pointer, and clears the transaction history.
Part 2: Dual-Engine External Log Memory Infrastructure
To enable sub-second log ingestion and allow downstream Agents (or a Meta-Inspector Agent) to run Reflection (Self-Correction), we deploy a hybrid storage topology: Relational (PostgreSQL) for transactional flow + Vector (Qdrant/Milvus) for semantic correlation.
[Agent JSON Output] ──> [Guardrails] ──> [Kafka/RabbitMQ Stream]
│
┌────────────────────────┴────────────────────────┐
▼ ▼
[PostgreSQL (TimescaleDB)] [Vector DB (Qdrant)]
- Structural Ledger & Audit Trail - Semantic Intent Clustered Logs
- Query: "Find price cuts for SKU X" - Query: "Find similar pricing failures"
1. Relational Engine (PostgreSQL / TimescaleDB): The Immutable Ledger
This serves as the deterministic source of truth. Because Agent logs are inherently time-series data, using TimescaleDB extensions allows lightning-fast execution analytics.
Table Schema Design
CREATE TABLE agent_execution_ledger (
trace_id UUID NOT NULL,
event_id VARCHAR(64) PRIMARY KEY,
parent_event_id VARCHAR(64),
tenant_id VARCHAR(64) NOT NULL,
timestamp TIMESTAMPTZ NOT NULL,
agent_role VARCHAR(32) NOT NULL,
digital_signature TEXT NOT NULL,
action_command VARCHAR(64) NOT NULL,
payload_json JSONB NOT NULL, -- Indexed via GIN for structural querying
is_interrupted_by_guardrail BOOLEAN DEFAULT FALSE
);
-- Indexing for sub-second lookups by trace and tenant (Data Isolation)CREATE INDEX idx_ledger_trace_tenant ON agent_execution_ledger(tenant_id, trace_id);CREATE INDEX idx_ledger_payload_gin ON agent_execution_ledger USING gin (payload_json);
2. Vector Engine (Qdrant): The Semantic Intuition Cache
While SQL handles structured queries ("Give me the price changes for SKU X"), the Vector Database allows an Agent to reflect on abstract mistakes ("Find instances where our ad bidding strategy was too aggressive during holiday promos").
Document Embedding Transformation Pattern
Do not embed the raw JSON directly; raw punctuation ruins cosine similarity scores. Instead, convert the structured JSON into an Asymmetric Semantic Log Block before embedding:
[Tenant: Alpha][Agent: AdAgent][Action: INCREASE_BUDGET]
Trace: tr_amz_us_99281a8c
Context: Budget increased by 40% for SKU HOODIE-BLK-XL because Competitor X raised their pricing.
Outcome: System flagged by Guardrail 30 minutes later due to sudden conversion rate drop (ROI crash).
- Vector Query Optimization: Store the trace_id and tenant_id as strict Payload Filters in Qdrant. When an Agent searches memory, it forces a hard filter matching its specific tenant_id, guaranteeing zero cross-customer data leakage.
Part 3: Running the Reflection Cycle (Agent Self-Correction Workflow)
How does an Agent actually use this setup to "reflect" in real-time? Let’s trace a live scenario in your system.
Scenario: The Ad Agent is underperforming (Low ROI)
- Trigger: The Operating Agent flags that SKU-01’s advertising ROI has dropped below 1.2 for 3 consecutive hours.
- The Reflection Prompt: The orchestrator wakes up the AdAgent and passes a localized memory footprint via a tool call:
System Instruction: You are underperforming on ROI. Review your past decisions.
[Structured Tool Results - Past 24 Hours via Postgres]:- 09:00: Increased bid to $2.50 (Reason: High inventory)- 12:00: Switched keywords to "Premium Hoodie" (Reason: Content Agent recommendation)
[Semantic Tool Results - Similar Historic Failures via Qdrant Search]:- "Trace_2025_BFCM: AdAgent raised bids on 'Premium Hoodie' during high-traffic windows, but because the Listing Agent had a typo in the sizing chart, conversion crashed, tanking ROI."
- The Realization (Execution Alignment): By reviewing the semantic history, the AdAgent realizes the root cause isn't its bidding engine—it’s an external alignment issue. It outputs a structured tool request: verify_listing_integrity(sku="HOODIE-BLK-XL").
- The Result: The Listing Agent finds a broken image link, fixes it, and the system resolves itself seamlessly—without a human ever logging into Seller Central.