OneAdvanced runs 50+ production AI agents on UK-sovereign AWS infrastructure. Every component (model inference, vector storage, orchestration runtime) stays within UK regional boundaries. This constraint shapes every architectural decision, from how they version agent definitions to how they route RAG queries across shared vector indexes.
The deployment exposes trade-offs that matter when you need strict data residency but still want multi-agent orchestration at scale. Self-hosting Llama 4 Maverick and Llama Guard 4 on SageMaker, building RAG pipelines on pgvector, and running the Strands Agents SDK on ECS creates specific failure modes and observability challenges.
Architecture: ECS Containers, Self-Hosted Models, Shared RAG
OneAdvanced’s stack has four layers:
- Model inference: Llama 4 Maverick (405B) and Llama Guard 4 on SageMaker AI endpoints in UK regions
- RAG pipeline: pgvector for embeddings and retrieval, Aurora PostgreSQL for vector storage
- Agent orchestration: Strands Agents SDK running on Amazon ECS
- State isolation: Per-agent containers with shared model endpoints but isolated execution contexts
The Strands SDK handles tool binding, conversation state, and inter-agent calls. Each agent runs in its own ECS task, but all tasks share the same SageMaker inference endpoints. This creates a shared-nothing execution model with shared-everything inference.
Why ECS Instead of Lambda or Fargate Spot
ECS gives OneAdvanced predictable latency for long-running agent conversations. Lambda cold starts would break multi-turn dialogues, and Fargate Spot’s interruption risk conflicts with agents that hold state across multiple tool calls.
ECS also simplifies tenant isolation. Each agent gets its own task definition, environment variables, and IAM role. When an agent needs to call another agent, it uses service discovery within the ECS cluster rather than exposing public endpoints.
RAG Pipeline: pgvector Query Routing and Index Updates
The RAG pipeline uses pgvector for semantic search across enterprise documents. All embeddings stay in Aurora PostgreSQL within UK regions. The challenge is routing queries when 50+ agents share the same vector index but need tenant-specific filtering.
OneAdvanced uses row-level security in PostgreSQL to enforce tenant boundaries. Each agent’s database connection includes a tenant_id context variable. Queries automatically filter to the correct document subset without application-layer logic.
Index updates happen asynchronously. When new documents arrive, a separate ECS task generates embeddings via the same SageMaker endpoint, then inserts vectors into Aurora. Agents see updated results within seconds, but there’s no synchronous guarantee. This creates a window where an agent might miss recently uploaded documents.
Vector Index Versioning
OneAdvanced versions vector indexes by schema, not by snapshot. When they change embedding models or chunking strategies, they create a new table in Aurora and backfill it. Agents switch to the new table via a feature flag in their task definitions.
This avoids the complexity of maintaining parallel indexes but requires careful coordination. If an agent references the old table during a migration, it gets stale results. They mitigate this with blue-green deployments: new agent versions point to the new table, old versions stay on the old table until traffic shifts.
Agent Definitions: CI/CD Boundary Between Models and Logic
Agent definitions live in Git as YAML files. Each file specifies:
- Agent name and description
- Tool bindings (API endpoints, database queries, file operations)
- Conversation templates
- Model parameters (temperature, max tokens, stop sequences)
The CI/CD pipeline deploys agent definitions to ECS by building Docker images. Each image includes the Strands SDK, the agent YAML, and environment-specific configuration. SageMaker model endpoints are referenced by ARN, so model updates don’t trigger agent redeployments.
This separation creates a versioning problem. If you update Llama 4 Maverick’s fine-tuning or change Llama Guard 4’s safety thresholds, existing agent definitions might behave differently. OneAdvanced handles this with integration tests that run against staging SageMaker endpoints before promoting model changes to production.
# Example agent definition (simplified)
agent:
name: contract-reviewer
model:
endpoint: arn:aws:sagemaker:eu-west-2:123456789012:endpoint/llama-4-maverick
guard: arn:aws:sagemaker:eu-west-2:123456789012:endpoint/llama-guard-4
temperature: 0.2
max_tokens: 2048
tools:
- name: search_contracts
type: rag
vector_table: contract_embeddings
top_k: 5
- name: extract_clauses
type: api
endpoint: https://internal.api/extract
conversation:
system_prompt: "You review contracts for compliance risks."
max_turns: 10Observability: Detecting Failures When Agents Share Infrastructure
Shared SageMaker endpoints create attribution problems. If inference latency spikes, which agent caused it? If Llama Guard 4 blocks a request, was it a legitimate safety issue or a false positive?
OneAdvanced uses CloudWatch Logs Insights to correlate agent task IDs with SageMaker invocation traces. Each agent tags its inference requests with a task_id dimension. When latency exceeds thresholds, they query CloudWatch to identify the agent and conversation context.
For RAG failures, they log every pgvector query with the agent ID, tenant ID, and result count. If an agent gets zero results when it should find documents, they replay the query manually to check for row-level security misconfigurations or stale indexes.
Failure Modes to Watch
| Failure Mode | Detection | Mitigation |
|---|---|---|
| SageMaker endpoint throttling | CloudWatch metric: ModelInvocationThrottles | Per-agent rate limits in SDK |
| pgvector query timeout | PostgreSQL slow query log | Query-specific indexes, connection pooling |
| Agent state desync | Missing conversation turns in logs | Idempotent tool calls, retry with exponential backoff |
| Cross-agent call deadlock | ECS task timeout | Circuit breakers, call depth limits |
| Llama Guard false positive | High block rate in CloudWatch | Manual review queue, guard threshold tuning |
Inter-Agent Communication: Service Discovery and Call Boundaries
When one agent needs to call another, it uses ECS service discovery. The Strands SDK resolves agent names to internal DNS records (e.g., contract-reviewer.agents.local). Calls go over HTTP within the VPC, never leaving UK regions.
Each agent exposes a /invoke endpoint that accepts a conversation context and returns a response. The calling agent includes its own task ID in the request headers so the receiving agent can log the dependency.
This creates a potential deadlock risk. If Agent A calls Agent B, and Agent B calls Agent A, both tasks wait indefinitely. OneAdvanced mitigates this with a call depth limit (default: 3) and a global timeout (default: 30 seconds). If an agent exceeds either limit, the SDK returns an error and logs the call chain.
Sovereign Infrastructure Constraints: What You Give Up
Running everything in UK regions eliminates some AWS services. OneAdvanced can’t use:
- Amazon Bedrock: Not available in all UK regions, and data residency guarantees vary by model
- Amazon Kendra: Managed search service doesn’t support UK-only deployments
- AWS Lambda@Edge: Runs in global edge locations, violates data residency
They also can’t use third-party model APIs (OpenAI, Anthropic, Cohere) because those services don’t guarantee UK data residency. Self-hosting Llama 4 on SageMaker is the only option.
This increases operational complexity. OneAdvanced manages model versioning, endpoint scaling, and fine-tuning pipelines that would be abstracted away by Bedrock. They also lose access to Bedrock’s built-in guardrails and knowledge base integrations.
Deployment Shape: Blue-Green with Staged Model Rollouts
OneAdvanced deploys agents using blue-green ECS service updates. New agent versions run alongside old versions until traffic shifts. This works well for agent logic changes but creates problems for model updates.
If they update the Llama 4 Maverick endpoint, all agents (blue and green) see the new model immediately. To avoid this, they maintain two SageMaker endpoints: llama-4-maverick-stable and llama-4-maverick-canary. New agent versions point to the canary endpoint. After validation, they promote the canary model to stable and update all agent definitions.
This doubles inference costs during rollouts but prevents model changes from breaking production agents. They also use SageMaker’s multi-model endpoints to reduce costs when running multiple Llama Guard 4 variants (different safety thresholds for different agent types).
Technical Verdict
Use this architecture when:
- Data residency requirements prohibit third-party model APIs
- You need 10+ agents with shared infrastructure (models, RAG, observability)
- Agent conversations span multiple turns and require stateful execution
- You have the operational capacity to manage SageMaker endpoints and pgvector indexes
Avoid this architecture when:
- You can use managed services like Bedrock (simpler, cheaper, faster iteration)
- Agents are short-lived or stateless (Lambda is cheaper and easier to scale)
- You need sub-second cold start latency (ECS tasks take 10-30 seconds to start)
- You lack expertise in PostgreSQL performance tuning and ECS networking
The sovereign infrastructure constraint forces OneAdvanced to self-host everything. If you don’t have that constraint, Bedrock Agents with Knowledge Bases is a better starting point. But if you do need strict data residency, this stack shows how to build multi-agent orchestration without leaving regional boundaries.