Symptoms:
• Messages are repeatedly becoming visible
• Lambda processing appears to restart
• Queue depth is growing
• Some messages are processed more than once
⚠️ Note for judges: This wording is intentionally DIFFERENT from INC-1832. No mention of "timeout" or "backlog". A keyword search would never connect these. Only semantic vector search can.
What happens when you click Start:
- HTTP POST to API Gateway → triggers opsmemory-orchestrator Lambda
- Orchestrator validates input, generates symptoms embedding text
- INSERT into CockroachDB incidents table
- Returns incident with status: "open"
What happens behind the scenes:
- Orchestrator invokes opsmemory-telemetry Lambda
- Telemetry Lambda calls CloudWatch
GetMetricDatafor SQS + Lambda metrics - Fetches: ApproximateNumberOfMessagesVisible, Duration, Errors, Throttles
- Calls
FilterLogEventsfor Lambda error logs - Parses anomalies: queue depth > 100, duration near timeout
- Returns structured telemetry summary for the reasoning engine
What happens (CockroachDB + Bedrock integration):
- Orchestrator invokes opsmemory-memory-search Lambda
- Lambda calls Amazon Bedrock Titan Embed Text v2 to generate 1024-dim embedding of current symptoms
- Embedding vector sent to CockroachDB via MCP Client interface
- SQL:
SELECT ... FROM incident_embeddings e JOIN incidents i ... ORDER BY e.embedding <=> $1::vector - HNSW vector index returns nearest neighbors by cosine distance
- Converts distance to similarity:
similarity = 1 - cosine_distance - Filters by threshold (0.5) — returns matches with full historical context
- INC-1832 found at 86.4% similarity despite zero keyword overlap!
🧠 OPERATIONAL MEMORY MATCH
Root cause found: SQS visibility timeout (60s) too short for Lambda processing time (~120s)
Remediation applied: Increased visibility timeout from 60s → 180s
Outcome: ✅ Successful — backlog cleared within 5 minutes
Lessons learned: Visibility timeout should be ≥6× average processing time
It reasons about the problem informed by what worked before.
What happens (Bedrock Nova Pro integration):
- Orchestrator invokes opsmemory-reasoning Lambda
- Lambda builds structured prompt with: incident data, CloudWatch telemetry, and memory matches
- Calls
bedrock:InvokeModelwith model amazon.nova-pro-v1:0 - System prompt instructs: "Cite historical incidents, provide confidence 0-1, be specific about AWS API calls"
- Temperature: 0.1 (low = consistent, reliable reasoning)
- Nova Pro outputs structured JSON: diagnosis, root cause, confidence, remediation with exact parameters
- Response parsed and validated; fallback if JSON extraction fails
Every destructive action requires explicit human approval. This is a core safety principle.
What happens:
- Agent state transitions to "awaiting_approval" — stored in CockroachDB agent_runs table
- Recommendation displayed to operator with full context
- Approval recorded in CockroachDB approvals table with: who approved, when, what was proposed
- Audit trail maintained — every action is traceable
- If rejected: agent stops, incident marked as "blocked", no infrastructure change
What happens (actual AWS change):
- Orchestrator invokes opsmemory-remediation Lambda
- Lambda validates action is in allowed registry (sqs_update_visibility_timeout)
- Captures previous state via
sqs:GetQueueAttributes(for rollback) - Executes
sqs:SetQueueAttributes→ VisibilityTimeout = 180 - Returns: {success: true, previousState: {timeout: 60}, newState: {timeout: 180}}
- IAM role scoped to ONLY the payment-queue ARN — least privilege
What happens (memory growth):
- Verification Lambda checks: SQS queue attributes match expected state
- CloudWatch metrics confirm: queue depth stabilizing, no new errors
- Memory-search Lambda generates new 1024-dim embedding for INC-1947 symptoms
- INSERT into CockroachDB incident_embeddings: new vector stored
- Incident marked "resolved" in CockroachDB
- Agent run completed with full audit trail
- Memory now contains 2 experiences — agent is smarter
- Future incidents matching either INC-1832 or INC-1947 will be recalled