Back to production requirements

Capacity overrides runbook

Status: Complete — runbook published
Priority: Must have
Audience: SRE, search platform, Elastic Product / Support / Engineering

Runbook URL: git-scale-elastic.vercel.app/docs/overrides

Purpose

This runbook covers how to request and apply Serverless capacity overrides (pre-scale floors) for Azure Central US projects. Overrides are used whenever autoscaling from a cold or undersized baseline would race load — POC scale-up errors, benchmarks, cutover, and other planned heavy windows.

Billing note: Once an override is applied, Search VCU (and related) charges start at the overridden floor, even if traffic is below that level. The customer must explicitly accept the cost impact before the override is applied.

Longer-term path: A customer-facing Reservation / pre-scale API is in development. Do not block pilot or Central US testing on that API — use this interim process now.

Use cases overrides cover

Use caseWhenWhat override does
A. Search capacity pre-scale (pilot / POC)Before production-like or benchmark loadSets a search-tier floor so tests do not race cold scale-up (POC historically hit errors without this).
B. Cutover / go-live windowTraffic shift onto ServerlessHolds enough search capacity through cutover spikes; remove or lower after steady state if agreed.
C. Planned heavy windowsLarge reindex, backfill, or ingest surgeTemporarily raises the relevant tier (search and/or ingest) for a dated window; schedule start/end.
D. Regression / soak testingCentral US validation campaignsMatches or exceeds the load profile under test so autoscaling noise does not invalidate results.
E. Event / peak readinessKnown calendar peaksPre-positions capacity ahead of predictable QPS jumps (still billed at the floor).

Most near-term work is A + B + D. Treat C and E with the same request packet and explicit end time so floors do not linger.

What you must collect (required packet)

Every override request — Slack or support ticket — must include all of the following:

  1. Project — which Serverless project (name + project ID / Cloud console link).
  2. Tier — which capacity tier to override (e.g. search; ingest if applicable — confirm with Product).
  3. Override value — the concrete floor (size / scale target). Start from the POC reference below and reconfirm with Product for the current Search Power / VCU model.
  4. When — date/time to add the override (and, if known, date/time to reduce or remove it).
  5. Customer acceptance — explicit confirmation that the customer understands the override increases VCU cost from the moment it is applied (billed at the floor).

Incomplete packets will be sent back; Engineering will not invent missing fields.

Interim request process (until Reservation API)

Requester (customer or Elastic AE/SA on their behalf)
        │
        ├─ Preferred: Support ticket with full packet (1–5)
        │         → Support routes to Engineering
        │         → Engineering ticket → manual apply
        │
        └─ Alternate: Slack — @ the owning engineer / on-call
                  with the same full packet (do not rely on an
                  unowned channel message)

Preferred path (recorded for the customer)

  1. Open a support ticket that includes the five required fields.
  2. Support routes to Engineering.
  3. Engineering creates an internal ticket and manually applies the override when the packet is complete.
  4. Confirm in the ticket: override applied, project, tier, value, effective time.
  5. Schedule follow-up to lower or remove the override when the window ends (unless Product agrees it stays for production).

Slack path (acceptable interim)

  • Post the full packet in the agreed channel and @ the responsible person so it is not missed.
  • Prefer also opening a support ticket so the customer has an official record — AE/SA may file on the customer’s behalf if all five fields are present.

Do not

  • Apply or request an override without customer acceptance of VCU cost.
  • Leave a temporary floor in place past the agreed end date without a new approval.
  • Block testing waiting for the Reservation API.

POC reference size (starting point only)

Search Engineering pre-scaled the original POC search deployment to 22 × 52 Gi for benchmarks. Use that as a conversation starter for Central US test projects — reconfirm the override value with Product for today’s architecture and Search Power / VCU model before applying.

Step-by-step (search pre-scale — primary path)

  1. Create or identify the Azure Central US Serverless project(s).
  2. Agree tier + override value with Elastic Product (start from 22×52 Gi; adjust).
  3. Obtain written customer acceptance of VCU billing at the floor.
  4. File the request via support ticket (preferred) or Slack with @ + full packet.
  5. Wait for Engineering confirmation that the override is live.
  6. Run benchmarks / production-like / cutover tests against the floor.
  7. After the window: request reduce/remove (same packet shape: project, tier, new value or “remove”, when, acceptance of any remaining cost posture).
  8. Track Reservation API as the follow-on self-serve path (not a pilot blocker).

Acceptance criteria

  • Overrides runbook published (/docs/overrides)
  • Interim request process documented (project, tier, value, when, VCU acceptance)
  • Use cases covered (pre-scale, cutover, heavy windows, soak, peaks)
  • Reservation API noted as follow-on (not a GA blocker if override works)
  • Per-window: Central US project(s), size, and customer VCU acceptance recorded when applied
  • Per-window: Engineering confirms override applied; reduce/remove when window ends

Template (copy into ticket or Slack)

OVERRIDE REQUEST
Use case: [ A pre-scale | B cutover | C heavy window | D soak test | E peak ]
Project name / ID:
Region: Azure Central US
Tier: [ search | ingest | other — specify ]
Override value:
Add at (date/time, timezone):
Remove or reduce at (date/time, timezone) — or “keep for production”:
Customer accepts VCU billing at this floor: [ Yes — name / date ]
Context / load profile (optional):

Related