Autoscaling, reindex, and ingest stability
Status: Testing in progress — Azure Central US
Priority: Must have
Audience: SRE, search platform
Why these belong together
These issues show up under the same production load profile: Serverless capacity changes interacting with search, backfills, and writes. Treat them as one hardening workstream before locking production SLOs.
| Theme | Symptom |
|---|---|
| Autoscaling | Scale oscillation hurts search latency / stability |
| Reindex | Long backfills interrupt when capacity changes |
| Ingest tail | Occasional extreme write latency |
Current state
Azure Central US is delivered. Production-like validation can now run in-region — no longer blocked on region availability.
Options
| Option | What it means | Tradeoff |
|---|---|---|
| A. Joint tuning + resilient patterns (recommended) | Cooldowns, pre-warm, capacity buffer; checkpointed reindex; measure and bound ingest tails before write SLOs | Medium effort; unblocks production without waiting on every platform fix |
| B. Wait for platform-only fixes | Rely on Elastic Serverless autoscaling / reindex / ingest improvements alone | Lower near-term ops work; cutover risk if SLOs are locked too early |
| C. Oversize Search capacity | Keep high minimum / Search Power so scale events are rare | Stabilizes latency; higher ongoing VCU cost |
| D. Defer production traffic | Hold live cutover until all three themes are closed | Lowest risk; delays value from Serverless |
| E. Accept interim exceptions | Documented lag / ingest / backfill exceptions with expiry dates | Fastest path to limited production; needs security / SRE sign-off |
Recommendation
Commit to option A, with production-like validation in Azure Central US:
- Autoscaling — Tune cooldowns, scale-down sensitivity, and buffer above minimum Search capacity; pre-warm ahead of known peaks; retest under representative QPS in Central US.
- Reindex — Use resumable / checkpointed backfills (slices,
updated_atwindows, idempotent bulk); automate resume after capacity changes. - Ingest tail — Measure p95 / p99 / max under production-like write rates; correlate with scale events; agree a mitigation or signed exception before formal write SLOs.
Use option C only as a temporary bridge during cutover if needed. Prefer A over B so you are not blocked on an open-ended platform calendar. Keep E narrow and time-boxed if anything remains open.
Detail by theme
1. Autoscaling tuning
Scale-up / scale-down that is too sensitive can sawtooth under production-like traffic.
- Capture baseline latency under representative QPS.
- Tune cooldowns, scale-down sensitivity, and buffer above minimum Search capacity.
- Pre-warm or schedule capacity ahead of known peaks when possible.
- Re-test and watch for oscillation in project telemetry.
2. Reindex under autoscaling
Long reindex / backfill jobs can break when nodes are replaced or capacity changes mid-job.
- Prefer resumable / checkpointed patterns (slices,
updated_atwindows, idempotent bulk). - Avoid single long-lived tasks that cannot survive a node change.
- Automate resume from the last successful checkpoint.
- Retest full backfill after related platform improvements ship.
3. Ingestion tail latency
Average ingest can look fine while p99 / max spikes miss write SLOs.
- Measure ingest percentiles under production-like write rates.
- Correlate spikes with autoscaling, bulk size, mapping changes, and concurrent search.
- Agree interim mitigations (batching, backoff, queue depth) if needed.
- Gate formal write SLOs on a closed root cause or signed exception.
Acceptance criteria
- No sustained scale oscillation under the agreed load profile in Central US
- Search p95 / p99 within SLO during scale events
- Backfill can pause / resume without a full restart
- Ingest tail characterized; fix or mitigation verified
- Capacity, reindex, and write SLO notes documented in one runbook section
- Chosen option (A–E) recorded with owners and target date
Related
- Search capacity pre-scale (override) — set the search floor before load tests
- Cold cache during deployments (also control-plane / capacity adjacent)
- Autosharding (index shape before load tests)
- Snapshot restore → Point-in-time restore (cutover windows)
- Azure Central US region (delivered)