Elastic Serverless
Running interactive search on Elastic Serverless
Move the interactive search workload to Serverless first — without lifting the entire historical corpus into the cloud. Private networking, autosharding, restore, audit logging, and cold-cache workarounds are complete. Azure Central US and operational stability remain underway. Production requirements are listed below.
- Audit logging: complete for GA via the Logstash bridge to Splunk / Azure Monitor; native export remains on the roadmap. Proposal
Production requirements
Each item has progress and priority. Open a detail page for options and recommendations.
- CompleteMust have
Private networking (Private Link)
Required for secure connectivity from your network to Elastic Serverless.
Document the approved path in your runbook; networking is ready for production use.
- CompleteMust have
Autosharding
Index layout for large interactive search on Serverless. Platform autosharding is available and accepted as complete for the production path.
Apply autosharding on interactive indices and keep layout documented in the cutover runbook.
- Complete (via workaround)Must have
Snapshot restore → Point-in-time restore
Disaster recovery for the Serverless search plane must meet your recovery objectives. The product path is point-in-time restore, not classic snapshot restore alone.
Accepted via workaround for GA. Align RPO/RTO for the interactive Serverless dataset and keep the interim restore runbook current.
- Complete (via workaround)Must have
Audit logging to external destinations
Security requires audit events in your domain (Splunk today; Azure Monitor later). Native Serverless destinations alone do not meet that for GA — external delivery is a future product enhancement.
Accepted for GA via the Logstash bridge: pull Serverless audit logs over Private Link and forward to Splunk / Azure Monitor. Native export remains on the roadmap.
- Complete (via workaround)Nice to have
Cold cache during deployments
Control-plane and node replacement events can clear caches and temporarily raise latency.
Accepted via workaround for GA. Continue to follow platform warming improvements as they ship.
- UnderwayMust have
Azure Central US region for Serverless
Primary Azure region for production clusters and the application monolith — Serverless must be available where the workload already runs for latency.
Target availability end of September. East US is not the production home. Cross-region replication is a last resort, not the plan.
- Underway (Pending Central)Must have
Autoscaling, reindex, and ingest stability
Under production-like load, scale oscillation, interrupted backfills, and ingest tail latency show up together — capacity changes interacting with search and writes.
Continue joint tuning once Azure Central US is available for production-like testing. Cooldowns, pre-warm, buffers, resumable reindex, and ingest-tail mitigation before locking write SLOs.
Recommended sequence
Phase 1
Confirm secure connectivity and region
Private networking and autosharding are complete. Finish Azure Central US so the search plane sits with the existing primary region.
Phase 2
Move interactive search to Serverless
Start with user-facing search on Autosharding-ready indices once Central US is live.
Phase 3
Restore, DR, and audit export
Point-in-time restore and audit logging to Splunk / Azure Monitor are accepted via workaround for GA.
Phase 4
Harden operations
Autoscaling, reindex, and ingest stability remain underway pending Central US for production-like validation.
Architecture & economics (optional)
Boost Window, Search AI Lake, and related background for architecture and cost discussions.
Expand
Architecture & economics (optional)
Boost Window, Search AI Lake, and related background for architecture and cost discussions.
2B docs
140 TB+ searchable corpus · 150k–200k docs/sec ingest
5 people · 1,300 nodes
Self-managed footprint today — Serverless is the ops off-ramp
last ~90 days
Search Boost Window (~10%) · lake for the rest at as low as $0.047/GB-mo
Why Search Boost Window cuts Serverless Search TCO
Elasticsearch Serverless Search bills VCUs (not ECUs) plus Search AI Lake storage. Search VCU baseline — including idle — scales with search-ready dataset size. Treating all 2B documents as search-ready is why compute looks prohibitive. Shorten the Boost Window; keep history in the lake.
Naive lift
~100%
Search VCU baseline · full corpus search-ready
- · 140 TB+ drives idle + active Search VCUs
- · 32-shard parity required
- · Snapshot = 28-day rebuild risk
Boost Window + lake
~10%
Search VCU · lake storage for ~90%
- · Search-ready tip ~15 TB · 6 shards
- · Lake at as low as $0.047/GB-mo · still queryable
- · Tip RTO 1–3 hours
Relative Search VCU baseline
Elastic Streams
MySQL CDC → replaces custom sync layer
Native pipeline · no DIY sync code
Search Boost Window
last ~90 days · ~200M docs · ~15 TB
Search VCU baseline on ~10% search-ready set
Search AI Lake
Historical · ~1.8B docs · bulk of 140 TB+
as low as $0.047/GB-mo · still searchable
Federated / full-corpus query
Users still search the full 2B-document corpus. Data outside the Boost Window stays searchable from the Search AI Lake; Federated Data Sources / CPS can span projects if the corpus is split. Note: federated queries may increase Search VCUs on the querying project.
Naive lift vs tiered Federated design
Same 2B-document / 140 TB+ code-search corpus — two Serverless Search economics. Toggle to compare Search VCU baseline, lake storage, shard needs, and restore risk.
Boost Window + lake + federation
Search-ready tip · lake for history · one query
Exploit temporal locality with Serverless Search economics: keep the last ~90 days in the Search Boost Window / search-ready set (~10% of corpus). Historical data remains in the Search AI Lake — fully searchable, billed as storage GB, with minimal Search VCU baseline impact. ES|QL data federation reads customer object storage (does not write — Streams ask fills that gap). Cross-project search (CPS) spans Serverless projects.
The EB reframe: shrink the search-ready set so Search VCU baseline collapses — and lake + federated query keep full-corpus search.
Key metrics (2B doc scenario)
- Search-ready set~10% (~200M docs)
Search Boost Window · ~15 TB
- Search VCU baseline~10%
Idle baseline ∝ boost window, not full lake
- Lake storage~90% of corpus
as low as $0.047/GB-mo · still searchable
- Shard need6 shards sufficient
Autosharding becomes non-blocking
- Search-ready RTO1–3 hours
~15 TB tip · lake data already object-stored
Pros
- Search VCU baseline priced on ~10% search-ready tip; lake holds history as storage GB
- Data outside the Boost Window stays searchable from the Search AI Lake
- Snapshot / rebuild shrinks to the search-ready tip — dissolves the #1 blocker without a product wait
- 6-shard Serverless default fits a 200M-doc hot tip
- Matches official Serverless levers: Search Power + search-ready dataset size
Cons
- Requires validating MySQL CDC path (Streams native vs Logstash/Debezium)
- ES|QL federation is read-only — managed Streams write to customer blob is a Product ask
- Queries that hit deep history may pull more Search VCUs than boost-window-only traffic
- CPS / federation queries can increase Search VCUs on the querying project
- Storage GB is still billed for retained lake data — this cuts compute, not all retention cost
Best for
- · Enterprise code search with strong recent-vs-historical locality
- · Teams blocked on snapshot RTO / autosharding at full-corpus scale
- · EB conversations that need a TCO reframe grounded in VCU + lake pricing
Side-by-side at a glance
| Dimension | Naive POC | Boost + Lake |
|---|---|---|
| Search VCU baseline | ~100% search-ready | ~10% Boost Window |
| Search AI Lake | Full corpus · compute-heavy | History as storage · as low as $0.047/GB-mo |
| Query UX | Single search-ready index | Lake + federated UX |
| Search-ready RTO | 28-day rebuild risk | 1–3 hours |
| Shards needed | 32 parity vs 6 default | 6 fits ~200M tip |
| Ingest layer | Custom CDC sync | Elastic Streams (native) |
How tiering dissolves active blockers
Same production criteria — different math when only ~10% of the corpus is search-ready (Search Boost Window).
Snapshot restore
Effectively eliminates the #1 blocker without waiting on a Serverless snapshot roadmap item.
Naive path
Full 140 TB+ search-ready → 28-day rebuild risk
Tiered + Federated
Search-ready tip ~15 TB → RTO 1–3 hours. Historical corpus already in Search AI Lake object storage — nothing to rebuild.
Autosharding
Complete — apply on interactive indices and document in the cutover runbook.
Naive path
32-shard parity was needed for a flat 2B-doc search-ready index; Serverless defaults were insufficient.
Tiered + Federated
Autosharding is available (confirmed 8/3) and accepted as complete for the production path.
Reserved / large SKUs
Scale reservation can target the search-ready tip, not 140 TB+ of always-on Search VCU baseline.
Naive path
Large capacity sized for the full search-ready corpus (and its Search VCU baseline).
Tiered + Federated
Small Boost Window tip removes the need for oversized capacity reservations.
EB conversation frame: the search-ready tip is small enough that snapshot restore is a non-issue — and Search VCU baseline collapses with it. Stronger than asking for a roadmap commitment on full-corpus Serverless restore.
Same shape as Observability — different Serverless meters
Conceptually: recent data expensive, history cheaper, one query UX. On Elasticsearch Serverless Search the meters are Search VCUs (baseline ∝ search-ready / Boost Window) and Search AI Lake GB (as low as $0.047/GB-mo) — not ECUs or classic frozen-tier SKUs.
Supporting Elastic references
Official Elastic pricing, docs, and blogs that support the VCU + Search Boost Window + Search AI Lake TCO story.
Published meters: Ingest / Search / ML VCUs + Search AI Lake storage GB (as low as $0.047/GB-mo). Compute and storage are billed separately.
Search VCU baseline is driven by Search Power + search-ready dataset size. For time series data, shortening the Search Boost Window is the direct cost lever; data outside the window stays searchable from the Search AI Lake.
Defines search-ready data: non–time-series indices are always search-ready; Search Boost Window applies to time-series docs with a concrete @timestamp field. Search Power tunes baseline vs cost.
VCU = Virtual Compute Unit (Search / Ingest / ML). ECU = Elastic Consumption Unit ($1 billing currency). Interactive dataset size for time series = Boost Window; for traditional search = entire dataset.
You are charged for each VCU consumed for Ingest, Search, and ML, plus GB/month for data in the Search AI Lake.
Search AI Lake architecture: object-store scale with a low-latency cache layer for recent / frequently queried data — the foundation for lake + boost economics.
Compute decoupled from storage; Search AI Lake as the cloud-native architecture behind Serverless.
Users define a Boost Window to prioritize recent data and choose Search Power (On-demand / Performant / High availability) — the operational controls behind VCU spend.
Increasing Search Power, Boost Window, or retention upscales Search VCU consumption — confirms the cost levers in product telemetry.
Official Serverless capability for searching across projects (single-pane UX). Prefer this name over informal “Federated Data Sources” unless Product confirms otherwise.
CPS federated queries are handled by Search VCUs on the origin project, which scale to accommodate cross-project workloads.
Streams is the Observability ingest / retention UI; MySQL CDC into Search is not Streams-native in public docs. Separate Product ask: Streams managed write of aged Search data to customer S3/blob so ES|QL federation can read it.
Documented Elastic-native path for MySQL → Elasticsearch sync when Streams CDC is not available for Search.
Technical Q&A
Tiered Serverless economics, Federated Data Sources, and how the architecture dissolves scale blockers.
Does Serverless Search use ECUs or VCUs?
Both, for different jobs. You consume Search / Ingest / ML VCUs (Virtual Compute Units) plus Search AI Lake storage GB. Elastic bills those in ECUs (Elastic Consumption Units, $1 nominal) as the commercial currency — see the Search Labs post “VCUs and ECUs explained.” Architecture and capacity planning should talk VCUs + search-ready size, not “ECU % of corpus.”
Does Search Boost Window apply to code-search indices?
Only if documents are time-series shaped: a concrete mapped @timestamp field (aliases do not qualify). Non–time-series indices are always 100% search-ready. For commits / PRs / issues, validate that indices use @timestamp so Boost Window can shrink the interactive dataset — otherwise the VCU lever does not apply. Source: Serverless project settings docs.
Why does a naive Serverless lift look so expensive?
A naive lift treats the entire historical corpus as search-ready. Search VCU baseline (including idle, once past small-workload thresholds) scales with search-ready dataset size and Search Power. Paying that baseline for all history is the wrong cost model when recent activity dominates query traffic. Search storage is metered on disk after compression — not Observability uncompressed volume.
How does the Search Boost Window change the bill?
For qualifying time-series data, shortening the Search Boost Window is the direct lever: data outside the window remains fully searchable from the Search AI Lake but has minimal impact on Search VCU baseline. Storage GB is still billed for retained lake data — this cuts compute, not all retention cost. See billing dimensions docs.
Does Serverless Search federate to S3 the way Observability does?
ES|QL data federation reads from customer S3/blob — it does not write. Data must be placed in a supported format on the blobstore. Availability is planned across Serverless project types and self-managed. Cross-project search (CPS) is separate: queries across Serverless projects.
Is it snapshot restore or point-in-time restore?
Same requirement, updated language: Snapshot restore → Point-in-time restore. Accepted as complete via workaround for GA. Align RPO/RTO against the interactive dataset and keep the interim restore runbook current.
Is autosharding available?
Yes — complete for the production path. Platform autosharding is available (confirmed 8/3). Apply it on interactive indices and document the layout in the cutover runbook.
Which Azure region does Serverless need?
Azure Central US is the primary region — where production clusters and the application monolith already run. Target availability is end of September. East US is not the production home. Longer-term failover / DR looks to East US 3 once Serverless is available there.
Can Serverless audit logs go to Splunk or Azure Monitor?
Yes for GA via an accepted workaround: Logstash (or similar) in your environment pulls Serverless audit data over Private Link and forwards to Splunk HEC or Azure Monitor. Native Serverless destinations alone do not meet the “in your domain” policy; native external delivery is a future enhancement and does not block cutover. See the audit logging proposal on this site.
What about ingest into Serverless Search from source systems?
Use an Elastic-native path for CDC / sync (Streams where it fits, otherwise connectors / Logstash JDBC).
Ops resilience lab (optional)
Legacy CCR failover / drift simulation on Serverless.
Expand
Ops resilience lab (optional)
Legacy CCR failover / drift simulation on Serverless.