Log Aggregation at Scale: ELK, Loki, and the ClickHouse Alternative

Every logging architecture eventually hits the same wall: cost. Here’s how three popular stacks handle the tradeoff between searchability and price at scale.

Logging is the observability pillar every team starts with, because console.log and print statements predate every other form of instrumentation by decades. It’s also the pillar that most reliably blows up cloud bills, because unlike metrics (which are pre-aggregated and bounded in cardinality by design) and traces (which are usually sampled), logs are unstructured, high-volume, and every engineer’s first instinct when debugging is to add more of them.

By the time a team is generating hundreds of gigabytes of logs per day, the choice of log aggregation architecture stops being an implementation detail and starts being one of the largest line items in the infrastructure budget. Understanding how the three dominant approaches — Elasticsearch-based stacks, Grafana Loki, and ClickHouse-based systems — actually store and index data is the difference between a bill that scales linearly with usage and one that scales linearly with your VP of Engineering’s blood pressure.

The ELK Stack: Index Everything, Pay for It Later

Elasticsearch (the “E” in ELK, alongside Logstash and Kibana) became the default choice for log aggregation because it does something genuinely powerful: it builds a full-text inverted index over every field in every log line, by default, at ingestion time. This means you can search for any word, in any field, and get results back in milliseconds, without needing to know in advance what you’d want to search for.

That power comes at a real cost, and the cost is not subtle. Building and maintaining an inverted index for every token in every log line multiplies your storage footprint significantly beyond the raw log data — it’s common to see index sizes reach several times the size of the original ingested logs once you include the primary index, replicas, and auxiliary data structures Elasticsearch maintains. The indexing process itself is also CPU-intensive, meaning your ingestion tier needs meaningfully more compute than the raw log volume alone would suggest, and that compute cost scales with every additional field you index, whether or not anyone ever actually searches on it.

The operational model that made ELK financially survivable at scale is the hot-warm-cold architecture: recent logs (say, the last 24-72 hours) live on fast, expensive, indexed “hot” storage, optimized for the flexible ad-hoc queries you actually need during active incident response. After that window, data rolls to “warm” nodes with less replication and slower disks, still searchable but at reduced performance. Eventually it rolls to “cold” or frozen storage — often object storage like S3 — where it’s cheap to retain for compliance purposes but slow and sometimes requires explicit rehydration before it can be queried again.

This tiering is not optional at any meaningful scale — teams that keep everything on hot, fully-indexed storage indefinitely are the ones who discover, usually via a shocked finance team, that their logging cluster costs more than the production services it’s monitoring.

Grafana Loki: Index the Metadata, Not the Content

Loki took a deliberately different bet: instead of indexing the full content of every log line, it indexes only a small set of labels (metadata like service, environment, pod, namespace) and stores the actual log content as compressed, unindexed chunks. Finding logs means first narrowing down by label (which is fast, because the label index is small) and then doing a linear scan — a grep, essentially — over the compressed chunks that match those labels.

The tradeoff is exactly what it sounds like: Loki’s storage footprint is dramatically smaller than an equivalent Elasticsearch index, because you’re not building an inverted index over every word in every log line, just over a handful of labels. Ingestion is also cheaper computationally for the same reason. The cost is query flexibility and speed on unindexed content — a search that has to scan every log line matching a broad label selector (say, all logs from an entire namespace over the last week) can be considerably slower than the equivalent Elasticsearch full-text query, because Loki is doing real work at query time that Elasticsearch already did at ingestion time.

This tradeoff makes Loki a strong fit for teams whose actual query pattern is “show me all logs from this specific pod during this specific ten-minute window” (label-based narrowing is fast, and the resulting scan window is small) and a weaker fit for teams whose pattern is “find every log line anywhere in the fleet containing this specific error string from the last month” (a query shape that plays directly to Elasticsearch’s strength and against Loki’s).

Label cardinality is Loki’s most common operational trap. Because Loki creates a separate stream per unique combination of label values, teams that carelessly add a high-cardinality label — a user ID, a request ID, a trace ID — as an indexed label rather than embedding it in the log content itself can accidentally create millions of tiny streams, which degrades performance in the opposite way Elasticsearch degrades: instead of one enormous index, you get an explosion of tiny, inefficient ones. The fix is disciplined label design: labels should have low, bounded cardinality (environment, service name, log level), and anything with unbounded cardinality (user IDs, request IDs) belongs in the unindexed log body, searchable via the scan, not the index.

The ClickHouse Alternative: Treat Logs as a Structured Analytics Problem

The newest entrant to this space takes a third approach entirely: treat logs not as text to be searched, but as structured, columnar data to be queried, using the same engine architecture built for large-scale analytics workloads. ClickHouse (and log-specific systems built on similar columnar principles) stores each log field as its own column, compressed independently, and uses techniques like sparse indexing and data skipping to avoid scanning irrelevant data, rather than building a full inverted index over every token.

This architecture tends to produce meaningfully better compression ratios than either ELK or Loki for structured log data, because columnar storage exploits the fact that values within a single column (all the HTTP status codes, all the service names) are far more repetitive and compressible than a full row of mixed data would be. Query performance for aggregation-heavy questions — “what’s the error rate broken down by service and region over the last six hours” — tends to be very strong, because that’s exactly the workload columnar analytics engines were built to answer efficiently.

The tradeoff shows up in flexibility and operational maturity for the “find me this specific needle in this haystack” use case that’s the bread and butter of incident response. Full-text search within a column is possible but generally not as fast or as flexible as Elasticsearch’s purpose-built inverted index, and the tooling ecosystem (query builders, alerting integrations, pre-built dashboards) is younger and less standardized across the industry than the decade-plus-mature Elasticsearch/Kibana ecosystem.

Structured Logging Changes the Calculus for Any Backend

Regardless of which storage architecture you choose, the single highest-leverage change most teams can make is emitting structured logs, meaning consistent, machine-parseable key-value fields, rather than free-form text strings. A log line like a request completed with status 500 for user 12345 requires string parsing or regex extraction to query reliably, while an equivalent structured entry with explicit fields for status, user ID, and message can be filtered, aggregated, and indexed far more efficiently by any of the three architectures discussed here. Structured logging doesn’t remove the tradeoffs between Elasticsearch, Loki, and ClickHouse-based systems, but it substantially reduces the cost and complexity of extracting value from whichever one you choose, and it’s a change application teams can make independently, well before any decision about backend architecture is finalized.

Choosing Between Them: The Questions That Actually Matter

The right choice isn’t about which system is “best” in the abstract — it’s about matching the architecture to your actual query patterns and cost tolerance.

What does your team actually search for during an incident? If the honest answer is “we grep for a specific error message or exception stack trace across the whole fleet,” you need real full-text search, which points toward Elasticsearch or a hybrid approach. If the honest answer is “we already know which service and pod we’re investigating, we just need to see its logs for a specific window,” Loki’s label-first model fits naturally and cheaply.

How much of your log volume is genuinely valuable versus generated out of habit? Most fleets, when audited, discover that a large fraction of logged volume is debug-level noise that’s never queried, ever, by anyone. Before optimizing the storage architecture, it’s almost always more cost-effective to audit and reduce what gets logged in the first place — sampling verbose debug logs, removing logging from hot loops, and reserving full-fidelity logging for error paths and business-critical events.

How long do you actually need to retain searchable logs, versus just retained for compliance? Compliance retention (often 1-7 years depending on industry) rarely needs to be instantly, fully searchable — it needs to be retrievable within a reasonable SLA if audited. This is exactly what cold/frozen tiering in Elasticsearch, or simple compressed object storage retention in Loki and ClickHouse-based systems, is built for. Conflating “must retain” with “must keep fully indexed and hot” is one of the most common unnecessary cost drivers in logging architecture.

What’s your team’s operational capacity for running the system itself? Elasticsearch clusters, especially at scale, require real operational expertise — shard management, index lifecycle policies, JVM heap tuning. Loki’s simpler storage model (leaning on object storage for the heavy lifting) generally requires less specialized operational knowledge. This is a genuine factor, not a minor one — the “best” architecture on paper that nobody on the team knows how to keep healthy at 2 AM is not actually the best choice for that team.

Alerting on Logs: A Different Cost Curve Entirely

A dimension that often gets overlooked when comparing these architectures is the cost of alerting directly on log content, as opposed to just supporting ad-hoc search during an investigation. Running a query every minute across a large, unindexed or lightly-indexed log volume to check for a specific error pattern is a fundamentally different, and often much more expensive, workload than an occasional human-initiated search during an incident. It’s a sustained, recurring cost multiplied by however many alert rules you define, running whether or not anything is actually wrong.

This is the reason most mature observability setups avoid alerting directly on raw log queries wherever an equivalent metric can be derived instead, for instance by incrementing a counter metric every time a specific error is logged, and alerting on that counter, rather than running a recurring full-text search across the log store itself. The log store remains the tool for deep, ad-hoc investigation once you already know something is wrong; the metric, cheaper to evaluate continuously, is what actually triggers the page. Conflating the two, using an expensive search-based system as your primary alerting mechanism, is a quiet but significant driver of both cost overruns and slow-to-fire alerts, since a heavy query scheduled too frequently against a large log volume can itself become a performance bottleneck on the very system you’re relying on to tell you something’s wrong.

A Pragmatic Middle Path

Many mature observability setups don’t pick a single system exclusively — they route logs based on value. High-value, frequently-searched logs (application errors, security-relevant events, anything customer-support-facing) go to a fully-indexed, full-text-searchable system where the cost is justified by the query value. High-volume, lower-value logs (routine access logs, verbose debug output) go to a cheaper, label-indexed or columnar system where the query pattern is narrower and more predictable.

This routing decision, made deliberately rather than by default, is usually where the largest cost savings in a logging architecture actually come from — not from picking the theoretically most efficient single system, but from refusing to pay full-text-index prices for data that will only ever be queried by label or time range.