chmonitorBlog
StarDashboard
All posts

Capacity planning with the disk-forecast and TTL advisor

A worked example of forecasting ClickHouse disk exhaustion and getting a recommend-only TTL adjustment before an out-of-space incident happens.

Case studyJuly 10, 2026

This is a composite worked example built for this post, not a real incident — the workflow and tool behavior described are exactly how chmonitor’s capacity-planning tools behave, run against a representative cluster. The scenario: disk usage on a high-ingest events table has been creeping up for weeks, and nobody wants to find out it hit 100% the hard way.

The problem

A team ingesting clickstream events into a MergeTree table notices disk usage alerts trending upward over several weeks, but no single day looks alarming enough to act on. Reactive capacity management — waiting for a disk-full page — is exactly the failure mode this workflow is meant to avoid.

Investigating

Step 1 — forecast when the disk actually runs out

Instead of eyeballing a disk-usage chart and guessing, ask the AI agent to forecast it directly. This calls the forecast_disk_capacity tool, which projects from system.part_log NewPart write volume over the last 30 days (plus the top contributing tables) and reports against a configurable horizon (default 90 days).

Forecast when this host’s disks will run out of free space.

The tool is explicit about its own limits: if system.part_log isn’t enabled on the cluster, it reports that clearly instead of fabricating a forecast — there’s no silent guess.

Step 2 — identify the retention floor

Before asking for a TTL suggestion, the actual business/compliance requirement has to be pinned down — how many days of this table’s data legally or operationally must be kept. This case assumes a 30-day floor (chmonitor’s own default when none is specified, though it’s meant to be overridden explicitly whenever the real requirement differs).

Step 3 — get a TTL recommendation

Suggest a TTL adjustment for default.events that keeps disk usage under control, with a 30-day retention floor.

This calls suggest_ttl_adjustment, which returns a suggested ALTER TABLE ... MODIFY TTL ... string aimed at keeping projected disk utilization at or under 80%, plus a risk note — and never suggests less than the retention floor given. Like the forecast tool, it reports a clear “part_log not available” message rather than a fabricated suggestion if the underlying data isn’t there.

Root cause

Sustained growth without a bounding TTL on a high-ingest table — normal and expected behavior for an ever-growing events table, but one that needs an explicit retention policy rather than “grow until the disk fills up.”

Resolution

The suggested ALTER TABLE ... MODIFY TTL ... is not applied automatically — this is a recommend-only tool, same as the query optimization advisor. Applying it is a deliberate, reviewed step: check the suggested cutoff against actual compliance/business requirements, run it in a maintenance window (TTL changes trigger background part expiry, not an instant deletion), and confirm the disk-usage trend flattens afterward by re-running the forecast or watching the Disks page.

Takeaway

Capacity planning works best as a forecast-then-recommend loop, not a reactive one: forecast_disk_capacity tells you when a problem becomes real, suggest_ttl_adjustment gives you a concrete, floor-respecting fix to review — but both stop short of touching your table. Reach for this pairing whenever a disk-usage trend looks concerning but isn’t yet an emergency; see what to do once it already is one.