Where the logs go, and what they cost

Writing good log lines is half the job

The previous lesson was about what to write. This one is about where it lands, how you search it, how long it survives, and why logging is the line item that surprises people on the bill.

The honest starting position: if you have one machine, journalctl and grep are a complete logging system, and you should not pay for anything until they stop working. They stop working at a predictable moment: the second machine. Two machines means a user’s request is in one of two files and you do not know which, so every question starts with SSHing into both.

Structured, so a machine can filter it

A line like user 41 failed login from 10.2.0.4 is readable and unqueryable. The same facts as JSON are both:

{"ts":"2026-09-06T09:14:02Z","level":"warn","msg":"login failed",
 "user_id":41,"ip":"10.2.0.4","request_id":"c9f2a1","reason":"bad_password"}

Now “all failed logins for this user in the last hour” is a filter rather than a regular expression. Every language has a structured logger and they are all free: pino or winston in Node, structlog or the standard library’s logging with a JSON formatter in Python, slog in Go, zerolog in Rust.

One rule that saves grief: log to stdout, as one line per event, and let something else decide where it goes. An application that writes its own files has to rotate them, and a full disk is an outage that also destroys the evidence of itself.

The pipeline, in three parts

1. Collect. An agent on the machine reads stdout or the journal and ships it. Free options: Vector, Fluent Bit, Promtail, or your platform’s built-in agent.
2. Store and index. This is where money goes. Free and self-hostable: Loki (indexes labels only, greps the rest, cheap), OpenSearch, ClickHouse. Hosted: Datadog, Better Stack, Papertrail, Axiom, your cloud provider’s own.
3. Query. Whatever the store gives you. The test of a logging setup is whether you can answer “show me every line with request ID c9f2a1, across all services, in order” in one query.

Why the bill is a surprise

Hosted log products charge by volume ingested and by retention. The numbers move, but the shape does not: roughly a few dollars per gigabyte ingested, held for a couple of weeks.

A busy small service produces more than people expect. One JSON access-log line is around 500 bytes. At 50 requests per second that is 25 KB/s: about 2 GB a day, 60 GB a month, before any application logging at all. Turn on debug logging “temporarily” during an incident and you can multiply that by ten while nobody is watching the bill.

The three controls, in order of effect:

• Sample the boring lines. Keep every error, every warning, and one in a hundred successful health checks and static asset requests. Most volume is 200s that nobody will ever read.
• Set retention deliberately. Two weeks of searchable logs plus cheap archived object storage for the rest is a common, sane split. Compliance may set a floor; nothing sets a ceiling except you.
• Do not log what you already have as a metric. A counter is a few bytes a minute; a log line per event is a few hundred bytes per event. If you only ever count them, count them.

The trap: high-cardinality fields in the wrong place

In a metrics system, a label with many distinct values explodes the cost. In a log index the equivalent mistake is indexing fields you never filter on. Loki’s design makes this explicit: it indexes only the labels you declare, and labels like request_id or user_id will destroy it, precisely because they are unique per line. Keep those in the message body, where full-text search finds them, and keep labels to things with a handful of values: service, environment, level.

What never goes in a log line

Passwords, tokens, session cookies, card numbers, one-time codes, full request bodies from authentication endpoints, and personal data you would not put in an email. Logs are copied, shipped to a third party, kept for months, and readable by everybody with dashboard access, which is a wider audience than your database.

A redaction list in the logger, applied to known field names, catches most of it. The rest is a code review habit: when you log an object, know what is in it.

The smallest setup that is enough

For a project with one or two machines and no budget:

• Log JSON to stdout, always with request_id, user_id where relevant, and the release SHA.
• Let systemd or the container runtime capture it.
• Run Grafana with Loki, either self-hosted on a small box or on Grafana Cloud’s free tier, and ship with Promtail or Vector.
• Set retention to 14 days and forget about it.

That costs nothing, survives a second machine, and answers the only question that matters at 2am: what happened to this one request.