{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"where-logs-go-and-what-they-cost","title":"Where the logs go, and what they cost","url":"https://addaly.com/learn/cloud-and-shipping/where-logs-go-and-what-they-cost","position":21,"of":74,"minutes":9,"answer":"Logs are a pipeline with a price per gigabyte, so log structured JSON to stdout, index only low-cardinality labels, sample the successful noise.","keyIdea":"Logs are a pipeline with a price per gigabyte, so log structured JSON to stdout, index only low-cardinality labels, sample the successful noise, and set a retention number on purpose rather than discovering one on an invoice."},"body":{"format":"markdown","text":"## Writing good log lines is half the job\n\nThe previous lesson was about what to write. This one is about where it lands, how you search it, how long it survives, and why logging is the line item that surprises people on the bill.\n\nThe honest starting position: if you have one machine, `journalctl` and `grep` are a complete logging system, and you should not pay for anything until they stop working. They stop working at a predictable moment: the second machine. Two machines means a user's request is in one of two files and you do not know which, so every question starts with SSHing into both.\n\n## Structured, so a machine can filter it\n\nA line like `user 41 failed login from 10.2.0.4` is readable and unqueryable. The same facts as JSON are both:\n\n```json\n{\"ts\":\"2026-09-06T09:14:02Z\",\"level\":\"warn\",\"msg\":\"login failed\",\n \"user_id\":41,\"ip\":\"10.2.0.4\",\"request_id\":\"c9f2a1\",\"reason\":\"bad_password\"}\n```\n\nNow \"all failed logins for this user in the last hour\" is a filter rather than a regular expression. Every language has a structured logger and they are all free: `pino` or `winston` in Node, `structlog` or the standard library's `logging` with a JSON formatter in Python, `slog` in Go, `zerolog` in Rust.\n\nOne rule that saves grief: log to stdout, as one line per event, and let something else decide where it goes. An application that writes its own files has to rotate them, and a full disk is an outage that also destroys the evidence of itself.\n\n## The pipeline, in three parts\n\n1. **Collect.** An agent on the machine reads stdout or the journal and ships it. Free options: Vector, Fluent Bit, Promtail, or your platform's built-in agent.\n2. **Store and index.** This is where money goes. Free and self-hostable: Loki (indexes labels only, greps the rest, cheap), OpenSearch, ClickHouse. Hosted: Datadog, Better Stack, Papertrail, Axiom, your cloud provider's own.\n3. **Query.** Whatever the store gives you. The test of a logging setup is whether you can answer \"show me every line with request ID c9f2a1, across all services, in order\" in one query.\n\n## Why the bill is a surprise\n\nHosted log products charge by volume ingested and by retention. The numbers move, but the shape does not: roughly a few dollars per gigabyte ingested, held for a couple of weeks.\n\nA busy small service produces more than people expect. One JSON access-log line is around 500 bytes. At 50 requests per second that is 25 KB/s: about 2 GB a day, 60 GB a month, before any application logging at all. Turn on debug logging \"temporarily\" during an incident and you can multiply that by ten while nobody is watching the bill.\n\nThe three controls, in order of effect:\n\n- **Sample the boring lines.** Keep every error, every warning, and one in a hundred successful health checks and static asset requests. Most volume is 200s that nobody will ever read.\n- **Set retention deliberately.** Two weeks of searchable logs plus cheap archived object storage for the rest is a common, sane split. Compliance may set a floor; nothing sets a ceiling except you.\n- **Do not log what you already have as a metric.** A counter is a few bytes a minute; a log line per event is a few hundred bytes per event. If you only ever count them, count them.\n\n## The trap: high-cardinality fields in the wrong place\n\nIn a metrics system, a label with many distinct values explodes the cost. In a *log index* the equivalent mistake is indexing fields you never filter on. Loki's design makes this explicit: it indexes only the labels you declare, and labels like `request_id` or `user_id` will destroy it, precisely because they are unique per line. Keep those in the message body, where full-text search finds them, and keep labels to things with a handful of values: service, environment, level.\n\n## What never goes in a log line\n\nPasswords, tokens, session cookies, card numbers, one-time codes, full request bodies from authentication endpoints, and personal data you would not put in an email. Logs are copied, shipped to a third party, kept for months, and readable by everybody with dashboard access, which is a wider audience than your database.\n\nA redaction list in the logger, applied to known field names, catches most of it. The rest is a code review habit: when you log an object, know what is in it.\n\n## The smallest setup that is enough\n\nFor a project with one or two machines and no budget:\n\n- Log JSON to stdout, always with `request_id`, `user_id` where relevant, and the release SHA.\n- Let systemd or the container runtime capture it.\n- Run Grafana with Loki, either self-hosted on a small box or on Grafana Cloud's free tier, and ship with Promtail or Vector.\n- Set retention to 14 days and forget about it.\n\nThat costs nothing, survives a second machine, and answers the only question that matters at 2am: what happened to this one request."},"check":{"question":"A team adds `request_id` as an indexed label in Loki so incidents are easier to search. Query performance collapses and storage grows sharply. Why?","options":[{"text":"Request IDs are random, so compression of the log body becomes ineffective","correct":false,"tempting":"Random strings do compress poorly, but one short field per line is a marginal cost, not the cause of a collapse."},{"text":"Loki cannot search unindexed text, so it falls back to scanning every chunk","correct":false,"tempting":"Backwards: Loki is designed to scan chunks for text, which is exactly why a request ID belongs in the body rather than in a label."},{"text":"Indexed labels are stored twice, doubling ingest volume for that field","correct":false,"tempting":"Duplication of a single small field cannot explain a large change; the problem is the number of distinct series, not the bytes."},{"text":"Loki's index is built per label combination, so a unique value per request creates a stream per request","correct":true}]},"prev":{"slug":"logs-at-2am","title":"Reading logs at 2am","url":"https://addaly.com/learn/cloud-and-shipping/logs-at-2am"},"next":{"slug":"metrics-that-mean-something","title":"Four numbers that describe any service","url":"https://addaly.com/learn/cloud-and-shipping/metrics-that-mean-something"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where the logs go, and what they cost', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/where-logs-go-and-what-they-cost (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where the logs go, and what they cost","courseTitle":"Shipping It","module":{"index":3,"slug":"knowing-what-it-is-doing","title":"Knowing what it is doing"},"answer":null,"keyIdea":"Logs are a pipeline with a price per gigabyte, so log structured JSON to stdout, index only low-cardinality labels, sample the successful noise, and set a retention number on purpose rather than discovering one on an invoice.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"writing-good-log-lines-is-half-the-job","runs":[{"text":"Writing good log lines is half the job"}]},{"t":"p","runs":[{"text":"The previous lesson was about what to write. This one is about where it lands, how you search it, how long it survives, and why logging is the line item that surprises people on the bill."}]},{"t":"p","runs":[{"text":"The honest starting position: if you have one machine, "},{"text":"journalctl","code":true},{"text":" and "},{"text":"grep","code":true},{"text":" are a complete logging system, and you should not pay for anything until they stop working. They stop working at a predictable moment: the second machine. Two machines means a user’s request is in one of two files and you do not know which, so every question starts with SSHing into both."}]},{"t":"h2","id":"structured-so-a-machine-can-filter-it","runs":[{"text":"Structured, so a machine can filter it"}]},{"t":"p","runs":[{"text":"A line like "},{"text":"user 41 failed login from 10.2.0.4","code":true},{"text":" is readable and unqueryable. The same facts as JSON are both:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"ts\":\"2026-09-06T09:14:02Z\",\"level\":\"warn\",\"msg\":\"login failed\",\n \"user_id\":41,\"ip\":\"10.2.0.4\",\"request_id\":\"c9f2a1\",\"reason\":\"bad_password\"}"},{"t":"p","runs":[{"text":"Now “all failed logins for this user in the last hour” is a filter rather than a regular expression. Every language has a structured logger and they are all free: "},{"text":"pino","code":true},{"text":" or "},{"text":"winston","code":true},{"text":" in Node, "},{"text":"structlog","code":true},{"text":" or the standard library’s "},{"text":"logging","code":true},{"text":" with a JSON formatter in Python, "},{"text":"slog","code":true},{"text":" in Go, "},{"text":"zerolog","code":true},{"text":" in Rust."}]},{"t":"p","runs":[{"text":"One rule that saves grief: log to stdout, as one line per event, and let something else decide where it goes. An application that writes its own files has to rotate them, and a full disk is an outage that also destroys the evidence of itself."}]},{"t":"h2","id":"the-pipeline-in-three-parts","runs":[{"text":"The pipeline, in three parts"}]},{"t":"ol","start":1,"items":[[{"text":"Collect.","b":true},{"text":" An agent on the machine reads stdout or the journal and ships it. Free options: Vector, Fluent Bit, Promtail, or your platform’s built-in agent."}],[{"text":"Store and index.","b":true},{"text":" This is where money goes. Free and self-hostable: Loki (indexes labels only, greps the rest, cheap), OpenSearch, ClickHouse. Hosted: Datadog, Better Stack, Papertrail, Axiom, your cloud provider’s own."}],[{"text":"Query.","b":true},{"text":" Whatever the store gives you. The test of a logging setup is whether you can answer “show me every line with request ID c9f2a1, across all services, in order” in one query."}]]},{"t":"h2","id":"why-the-bill-is-a-surprise","runs":[{"text":"Why the bill is a surprise"}]},{"t":"p","runs":[{"text":"Hosted log products charge by volume ingested and by retention. The numbers move, but the shape does not: roughly a few dollars per gigabyte ingested, held for a couple of weeks."}]},{"t":"p","runs":[{"text":"A busy small service produces more than people expect. One JSON access-log line is around 500 bytes. At 50 requests per second that is 25 KB/s: about 2 GB a day, 60 GB a month, before any application logging at all. Turn on debug logging “temporarily” during an incident and you can multiply that by ten while nobody is watching the bill."}]},{"t":"p","runs":[{"text":"The three controls, in order of effect:"}]},{"t":"ul","items":[[{"text":"Sample the boring lines.","b":true},{"text":" Keep every error, every warning, and one in a hundred successful health checks and static asset requests. Most volume is 200s that nobody will ever read."}],[{"text":"Set retention deliberately.","b":true},{"text":" Two weeks of searchable logs plus cheap archived object storage for the rest is a common, sane split. Compliance may set a floor; nothing sets a ceiling except you."}],[{"text":"Do not log what you already have as a metric.","b":true},{"text":" A counter is a few bytes a minute; a log line per event is a few hundred bytes per event. If you only ever count them, count them."}]]},{"t":"h2","id":"the-trap-high-cardinality-fields-in-the-wrong-place","runs":[{"text":"The trap: high-cardinality fields in the wrong place"}]},{"t":"p","runs":[{"text":"In a metrics system, a label with many distinct values explodes the cost. In a "},{"text":"log index","i":true},{"text":" the equivalent mistake is indexing fields you never filter on. Loki’s design makes this explicit: it indexes only the labels you declare, and labels like "},{"text":"request_id","code":true},{"text":" or "},{"text":"user_id","code":true},{"text":" will destroy it, precisely because they are unique per line. Keep those in the message body, where full-text search finds them, and keep labels to things with a handful of values: service, environment, level."}]},{"t":"h2","id":"what-never-goes-in-a-log-line","runs":[{"text":"What never goes in a log line"}]},{"t":"p","runs":[{"text":"Passwords, tokens, session cookies, card numbers, one-time codes, full request bodies from authentication endpoints, and personal data you would not put in an email. Logs are copied, shipped to a third party, kept for months, and readable by everybody with dashboard access, which is a wider audience than your database."}]},{"t":"p","runs":[{"text":"A redaction list in the logger, applied to known field names, catches most of it. The rest is a code review habit: when you log an object, know what is in it."}]},{"t":"h2","id":"the-smallest-setup-that-is-enough","runs":[{"text":"The smallest setup that is enough"}]},{"t":"p","runs":[{"text":"For a project with one or two machines and no budget:"}]},{"t":"ul","items":[[{"text":"Log JSON to stdout, always with "},{"text":"request_id","code":true},{"text":", "},{"text":"user_id","code":true},{"text":" where relevant, and the release SHA."}],[{"text":"Let systemd or the container runtime capture it."}],[{"text":"Run Grafana with Loki, either self-hosted on a small box or on Grafana Cloud’s free tier, and ship with Promtail or Vector."}],[{"text":"Set retention to 14 days and forget about it."}]]},{"t":"p","runs":[{"text":"That costs nothing, survives a second machine, and answers the only question that matters at 2am: what happened to this one request."}]}],"check":{"question":[{"text":"A team adds "},{"text":"request_id","code":true},{"text":" as an indexed label in Loki so incidents are easier to search. Query performance collapses and storage grows sharply. Why?"}],"options":[{"text":[{"text":"Request IDs are random, so compression of the log body becomes ineffective"}],"correct":false,"tempting":[{"text":"Random strings do compress poorly, but one short field per line is a marginal cost, not the cause of a collapse."}]},{"text":[{"text":"Loki cannot search unindexed text, so it falls back to scanning every chunk"}],"correct":false,"tempting":[{"text":"Backwards: Loki is designed to scan chunks for text, which is exactly why a request ID belongs in the body rather than in a label."}]},{"text":[{"text":"Indexed labels are stored twice, doubling ingest volume for that field"}],"correct":false,"tempting":[{"text":"Duplication of a single small field cannot explain a large change; the problem is the number of distinct series, not the bytes."}]},{"text":[{"text":"Loki’s index is built per label combination, so a unique value per request creates a stream per request"}],"correct":true,"tempting":null}],"order":[2,3,0,1],"why":null},"prev":{"slug":"logs-at-2am","title":"Reading logs at 2am","minutes":7},"next":{"slug":"metrics-that-mean-something","title":"Four numbers that describe any service","minutes":9}}}