{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"uploads-and-object-storage","title":"Files people send you","url":"https://addaly.com/learn/cloud-and-shipping/uploads-and-object-storage","position":37,"of":74,"minutes":9,"answer":"Uploads belong in object storage, written directly by the browser through a short-lived presigned URL with the key, type and size constrained, and everything the client tells you about the file.","keyIdea":"Uploads belong in object storage, written directly by the browser through a short-lived presigned URL with the key, type and size constrained, and everything the client tells you about the file, including its name and content type, is attacker-controlled until you check the bytes."},"body":{"format":"markdown","text":"## The local disk is not storage\n\nThe first version of file upload always writes to a folder next to the code. It works on your laptop and it fails in production for four separate reasons: a second instance cannot see the file, a container's filesystem disappears on restart, the disk fills and takes the whole machine down with it, and nothing about it is backed up.\n\nThe replacement is object storage: a service that holds blobs by key over HTTP. Amazon S3 defined the interface, and almost everything else speaks the same API: Cloudflare R2, Backblaze B2, DigitalOcean Spaces, Hetzner, Wasabi, and MinIO if you want to run it yourself for free. That compatibility is worth insisting on, because it means moving provider is a configuration change rather than a rewrite.\n\nThe price shape is the thing to learn: storage is cheap (roughly 1.5 to 2.5 US cents per gigabyte-month), requests are almost free, and **egress is where the difference is**: the large clouds meter it at several cents per gigabyte, while R2 and B2 charge nothing or almost nothing to serve it out. For a site serving images, that single difference can be the whole hosting bill.\n\n## Do not route the bytes through your app\n\nThe obvious design has the browser POST the file to your server, which then forwards it to the bucket. It doubles the bandwidth, ties up a worker for the length of a mobile upload, and puts a 200 MB video through a process sized for 2 KB JSON requests.\n\nThe standard fix is a presigned URL. Your server authorises the upload and hands back a short-lived URL; the browser sends the bytes straight to the bucket.\n\n```python\nurl = s3.generate_presigned_url(\n    \"put_object\",\n    Params={\"Bucket\": \"uploads\", \"Key\": key,\n            \"ContentType\": \"image/jpeg\"},\n    ExpiresIn=300)      # five minutes is plenty\n```\n\nThree things to constrain when you sign: the key (so a user cannot overwrite somebody else's object), the content type, and the maximum size: with presigned POST policies you can enforce a content-length range, which is the only way to stop somebody uploading 5 GB to a form meant for avatars.\n\nAfter the upload, the browser tells your server it finished. Never trust that message alone: check the object exists and its size before you record it, because the client controls what it claims.\n\n## What arrives is not what was promised\n\nEverything about an uploaded file is attacker-controlled, including the parts that look like metadata.\n\n- **The filename.** `../../etc/passwd` and a 2,000-character Unicode name are both things you will receive. Never use the supplied name as a path. Generate your own key (a UUID, or a hash of the content) and keep the original name as a display label only.\n- **The content type.** The browser sends whatever it likes. Check the actual bytes: `file --mime-type`, or a magic-number library. A file named `.jpg` can be an HTML page, and if you serve it back from your own domain with the type the client claimed, you have a stored cross-site scripting hole.\n- **The size.** Enforce it at the edge, at the bucket policy, and in your own record. A limit implemented only in JavaScript is decoration.\n- **The contents.** Images can carry EXIF data including the exact GPS coordinates where a photo was taken. If your users are posting personal photographs, strip metadata on ingest: this is a privacy obligation, not an optimisation. `exiftool`, ImageMagick and libvips all do it, all free.\n\nServe user content from a different domain from your application, or at minimum with `Content-Disposition: attachment` and a strict content type. Same-origin user files are how one uploaded HTML page becomes a session-stealing page on your own domain.\n\n## Public buckets, the classic accident\n\nMisconfigured buckets are one of the most reliable sources of data leaks in the industry, and the mistake is almost always the same: making the bucket public because a private URL was inconvenient.\n\nKeep the bucket private and serve reads through presigned GET URLs, or through a CDN configured with a signing key. Both are a few lines. If content is public (a logo, a course image) a public bucket is fine; the rule is that publicness is a decision per object, not a shortcut taken once at three in the morning.\n\n## Derivatives, and the work you did not plan\n\nAn uploaded 8 MB phone photo should not be what a visitor downloads. You need thumbnails, and there are three honest options: resize on upload in a background job (predictable, storage-heavy), resize on first request and cache (lazier, needs a cache), or use an image CDN that transforms on the fly by URL parameter. libvips is the fast free library, `sharp` wraps it for Node, and Thumbor and imgproxy are free self-hosted services that do the whole job.\n\nDo the resizing in a queue, not in the request. A large image processed inline is exactly the kind of slow, memory-hungry work that the queues lesson exists to move.\n\n## Storage grows and never shrinks by itself\n\nDeleted rows leave orphaned objects. Failed uploads leave partial ones. Set a lifecycle rule on the bucket to expire incomplete multipart uploads after a day and to move old derivatives to a cheaper class, and write a small job that reconciles objects against the database occasionally. Nobody does this until the bill arrives, and the fix is a configuration setting that takes two minutes."},"check":{"question":"An app stores uploads in a private bucket, generates its own UUID keys, and serves files back from `app.example.com/files/<uuid>` using the content type the browser reported at upload. Where is the hole?","options":[{"text":"UUID keys are guessable, so one user can enumerate another user's files","correct":false,"tempting":"Version-4 UUIDs have enough entropy that enumeration is not the practical risk here; the danger is what happens when a legitimate URL is opened."},{"text":"Presigned URLs expire, so the served links will break for users after a few minutes","correct":false,"tempting":"Expiry is a real operational detail, but a broken link is an availability annoyance, not the security flaw the design contains."},{"text":"Private buckets cannot be served through an application without making them public","correct":false,"tempting":"They can (the app proxies or presigns) and privateness is not what fails in this design."},{"text":"A file claiming to be an image can actually be HTML, and serving it from the app's own domain with that type executes script in the site's origin","correct":true}]},"prev":{"slug":"rate-limits-and-abuse","title":"The traffic that is not customers","url":"https://addaly.com/learn/cloud-and-shipping/rate-limits-and-abuse"},"next":{"slug":"email-that-arrives","title":"Why your email went to spam","url":"https://addaly.com/learn/cloud-and-shipping/email-that-arrives"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Files people send you', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/uploads-and-object-storage (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Files people send you","courseTitle":"Shipping It","module":{"index":4,"slug":"load-cost-and-abuse","title":"Load, cost and the traffic that is not customers"},"answer":null,"keyIdea":"Uploads belong in object storage, written directly by the browser through a short-lived presigned URL with the key, type and size constrained, and everything the client tells you about the file, including its name and content type, is attacker-controlled until you check the bytes.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-local-disk-is-not-storage","runs":[{"text":"The local disk is not storage"}]},{"t":"p","runs":[{"text":"The first version of file upload always writes to a folder next to the code. It works on your laptop and it fails in production for four separate reasons: a second instance cannot see the file, a container’s filesystem disappears on restart, the disk fills and takes the whole machine down with it, and nothing about it is backed up."}]},{"t":"p","runs":[{"text":"The replacement is object storage: a service that holds blobs by key over HTTP. Amazon S3 defined the interface, and almost everything else speaks the same API: Cloudflare R2, Backblaze B2, DigitalOcean Spaces, Hetzner, Wasabi, and MinIO if you want to run it yourself for free. That compatibility is worth insisting on, because it means moving provider is a configuration change rather than a rewrite."}]},{"t":"p","runs":[{"text":"The price shape is the thing to learn: storage is cheap (roughly 1.5 to 2.5 US cents per gigabyte-month), requests are almost free, and "},{"text":"egress is where the difference is","b":true},{"text":": the large clouds meter it at several cents per gigabyte, while R2 and B2 charge nothing or almost nothing to serve it out. For a site serving images, that single difference can be the whole hosting bill."}]},{"t":"h2","id":"do-not-route-the-bytes-through-your-app","runs":[{"text":"Do not route the bytes through your app"}]},{"t":"p","runs":[{"text":"The obvious design has the browser POST the file to your server, which then forwards it to the bucket. It doubles the bandwidth, ties up a worker for the length of a mobile upload, and puts a 200 MB video through a process sized for 2 KB JSON requests."}]},{"t":"p","runs":[{"text":"The standard fix is a presigned URL. Your server authorises the upload and hands back a short-lived URL; the browser sends the bytes straight to the bucket."}]},{"t":"code","lang":"python","label":"Python","text":"url = s3.generate_presigned_url(\n    \"put_object\",\n    Params={\"Bucket\": \"uploads\", \"Key\": key,\n            \"ContentType\": \"image/jpeg\"},\n    ExpiresIn=300)      # five minutes is plenty"},{"t":"p","runs":[{"text":"Three things to constrain when you sign: the key (so a user cannot overwrite somebody else’s object), the content type, and the maximum size: with presigned POST policies you can enforce a content-length range, which is the only way to stop somebody uploading 5 GB to a form meant for avatars."}]},{"t":"p","runs":[{"text":"After the upload, the browser tells your server it finished. Never trust that message alone: check the object exists and its size before you record it, because the client controls what it claims."}]},{"t":"h2","id":"what-arrives-is-not-what-was-promised","runs":[{"text":"What arrives is not what was promised"}]},{"t":"p","runs":[{"text":"Everything about an uploaded file is attacker-controlled, including the parts that look like metadata."}]},{"t":"ul","items":[[{"text":"The filename.","b":true},{"text":" "},{"text":"../../etc/passwd","code":true},{"text":" and a 2,000-character Unicode name are both things you will receive. Never use the supplied name as a path. Generate your own key (a UUID, or a hash of the content) and keep the original name as a display label only."}],[{"text":"The content type.","b":true},{"text":" The browser sends whatever it likes. Check the actual bytes: "},{"text":"file --mime-type","code":true},{"text":", or a magic-number library. A file named "},{"text":".jpg","code":true},{"text":" can be an HTML page, and if you serve it back from your own domain with the type the client claimed, you have a stored cross-site scripting hole."}],[{"text":"The size.","b":true},{"text":" Enforce it at the edge, at the bucket policy, and in your own record. A limit implemented only in JavaScript is decoration."}],[{"text":"The contents.","b":true},{"text":" Images can carry EXIF data including the exact GPS coordinates where a photo was taken. If your users are posting personal photographs, strip metadata on ingest: this is a privacy obligation, not an optimisation. "},{"text":"exiftool","code":true},{"text":", ImageMagick and libvips all do it, all free."}]]},{"t":"p","runs":[{"text":"Serve user content from a different domain from your application, or at minimum with "},{"text":"Content-Disposition: attachment","code":true},{"text":" and a strict content type. Same-origin user files are how one uploaded HTML page becomes a session-stealing page on your own domain."}]},{"t":"h2","id":"public-buckets-the-classic-accident","runs":[{"text":"Public buckets, the classic accident"}]},{"t":"p","runs":[{"text":"Misconfigured buckets are one of the most reliable sources of data leaks in the industry, and the mistake is almost always the same: making the bucket public because a private URL was inconvenient."}]},{"t":"p","runs":[{"text":"Keep the bucket private and serve reads through presigned GET URLs, or through a CDN configured with a signing key. Both are a few lines. If content is public (a logo, a course image) a public bucket is fine; the rule is that publicness is a decision per object, not a shortcut taken once at three in the morning."}]},{"t":"h2","id":"derivatives-and-the-work-you-did-not-plan","runs":[{"text":"Derivatives, and the work you did not plan"}]},{"t":"p","runs":[{"text":"An uploaded 8 MB phone photo should not be what a visitor downloads. You need thumbnails, and there are three honest options: resize on upload in a background job (predictable, storage-heavy), resize on first request and cache (lazier, needs a cache), or use an image CDN that transforms on the fly by URL parameter. libvips is the fast free library, "},{"text":"sharp","code":true},{"text":" wraps it for Node, and Thumbor and imgproxy are free self-hosted services that do the whole job."}]},{"t":"p","runs":[{"text":"Do the resizing in a queue, not in the request. A large image processed inline is exactly the kind of slow, memory-hungry work that the queues lesson exists to move."}]},{"t":"h2","id":"storage-grows-and-never-shrinks-by-itself","runs":[{"text":"Storage grows and never shrinks by itself"}]},{"t":"p","runs":[{"text":"Deleted rows leave orphaned objects. Failed uploads leave partial ones. Set a lifecycle rule on the bucket to expire incomplete multipart uploads after a day and to move old derivatives to a cheaper class, and write a small job that reconciles objects against the database occasionally. Nobody does this until the bill arrives, and the fix is a configuration setting that takes two minutes."}]}],"check":{"question":[{"text":"An app stores uploads in a private bucket, generates its own UUID keys, and serves files back from "},{"text":"app.example.com/files/<uuid>","code":true},{"text":" using the content type the browser reported at upload. Where is the hole?"}],"options":[{"text":[{"text":"UUID keys are guessable, so one user can enumerate another user’s files"}],"correct":false,"tempting":[{"text":"Version-4 UUIDs have enough entropy that enumeration is not the practical risk here; the danger is what happens when a legitimate URL is opened."}]},{"text":[{"text":"Presigned URLs expire, so the served links will break for users after a few minutes"}],"correct":false,"tempting":[{"text":"Expiry is a real operational detail, but a broken link is an availability annoyance, not the security flaw the design contains."}]},{"text":[{"text":"Private buckets cannot be served through an application without making them public"}],"correct":false,"tempting":[{"text":"They can (the app proxies or presigns) and privateness is not what fails in this design."}]},{"text":[{"text":"A file claiming to be an image can actually be HTML, and serving it from the app’s own domain with that type executes script in the site’s origin"}],"correct":true,"tempting":null}],"order":[1,0,2,3],"why":null},"prev":{"slug":"rate-limits-and-abuse","title":"The traffic that is not customers","minutes":9},"next":{"slug":"email-that-arrives","title":"Why your email went to spam","minutes":9}}}