Requirements for a File Upload Endpoint

A file upload endpoint looks like the simplest thing in an API. Take bytes, store bytes, return a URL. Teams ship one in an afternoon, and then spend two years discovering everything that first afternoon missed. The endpoint itself is small. The requirements around it are where the real work lives, and most of that work is defensive.

This is a walk through what a production upload endpoint actually has to handle. Not a tutorial for a framework, not a code sample. A list of the obligations that accumulate once real users with real files start hitting the route.

Size limits belong in three places, not one

A single max-size check in application code is the classic mistake. By the time the check runs, the server has already buffered the whole body into memory. A hundred concurrent requests with a hundred-megabyte limit becomes ten gigabytes of RAM in flight, and the process dies before it ever rejects anyone.

The limit needs to live at the reverse proxy (nginx's client_max_body_size, for instance), at the framework's body-parsing layer, and in the application logic that decides what to do with the file. The proxy rejects early and cheaply. The framework enforces a hard ceiling. The application applies business rules, like a ten-megabyte limit for avatars and a two-gigabyte limit for video. Each layer catches a different failure.

There is also the question of streaming versus buffering. A large upload should stream to disk or object storage as it arrives. A small one can sit in memory. The threshold between them deserves a deliberate number, not whatever the framework defaults to.

The filename is hostile input

Filenames arrive from the client and can contain path separators, null bytes, control characters, right-to-left override marks that disguise extensions, and lengths that overflow naive buffers. A file called ../../etc/passwd is a real thing that shows up in logs. So is a name with a newline in the middle, which corrupts line-oriented log parsers downstream.

The safe pattern is to generate a storage key server-side, using a UUID or a content hash, and keep the original filename as metadata only. Never join a client-supplied string onto a storage path. Sanitize and truncate the display name separately, and assume it will be rendered in HTML somewhere, so it needs escaping at that boundary too.

Content-Type headers are equally untrustworthy. A client can declare image/png and send a zip archive. The declared type is a hint for routing, nothing more.

Detecting the real file type means reading the bytes

File signatures (magic numbers) at the head of the stream reveal what a file claims to be. A JPEG starts with FF D8 FF, a PNG with 89 50 4E 47, a PDF with 25 50 44 46. A sniffer library can compare the declared type, the file extension, and the actual signature, and reject anything that disagrees.

Signature checks are not storage-grade either. Polyglot files exist that are valid in two formats at once. The honest position is that type detection is a filter, not a guarantee, and the real protection comes from how the file is later served.

Serve uploads from a different origin

The nastiest upload vulnerabilities come from serving user files back on the same domain as the application. An uploaded HTML file that contains a script, served from the app's own origin, runs with the app's cookies. Upload an SVG with an embedded script tag and the same thing happens, because browsers execute SVG as markup.

The fix is structural. Host user content on a separate domain, ideally a dedicated bucket or CDN hostname, so a hostile file lives in a different origin than the session cookie. Add Content-Disposition: attachment for anything that is not an image, and set Content-Security-Policy: sandbox on the response. Force a conservative Content-Type on the way out. These headers are cheap and they close a whole class of attacks.

Chunked and resumable uploads for anything large

A single POST that carries a two-gigabyte video over a hotel wifi connection will fail. It will fail repeatedly, and the user will retry from zero each time, and the product will look broken. Resumable uploads split the file into chunks, track which chunks have arrived, and let the client pick up where it stopped. The protocol can be as simple as a session ID plus a chunk index, or as formal as the S3 multipart API.

Chunked uploads also make concurrency manageable. Several chunks can arrive in parallel, each one small enough to validate and store without holding the whole file. The endpoint needs an idempotency rule per chunk, so a retried chunk does not corrupt the final assembly.

Authentication and per-tenant quotas

An open upload endpoint is a disk-filling service. Every upload must be tied to an authenticated principal, and every principal needs a quota: bytes stored, files per day, concurrent uploads in flight. Without quotas, one compromised account or one bored user can fill the bucket and take down the service for everyone.

Quotas also belong at the tenant level, not just the user level, because a shared workspace is the natural unit of abuse. The accounting needs to be approximate and fast, not perfectly consistent. A counter in Redis with a periodic reconciliation against the database is enough; a transactional ledger that locks on every byte is not worth the latency.

Storing the metadata the endpoint will need later

The bytes are the easy part. The record around them is what makes the endpoint useful six months later. It should carry the storage key, the original filename, the sniffed type, the byte size, a checksum, the uploader's identity, the timestamp, and the access policy. A checksum in particular pays for itself: deduplication becomes possible, corruption becomes detectable, and a failed transfer can be verified end to end.

The access policy field matters because uploads are rarely public by default. A private-by-default flag, with sharing as an explicit later action, is a safer foundation than a public bucket with a hope that nobody guesses the path.

Antivirus scanning as an asynchronous step

Scanning every file inline makes the endpoint slow and brittle. A better design accepts the upload, marks the file as pending, and runs a scan job asynchronously. Until the scan completes, the file is not served to other users. The response returns a file ID and a status, and the client polls or subscribes for the transition to clean.

This pattern has a cost: the file exists in the system before anyone knows whether it is safe. The mitigating rule is that pending files are only visible to their uploader, and every read path checks the status first. A file that fails the scan gets quarantined or deleted, and the uploader gets a clear error.

Errors that tell the client something actionable

Upload failures are notoriously opaque. The endpoint should distinguish between a body too large, a type not allowed, a quota exceeded, a checksum mismatch, and a storage outage, each with its own status code and a message that names the specific problem. A generic 400 for all of them wastes the client's retry logic and the user's patience.

The response should also be shaped for partial success. In a multipart upload, one chunk failing should not invalidate the others. The answer needs to say which parts landed and which did not, so the client can resend only the gaps.

A GET for the file and a DELETE to clean up

Uploads accumulate. The endpoint needs a delete path that removes the bytes and the metadata together, and a policy for what happens when a user account closes or a tenant is offboarded. Orphaned bytes in a bucket are invisible until the storage bill arrives, and then they are very visible.

Deletion should be soft by default, with a retention window, because users delete things by accident and support requests follow. A scheduled job then hard-deletes past the window. The lifecycle rules on the storage bucket should mirror the application's policy so bytes do not outlive their records.

The rate limit that protects the storage bill

Even with per-user quotas, a single client hammering the endpoint with tiny files can generate enormous metadata churn and request costs. A token-bucket rate limit on upload requests per minute per principal, separate from the byte quota, keeps the cheap-but-numerous pattern in check. The limit should be generous enough for a legitimate batch import and tight enough that a script cannot create a million rows overnight.

The endpoint also deserves a concurrency cap. Ten simultaneous uploads from one user is reasonable; ten thousand is an incident. Rejecting the eleventh with a clear 429 is a better outcome than a queue that grows without bound.

Logging that survives an audit

Every upload and every rejected upload should leave a log line with the principal, the byte count, the sniffed type, the outcome, and the request ID. That log is what answers the question after an incident: who put this file here, and when. It is also what makes abuse patterns visible before they become expensive.

The logs must not contain the file contents or the full original filename if it might carry personal data. A hash of the filename, or the sanitized display name, is enough for correlation without leaking.

The endpoint is a contract, not a route

None of these requirements are exotic. Each one is a known failure that has bitten a real service somewhere. The cumulative weight of them is what turns a trivial route into a piece of infrastructure with a contract: what it accepts, what it rejects, what it stores, and what it promises to the rest of the system.

A team that writes the endpoint in an afternoon and then spends a week on these requirements will ship something that survives contact with the internet. A team that skips them will ship something that works in the demo and fails in the first month of real traffic, usually at the worst possible time. The bytes are easy. The obligations around them are the job.