Video Hosting Architecture: Hard-Won Notes on Building for Scale

Adding video to an app feels simple at first. Setup a storage bucket, POST endpoint, sprinkle a little ffmpeg, done... But designing a scalable video hosting architecture is rarely that straightforward.

Then a user uploads a 400MB mystery file on LTE, your server sits around holding the connection hostage, or playback is mysteriously slow. Suddenly your "simple MVP feature" is falling apart.

I built Hyperserve because I realized how hard and esoteric it is to build a DIY video backend that actually works in your app. Shipping video features shouldn't require a side quest into media engineering.

This post walks through the full lifecycle of a video, the performance pitfalls in each stage, and the architecture that keeps everything fast and predictable. These are the early lessons I learned the hard way.

Every video travels the same path:

Client → Upload → Storage → Processing → Output Storage → CDN → Client Playback

Performance problems can pop up anywhere, so let's shine some light on them.

Most developers start with the classic flow:

That works well early on, but then it inevitably starts to strain under the pressure:

A better approach for your video infrastructure is direct-to-storage uploads using pre-signed URLs. The file bypasses your API and your API just provides orchestration on where to send the file. This improves all of the issues mentioned above.

It also unlocks chunked & resumable uploads which are better suited for larger files or situations where the upload is unpredictable.

Right now Hyperserve supports POST uploads because it keeps integration easy. But I'm migrating toward direct-to-storage as part of optimizing the system for the best performance.

Once the video arrives in storage, the processing worker takes over. We can take a big bite out of the performance problem here by optimizing the video file.

The worker needs to:

The only way I know of to get reliable metadata is to run ffprobe on the actual uploaded file. It works well, but some fields can still be missing or misleading depending on how the video was created or edited. So you shouldn't treat any single metadata field as guaranteed.

Storage and delivery are tightly linked. You store the optimized files, and the CDN serves them. A few simple choices here make a big difference for performance:

It's crucial to design these pieces correctly as they're really the workhorse of a scalable video architecture.

By the time the video hits the client, two things matter:

A thumbnail or poster is the fastest performance win you can get. If the first frame loads quickly, you can avoid layout shift and contentful paint issues all together. Of course if you're autoplaying your video this doesn't help at all.

You can give the browser a hint as to what resources you want it to prioritize with the <link> tag. If you have video above the fold that you want to load as fast as possible it's recommended that you use link preload:

This one's often overlooked in my experience—a lot of apps serve videos far larger than the size they're actually rendered.

This can really be a performance hit as the video(s) will be the largest network transit. Specifically this can affect:

This is why Hyperserve lets you generate multiple resolution variants, from 8k down to tiny sizes like 144p. It's a standard pattern in modern video hosting architecture to reduce bandwidth. Not because anyone "watches" these very small sizes but because:

It's a simple but meaningful optimization, especially for mobile apps where device performance or internet speed is questionable.

Adding video isn't the hard part—it's the video hosting architecture surrounding it that is.

So what actually matters early on?

Once you get the upload flow, processing, storage strategy, and CDN delivery sorted, adding video becomes a manageable problem instead of a mystery. You don't need to be a media engineer—you just need the right architecture.

If you'd rather skip the pain and fast forward the build vs. buy debate, Hyperserve is here for you. It handles the entire video backend, so you can add video to your app almost as easily as adding images. Letting you get back to building the product you set out to build.