Supabase Storage is receiving a major upgrade, implementing many of the most requested features from our users: Resumable Uploads, Quality Filters, Next.js support, and WebP support.

The key feature: Resumable Uploads! With Resumable Uploads, you can continue uploading a file from where you left off, even if you lose internet connectivity or accidentally close your browser tab while uploading.

Resumable uploads divides the file into chunks before uploading them, emitting progress events during upload.

With this release, users on the Pro Plan or higher can now upload files as large as 50GB! This substantial upgrade from the previous limit of 5GB offers even more flexibility for your file uploads.

To build this feature, we implemented Postgres Advisory locks which solved some gnarly concurrency problems. We can now handle edge-cases, like two clients uploading to the same location. We’ll deep dive into how we implemented Advisory locks later in the post.

Storage v3 introduces a number of new features.

More image transformations options#

We introduced image resizing last Launch Week. This time, we’ve added the ability to specify quality and format filters when downloading your image. When you request images via the transform endpoint, by default we render it as Webp, if the client supports it.

Next.js loader#

You can serve images from Storage with a simple Next.js loader for the Image component. Check out our docs on how to get started.

Presigned upload URLs#

Authenticated users can now generate presigned URLs.

These URLs can then be shared with other users who can then upload to storage without further authorization. For example, you can generate a presigned URL on your server (ahem, Edge Function).

Shoutout to community members @abbit and @MagnusHJensen who implemented this feature on the Storage server and @Rawnly for the client library bindings 🎉.

Size and file type limits per bucket#

You can now restrict the size and type of objects on a per-bucket basis. These features make it easy to upload to Storage from the client directly, without requiring validation from an intermediary server.

For example, you can restrict your users a 1 MB and image/* files when uploading their profile images:

Let’s get into the nuts-and-bolts of how we implemented Resumable Uploads.

First, why do we need Resumable Uploads, when the HTTP protocol has a standard method for uploading files - multipart/form-data ? This approach works well for small files, since the file is streamed to the server in bytes over the network. For medium to large files this method becomes problematic, especially on spotty connections like mobile networks. Uploads that are interrupted need to be restarted from the beginning.

TUS - Resumable Protocol#

We use S3 to store your files and it implements a proprietary protocol for resumable uploads. At Supabase, we support existing open source communities when possible and so, instead of exposing the S3 protocol to our users, we implemented TUS (historically an acronym for Transloadit Upload Server, later renamed to The Upload Server). TUS is an open protocol for resumable file uploads. By leveraging an open protocol, developers can use existing libraries with Supabase Storage.

TUS is a powerful protocol. It’s built on top of HTTP, making it easy to integrate your browser and mobile applications. Because of its open nature, a variety of powerful, drop-in clients and open-source libraries have been built around it. For example, at Supabase, we love Uppy.js, a multi-file uploader for TUS.

Using Uppy with Supabase Storage looks like this:

And there you have it, with a few lines of code, you can support parallel, resumable uploads of multiple files, with progress events!

There were a few technical challenges we faced while implementing TUS in Supabase Storage.

Storage is powered by our Storage-API service, a Node.js server that interfaces with different storage backends (like AWS S3). It is fully integrated with the Supabase ecosystem, making it easy to protect files with Postgres RLS policies.

To implement the TUS protocol, we use tus-node-server, which was recently ported to Typescript. It was only missing a few features we needed:

• Ability to limit the upload to files of a certain size
• Ability to run multiple instances of TUS (more on this later)
• Ability to expire upload URLs after a certain amount of time

We will be contributing these features back to TUS with discussions and PRs after Launch Week.

Scaling TUS#

One of the biggest challenges we faced was the ability to scale TUS by running multiple instances of the server behind a load balancer. The protocol divides the file into chunks and sends it to any arbitrary server. Each chunk can be processed by a different server. Cases like these can lead to corrupted files with multiple servers trying to buffer the same file to S3 concurrently.

The TUS documentation gives 2 work-arounds:

1. Use Sticky sessions to direct the client to the same server the upload was originally started.
2. Implement some sort of distributed locking to ensure exclusive access to the storage backend.

Option 1 would have affected the even distribution of requests across servers. We decided to go with option 2 - Distributed Locking. Storage uses Postgres as a database, a queue, and now as a lock manager.

Enter Postgres Advisory Locks#

Postgres advisory locks offer a way for defining locking behaviour of resources outside of the database. These are called advisory locks because Postgres does not enforce their use - it is up to the application to acquire and release the locks when accessing the shared resource. In our case, the shared resource is an object in S3. Advisory locks can be used to mediate concurrent operations to the same object.

With advisory locks, we’ve been able to utilize Postgres as a key part of the Supabase Stack to solve difficult concurrency problems.

Roll out#

Because this is a major update, we’re rolling it out gradually over the next month. You will receive a notification in your dashboard when the feature is available for project. Reach out to us if you want early access to this feature.

We’ve got an exciting roadmap for the next few Storage releases:

• Presigned upload URLs for TUS
• Increasing max file size limit to 500 GB
• Transform images stored outside Supabase Storage
• Smart CDN v2 with an even higher cache hit rate

Reach out on Twitter or Discord to share anything else you need to build amazing products.