Handling Large File Uploads (20GB+) in Fast API with S3 Multipart Upload Using Signed URLs

Ever tried uploading massive files (20GB–30GB) through your backend server? Memory spikes, timeouts, server crashes—it's a mess. That's why you need Amazon S3 Multipart Upload with Signed URLs.

Here's how to implement a direct-to-S3 large file uploader using:

• Python + AWS SDK for the backend
• Fast API for the server
• React + Vite for the frontend
• MongoDB for tracking uploads

🧠 Why This Approach?

Instead of uploading files directly to your backend server, we’ll use S3 Multipart Upload. This method breaks the file into smaller chunks, uploads them in parallel, and then combines them on the server. This approach is much more efficient and scalable.

How does this work?

1. Client: The client (browser) initiates the upload process.
2. Backend: The backend generates a signed URL for each chunk.
3. Client: The client uploads each chunk directly to S3 using the signed URL.
4. Backend: The backend combines the chunks on S3.

This makes the system:

• Scalable: Can handle large files without overloading the server.
• Secure: Uses signed URLs to ensure only authorized users can upload files.
• Efficient: Uploads chunks in parallel, reducing the overall upload time.

🛠️ Tech Stack

• Backend
  • Python: For the backend server.
  • AWS SDK for Python: For interacting with S3.
  • Fast API: For building the server.
• Frontend
  • React + Vite: For building the frontend.
• Database
  • MongoDB: For tracking uploads.

🔁 Flow Overview

1. User selects file in frontend.
2. Frontend sends filename to backend → backend creates multipart upload (S3) and stores metadata (MongoDB).
3. Frontend splits file into chunks.
4. For each chunk:
   • Ask backend for signed URL.
   • Upload chunk to S3 using that URL.
   • Notify backend with ETag + part number.
5. After all chunks are uploaded:
   • Frontend tells backend to finalize upload.
   • Backend completes the multipart upload using S3 API.

The Flow (Backend Perspective):

1. Client → API: POST /api/start-upload
2. API → S3: Create multipart upload → Get upload_id
3. API → MongoDB: Record upload session
4. API → Client: Return upload_id, key

For each file part:

• Client → API: GET /api/get-signed-url
• API → S3: Generate presigned URL
• API → Client: Return signed_url
• Client → S3: PUT part directly using signed_url → Get ETag
• Client → API: POST /api/upload-part with ETag
• API → MongoDB: Record part info

Complete upload:

• Client → API: POST /api/complete-upload
• API → MongoDB: Get all parts
• API → S3: Complete multipart upload → Get file location
• API → MongoDB: Update status to completed
• API → Client: Return success + file location

✅ Final Thoughts

Handling large file uploads efficiently is crucial for modern web applications. By using S3 Multipart Upload with Signed URLs, you can ensure that your system remains scalable, secure, and efficient. This approach is widely used in industries ranging from media to e-commerce, making it a reliable choice for handling large file uploads.

For those who like bullet points to succinctly summarize the key takeaways:

• Use signed URLs to offload heavy uploads from your backend.
• Track everything in MongoDB for status, retries, resumability.
• Only finalize uploads once all parts are done.

This is the exact approach I've used in production for multi-GB file uploads. It’s stable, efficient, and easy to extend.

The code in this article is focused on demonstrating the overall flow and concept. It’s not fully production-structured, but feel free to explore the full implementation and adapt it to your needs — GitHub repo.