Uploading large files can be a tedious and unpredictable task. Flaky networks and faulty file servers can cause uploads to fail unexpectedly, forcing users to reupload the files and costing both time and money. Conventional approaches to file uploads (such as utilizing regular HTTP requests that transmit files via
multipart/form-data) are not efficient because they rely on single requests to upload the entire file. If such a request fails, the entire file has to be uploaded again, which is time-consuming.
In this article, we'll walk through how to easily upload large files (such as videos) to remote servers using UpChunk and Airplane.
What is UpChunk?
The module splits the file into chunks of a defined length and then makes a
PUT request to the server for each chunk with the correct
Content-Range headers. If a chunked upload fails, it retries automatically. And, if needed, users can pause and resume an upload.
Resumable file uploads help upload large files efficiently in cases like cloud backups of app databases or when sharing large files with colleagues or clients. In this tutorial, we'll use the Mux Video API as the data store to upload video files. As mentioned previously, we can use any file server as long as it supports resumable file uploads, but Mux has been chosen here because it integrates seamlessly with UpChunk.
Uploading large files with UpChunk using Airplane
Airplane is the developer platform for building internal tools. With Airplane, users can transform scripts, queries, and more into powerful internal UIs and workflows. Airplane Tasks are single or multi-step functions that anyone on your team can use. Airplane also offers Views, its React-based platform to build UIs without having to deal with the boilerplate that a native React project comes with.
In the following section, we'll walk through how to set up a View in Airplane that allows users to upload large video files to the Mux Video API (the example data store for this tutorial) using UpChunk with the help of an Airplane Task.
To build the project, you'll need the following:
- Node.js set up on your system to develop and test the project (please use one of these versions: ^14, ^15, ^16, ^17, or ^18)
- npm, the official package manager for Node.js
- An Airplane account to deploy the project when it's ready
- The Airplane CLI set up on your system to develop the project
- A Mux account to use as a video data store
Once you're done setting up the prerequisites, it's time to get started with the initial configuration of the project. You can always refer to this GitHub project to view the complete source code for the tutorial.
To begin, we need to create a new Airplane project using the Airplane CLI. This project will hold the source code, the View, and the Task. Start by logging in to the CLI with the following command:
This command will generate the following output:
Once we press Enter, we'll be redirected to the login page in the web browser. After we log in, we should see the following screen:
The CLI will be able to automatically access the Airplane credentials from the browser login, and we'll see the following output in the terminal:
Next, create a new directory for this Airplane project, change the terminal's working directory to it, and create a new Airplane project:
The Airplane CLI will ask us a series of questions related to the new project. Enter the following answers:
This completes the setup of a new Airplane project with a View in our system. Try running the project locally using the following command:
We should see something like this in the browser:
Next, we need to set up the UpChunk package in the project.
Setting up UpChunk
Setting up Mux
We are going to use Mux as the data store for the uploads because of its seamless integration with UpChunk. Mux is a video platform that enables developers to build video streaming and video-on-demand features into their applications. We'll use Mux's upload API as an example file upload server to use with UpChunk.
To use the Mux upload API, we need to generate an API token and secret key. Navigate to the Mux dashboard, click the Settings option at the bottom of the left navigation pane, and then click Access Tokens:
Click Generate new token to create a new API token:
A form asking for the details of the new access token should appear. Enter the following information and click Generate token:
Once the access token is created, the token and the secret key will be shown to us. Copy and store these in a safe location, as we'll need them later when setting up the Airplane project.
Once done, click Continue and close the window.
Building the project
At this point, we've set up everything we need to build the project. Next, we'll create a Task and a View to start uploading large files with UpChunk.
The Task will contain an SDK call that generates an authenticated upload URL for the UpChunk package to upload the video. Please note that this is specific to the data store we're currently using (i.e., Mux). Each data store may have its own unique method of authenticating or creating upload URLs.
Building the Task
upload-upchunk-task task by running the following command:
Then, open the
upload_upchunk_task.airplane.ts file in the IDE and paste the following code snippet into it:
There are inline comments above to help us understand this code, but in summary, the purpose of the code is to use the Mux Node SDK to create an authenticated upload URL for the UpChunk package using the Mux API token and secret key.
Building the View
To update the View, paste the following code snippet into the file
There are inline comments to help us understand this code.
At this point, we've completed setting up the project. We can test it out locally by running the following command:
Once completed, the following screen will be displayed in the browser:
While the View loads, it calls the Task to generate an upload URL, but the call fails since the Task depends on environment variables to access the MUX API token and secret key. To add those, navigate to the Configs page located on the left of the page, and add the following config variables:
Here's how the Configs page should look once we've added all the config variables:
If we try running the View now, it should be able to call the task successfully:
Try clicking the Choose file button to select a file. Once we select a file from the system's file picker, the upload will begin automatically, and the progress will be shown via the progress bar at the bottom of the page. Once the upload is complete, the View will look like this:
Then, navigate to the /Video > Assets page in the Mux dashboard to see the uploaded file:
Deploying and testing
The final step in this tutorial is to deploy the project to Airplane's remote deployment environment.
Deploying the setup to Airplane
Before deploying, we'll need to define the values for the config vars in the Airplane account.
The environment variables we previously set up through Airplane Studio while running the project locally are only set for the dev environment. For the remote deployment environment, we must set the values using the following commands:
Once that's done, deploying an Airplane project is very straightforward. All we need to do is run the following command:
Y when asked whether to continue deploying to the default environment. Once the CLI completes the deploy operation, it will print an output that should end similarly to this:
We can copy and paste the link printed in this output in the browser to see the result of the deployment:
Click the upload-upchunk-view chip in the Entities column to navigate to the deployed View:
Then, try uploading a file with the Choose file button. The file should upload to the Mux data store successfully:
And that's how to utilize UpChunk and Airplane to upload large files to the Mux Video API. All of the code used for this project is available on GitHub.
Building an internal tool to upload large files can be quite cumbersome. That's where Airplane can help. Airplane Views offers a prebuilt set of components to quickly scaffold a UI, and Airplane Tasks helps you easily run background jobs and in-view operations. With Tasks and Views in Airplane, you can go from ideas to a working tool within hours.
Author: Kumar Harsh
Kumar Harsh is a technical writer based in India. He loves to build robust and scalable MERN applications to solve problems and automate repetitive tasks. He enjoys learning and writing content on technical topics.