We've got a number of questions recently about how to transfer data into pods, and while do we have some transfer methods in our docs we'd like to go into a bit more detail on how to upload files into your pod.

In general, for large files, you'll want to ensure that they are hosted on the Internet before sending to your pod - simply because if you upload from your local computer you will be limited to whatever upload speed your local internet connection is capable of, and for home-based Internet this may not be fast enough to transfer multiple gigabyte files, especially on a regular basis. This may work fine for smaller files or as one-offs, but for routinely sending large models, for example, we would recommend uploading to Huggingface once so you can repeatedly pull files from there instead.

Without further ado, here's a rundown of the most popular methods.

runpodctl

runpodctl is our simple little official commandline utility that you can use to send files back and forth between the pod and your local computer, or to send files back and forth between pods.

- Instructions for Windows, Linux, and macOS

- Video tutorial for Stable Diffusion via Web UI on Runpod cloud services

To summarize, you use runpodctl with a send command in the terminal where the file is:

It will give you a command and a passphrase to run where the file is to be sent:

Note that you do not need to open any HTTP or TCP ports to use runpodctl - terminal access is enough. You don't need to download locally from one pod to send to another - you can just run runpodctl on the pod itself to send files directly without using your local computer as a middleman.

Drag and drop through VSCode / Jupyter Notebook

If your code has one of these editors enabled, you can drag and drop straight from your desktop straight into the file structure of the pod with a supported browser- or failing that, use the upload function from the menu in the editor.

wget

If the file is out there retrievable on the Internet, you can use wget in the pod terminal to pull the file straight from that server.

Note that certain sites like CivitAI require an API key to download files from their servers, so special utilities like the CivitAI downloader may needed instead.

git

You can git clone repositories from git or Huggingface including large models by cloning the repo:

Remember to install and initialize git lfs if you intend on bringing down any files over 100mb:

scp / rsync

The general syntax for sending files to a Pod with SCP is as follows (execute this on your local machine, and replace the x's with your Pod's external TCP port and IP; for this example, they are 43201 and 194.26.196.6, respectively):

Example of sending a file to a Pod:

If you want to receive a file from your Pod, switch the source and destination arguments:

The general syntax for sending files to a Pod with rsync is as follows (execute this on your local machine, and replace the x's with your Pod's external TCP port and IP):

Learn more about scp and rsync.

Package-specific methods

If you're using a pre-built package, these often have methods to download files provided. This will vary depending on the package. For example, text-generation-webui has a download-model.py you can call with a neat threaded downloader:

RFC 1149

If you've got enough carrier pigeons or other suitably equipped birds available, feel free to print packet data out, attach it to their leg, and direct them towards your host's server's physical location. Note that this may not be particularly viable if you are on a different continent than the host as severe packet loss may occur.

(... okay, no, that's not a thing. There are no plans to implement RFC 1149 at Runpod at this time.)

But if you have any questions on specific transfer methods, feel free to refer to our docs!

‍

Author profile: Brendan McKeag