Skip to main content

Knowledge Article: Moving Data Between Data Centers

Geovany Jimenez
Updated

 

Overview
This article provides a step-by-step guide on how to move data between data centers using network volumes, SSH, and the `rsync` command. The process involves creating network volumes, deploying pods, setting up SSH keys, and using `rsync` to transfer data securely.
 

Prerequisites
- Access to the source and target data centers.
- Permissions to create network volumes and deploy pods.
- Basic knowledge of SSH and terminal commands.

Steps to Move Data Between Data Centers

1. Create a New Network Volume in the Target Data Center
- Log in to the target data center's management interface.
- Create a new network volume where the data will be stored.

2. Deploy Pods in Both Data Centers
- Deploy two pods using the default RunPod PyTorch template.
- Source Pod: Mount the source network volume.
- Target Pod: Mount the target network volume.
1.jpeg

3. Start the Web Terminal in Both Pods
- Access the web terminal for both the source and target pods.
- Ensure both terminals are ready for command execution.
2.jpeg

4. Generate a pair of ssh key on source machine

apt update && apt install -y vim rsync tmux openssh-client && \
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -q && \
cat ~/.ssh/id_ed25519.pub

5. Copy the above public key and add to the destination machine authorized_keys

apt update && apt install -y vim rsync openssh-client && \
ip=$(printenv RUNPOD_PUBLIC_IP) && \
port=$(printenv RUNPOD_TCP_PORT_22) && \
echo "rsync -avzP --inplace -e \"ssh -p $port\" /workspace/ root@$ip:/workspace" && \
vi ~/.ssh/authorized_keys

6. After you run above command and ingest the public key, you will get rsync command

P.S - Copy this command and run it on source machine or follow below instruction to manually set it up.

7. Take the destination Pod IP and Port number, Run the rsync command on source machine

3.jpeg

# start a new tmux session, name it "transfer"
tmux new -s transfer

# sync files between source /workspace to destination /workspace
# using destination machine's IP address and port number in below command
rsync -avzP --inplace -e "ssh -p destination_port_number" /workspace/ root@destination_ip_address:/workspace
# Example: 
rsync -avzP --inplace -e "ssh -p 18598" /workspace/ root@157.66.254.13:/workspace

# ctrl + b
# press d

# which will detach you, close the terminal and can come back and re attach
tmux attach -t transfer

Use `du -sh /workspace` to check disk size

If you have further questions, please reach out to our Support center via the options below -

Related articles