Speed matters

I often see my clients downloading files to their own PC when asked to migrate a database or a bunch of files to a different server. While you may think it’s not worth bothering to learn something new for this purpose, the size of said files or the quality of your connection could make it advisable at some point. Migrating a multi-gigabyte database is going to be a completely different affair once we learn the proper way to do it.

As it often happens, searching for SCP online will yield a tortuous syntax reference with little real world meaning. But fear not as I will bring you very precise instructions to do exactly what you need to do, and nothing else. These instructions work just as well in either FreeBSD or Linux.

A tar pit

The first thing you should do is open a terminal such as Putty or my favourite Termius on the server where the files reside and create an archive with the files you want to move. While in Windows we are used to tools such as Zip and its derivates (why is people still using WinRAR in 2023?) in the Unix world we use two different tools to achieve this result: tar (abbreviation for tape archive) creates a file out of several ones (known as a tarball; the extension is .tar), while gzip is used to compress them and reduce its size (extension .gz). The syntax to combine both is as follows:

tar czvf outfile.tgz infolder/

This will create a gzipped tarball (.tgz) out of the folder “infolder”. We can compress an arbitrary number of files and/or folders in this fashion:

tar czvf outfile.tgz somefile.sql anotherfile.sql yetanother.sql

From server to server

Now that our files are ready to move, let’s proceed to the copy itself. In order to copy files to another server, we will need to authenticate on it from our server, either by using a password or a ssh key.

If you are like me you will have disabled password authentication everywhere, so let’s explain how to use a ssh key to login to another server. The simplest way is to copy the origin server’s public key into the destination server’s authorized key file.

Let’s say we are the root user in both servers. In our origin server, we run this command:

cat /root/.ssh/id_rsa.pub

The gibberish you just have seen happens to be our origin server’s public key for the root user. If this file doesn’t exist, we can create it with this simple command:

ssh-keygen

(Just press enter three times here to accept all the defaults)

We can copy the output of the previous cat command manually by selecting it with our mouse as you would do in a text editor and pressing ctrl+c. Now switch to a shell in the destination server:

ee /root/.ssh/authorized_keys

Paste the public key here by pressing the right mouse button and save the file with ESC. Now the destination server will allow the origin server’s root user to login automatically.

Finally, let’s do the copying proper. Here’s a simplified syntax:

scp <file> <user>@<host>:/<directory>

Example for port 22:

scp outfile.tgz [email protected]:/root

Example with a custom ssh port:

scp -P 13344 outfile.tgz [email protected]:/root

The first time we login to a new server we will be asked if we accept the host key, to which we will answer yes. Now watch the magic happen:

[email protected]:/home/backup # scp -P 55100 db.tgz [email protected]:/home/backup
The authenticity of host '[aries.doomsday.xyz]:55100 ([137.17.53.7]:55100)' can't be established.
ED25519 key fingerprint is SHA256:Q8U22lOogln0BVCO9kLg1GafKI3f5GZJIREydWI8XVM.
No matching host key fingerprint found in DNS.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[aries.doomsday.xyz]:55100' (ED25519) to the list of known hosts.
db.tgz 100% 1614MB 8.6MB/s 03:07