How to use Rsync on Windows using SSH connection?

Using Ubuntu Linux subsystem with Windows (source)

Introduction

Remote Sync is known as Rsync that allows you to transfer and synchronise files/data between systems. Using SSH (Secure Shell or Secure Socket Shell) you can copy your data securely from one system to another machine. Rsync is used widely for syncing files for many years.

The purpose of writing this story is to share the experience and issues faced during its implementation with Windows.

Rsync is typically used for synchronizing files and directories between two different systems. You may design a system that has constraints such as intermittent connectivity and limited bandwidth. Rsync could be the best solution to replication your data from one location to another location. The utility only transfers new or updated files saving on bandwidth and bringing faster transfer times.

Syntax of Rsync :

PULL: $ rsync [options] <UserName>@<Romote_Host_Server>:<Source_File_Dir> <Target>

PUSH: $ rsync [options] <Source_File_Dir> <UserName>@<Romote_Host_Server>:<Target>

Some of the commonly used options are given below:

• -v | -verbose → Verbose output
• -n | -dry-run → Performing a trial run without synchronization
• -e | –rsh=COMMAND → TO use the SSH
• -a |–archive → Archive files and directory while synchronizing
• –progress → Show the sync progress during file transfer
• -r | –recursive → Sync files and directories recursively
• -b |–backup → Take the backup during synchronization
• -u |–update → Don’t copy the files if destination files are newer
• -l | –links → Copy symlinks as symlinks during the sync
• -z |–compress → Compress file during the transfer
• -h | –human-readable → Display the output in a human-readable format

Let’s implement the Rsync with Windows

1. Prerequisite :

Assuming you have 64-bit Windows and the latest Windows Operating system. First, we need to enable the Windows subsystem (WSL) and then need to install the latest version of ubuntu on client and server machines.

Enable the Windows subsystem (WSL)

• Click on the Start menu of Windows and navigate to open the Control Panel → Programs → Turn Windows Features On/Off.
• Enable the “Windows Subsystem for Linux” option in the list, and then click the “OK” button

Perform the action on the client and remote machine (Remote Server).

Install the Ubuntu from Microsoft Store

Get Ubuntu — Microsoft Store

Once you install ubuntu on both client and remote machines, you can start using $wsl <Linux command> or type $bash on command prompt then you can use Linux commands.

wsl stands for Windows subsystem for Linux

To avoid any issue with OpenSSH set up, we need to set up the ssh server and ssh client on a Window system, not on WSL (As now you have both on your machine). You can also enable ssh setup on WSL and change Rsync command accordingly. I faced many issues hence recommend the following steps with no issue.

2. Install OpenSSH Server at Server

Once you installed ubuntu on the server then we need to install the Open SSH server.

Installing OpenSSH server with PowerShell as an Administrator

Step 1 - Check if the OpenSSH Server is already installed

Get-WindowsCapability -Online | ? Name -like 'OpenSSH*'

Step 2- Install the OpenSSH Server

Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0

Step 3 - Start the SSH Open Server

Start-Service sshd

Step 4 - This command is optional but recommended to make the Open SSH service automatic

Set-Service -Name sshd -StartupType 'Automatic'

Step 5 - Confirm the Firewall rule is configured ideally it should be created by setup. There should be a firewall rule named “OpenSSH-Server-In-TCP”, which should be enabled

Get-NetFirewallRule -Name *ssh*

Check if port 22 is listening or not using the below command

> netstat -a  
Results:
TCP    [::]:22                <server ip>:0      LISTENING

Step 6- Copy public key to the Server-side (Once created or have key pair)

C:\users\<users>\.ssh\authorized_keys

Step 7- Update the sshd_config file at the server end. We need to understand the exact location of sshd_config and where we need to update it. Here my case is Windows Sub System (Linux) or SSH client -> Windows 2019 Server /SSH Server then the path would be C:\ProgramData\ssh\

In some cases, we need WSL rsync command Linux to Linux in that case you need to update sshd_config at SSH server-side at this path wsl /etc/ssh/

Make the below changes to make the passwordless ssh connection.

PubkeyAuthentication yes
PasswordAuthentication no

Also, comment last two settings

#Match Group administrators
# AuthorizedKeysFile __PROGRAMDATA__/ssh/Authorized_keys

You might face a problem editing this file due to permission hence open notepad as administrator and then open this file in notepad to update it.

3. Generate the Key-pair keys to enable the passwordless connection over SSH

Or use existing Key-pair keys

Generate an RSA key by using the command SSH-KEYGEN to create a key

wsl ssh-keygen

wsl ssh-keygen -f ~/.ssh/id_rsa -q -P ""

wsl ssh-keygen

It would generate the private and public RSA key then follow the above Step 6 for Copying public key to the Server-side by editing ‘.pub’ file and cut and paste the key value into ‘authorized_keys’ file on the SERVER.

C:\users\<users>\.ssh\authorized_keys

4. SSH Setup on the Client machine

Once you installed ubuntu on the server then we need to install the Open SSH client on the client machine.

Installing OpenSSH client with PowerShell as an Administrator

Step 1- Check if the OpenSSH Server is already installed

Get-WindowsCapability -Online | ? Name -like 'OpenSSH*'

Step 2- Install the OpenSSH client

Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0

Step 3- Create a config file in the C:\users\<username>\.ssh\config
with the following content.

Host 20.0.0.1
HostName 20.0.0.1
User <username>
IdentityFile /users/<username>/.ssh/id_rsa

Please note, if you want to connect multiple servers then add an entry for each host server in the config file. Also, if you want to use different key-pair then update the identy File /users/<username>/.ssh/id1_rsa.

5. Test the ssh connection (passwordless)

Test the Openssh Server connection using below command

Wsl ssh username@servername

The first ssh connection to any server will result in a message:

The authenticity of host <serverName> can’t be established.ECDSA key fingerprint is SHA256:(<a large string>). Are you sure you want to continue connecting (yes/no)?

The answer must be either “yes” or “no”. Answering Yes will add that server to the local system’s list of known ssh hosts. Known_hosts’ file will be at the client machine.

6. Let’s use the rsync command for transferring files to and from the server.

To get detailed information about rsync commands, use the below command for help.

wsl man rsync

Files transfer with Rsync over SSH

wsl rsync [OPTION] SourceDirectory_or_filePath user@Server:Target

wsl rsync -avn — timeout=240 /mnt/c/sourceDir/out /mnt/c/Destdir/in/

Worth knowing:

Rsync is always been used for Linux to Linus (Apple to Apple). However, now Rsync will be used at Windows(Orange). Windows ( runs Apple command ) → connecting to an orange. It won’t work as Rsync only works with apple hence we either need a service running on the remote Orange that will connect inner apple.

Therefore, while writing your wsl rsync command then make sure you used — rsync-path=’wsl rsync’ to make orange (run apple) to orange (with inner apple) conection.

You can take the reference of the below commands for PULL and PUSH files.

PUSH to Remote_Server:

wsl rsync -e ssh -avvi /mnt/c/rep/out — remove-sent-files testUser@1.0.1.1:/mnt/c/repl/in/ oorsync-path='wsl rsync'

PULL from Remote_Server:

wsl rsync -e ssh -avvi testUser@1.0.1.1:/mnt/c/repl/out/ --rsync-path='wsl rsync' — remove-sent-files /mnt/c/rep/in/

— remove-sent-files is used to remove sent files form source directoy

Delete files from the Remotely:

You can remote specific file remotely using the below command.

PUSH to Remote_Server:

wsl ssh testUser@10.0.1.1: 'wsl rm -f' /mnt/c/in/file1 /mnt/c/in/file2

Summary

This article enables you to use Rsync with Windows and help to sync the files to and from the remote server. Also, help how to install the OpenSSH server and resolve the issues while making ssh connection to the remote server.

Please give your feedback to make better content for future usage.