Ssh Sftp



SSH authentication in SFTP. SSH key base authentication is an access credential authentication using SSH protocol. It’s similar to username/password based authentication but here the SSH key is used instead of password. Now it become a norm to use SSH authentication in SFTP. The key-based authentication mechanism in SSH is called public key. The sftp class provided by David Barnes works great. However, if you get errors about fopen and it failing to open a stream, try the fully qualified path on the remote server. Both sftp-server and internal-sftp are part of OpenSSH. The sftp-server is a standalone binary. The internal-sftp is just a configuration keyword that tells sshd to use the SFTP server code built-into the sshd, instead of running another process (what would typically be the sftp-server).

Many Unix environments have the command-line SSH and SFTP client software tools installed. This page is intended as a guide to just the basics of using these command-line tools. They have many more features than what is described here -- but these instructions should get you started.

Where would I use these?

CS account

  • If you have a CS account, you have access to at least three machines you can log into:
    • shell.cs.fsu.edu
    • program.cs.fsu.edu
    • linprog.cs.fsu.edu
    These machines all mount your home directory, so there's no need to SFTP between them.
  • You might also have an account on websrv2.cs.fsu.edu. This would not mount your normal CS account directory, so you might need to file transfer between this machine and others

Mac OS X home computer / laptop

  • If you have a Macintosh laptop or home computer running OS X, you cannot use the Windows SSH client unless you're running Windows emulation or have booted into Windows
  • In this case, you can use the terminal application (which takes you into a Unix prompt on your Mac) and run the command line SSH and SFTP programs from there
  • This is useful for logging into your CS account remotely, as well as doing file transfers between your Mac and your CS account
  • To open the terminal application on a Mac:
    • Go to the spotlight icon in the upper right of your desktop
    • In the search box, type 'terminal'
    • A menu choice for the terminal application should appear. Select this

Using command-line SSH

Ssh SftpLogin format:In this format, username refers to your user name on the remote account you are logging into, and host_name refers to the name of the machine (usually along with doman) that you are logging into

If you leave out the username, the command will assume that you are logging into another machine with the same username as the machine you're currently on.

Examples

  • Suppose I'm on a Mac, logged in with user account 'john'.
    This command would log me into shell.cs.fsu.edu as username 'smith':This command would log me into shell.cs.fsu.edu, attempting username 'john' (probably will fail, since john is probably not your CS account username!):
  • You can also log into the machines program or linprog:

Using Command-Line SFTP

SFTP is Secure File Transfer Protocol. It is similar to ssh, but its primary purpose is to enable file transfers between a local machine and a remote machine, whereas the ssh (Secure Shell) protocol is for opening up a general command shell on a remote machine where you have an account.

The login format for command line sftp is exactly the same as with the ssh command, but with the 'sftp' command:Using the same basis of the prior example, this command would log me into shell.cs.fsu.edu with the sftp (file transfer) program as username 'smith':

Local vs. Remote machines

Before you transfer files, make sure you know the difference between the local and remote machines:
  • The local machine is the one you started from -- i.e. the machine you were on when you typed the 'sftp' command. This might be your actual physical computer, like a MacBook, if you're using the terminal application. This might also be a machine you are logged into through ssh
  • The remote machine is the one you just logged into via the sftp command

Examples

  • Suppose I'm on my MacBook, with username john. I open the terminal application and sftp into linprog.cs.fsu.edu with username smith:In this case, the local machine is my MacBook, and the remote machine is linprog
  • Suppose I've already logged into user account smith on shell.cs.fsu.edu through an ssh command (or any SSH client). From my CS account, I type this command:Now I have logged into the machine websrv2, with the same user account smith. In this case, the local machine is shell.cs.fsu.edu, and the remote machine is websrv2.cs.fsu.edu.

Basic file transfer commands

There are more commands available than this, but the primary commands you will need are:
  • put -- copy a file from the local machine to the remote machine
  • get -- copy a file from the remote machine to the local machine
  • ls -- get a directory listing on the remote machine
  • cd -- change your current working directory on the remote machine
  • lls -- get a directory listing on the local machine
  • lcd -- change your current working directory on the local machine
Note here that the ls (list files) and cd (change directory) commands work exactly as you are used to them from a regular unix shell. Except when you use them as-is, you are requesting a listing or a change directory operation on the remote machine -- i.e. the machine you just sftp-ed into.

If you want to get a directory listing or change directories on the local machine, use the lls (local list files) and lcd (local change directory) commands instead.

I'm not going to describe these commands (listing and change directories) any further, as readers of this file should already be familiar with these unix commands

put and get commands

Once you have used the listing and navigation commands to put yourself in the correct directories, use get to retrieve files from the remote machine to the local account, and use put to send files from your local account to the remote account. The formats of these commands are simple, and you may use unix wildcards.

Format:Note that the second parameter of each command is optional -- and it will serve to specify a different name or destination directory for the transferred file. Most of the time, it's easiest to simply use put and get with a single parameter, and the destination will default to the current remote or local directory, respectively.

Examples

  • This command will copy the file mycode.cpp from the remote machine to the local one:
  • This command will copy the file mypaper.doc from the local machine to the remote one:
  • This command will copy the file assign1.cpp from the remote machine, and save it as smith1.cpp on the local machine:
  • This command will copy all files ending in .cpp in the current directory on the local machine to the remote machine:

Other SFTP commands

SshTo see a full list of SFTP commands and their formats, you can type help when you are logged in via sftp, and it will give you a list of available commands.

To exit the sftp login, you can use either the quit or the exit command.

SFTP

SFTP is the Secure (or SSH) File TransferProtocol.

The SFTP backend can be used with a number of different providers:

  • C14
  • rsync.net

SFTP runs over SSH v2 and is installed as standard with most modernSSH installations.

Paths are specified as remote:path. If the path does not begin witha / it is relative to the home directory of the user. An empty pathremote: refers to the user's home directory.

'Note that some SFTP servers will need the leading / - Synology is agood example of this. rsync.net, on the other hand, requires users toOMIT the leading /.

Here is an example of making an SFTP configuration. First run

This will guide you through an interactive setup process.

This remote is called remote and can now be used like this:

See all directories in the home directory

Make a new directory

List the contents of a directory

Sync /home/local/directory to the remote directory, deleting anyexcess files in the directory.

SSH Authentication

The SFTP remote supports three authentication methods:

  • Password
  • Key file, including certificate signed keys
  • ssh-agent

Key files should be PEM-encoded private key files. For instance /home/$USER/.ssh/id_rsa.Only unencrypted OpenSSH or PEM encrypted files are supported.

The key file can be specified in either an external file (key_file) or contained within therclone config file (key_pem). If using key_pem in the config file, the entry should be on asingle line with new line ('n' or 'rn') separating lines. i.e.

key_pem = -----BEGIN RSA PRIVATE KEY-----nMaMbaIXtEn0gAMbMbaSsdnMbaassn-----END RSA PRIVATE KEY-----

This will generate it correctly for key_pem for use in the config:

If you don't specify pass, key_file, or key_pem then rclone will attempt to contact an ssh-agent.

You can also specify key_use_agent to force the usage of an ssh-agent. In this casekey_file or key_pem can also be specified to force the usage of a specific key in the ssh-agent.

Using an ssh-agent is the only way to load encrypted OpenSSH keys at the moment.

If you set the --sftp-ask-password option, rclone will prompt for apassword when needed and no password has been configured.

If you have a certificate then you can provide the path to the public key that contains the certificate. For example:

If you concatenate a cert with a private key then you can specify themerged file in both places.

Note: the cert must come first in the file. e.g.

Host key validation

By default rclone will not check the server's host key for validation. Thiscan allow an attacker to replace a server with their own and if you usepassword authentication then this can lead to that password being exposed.

Host key matching, using standard known_hosts files can be turned on byenabling the known_hosts_file option. This can point to the file maintainedby OpenSSH or can point to a unique file.

e.g.

There are some limitations:

  • rclone will not manage this file for you. If the key is missing orwrong then the connection will be refused.
  • If the server is set up for a certificate host key then the entry inthe known_hosts file must be the @cert-authority entry for the CA
  • Unlike OpenSSH, the libraries used by rclone do not permit (at timeof writing) multiple host keys to be listed for a server. Only the firstentry is used.

If the host key provided by the server does not match the one in thefile (or is missing) then the connection will be aborted and an errorreturned such as

or

If you see an error such as

then it is likely the server has presented a CA signed host certificateand you will need to add the appropriate @cert-authority entry.

The known_hosts_file setting can be set during rclone config as anadvanced option.

ssh-agent on macOS

Note that there seem to be various problems with using an ssh-agent onmacOS due to recent changes in the OS. The most effective work-aroundseems to be to start an ssh-agent in each session, e.g.

And then at the end of the session

These commands can be used in scripts of course.

Ssh Sftp File Transfer

Modified time

Modified times are stored on the server to 1 second precision.

Modified times are used in syncing and are fully supported.

Some SFTP servers disable setting/modifying the file modification time afterupload (for example, certain configurations of ProFTPd with mod_sftp). If youare using one of these servers, you can set the option set_modtime = false inyour RClone backend configuration to disable this behaviour.

Standard Options

Here are the standard options specific to sftp (SSH/SFTP Connection).

--sftp-host

SSH host to connect to

  • Config: host
  • Env Var: RCLONE_SFTP_HOST
  • Type: string
  • Default: '
  • Examples:
    • 'example.com'
      • Connect to example.com

--sftp-user

SSH username, leave blank for current username, $USER

  • Config: user
  • Env Var: RCLONE_SFTP_USER
  • Type: string
  • Default: '

--sftp-port

SSH port, leave blank to use default (22)

  • Config: port
  • Env Var: RCLONE_SFTP_PORT
  • Type: string
  • Default: '

--sftp-pass

SSH password, leave blank to use ssh-agent.

NB Input to this must be obscured - see rclone obscure.

  • Config: pass
  • Env Var: RCLONE_SFTP_PASS
  • Type: string
  • Default: '

--sftp-key-pem

Raw PEM-encoded private key, If specified, will override key_file parameter.

  • Config: key_pem
  • Env Var: RCLONE_SFTP_KEY_PEM
  • Type: string
  • Default: '

--sftp-key-file

Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent.

Leading ~ will be expanded in the file name as will environment variables such as ${RCLONE_CONFIG_DIR}.

  • Config: key_file
  • Env Var: RCLONE_SFTP_KEY_FILE
  • Type: string
  • Default: '

--sftp-key-file-pass

The passphrase to decrypt the PEM-encoded private key file.

Only PEM encrypted key files (old OpenSSH format) are supported. Encrypted keysin the new OpenSSH format can't be used.

NB Input to this must be obscured - see rclone obscure.

  • Config: key_file_pass
  • Env Var: RCLONE_SFTP_KEY_FILE_PASS
  • Type: string
  • Default: '

--sftp-pubkey-file

Optional path to public key file.

Set this if you have a signed certificate you want to use for authentication.

Leading ~ will be expanded in the file name as will environment variables such as ${RCLONE_CONFIG_DIR}.

  • Config: pubkey_file
  • Env Var: RCLONE_SFTP_PUBKEY_FILE
  • Type: string
  • Default: '

--sftp-key-use-agent

When set forces the usage of the ssh-agent.

When key-file is also set, the '.pub' file of the specified key-file is read and only the associated key isrequested from the ssh-agent. This allows to avoid Too many authentication failures for *username* errorswhen the ssh-agent contains many keys.

  • Config: key_use_agent
  • Env Var: RCLONE_SFTP_KEY_USE_AGENT
  • Type: bool
  • Default: false

--sftp-use-insecure-cipher

Enable the use of insecure ciphers and key exchange methods.

This enables the use of the following insecure ciphers and key exchange methods:

  • aes128-cbc
  • aes192-cbc
  • aes256-cbc
  • 3des-cbc
  • diffie-hellman-group-exchange-sha256
  • diffie-hellman-group-exchange-sha1

Those algorithms are insecure and may allow plaintext data to be recovered by an attacker.

  • Config: use_insecure_cipher
  • Env Var: RCLONE_SFTP_USE_INSECURE_CIPHER
  • Type: bool
  • Default: false
  • Examples:
    • 'false'
      • Use default Cipher list.
    • 'true'
      • Enables the use of the aes128-cbc cipher and diffie-hellman-group-exchange-sha256, diffie-hellman-group-exchange-sha1 key exchange.

--sftp-disable-hashcheck

Disable the execution of SSH commands to determine if remote file hashing is available.Leave blank or set to false to enable hashing (recommended), set to true to disable hashing.

  • Config: disable_hashcheck
  • Env Var: RCLONE_SFTP_DISABLE_HASHCHECK
  • Type: bool
  • Default: false

Advanced Options

Here are the advanced options specific to sftp (SSH/SFTP Connection).

--sftp-known-hosts-file

Optional path to known_hosts file.

Set this value to enable server host key validation.

Leading ~ will be expanded in the file name as will environment variables such as ${RCLONE_CONFIG_DIR}.

  • Config: known_hosts_file
  • Env Var: RCLONE_SFTP_KNOWN_HOSTS_FILE
  • Type: string
  • Default: '
  • Examples:
    • '~/.ssh/known_hosts'
      • Use OpenSSH's known_hosts file

--sftp-ask-password

Allow asking for SFTP password when needed.

If this is set and no password is supplied then rclone will:

  • ask for a password

  • not contact the ssh agent

  • Config: ask_password

  • Env Var: RCLONE_SFTP_ASK_PASSWORD

  • Type: bool

  • Default: false

--sftp-path-override

Override path used by SSH connection.

This allows checksum calculation when SFTP and SSH paths aredifferent. This issue affects among others Synology NAS boxes.

Shared folders can be found in directories representing volumes

Home directory can be found in a shared folder called 'home'

Sftp
  • Config: path_override
  • Env Var: RCLONE_SFTP_PATH_OVERRIDE
  • Type: string
  • Default: '

--sftp-set-modtime

Set the modified time on the remote if set.

  • Config: set_modtime
  • Env Var: RCLONE_SFTP_SET_MODTIME
  • Type: bool
  • Default: true

--sftp-md5sum-command

The command used to read md5 hashes. Leave blank for autodetect.

  • Config: md5sum_command
  • Env Var: RCLONE_SFTP_MD5SUM_COMMAND
  • Type: string
  • Default: '

--sftp-sha1sum-command

Ssh/sftp

The command used to read sha1 hashes. Leave blank for autodetect.

  • Config: sha1sum_command
  • Env Var: RCLONE_SFTP_SHA1SUM_COMMAND
  • Type: string
  • Default: '

--sftp-skip-links

Set to skip any symlinks and any other non regular files.

Ssh Sftp Port

  • Config: skip_links
  • Env Var: RCLONE_SFTP_SKIP_LINKS
  • Type: bool
  • Default: false

--sftp-subsystem

Specifies the SSH2 subsystem on the remote host.

  • Config: subsystem
  • Env Var: RCLONE_SFTP_SUBSYSTEM
  • Type: string
  • Default: 'sftp'

--sftp-server-command

Specifies the path or command to run a sftp server on the remote host.

The subsystem option is ignored when server_command is defined.

  • Config: server_command
  • Env Var: RCLONE_SFTP_SERVER_COMMAND
  • Type: string
  • Default: '

--sftp-use-fstat

If set use fstat instead of stat

Some servers limit the amount of open files and calling Stat after openingthe file will throw an error from the server. Setting this flag will callFstat instead of Stat which is called on an already open file handle.

It has been found that this helps with IBM Sterling SFTP servers which have'extractability' level set to 1 which means only 1 file can be opened atany given time.

  • Config: use_fstat
  • Env Var: RCLONE_SFTP_USE_FSTAT
  • Type: bool
  • Default: false

--sftp-disable-concurrent-reads

If set don't use concurrent reads

Normally concurrent reads are safe to use and not using them willdegrade performance, so this option is disabled by default.

Some servers limit the amount number of times a file can bedownloaded. Using concurrent reads can trigger this limit, so if youhave a server which returns

Then you may need to enable this flag.

If concurrent reads are disabled, the use_fstat option is ignored.

  • Config: disable_concurrent_reads
  • Env Var: RCLONE_SFTP_DISABLE_CONCURRENT_READS
  • Type: bool
  • Default: false

--sftp-idle-timeout

Max time before closing idle connections

If no connections have been returned to the connection pool in the timegiven, rclone will empty the connection pool.

Set to 0 to keep connections indefinitely.

  • Config: idle_timeout
  • Env Var: RCLONE_SFTP_IDLE_TIMEOUT
  • Type: Duration
  • Default: 1m0s

Limitations

Ssh Sftp Port

SFTP supports checksums if the same login has shell access and md5sumor sha1sum as well as echo are in the remote's PATH.This remote checksumming (file hashing) is recommended and enabled by default.Disabling the checksumming may be required if you are connecting to SFTP serverswhich are not under your control, and to which the execution of remote commandsis prohibited. Set the configuration option disable_hashcheck to true todisable checksumming.

SFTP also supports about if the same login has shellaccess and df are in the remote's PATH. about willreturn the total space, free space, and used space on the remotefor the disk of the specified path on the remote or, if not set,the disk of the root on the remote.about will fail if it does not have shellaccess or if df is not in the remote's PATH.

Note that some SFTP servers (e.g. Synology) the paths are different forSSH and SFTP so the hashes can't be calculated properly. For themusing disable_hashcheck is a good idea.

The only ssh agent supported under Windows is Putty's pageant.

The Go SSH library disables the use of the aes128-cbc cipher bydefault, due to security concerns. This can be re-enabled on aper-connection basis by setting the use_insecure_cipher setting inthe configuration file to true. Further details on the insecurity ofthis cipher can be foundin this paper.

SFTP isn't supported under plan9 until thisissue is fixed.

Note that since SFTP isn't HTTP based the following flags don't workwith it: --dump-headers, --dump-bodies, --dump-auth

Note that --timeout isn't supported (but --contimeout is).

C14

C14 is supported through the SFTP backend.

See C14's documentation

Ssh Sftp Protocol

rsync.net

rsync.net is supported through the SFTP backend.

See rsync.net's documentation of rclone examples.