Data Management and Transfer¶

This section covers the storage and file systems available on the system and the different ways that you can transfer data to and from Cirrus.

In all cases of data transfer, users should use the Cirrus login nodes.

Cirrus file systems and storage¶

There are three different data storage types available to users:

• Home file system (CephFS)
• EPCCfs (VAST)
• Work file system (Lustre)

Each type of storage has different characteristics and policies, and is suitable for different types of use.

There are also two different types of node available to users:

• Login nodes
• Compute nodes

Each type of node sees a different combination of the storage types. The following table shows which storage options are available on different node types:

Home file system¶

Every project has an allocation on the home file system and your project's space can always be accessed via the path /home/[project-code]. The home file system is approximately 1.5 PB in size and is implemented using the Ceph technology. This means that this storage is not particularly high performance but are well suited to standard operations like compilation and file editing. This file systems is visible from the Cirrus login nodes.

Quotas on home file system¶

All projects are assigned a quota on the home file system. The project PI or manager can split this quota up between groups of users if they wish.

You can view any home file system quotas that apply to your account by logging into SAFE and navigating to the page for your Cirrus login account.

1. Log into SAFE
2. Use the "Login accounts" menu and select your Cirrus login account
3. The "Login account details" table lists any user or group quotas that are linked with your account. (If there is no quota shown for a row then you have an unlimited quota for that item, but you may still may be limited by another quota.)

Quota and usage data on SAFE is updated twice daily so may not be exactly up to date with the situation on the system itself.

From the command line¶

Some useful information on the current contents of directories on the /home file system is available from the command line by using the Ceph command getfattr. This is to be preferred over standard Unix commands such as du for reasons of efficiency.

For example, the number of entries (files plus directories) in a home directory can be queried via

[auser@login01]$ cd
[auser@login01]$ getfattr -n ceph.dir.entries .
# file: .
ceph.dir.entries="33"

The corresponding attribute rentries gives the recursive total in all subdirectories, that is, the total number of files and directories:

[auser@login01]$ getfattr -n ceph.dir.rentries .
# file: .
ceph.dir.rentries="1619179"

Other useful attributes (all prefixed with ceph.dir.) include files which is the number of ordinary files, subdirs the number of subdirectories, and bytes the total number of bytes used. All these have a corresponding recursive version, respectively: rfiles, rsubdirs, and rbytes.

A full path name can be specified if required.

Work file system¶

Every project has an allocation on the work file system and your project's space can always be accessed via the path /work/[project-code]. The work file system is approximately 1 PB in size and is implemented using the Lustre parallel file system technology. They are designed to support data in large files. The performance for data stored in large numbers of small files is probably not going to be as good.

Ideally, the work file system should only contain data that is:

• actively in use;
• recently generated and in the process of being saved elsewhere; or
• being made ready for up-coming work.

In practice it may be convenient to keep copies of datasets on the work file system that you know will be needed at a later date. However, make sure that important data is always backed up elsewhere and that your work would not be significantly impacted if the data on the work file system was lost.

If you have data on the work file system that you are not going to need in the future please delete it.

Quotas on the work file system¶

As for the home file system, all projects are assigned a quota on the work file system. The project PI or manager can split this quota up between groups of users if they wish.

You can view any work file system quotas that apply to your account by logging into SAFE and navigating to the page for your Cirrus login account.

1. Log into SAFE
2. Use the "Login accounts" menu and select your Cirrus login account
3. The "Login account details" table lists any user or project quotas that are linked with your account. (If there is no quota shown for a row then you have an unlimited quota for that item, but you may still may be limited by another quota.)

Quota and usage data on SAFE is updated twice daily so may not be exactly up to date with the situation on the system itself.

You can also examine up to date quotas and usage on the Cirrus system itself using the lfs quota command. To do this:

Change directory to the work directory where you want to check the quota. For example, if I wanted to check the quota for user auser in project t01 then I would:

[auser@login01:~]$ cd /work/t01/t01/auser

[auser@login01:auser]$ lfs quota -hu auser .
Disk quotas for usr auser (uid 68826):
        Filesystem    used   quota   limit   grace   files   quota   limit   grace
                .  5.915G      0k      0k       -   51652       0       0       -
uid 68826 is using default block quota setting
uid 68826 is using default file quota setting

the quota and limit of 0k here indicate that no user quota is set for this user.

To check your project quota, you would use the command:

[auser@login01:auser]$ lfs quota -hp $(id -g)'01' .
Disk quotas for prj 3773301 (pid 3773301):
    Filesystem    used   quota   limit   grace   files   quota   limit   grace
            .   958.3G     0k  13.57T       - 9038326       0       0       -
pid 3773301 is using default file quota setting

EPCCfs storage¶

Every project has an allocation on the EPCCfs storage and your project's space can always be accessed via the path /epccfs/[project-code]. The EPCCfs storage provides a large capacity (more than 30 PB) and is currently implemented using the VAST technology.

You can find your directory on the EPCCfs at:

/epccfs/<project code>/<project code>/<username>

For example, if your username is auser and you are in the e05 project, then your EPCCfs directory will be at:

/epccfs/e05/e05/auser

Copying data to /epccfs from /work file systems¶

You can use the standard Linux cp command to copy data from other Cirrus file systems to EPCCfs or vice versa. For example, to transfer the file important-data.tar.gz from the /work file system to /epccfs you would use the following command (assuming you are user auser in project e05):

cp /work/e05/e05/auser/important-data.tar.gz /epccfs/e05/e05/auser/

(remember to replace the project code and username with your own username and project code).

Quotas on EPCCfs¶

As for the other Cirrus storage systems, all projects are assigned a quota on EPCCfs. The project PI or manager can split this quota up between users or groups of users if they wish.

You can view any EPCCfs quotas that apply to your account by logging into SAFE and navigating to the page for your Cirrus login account.

1. Log into SAFE
2. Use the "Login accounts" menu and select your Cirrus login account
3. The "Login account details" table lists any user or group quotas that are linked with your account. (If there is no quota shown for a row then you have an unlimited quota for that item, but you may still may be limited by another quota.)

You can also query quotas that apply to your current EPCCfs directory from the the command line using the df -h $PWD command, for example:

auser@uan01:/epccfs/e05/e05/auser> df -h $PWD
Filesystem                          Size  Used Avail Use% Mounted on
fs02.naidin.epcc.ed.ac.uk:/cirrus  8.8T  1.8G  8.8T   1% /mnt/nfs/epccfs

Archiving¶

If you have related data that consists of a large number of small files it is strongly recommended to pack the files into a larger "archive" file for ease of transfer and manipulation. A single large file makes more efficient use of the file system and is easier to move and copy and transfer because significantly fewer meta-data operations are required. Archive files can be created using tools like tar and zip.

tar¶

The tar command packs files into a "tape archive" format. The command has general form:

tar [options] [file(s)]

Common options include:

• -c create a new archive
• -v verbosely list files processed
• -W verify the archive after writing
• -l confirm all file hard links are included in the archive
• -f use an archive file (for historical reasons, tar writes its output to stdout by default rather than a file).

Putting these together:

tar -cvWlf mydata.tar mydata

will create and verify an archive.

To extract files from a tar file, the option -x is used. For example:

tar -xf mydata.tar

will recover the contents of mydata.tar to the current working directory.

To verify an existing tar file against a set of data, the -d (diff) option can be used. By default, no output will be given if a verification succeeds and an example of a failed verification follows:

[auser@login01:auser]$ tar -df mydata.tar mydata/*
mydata/damaged_file: Mod time differs
mydata/damaged_file: Size differs

zip¶

The zip file format is widely used for archiving files and is supported by most major operating systems. The utility to create zip files can be run from the command line as:

zip [options] mydata.zip [file(s)]

Common options are:

• -r used to zip up a directory
• -# where "#" represents a digit ranging from 0 to 9 to specify compression level, 0 being the least and 9 the most. Default compression is -6 but we recommend using -0 to speed up the archiving process.

Together:

zip -0r mydata.zip mydata

will create an archive.

The corresponding unzip command is used to extract data from the archive. The simplest use case is:

unzip mydata.zip

which recovers the contents of the archive to the current working directory.

Files in a zip archive are stored with a CRC checksum to help detect data loss. unzip provides options for verifying this checksum against the stored files. The relevant flag is -t and is used as follows:

[auser@login01:auser]$ unzip -t mydata.zip
Archive:  mydata.zip
    testing: mydata/                 OK
    testing: mydata/file             OK
No errors detected in compressed data of mydata.zip.

Data transfer¶

Before you start¶

Read Harry Mangalam's guide on How to transfer large amounts of data via network. This tells you all you want to know about transferring data.

Data Transfer via SSH¶

The easiest way of transferring data to/from Cirrus is to use one of the standard programs based on the SSH protocol such as scp, sftp or rsync. These all use the same underlying mechanism (ssh) as you normally use to login to Cirrus. So, once the command has been executed via the command line, you will be prompted for your password for the specified account on the remote machine.

To avoid having to type in your password multiple times you can set up a ssh-key as documented in the User Guide at connecting

SSH Transfer Performance Considerations¶

The ssh protocol encrypts all traffic it sends. This means that file-transfer using ssh consumes a relatively large amount of CPU time at both ends of the transfer. The encryption algorithm used is negotiated between the ssh-client and the ssh-server. There are command line flags that allow you to specify a preference for which encryption algorithm should be used. You may be able to improve transfer speeds by requesting a different algorithm than the default. The arcfour algorithm is usually quite fast assuming both hosts support it.

A single ssh based transfer will usually not be able to saturate the available network bandwidth or the available disk bandwidth so you may see an overall improvement by running several data transfer operations in parallel. To reduce metadata interactions it is a good idea to overlap transfers of files from different directories.

In addition, you should consider the following when transferring data.

• Only transfer those files that are required. Consider which data you really need to keep.
• Combine lots of small files into a single tar archive, to reduce the overheads associated in initiating many separate data transfers (over SSH each file counts as an individual transfer).
• Compress data before sending it, e.g. using gzip.

scp command¶

The scp command creates a copy of a file, or if given the -r flag, a directory, on a remote machine.

For example, to transfer files to Cirrus:

scp [options] source user@login.cirrus.ac.uk:[destination]

(Remember to replace user with your Cirrus username in the example above.)

In the above example, the [destination] is optional, as when left out scp will simply copy the source into the user's home directory. Also the source should be the absolute path of the file/directory being copied or the command should be executed in the directory containing the source file/directory.

If you want to request a different encryption algorithm add the -c [algorithm-name] flag to the scp options. For example, to use the (usually faster) aes128-ctr encryption algorithm you would use:

scp [options] -c aes128-ctr source user@login.cirrus.ac.uk:[destination]

(Remember to replace user with your Cirrus username in the example above.)

rsync command¶

The rsync command can also transfer data between hosts using a ssh connection. It creates a copy of a file or, if given the -r flag, a directory at the given destination, similar to scp above.

Given the -a option rsync can also make exact copies (including permissions), this is referred to as mirroring. In this case the rsync command is executed with ssh to create the copy on a remote machine.

To transfer files to Cirrus using rsync the command should have the form:

rsync [options] -e ssh source user@login.cirrus.ac.uk:[destination]

(Remember to replace user with your Cirrus username in the example above.)

In the above example, the [destination] is optional, as when left out rsync will simply copy the source into the users home directory. Also the source should be the absolute path of the file/directory being copied or the command should be executed in the directory containing the source file/directory.

Additional flags can be specified for the underlying ssh command by using a quoted string as the argument of the -e flag. e.g.

rsync [options] -e "ssh -c aes128-ctr" source user@login.cirrus.ac.uk:[destination]

(Remember to replace user with your Cirrus username in the example above.)

Data transfer using rclone¶

Rclone is a versatile command-line program for transferring data. It can interface with various cloud storage services, such as MS OneDrive and Dropbox, as well as copying files locally or over SFTP. The program preserves timestamps and verifies checksums at all times.

rclone is available via the rclone module on Cirrus:

module load rclone

Note that rclone supports many options and sub-commands that are not covered here. Please refer to the official rclone documentation for full details.

Cloud storage¶

First we will cover using rclone to copy data to a cloud storage system. Configure rclone using ./rclone config. This will guide you through an interactive setup process where you can make a new remote (called remote). See the following for detailed instructions for:

• Microsoft OneDrive.
• Dropbox.

Please note that a token is required to connect from Cirrus to the cloud service. You need a web browser to get the token. The recommendation is to run rclone in your laptop using rclone authorize, get the token, and then copy the token from your laptop to Cirrus. The rclone website contains further instructions on configuring rclone on a remote machine without web browser.

Once all the above is done, you're ready to go. If you want to copy a directory, please use:

./rclone copy <cirrus_directory> remote:<cloud_directory>

Please note that "remote" is the name that you have chosen when running rclone config. To copy files, please use:

./rclone copyto <cirrus_file> remote:<cloud_file>

Local file transfer¶

It is possible to use rclone to copy files locally, for example from one of Cirrus' file systems to another. One advantage of using rclone for this is that it supports parallel data transfer, copying multiple files simultaneously for better performance. To copy local files, simply specify the source and destination directory paths:

./rclone copy <cirrus_directory> <another_cirrus_directory>

By default rclone will run 4 transfers simultaneously. However this can be customised using the --transfers option:

./rclone --transfers 8 copy <cirrus_directory> <another_cirrus_directory>

SFTP transfers¶

Using its SFTP storage type, rclone can transfer files to and from other systems (for example, other HPC services or compute clusters) that support SSH. As described above for cloud storage access, SFTP must be configured by adding a new remote to rclone before it can be used, by running ./rclone config. This command will guide you through the interactive setup process. For SFTP, the most important items are the hostname, the username, and the password if required. If the system you are accessing requires an SSH key, you can either provide the path to the private key and its passphrase as part of the setup process, or you can use ssh-agent to make the key available.

For many of the options, the default value will usually work fine. For full details of the SFTP setup process, see rclone's SFTP documentation.

Once you have set up the SFTP remote, it can be used in the same way as a cloud remote:

./rclone copy <cirrus_directory> remote:<sftp_directory>

"remote" is the name you have chosen when running rclone config.

If your transfers are taking a long time, you can pass the --progress option to see how they are progressing:

./rclone --progress copy <cirrus_directory> remote:<sftp_directory>

You may also want to tune the --transfers option which specifies how many files rclone will attempt to transfer in parallel (default 4). With some hosts you may have to reduce this number for the transfers to operate reliably, due to poor network connectivity, or security policies that limit the number of connections allowed. Conversely, if you have a good connection to the remote server, you may wish to increase this number to speed up your transfer:

./rclone --progress --transfers 8 copy <cirrus_directory> remote:<sftp_directory>