Staging with srm commands

Staging with srm commands

Staging a single file

• Check the status of a file:

srmls -l srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/projects.nl/tropomi/natalie/zap.tar | grep locality
#locality:ONLINE_AND_NEARLINE

ONLINE: The file is only on disk
NEARLINE: The file is only on tape
ONLINE_AND_NEARLINE: The file is both on disk and on tape

• Submit a staging request for a single file:

After submitting the staging command below, the prompt is waiting for the file to get online. Once the file gets online, a unique request id is returned.

The pin lifetime is set in seconds, in this examples the requested pin time is a day (or 86400 sec).

srm-bring-online -request_lifetime=86400 srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/projects.nl/tropomi/natalie/zap.tar
#srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/projects.nl/tropomi/natalie/zap.tar brought online, use request id 897617461 to release

• Unpin a file:

After submitting the unpinning command below, the file will remain cached but purgeable until new requests will claim the available space.

srm-release-files srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/projects.nl/tropomi/natalie/zap.tar

Staging a list of files

Here is an example to stage a list of files. Let's say that you want to stage all the .tgz files in a certain dCache directory like /pnfs/grid.sara.nl/data/projects.nl/tropomi/natalie/. We will use the command srm-bring-online. However, when you run this command the prompt will hang until the file is actually staged. So you can start first a screen shell to make sure your copying process continues when you accidentally lose connection to the server.

• Start screen on the trui:

screen

• Create the file list and note that you need to use a SURL for your filepaths:

FILES=`srmls srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/projects.nl/tropomi/natalie/ | egrep 'tgz' | awk '{print "srm://srm.grid.sara.nl:8443"$2}'`

• Check if the list looks OK. All filenames should be split with spaces in one line:

echo $FILES
#srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/projects.nl/tropomi/natalie/openstack_test_dataset.tgz srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/projects.nl/tropomi/natalie/output_dcache_token_2.tgz srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/projects.nl/tropomi/natalie/output_dcache_token_3.tgz srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/projects.nl/tropomi/natalie/output_dcache_token_1.tgz

• Check the status of the files to see how many are online (Disk) and how many are nearline (Tape):

srmls -l $FILES | grep locality | awk '/ONLINE/{i++};/:NEARLINE/{j++}; END{print "Disk:", i, ", Tape:",  j}'
#Disk:  , Tape: 4

• Submit the stage command to request staging the bulk of files. You can store the output in a file stage.log to save the request IDs. The pin lifetime here is set to 1 week, but note that this counts from the moment you submit the request independent to the actual time that the files are on disk:

srm-bring-online $FILES -request_lifetime=604800 > stage.log
# prompt will hang until operation is complete for all the files

• Once processing of the requested files is done, you can release the bulk of files so that the pin is removed and the staging read-pool is free for other data:

srm-release-files $FILES