File system restructuring

In the near future, the home, scratch, project and hpss file systems  at SciNet will be restructured.   As a consequence, users’ files will reside in  different (absolute) locations than before.  The main objective of this move is to reduce the access contention to the root directory of  these file systems, which we have found can cause trouble to the gpfs file system.  The new file system layout will be group-based, and groups will furthermore be clustered by the initial letter of the group name.

To be more specific, the move will involve the following renaming scheme:

Old Location Old Location Environment Variable
/scratch/USER /scratch/G/GROUP/USER $SCRATCH

(Note: not all users have access to the latter two).  The “G” in the new location stands for the first letter of the group’s name. For instance, the current home directory of user ‘resu’ in group ‘puorg’ would move from /home/resu to /home/p/puorg/resu. Likewise, /scratch/resu would be moved to /scratch/p/puorg/resu.

To facilitate the transition, we ask the user’s cooperation in making sure all their scripts and applications only use relative paths, or use the predefined variables $HOME, $SCRATCH and $PROJECT. These variables are already defined now, and are set to the current (old) location.

Here are some examples of the type of changes you may need to make:

  • If a job script calls an executable in your home directory and refers to a location in your scratch directory, e.g.
    #PBS -l nodes=1:ppn=8;walltime=24:00:00
    #PBS -N runexecutable
    /home/rzon/code/executable > /scratch/rzon/out

    then you should change the last line to

    $HOME/code/executable > $SCRATCH/out
  • If you add a directory in your project directory to the PATH variable, e.g.
    export PATH=$PATH:/project/group/rzon/bin then you should change that to
    export PATH=$PATH:$PROJECT/bin.
  • If you add a directory in your group’s project directory to the PATH variable, e.g.
    export PATH=$PATH:/project/group/bin

    then you should change that to

    export PATH=$PATH:${PROJECT%/*}/bin

    , which uses a bash trick to remove the user part from the directory

  • If you have a symbolic link called scratch in your home directory pointing to your scratch directory, you will need to remove it and relink it, e.g.
    rm $HOME/scratch
    ln -s $SCRATCH $HOME/scratch

    You’ll only need to do this once, after the file system change has
    been made.

The locations of executables and libraries in SciNet’s modules will not change, but in any case, you should be using the environment variables that the modules defines ($SCINET_[module]_BASE, $SCINET_[module]_INC and $SCINET_[module]_LIB).

It is worth stressing that you do not have to move any files yourself; we will move them by renaming their top directory. Your only responsibility in this matter is to make sure your scripts work in the new setting.

The exact time line for this change is not yet set but it will likely coincide with a downtime somewhere between two weeks from now and the end of the year.

Note that we are here to help you if you experience difficulties in adapting your scripts to use these environment variables; just email support.  In addition, in the next SNUG on Nov 9th, we will discuss some of the common cases.

Yours sincerely,
The SciNet Team