Like a virtual factory, the SciNet computing  systems run twenty-four hours a day, seven days a week; each second it is running, it completes up to 300,000,000,000,000 mathematical operations. Researchers across the country use the internet to construct their simulation or data analysis task remotely; then the supercomputer assigns it to a collection of its 40,000 processors when they become available. The systems tackle such tasks as biomedical research including studying Alzheimer’s and brain function; aerospace research such as finding cleaner-burning mixes of biofuels; and in astronomy for finding signals in the very first light to travel through the Universe.

“We’re enormously pleased that our centre, its people, and its facilities have been in such high demand from researchers across Ontario and all of Canada”, said Dr. Chris Loken, Chief Technical Officer of SciNet. “To have built something that has proven so essential for so many scientists, engineers, and others that it’s been asked to provide ten million compute `jobs’ in just two and a half years is remarkable.”

The science behind the ten-millionth job involves some of the most fundamental physics possible, the search to understand the basic properties of matter and the forces that govern the universe. ATLAS, an international experiment based at the Large Hadron Collider in Geneva, Switzerland is one of SciNet’s biggest users. “The SciNet facility provides the largest ATLAS Tier-2 Analysis Facility in Canada and we now run about 8,000 jobs a day for ATLAS, about 2.2 million in the last 12 months”, said Dr. Leslie Groer, in charge of the ATLAS project at SciNet. “Simulation calculations like this one are vital to understand the ATLAS detector and to analyze the physics coming from the largest experiment in the world.”

This article was featured on InsideHPC.

–

About SciNet:

SciNet is Canada’s largest supercomputer centre, providing Canadian researchers with the computational resources and expertise necessary to perform their research on scales not previously possible in Canada, from the biomedical sciences and aerospace engineering to astrophysics and climate science. More information is available at http://www.scinet.utoronto.ca.

About Compute Canada

Compute Canada is a national platform of advanced computing resources across the country, bringing together computer and data resources, academic researchers, and computational expertise to tackle some of the Canada’s biggest research questions. Compute Canada has built a user community across Canada in disciplines ranging from the sciences and engineering to arts and humanities. The Compute Canada Resource Allocation Committee annually awards supercomputing time to projects on the basis of scientific merit. For more information about Compute Canada or this year’s allocations, see https://computecanada.org.

We’ll discuss using the gnu compilers here; other compiler suites have similar options.

Let’s take a look at the following Fortran code:

program nantest
    real :: a, b, c
 
    a = 1.
    b = 2.
 
    c = a/b
    print *, c,a,b
 
    a = 0.
    b = 0.
 
    c = a/b
    print *, c,a,b
 
    a = 2.
    b = 1.
 
    c = a/b
    print *,c,a,b
end program nantest

If we compile this code with -ffpe-trap=invalid (I usually add ,zero,overflow , and even underflow if I think that’s causing me a problem in intermediate results), then the debugger can tell us the line where it all goes wrong:

$ gfortran -o nantest nantest.f90 -ffpe-trap=invalid,zero,overflow -g -static
$ gdb nantest
[...]
(gdb) run
Starting program: /scratch/ljdursi/Testing/fortran/nantest 
  0.50000000       1.0000000       2.0000000    
 
Program received signal SIGFPE, Arithmetic exception.
0x0000000000400384 in nantest () at nantest.f90:13
13          c = a/b
Current language:  auto; currently fortran

It's a little tricker with C code; we have to actually insert a call to "feenableexcept()", which enables floating point exceptions, and is defined in fenv.h;

#include <stdio.h>
#include <fenv.h>
 
int main(int argc, char **argv) {
    float a, b, c;
    feenableexcept(FE_DIVBYZERO | FE_INVALID | FE_OVERFLOW);
 
    a = 1.;
    b = 2.;
 
    c = a/b;
    printf("%f %f %f\n", a, b, c);
 
    a = 0.;
    b = 0.;
 
    c = a/b;
    printf("%f %f %f\n", a, b, c);
 
    a = 2.;
    b = 1.;
 
    c = a/b;
    printf("%f %f %f\n", a, b, c);
 
    return 0;
}

but the effect is the same:

$ gcc -o nantest nantest.c -lm -g
$ gdb ./nantest
[...]
(gdb) run
Starting program: /scratch/s/scinet/ljdursi/Testing/exception/nantest 
1.000000 2.000000 0.500000
 
Program received signal SIGFPE, Arithmetic exception.
0x00000000004005d0 in main (argc=1, argv=0x7fffffffe4b8) at nantest.c:17
17	    c = a/b;

either way, you have a much better handle on where the errors are occuring.



Scientists refer to the talent shortage as the “missing middle,” meaning there are enough specialists to run the handful of world-beating supercomputers that cost a few hundred million dollars, and plenty of people who can manage ordinary personal computers and server computer—but there are not nearly enough people who know how to use the small and mid-sized high-performance machines that cost anywhere from $1 million to $10 million.

From The Daily Beast, 28 Dec.

Interested in learning about programming supercomputers?   SciNet offers regular courses on a variety of topics!

Another good work for scientists wanting to understand the realities of high performance computing is How To Write Fast Numerical Code: A Small Introduction by Chellappa, Franchetti, and Puschel at CMU.  This smaller, more focussed tutorial comes out of a course one of the authors taught in 2005 and again in 2008, and the course materials themselves are an interesting resource.

The group of 30 U of T researchers, including 16 graduate students, has used the SciNet super-computing resources at U of T to sift through the ATLAS data to identify collisions containing Higgs boson candidates.

Find out more about how SciNet helped power Atlases search for the Higgs boson!

The students had a million questions, and we — including CTOs Chris Loken and Danny Gruner — enjoyed answering their questions and taking them on a tour of the facilities.   It gets loud (and cold!) in our data centre, but the students were eager:

Afterwards,the students worked on simulations of their own, to see what it is our researchers do on these huge machines.

And, most importantly, we all had a great time!

We’ll consider a simple numerical problem; imagine a projectile launched from height h=0 with velocity v₀=5000 m s^-1, and subject to the Earth’s gravitational accelleration, g = 9.81 m s^-2. We’re going to ask when the (first) time is that the projectile hits a height h.

This is going to be an application of our friend the quadratic equation:
.
Now, because of the repeated subtraction, a naive implementation of this equation is known to undergo catastrophic cancellation near b²=4ac, or for where the discriminant is much less than b — in our case, near the ends and the peak of the projectile’s trajectory.   We’re going to demonstrate that below.

Now, before we show that such sensitivity can happen, we should ask — why would we care? If we test our code and know it gives “good enough” answers under the conditions that matter to us, does it really matter what could happen in other circumstances? The answer, of course, is yes. There are a lot of things we could want to do — increase the agressiveness of compiler optimizations when compiling our code, for instance — which will have the effect of numerically perturbing our computation; and we need to know if those small perturbations will have small, or large, effects on our answers.

It turns out that IEEE 754, the standard for floating point numbers, can give us some help with this. (Everyone who does numerical work should know at least a little bit about the floating point standard, or at least the issues involved with floating point numbers. What every computer scientist should know about floating point, particularly the first few sections, is an essential guide). The floating point standard – which almost all widely-used computing hardware should support – allows you to set certain properties of the mathematics “on the fly”. One particularly useful feature is the ability to set how the last digit of all floating point operations are rounded – to nearest (the default), to zero (eg, always truncate), to positive infinity (eg, always round up) or to negative infinity (always round down). In the C99 standard, this is implemented in the “fenv.h” header and the math library; in Fortran2003, this is part of the intrinsic IEEE_ARITHMETIC module, where you can call IEEE_SET_ROUNDING_MODE.

By changing the rounding, you are perturbing every floating point operation in your calculation. If this perturbation results in significant changes in your result, then your calculation is very fragile, and you may have to look into re-writing the calculation, using another algorithm, or resorting to using higher precision for that calculation (which will push the perturbations to less significant decimal places). If not, then you have some evidence that your calculation is robust to perturbations, at least in the last bit.

Below we have an example of how you’d do this in C. We have a simple routine which uses the obvious implementation of the quadratic equation to calculate the time when the projectile is at one meter, and we perform this calculation with all available rounding modes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

#include <stdio.h>
#include <math.h>
#include <fenv.h>
 
const int NOSOLN=-1;
const int SOLN  = 0;
 
int time(const float vo, const float g, const float ho, float *time) {
    float disc  = (vo*vo - 2.*g*ho);
 
    if (disc < 0) return NOSOLN;
 
    disc = sqrt(disc);
    float root1 = (vo + disc)/g;
    float root2 = (vo - disc)/g;
 
    if ((root2 >= 0.) && root2 < root1)
        *time = root2;
    else
        *time = root1;
 
    return SOLN;
}
 
 
int main(int argc, char **argv) {
 
    const float g =9.81;
    const float vo=5000.;
    const int   ho=1.;
 
    int nroundings=4;
    int roundings[]={FE_TONEAREST, FE_UPWARD, FE_DOWNWARD, FE_TOWARDZERO};
    char *names[]  ={"To nearest", "To +inf", "To -inf", "To zero"};
 
    for (int r=0; r<nroundings; r++) {
        int status = fesetround(roundings[r]);
        if (status) {
            fprintf(stderr,"Could not set rounding to '%s'.\n", names[r]);
        } else {
            float soln;
            time(vo, g, ho, &soln);
            printf("%s: %f\n", names[r], soln);
        }
    }
 
    return 0;
}

We compile the code with gcc (any C99 compiler should work):

$ gcc -O0 -Wall -std=c99 quadratic.c -o quadratic -lm

Note that we need to explicitly link in the math library, and to turn off optimization (so that the compiler doesn’t replace the repeated calls to time() with a single call). Running this, we find:

$ ./quadratic
To nearest: 0.000199
To +inf: 0.000149
To -inf: 0.000249
To zero: 0.000249

Changing the rounding modes changes the result by 50%! This shows that our current implementation – which is not giving obviously wrong answers – is extremely fragile in the presence of numerical noise, and we should exercise extreme caution with compiler flags, etc. (We’ll leave it as an exercise for the reader to come up with a better approach!)

To prepare for this transition, ten (ethernet) nodes are currently reserved for test purposes already, and have the most common modules already installed. We encourage all users to try and compile their code there and to submit short test runs to the centos6 compute nodes.

Here’s how you can access the centos 6 nodes:

• Log in to one of the centos6 development nodes, gpc05 and gpc06.
• Make sure the right modules are loaded, as they likely will have changed in the upgrade.
• Recompile your code (don’t forget a “make clean” or equivalent).
• Submit test jobs to the centos6 compute nodes using an extra flag “os=centos6computeA” in the “-l” argument, i.e. “-l nodes=2:ppn=8,walltime=1:00:00,os=centos6computeA”

To make your .bashrc work for both versions of CentOS, check out the example .bashrc on the wiki page on how to distiguish the two OSes.

Note that most modules that were available under the CentOS 5 are available with CentOS 6, but often their version number has changed. Use “module avail” on the centos6 compute nodes to see what module are available, or check the Software & Libraries wiki page .

Let us know if there is anything that is not working as it should, or if there are modules that you need which are missing.

The goal of the first course is to have students learn the basics of best practice for writing maintainable, modular scientific programs. At the end of minicourse I, “Scientific Software Development”, students arriving with fairly modest scientific programming knowledge will leave being able to:

• Do basic software development in a linux-like environment
• Write a modular scientific program in C
• Read and write Makefiles for building large pieces of software
• Be able to install simple libraries on their systems
• Use git for basic version control for software development (or paper writing)
• Use Python for basic visualization and data manipulation
• Debug C programs with gdb/ddd
• Write simple C++ and Python programs
• Use gprof or python profiling to find performance “hot spots”

The course will require 4-6 hours each week spent on reading and homework.

Each lecture will have a hands-on component; students are strongly encouraged to bring laptops. (Contact us if this will be a problem). Windows, Mac, or Linux laptops are all fine, but some software will have to be installed *before* the first lecture:

On windows laptops only, Cygwin will have to be installed ; ensure that development tools (gcc/g++/gfortran, gdb), git, and the X environment (Xorg) is installed.

On Mac laptops, ensure that the development tools (Xcode) is installed.

On Linux, ensure that packages for the gcc compiler suite (gcc/g++/gfortran), gdb, and git are installed

On all platforms, the Enthought python distribution must be installed.

Students who aren’t already comfortable with working in a shell / terminal environment should work through at least the first three 10-minute lectures at Software Carpentry .

Course outline:

Week 1 – Intro to software developemnt: basics of C, make, git.
- HW1: modular, multifile programming and make.

Week 2 – Simple parabolic PDEs; modular programming, refactoring, and testing. Simple visualization using python.
- HW2: Refactoring, testing, and debugging a simple PDE solver

Week 3 – Structures in C; simple ODE solvers and interpolation.
- HW3: tracer particle evolution

Week 4 – Going further with C++ and Python; profiling
- HW4: “porting” tracer particle evolution to C++ or python.

Evaluation will be based entirely on the four homeworks, with equal weighting given to each.

SciNet may be willing to work with you or your school on similar projects – to inquire, contact us.

Building a Pint-Sized Supercomputer

Big computing — whether for supercomputing simulations of black holes, or business analytics for Walmart sales data, or Facebook’s data centres — is done on clusters, large computers assembled from smaller ones. The challenge is in getting the smaller computers to work together on single tasks.

In this project, a team of students, working with a number of experts from SciNet, will assemble a working cluster from scratch from existing PCs, and will examine the effects of different hardware and software choices on the performance of various pieces of parallel software.

Recommended Background:
Strong computer skills.

Estimated Number of Hours:
One to two hours per week working with mentor, with additional reading and computer homework.

Number of Students:
One to three students will be accepted.

New Models for Parallel Scientific Computing

Scientific, Technical, and Data-Intensive computing, to give timely results and to be on the cutting edge, have to make use of parallel computing: using many CPUs at the same time. But writing such software — breaking a problem into individual parts which can be solved separately and then recombined — is very challenging. What tools and languages there are for writing such software is usually either quite old, or is designed for problems very different than scientific programming.

In this project, students will take existing serial programs for simplified but relevant scientific computations, and use emerging tools to re-write them for parallel computing on clusters of PCs, or on graphics cards. The pros and cons of the different tools will be compared. The results will likely be written up as a case study and published. The problems, and the tools used, will be chosen based on the interests and knowledge of the students.

Recommended Background:
Strong programming skills in a language like C++ or Java; strong math skills.

Estimated Number of Hours:
One to two hours per week working with mentor, with additional reading and computer homework.

Number of Students:
One to three students will be accepted.