As computing grows more important — for creating the next Facebook or Google, or for designing bicycles, studying black holes, or improving Canadians’ health — using a single computer just isn’t enough.  More and more, firms and researchers turn to “cluster computing”, building a single supercomputer out of many individual computers.

Three of the students wrestling with an OpenMP problem

Working with staff from SciNet, Canada’s largest open supercomputing centre, the students at the SATEC school, (many of whom have been part of the school’s ICT Specialist High Skills Major program) with the help of their teacher Sacha Noukhovitch and their ICT department, learned how to write parallel programs to tackle the biggest computations, faster.   They learned to use OpenMP to make use of all the processors on modern motherboards, and MPI to communicate between different nodes within a cluster.

Some of the students are cabling up what will be the "Goliath" cluster, a cluster of 3 old single-core Pentium-4 desktops with 100Mb ethernet.

Learning the theory wasn’t enough for these students, however.  They started building their own cluster, first using old PCs in one of the school’s computer labs, networking them together and installing BCCD Linux on the systems to start running them as one single compute resource.

Once this computer, “Goliath” was up and running the students started assembling a LittleFe cluster, “David” a cluster with 12 Intel Atom processors and 6 NVIDIA CUDA-capable GPUs, with a top speed  of 13.6 GFLOPS (13.6 billion mathematical operations per second– a power which would have made it one of the worlds fastest computers in 1997).

All the parts for a LittleFe gathered together. Doesn't look much like a supercomputer at this point.

The LittleFe cluster construction was one part computer engineering and one part carpentry, assembling the aluminum frame, running cables and wiring, mounting the six motherboards to plates which slide into the finished frame.   Once the machine was physically assembled, which took about a month, the software side began, installing BCCD Linux on to the “head node” and then having the “client nodes” boot from the head node.

It's Alive!

On May 9th, the David cluster powered up for the first time. Then the experiments began – how much faster was this mini-supercomputer than Goliath, or their desktops?   What sorts of problems work well in parallel on this cluster?

The students examined the scaling behaviour of three different computer programs on the cluster: Galaxsee, which simulates gravitational N-body dynamics such as early models of galaxy formation; Life, a parallel implementation of Conway’s game of life; and Monkey, software written by one of the SATEC students investigating the “infinite monkey theorem”, measuring the rate at which a sequence of randomly-generated characters matches text from Hamlet.

The team at their booth at University College

The students presented their results on May 17th at University College at the University of Toronto, at a poster session for high school students working with University of Toronto researchers.  The presented their scaling results and summary of what they learned, and had both the Goliath and David clusters up and running.  The project generated a huge amount of interest as visitors flocked to the booth to ask questions.

This year explorers of all ages at our booth found out how researchers use computers for discovery.  They saw how even simple computer simulations that you can run in your web browser can teach them important facts about how complex systems behave.

A Science Rendezvous explorer learns how computer simulations can teach us about how forest fires spread

Canada is a leader in forest fire research; we have huge stands of forests and we must understand how fires behave if we’re to prevent them.    Students explored the Forest Fire application by Shodor and saw how wind speed and forest density effect forest fires in  a simplified model of how forest fires spread.

Students also crashed galaxies together with Galcrash; processes that take billions of years unfolded as they watched and they saw how different galaxy orientations — and even turning “on” and “off” effects of gravitational dark matter — change the outcome of these enormous events.

Students use computer simulations to explore how galaxy collisions - which unfold over billions of years - take place

But as researchers ask harder questions – what if we wanted to include topography in our forest fire simulation?  Or more realistic galaxies in our galaxy collision simulations? – they quickly outgrow what can be done on a single computer.  That’s where we come in!

SciNetter Danny Gruner explains cluster computing and our Little Fe cluster at Science Rendezvous 2012

To show how supercomputers work,  we also debuted our LittleFe cluster, showing how networking together ordinary computers can, with carefully written software, make use of the power of many processors and memory to tackle bigger problems than you could with a single computer.

Many thanks the organizers who made this possible, and to everyone who turned out on a Saturday to discover science!

• SciNet is pleased to announce that it will host and run a new Blue Gene/Q system for the newly-announced Southern Ontario Smart Computing and Innovation Partnership (SOSCIP). The system will consist of at least 40,000 cores with a peak theoretical speed exceeding 500 TFlops and should be operational in the fall of this year. This will be the first Blue Gene system in Canada. For more information see the University of Toronto release and elsewhere on this site.Renovations needed to expand the SciNet machine room will be ongoing for the next 3-4 months and will necessitate some shutdowns for
  electrical and plumbing work. Great care is being taken to minimize the interruptions to SciNet researchers but at this stage we anticipate two shutdowns of 1-2 days each.The first shutdown associated with the machine room expansion will take place next week on Wed 18 April. We were already planning a shutdown in order to make changes to GPC networking and we will now
  combine those into a single, but longer shutdown. Details will be sent in a separate email but we expect systems will go down on the morning of 18 April and come back in the evening of 19 April.
• GPC: In the last two weeks, there was a problem which caused quite a lot of multinode jobs to crash and caused file system issues. It
  took a lot of hard work to get the gpc in a stable state because the issue was caused by the unique interplay of the particular
  configurations of the gpc (involving hardware and software issues).  A lot of mainly smaller jobs still ran successfully, so rather than
  shutting down the gpc for an undetermined time, solutions were rolled out by rebooting nodes after jobs ended. This improved things for some users, while in a few other cases, we were able to provide a temporary work-around.Currently, we have found what seems to be a stable configuration for the gpc, and expect users to be able to run jobs as usual. However, if you encounter any further trouble, please contact us.
• GPC: Due to the network changes we are making to the GigE nodes, if you run multinode ethernet MPI jobs, you still need to explicitly request the ethernet interface in your mpirun command:For Openmpi: mpirun –mca btl self,sm,tcp
  For IntelMPI: mpirun -env I_MPI_FABRICS shm:tcpThere is no need to do this if you run on IB, or if you run single node mpi jobs on the ethernet (GigE) nodes. Please check the wiki
  page on ‘GPC MPI Versions’ for more details. We expect these changes to be finished by the end of the month.

EVENTS COMING UP

Unless stated otherwise, all events take place at the SciNet Headquarters, Rm 235 of 256 McCaul Street, Toronto. All events below are free but we ask that you sign up on the courses website: https://support.scinet.utoronto.ca/courses.

• Apr 16/24: DEADLINES COMPUTE CANADA SPECIAL RESOURCE ALLOCATION CALLEvery fall, the Compute Canada Resource Allocation Committee issues
  an annual Call for Proposals for allocations of its distributed
  computing and data storage systems. This spring, two specialized
  Calls for Proposals requests are being issued outside of the regular
  cycle:
  1. Large Shared Memory System: Deadline: Monday, April 16, 2012 at 3:00 pm Eastern
  2. Humanities and Social Sciences Researchers: Deadline: Tuesday, April 24, 2012 at 3:00 pm Eastern

  See https://computecanada.org for details.

• Wed Apr 11, noon: SCINET USER GROUP (SNUG) MEETINGThe SciNet Users Group (SNUG) meetings are every month on the second Wednesday, and involve pizza, user discussion, feedback, and one or two short talks on topics or technologies of interest to the SciNet community.This time, we will have:
  • TechTalk by Scott Northrup (SciNet) on “Infiniband on the GPC”
  • User discussion
  • Pizza!

  Sign up at https://support.scinet.utoronto.ca/courses/?q=node/49

• Thu Apr 12: SCHEDULED SHUTDOWN OF THE TCSThe TCS will be shutdown Thursday, 12 April in order to do some maintenance on the InfiniBand network. The system will go down at 11 am and should be available again by 5 pm.
• Wed Apr 18-19: SCHEDULED SHUTDOWN OF ALL SCINET SYSTEMSThis is the first shutdown needed to expand the capacity of the SciNet datacentre for the coming Blue Gene/Q system. We expect systems will go down on the morning of 18 April and come back in the evening of 19 April.
• Mon Apr 23: INTRODUCTION TO SCIENTIFIC C++This is a one-day course that will introduce you to the various features of C++ with a focus on those that are useful for scientific software development. We will take the C-to-C++ route, so familiarity with C, in particular with pointers, is a prerequisite.We will cover:
  • a basic refresher of C;
  • the nice features of C++ (“a better C”);
  • object oriented programming (classes, inheritance, …);
  • very basic generic programming with templates;
  • a discussion of some useful libraries out there.

  Sign up at our courses website.

• Wed May 9, 10:30 am – 12:00 am: INTRO TO SCINETLearn what SciNet resources are available, how to recompile your code and how to use the batch system, in approximately 90 minutes.Intended for new users, but experienced users may still pick up some valuable pointers.

  Sign up at our courses website.

  Note that attendants to the intro may be interested in the immediately following event:

• Wed May 9, 12:00 am: MAY SNUG MEETINGWe are still looking for users willing to giving a short talk (20-30 minutes) at the May SNUG about interesting work that they did on SciNet clusters and how they did it! If you are up for it, email support@scinet.utoronto.ca.More info on future SNUGs and sign-up at https://support.scinet.utoronto.ca/courses/?q=node/51
• May 14-18: SCICOMPSciNet will host the annual meeting of ScicomP, the IBM HPC Systems Scientific Computing User Group. This meeting (which is part of the meeting of SPXXL, the user group of large IBM installations) is open to users and deals mostly with applications and science rather than just with technical aspects of the computers.This event will not be held at the SciNet Headquarters. For more information on this event, its schedule, location and registration, go to http://spscicomp.org/scicomp2012 .

ADDED TO THE WIKI IN MARCH

All new wiki content below is listed and linked on the main page:

http://wiki.scinethpc.ca/wiki/index.php/SciNet_User_Support_Library#What.27s_New_On_The_Wiki)

• Slides of the fourth lecture of High Performance Scientific Computing
• Slides of the TechTalk on the Intel Math Kernel Library
• Software installed on the Power 7 Linux cluster
• A FAQ entry on how to deal with ib memory problems

WHAT ELSE HAPPENED AT SCINET IN MARCH?

• Mar 12: SNUG meeting was held, with a TechTalk by Ramses van Zon on “The Intel Math Kernel Library”
• Mar 26: “Introduction to the Linux Shell” was given.
• Mar 28: “Intro to SciNet” was given.

SciNet announced today that, as part of the new Southern Ontario Smart Computing and Innovation Partnership (SOSCIP) with the IBM, the Federal Government, Government of Ontario, the University of Toronto and six other Ontario Universities, it will be running what is expected to be Canada’s largest computer, an IBM Blue Gene/Q which will be able to execute almost a half quadrillion mathematical operations per second.

Such high-end computing power used to be used exclusively for flagship government or industrial projects as designing racing cars, airplane engines, or nuclear reactors. The increasingly widespread availability of computing resources means that high performance computing (HPC) is now also used for designing plastic coffee containers, bicycle wheels, and golf clubs. And indeed, a major thrust of the SOSCIP project will be to work with Ontario small and medium-sized enterprises to speed product research and development, to speed innovation in the Ontario economy.

SciNet is one of seven Compute Canada high performance computing consortia across Canada, and is already a local hub for technical and high-performance computing expertise; but this new project increases the amount of resources available to work with Ontario’s flourishing innovative technology sector. “This provides an excellent model for how the Compute Canada consortia can be leveraged to support innovation and excellence across Canada,” said Jill Kowalchuck, Executive director of Compute Canada.

The new system is expected to be up and running in the Fall, with necessary upgrades to the SciNet data centre already starting. “We’re very excited about our role in this new project,” said Dr. Chris Loken, CTO of SciNet. “This incredible new resource, plus our existing experience in working with a wide range of clients solving their research and development computing problems, is going to enable us to help many more Ontario innovators use computing to solve their R&D problems.”

About SciNet

SciNet is Canada’s largest supercomputer centre, providing Canadian researchers with the computational resources and expertise necessary to perform their research on scales not previously possible in Canada, from the biomedical sciences and aerospace engineering to astrophysics and climate science. SciNet is funded in part by the Canada Foundation for Innovation, the Province of Ontario, and the University of Toronto. More information is available at http://www.SciNetHPC.ca , or contact us.

About Compute Canada

Compute Canada is a national platform of advanced computing resources across the country, bringing together computer and data resources, academic researchers, and computational expertise to tackle some of the Canada’s biggest research questions. Compute Canada has built a user community across Canada in disciplines ranging from the sciences and engineering to arts and humanities. For more information about Compute Canada or this year’s allocations, see https://computecanada.org .

For More Information

More information will be posted shortly; for now, releases are available from

• IBM,
• University of Toronto,
• Western University
• The government of Ontario, and
• The Federal government.

SciNet is unique – a one-stop shop with research scientists, technical computing and cluster experts, and High Performance Computing (HPC) resources all available to you. That means you can come in, work with us to improve your software, get accounts, and start getting your computing done today. The same experts working with you in front of a whiteboard, or teaching you in HPC training sessions, can also be the ones to monitor your first few computations on our massive cluster and make sure that they are working the way they should. We’ll even help you design your own computer system, or move your computations to the Amazon cloud, if that’s what you prefer.

Who we are: SciNet is the largest concentration of academic HPC research expertise in Canada, at the University of Toronto, and we run Canada’s largest supercomputing centre. We have eleven technical staff, including ten research scientists and engineers with nine PhDs among us, and have expertise in chemistry, physics fluid dynamics, and bio-informatics; in massive storage, numerical methods, and in all manner of technical computing methods, and on systems from desktops and GPUs to clusters and specialized supercomputers.

What we have: Canada’s largest open research computer facility (soon 75,000 processors, to include Canada’s only BlueGene system); more storage than you could ever use (6PB and growing); and a mandate to help drive innovation and discovery in Toronto and across Canada.

What we do: We offer a wide range of services to clients in academia and industry. We consult on complex technical computing problems; we offer our compute resources at extremely competitive rates to Canadian researchers and innovators; we teach training sessions on computational and data-enabled science at scale; and we can even help develop or customize the software you need to take your work to the next level.

How we’ve helped other clients get better results, faster:

• A chemist visited us who wanted to do hundreds of Matlab computations, each of which took hours on her workstation. One of our consultants sat down with her and, in an afternoon, reworked the calculation to be 200x faster, so that everything could be done over a weekend on her desktop, without even using our cluster; she ended up with a better code and got all her work done faster.

• A molecular genetics researcher wanted to hugely scale up their BLAST sequence alignment analyses to consider thousands of possible cases; we worked with them to enable these enormous searches quickly and efficiently on our cluster and others.

• One established engineering firm was outgrowing their in-house computational resources and needed bigger, faster computing for a few months without buying and installing a whole new system. Within days they were up and running on our cluster, developing better, more competitive products.

We thrive on helping drive innovation and discovery, in academia, with small businesses, and in large, established firms. How can we help you? Contact us to find out.

To give you an idea of what’s covered, here’s the table of contents:

• HPC, SciNet, and You
• SciNet’s HPC Innovation Services
• Our Computing Resources
• Storage and Data Security
• Working With SciNet.
• Our Staff
• HPC Training
• Success Stories
• Getting Started with SciNet

Once you’ve read our booklet, feel free to Contact Us and ask us any questions!

• Is Canada’s largest open supercomputing centre
• Operates two (and soon three) of Canada’s largest computers
• Headquarters downtown on University of Toronto’s St. George campus; data centre in Vaughan
• Runs one of Canada’s greenest data centres, taking full advantage of its size and Canada’s cold winters to cut power needed for cooling
• Once BG/Q is installed, will have almost one petaflop (1 quadrillion mathematical operations per second) of installed compute power, 2 petabytes (28 years worth of HDTV content) of disk storage, and double that of archival tape storage
• Has over one thousand clients using our systems and expertise to power research and development in aerospace, astrophysics, biomedicine, forestry, natural water systems, climate science, and any other problem area you could imagine.
• Runs the entire centre with only thirteen staff, including eleven technical staff with 9 PhDs
• Is consortium of the University of Toronto and associated research hospitals
• Is part of Compute Canada, a national platform of seven such regional consortia
• Is Funded by the Canada Foundation for Innovation, the Province of Ontario, NSERC, and the University of Toronto faculties of Arts and Science, Engineering, and Medicine, and the University of Toronto Scarborough

For more information about SciNet, see our “About SciNet” page.

The GPC consists of 3,780 nodes (IBM iDataPlex DX360M2) with a total of 30,240 cores (Intel Xeon E5540) at 2.53GHz, with 16GB RAM per node (2GB per core) with some larger-memory nodes up to 32GB.   The nodes run Linux.   Approximately one quarter of the cluster is interconnected with non-blocking 4x-DDR InfiniBand while the rest of the nodes are connected with gigabit ethernet. The compute nodes are accessed through a queuing system that allows jobs with a maximum wall time of 48 hours.

A “quickstart” guide to using SciNet’s GPC can be found on our technical documentation wiki.

The devel nodes for this cluster are tcs01 and tcs02, and the compilers we recommend are IBM’s xl compilers.   A “quickstart” guide to running on SciNet’s TCS cluster can be found on our technical documentation wiki.