SciNet’s publication about Niagara deployment

August 3, 2019 in blog, blog-general, blog-technical, for_press, for_researchers, for_users, frontpage, news, Road_to_Niagara

Have you ever wondered how a supercomputer is designed and brought to life?
Read SciNet’s latest paper on the deployment of Canada’s fastest supercomputer: Niagara.

Niagara is currently the fastest supercomputer accessible to academics in Canada.
In this paper we describe the transition process from our previous systems, the TCS and GPC, the procurement and deployment processes, as well as the unique features that make Niagara a one-of-a-kind machine in Canada.

Please cite this paper when using Niagara to run your computations, simulations or analysis:
“Deploying a Top-100 Supercomputer for Large Parallel Workloads: the Niagara Supercomputer”, Ponce et al, “Proceedings of PEARC’19: Practice and Experience in Advanced Research Computing on Rise of the Machines (Learning)”, 34 (2019).

Learn more about SciNet’s research and publications by visiting the following link.

SciNet News June 2019

June 4, 2019 in for_researchers, for_users, newsletter

Summary

  • Increased security of the SciNet data centre
  • Decommissioning of the SOSCIP BlueGene/Q at the end of June.
  • Announcement of several training opportunities.

Details can be found below and are also available on the SciNet education website https://courses.scinet.utoronto.ca and the SciNet wiki https://docs.scinet.utoronto.ca.

System News

  • The security level of the data centre has been increased by disabling certain weak encryption algorithms and certain weak public key types and by regenerating Niagara’s host keys. The latter requires users to replace their ssh clients stored host key for Niagara upon first login after the shutdown. If you have not done so yet, the details of how to do this can be found on https://docs.scinet.utoronto.ca/index.php/SSH_Changes_in_May_2019.
  • The SOSCIP BGQ will be decommissioned on June 30, 2019. The P7 cluster will be decommisioned at the same time. The file system of the BGQ, which is shared with the SOSCIP GPU Cluster, will remain active, and the BGQ front-end node (bgqdev.scinet.utoronto.ca) will remain up for the foreseeable future to allow users to access their files. If you have further questions regarding the decommisioning of the BGQ, write to support@scinet.utoronto.ca. For general questions regarding SOSCIP, write to info@soscip.org.

Events Coming Up

Unless stated otherwise, all events listed below are free of charge and take place at the SciNet Teaching Room or Boardroom at our offices on the eleventh floor of the MaRS West Tower, suite 1140A (661 University Avenue, Toronto ON M5G 1M1).

Most events are recorded and some are broadcast, but only some of the courses can be taken remotely for SciNet certificate credits.

Registration for SciNet courses should be done by logging into https://courses.scinet.utoronto.ca with your Compute Canada account (the same one that you use to log into Niagara).

  • INTRO TO SCINET AND NIAGARA Wednesday June 12, 10:00 am – 11:30 am SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    This is a class of approximately 60-90 minutes to introduce SciNet and the Niagara supercomputer and teach you how to use it.

    Participation counts towards the SciNet HPC Certificate.

    For more information and registration, go to https://scinet.courses/465

  • SCINET USER GROUP MEETING Wednesday June 12, 12:00 noon – 1:00 pm SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    Pizza, user discussion, and a tech talk by Ramses van Zon on

    “No Conda: Using Python, Installing Packages, and Accessing Jupyter Notebooks on Niagara”

    Abstract: If you want to know how to use install specific Python packages or how to use jupyter notebooks, most instructions that you find these days will tell you to use anaconda or miniconda. While very convenient on your own personal computer, conda comes with substantial storage inefficiencies and conflicts with other software, and is therefore not the recommended option on supercomputers. In this techtalk, we will explain and demonstrate how you can instead use virtual environments instead of conda environments, and use Niagara’s JupyterHub instance (which also supports R and can be made to work with Julia as well).

    For sign up and more information, see https://scinet.courses/471

  • COMPUTE ONTARIO SUMMER SCHOOL West – Hamilton June 10-14, 2019 Central – Toronto, June 24-28, 2019 East – Ottawa, July 8-12, 2019

    The Compute Ontario Summer School on Scientific and High Performance Computing is an annual educational event for graduate/undergraduate students, postdocs and researchers to learn and share knowledge and experience in high performance and technical computing on modern HPC platforms.

    As in previous years, the 2019 Summer School will have three instances, “West”, “Central”, and “East”. Each will be a week-long event with multiple parallel streams.

    Registration for the central instance will be open until June 17, 2019, see https://scinet.courses/438

  • SCALING TO PETASCALE INSTITUTE August 19-23, 2019

    This is a free virtual advanced HPC summer school, organized by a number of the US XSEDE sites and SciNet. include OpenMP, MPI, CUDA, and OpenACC.

    More details can be found at https://bluewaters.ncsa.illinois.edu/petascale-computing-2019

  • Intro to GIT Version Control Friday October 4, 2019, 1pm – 4pm Teaching Room 1140A (MaRS West Tower, 661 University Ave, Toronto)

    Introductory workshop to get started in the usage of version control GIT. This workshop is held in collaboration with UofT-Libraries and UofT graduate students could gain GPS credits.

    For sign up and more information, see https://scinet.courses/467

  • INTRODUCTION TO PROGRAMMING October 2019 Tuesdays and Thursdays, 1:00 pm – 2:00 pm Four weeks starting Oct 1. Teaching Room 1140A (MaRS West Tower, 661 University Ave, Toronto)

    New to programming? Learn the basics of programming using the python programming language in eight one-hour sessions over the course of four weeks. Sessions will consist of a mix of lectures and hands-on exercises.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://scinet.courses/472

  • NUMERICAL COMPUTING WITH PYTHON November 2019 Tuesdays and Thursdays, 1:00 pm – 2:00 pm Four weeks, starting Nov 5, 2019 Teaching Room 1140A (MaRS West Tower, 661 University Ave, Toronto)

    Learn about research computing even with little programming experience. Covers programming in python, best practices and visualization. Experience with python is required. Four home work sets will be the basic of the evaluation.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://scinet.courses/473

  • SCIENTIFIC COMPUTING FOR PHYSICISTS (PHY1610) Winter 2019 Teaching Room 1140A (MaRS West Tower, 661 University Ave, Toronto)

    This course is aimed at reducing your struggle in getting started with computational projects, and make you a more efficient computational scientist. Topics include well-established best practices for developing software as it applies to scientific computations, common numerical techniques and packages, and aspects of high performance computing. While we will introduce the C++ language, in one language or another, students should already have some programming experience. Despite the title, this course is suitable for many physical scientists (chemists, astronomers, …).

    This course is part of the physics graduate program. Students willing to take the course as part of their graduate program have to enroll through Acorn/ROSI.

    For more information, see https://scinet.courses/468

2019 Compute Ontario Summer School Central

May 14, 2019 in blog, for_educators, for_press, for_researchers, for_users, frontpage, news

The Compute Ontario Summer School on Scientific and High Performance Computing is an annual educational event for graduate/undergraduate students, postdocs and researchers who are engaged in a compute intensive research. Held geographically in the west, centre and east of the province of Ontario, the summer school provides attendees with the opportunity to learn and share knowledge and experience in high performance and technical computing on modern HPC platforms.

Each site will have a slightly different list of courses. The summer school will include both in-class lectures and hands-on labs (done on the participants’ laptops). Those who attend at least three full days cumulatively will receive an official certificate in HPC training (i.e., a total of 6 full morning and afternoon sessions).

Instructors for this school have been provided by SciNet, CAMH and SHARCNET. Break refreshments are provided courtesy of Compute Ontario.

Registration for the central installment in Toronto from June 24-28, 2019 is now open!

The registration is free and is aimed at Compute Canada users as well as students, post-docs and other researchers from academic institutions. You do not need to have a SciNet or Compute Canada account (although you can use that). Please be advised that seats are limited and tend to fill up.

SciNet News March 2019

March 4, 2019 in for_researchers, for_users, newsletter

Summary

  • Two additional ways to interact with the nearline/archive HPSS system: Globus and a Virtual File System.
  • Compute jobs on Niagara now have access to temporary burst buffer space.
  • Announcement of several techtalks at the SciNet User Group Meetings.

Details can be found below and are also available on the SciNet education website courses.scinet.utoronto.ca and the SciNet wiki docs.scinet.utoronto.ca.

System News

  • The Globus interface to the nearline/archive HPSS system is now operational. The endpoint name is computecanada#hpss. See https://docs.scinet.utoronto.ca/index.php/Globus and https://docs.scinet.utoronto.ca/index.php/HPSS for more details.
  • A Virtual File System (VFS) interface to the nearline/archive HPSS system is now available on a special dedicated node which you access through the scheduler. This interface makes HPSS seem like a regular file system, which is really only suited for small scale data management. See https://docs.scinet.utoronto.ca/index.php/HPSS for more details.
  • Compute jobs on Niagara now get access to temporary space on the Burst Buffer (a faster file system than scratch). For every job on Niagara, the scheduler now creates a temporary directory on the burst buffer called $BB_JOB_DIR. The $BB_JOB_DIR directory will be empty when your job starts and its content gets deleted after the job has finished. The directory is shared among the nodes of a job. This can be useful for I/O heavy jobs whose temporary files do not fit in ramdisk. See https://docs.scinet.utoronto.ca/index.php/Burst_Buffer for more details.
  • For consistency with the general purpose clusters Cedar and Graham, the environment variable $SLURM_TMPDIR will be set in Niagara compute jobs. This variable will point to ramdisk, not to local hard drives (since Niagara has none).

Events Coming Up

Unless stated otherwise, all events listed below are free of charge and take place at the SciNet Teaching Room or Boardroom at our offices on the eleventh floor of the MaRS West Tower, suite 1140A (661 University Avenue, Toronto ON M5G 1M1).

Most events are recorded and some are broadcast, but only some of the courses can be taken remotely for SciNet certificate credits.

Registration for SciNet courses should be done by logging into https://courses.scinet.utoronto.ca with your Compute Canada account (the same one that you use to log into Niagara).

  • INTRO TO SCINET AND NIAGARA Wednesday February 13, 10:00 am – 11:30 am SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    This is a class of approximately 60-90 minutes to introduce SciNet and the new supercomputer Niagara and teach you how to use Niagara.

    Participation counts towards the SciNet HPC Certificate.

    For more information and registration, go to https://courses.scinet.utoronto.ca/419

  • SCINET USER GROUP MEETING Wednesday February 13, 12:00 noon – 1:00 pm SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    Pizza, user discussion, and a tech talk by Mubdi Rahman, Ph.D, on

    “The Grammar and Tools of Data Visualization in the Era of Big Data”

    For sign up and more information, see https://courses.scinet.utoronto.ca/424

  • ADVANCED LINUX SHELL PROGRAMMING Wednesday February 13, 1:00 pm – 4:00 pm SciNet Teaching Room

    Learn how to write bash script, use environment variables, how to control process, and much more. Requires some linux basic command line experience.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/431

  • INTRO TO SCINET AND NIAGARA Wednesday March 13, 10:00 am – 11:30 am SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    This is a class of approximately 60-90 minutes to introduce SciNet and the new supercomputer Niagara and teach you how to use Niagara.

    Participation counts towards the SciNet HPC Certificate.

    For more information and registration, go to https://courses.scinet.utoronto.ca/420

  • SCINET USER GROUP MEETING Wednesday March 13, 12:00 noon – 1:00 pm SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    Pizza, user discussion, and a tech talk TBA.

    For sign up and more information, see https://courses.scinet.utoronto.ca/425

  • INTRODUCTION TO THE LINUX SHELL Wednesday March 13, 1:00 pm -4:00 pm SciNet Teaching Room

    Learn the basics of how to use the unix shell in two hours. Very useful for new users of SciNet that have little or no experience with unix or linux.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/429

  • INTRO TO SCINET AND NIAGARA Wednesday April 10, 10:00 am – 11:30 am SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    This is a class of approximately 60-90 minutes to introduce SciNet and the new supercomputer Niagara and teach you how to use Niagara.

    Participation counts towards the SciNet HPC Certificate.

    For more information and registration, go to https://courses.scinet.utoronto.ca/420

  • SCINET USER GROUP MEETING Wednesday April 10, 12:00 noon – 1:00 pm SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    Pizza, user discussion, and a tech talk by Anjali Silva on

    “Bayesian Model-Based Clustering Approaches for Discrete-Valued Gene Expression Data”

    For sign up and more information, see https://courses.scinet.utoronto.ca/425

  • ADVANCED LINUX COMMAND LINE Wednesday April 10, 1:00 pm – 4:00 pm SciNet Teaching Room

    Increase you Linux (bash) command line productivity. Requires some basic Linux command line experience.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/432

  • INTRODUCTION TO NEURAL NETWORK PROGRAMMING Starting April 23, 6 weeks, Tuesdays and Thursdays 11:00 am – 12:00 noon SciNet Teaching Room

    This six-week class will introduce neural network programming concepts, theory and techniques. The class material will begin at an introductory level, intended for those with no experience with neural networks, eventually covering intermediate-to-advanced concepts. The programming language will be Python 3.7; experience with Python programming will be assumed. The Keras neural network framework will be used for neural network programming; no experience with Keras will be expected.

    Participation counts towards the SciNet Data Science Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/417

  • RELATIONAL DATABASE BASICS Wednesday May 1, 1:00 pm – 4:00 pm SciNet Teaching Room

    Principles and uses of relational databases with practical examples using python and sqlite.

    Participation counts towards the SciNet Data Science Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/433

  • INTRO TO SCINET AND NIAGARA Wednesday May 8, 10:00 am – 11:30 am SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    This is a class of approximately 60-90 minutes to introduce SciNet and the new supercomputer Niagara and teach you how to use Niagara.

    Participation counts towards the SciNet HPC Certificate.

    For more information and registration, go to https://courses.scinet.utoronto.ca/422

  • SCINET USER GROUP MEETING Wednesday May 8, 12:00 noon – 1:00 pm SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    Pizza, user discussion, and a tech talk TBA.

    For sign up and more information, see https://courses.scinet.utoronto.ca/427

  • INTRODUCTION TO THE LINUX SHELL Wednesday May 8, 1:00 pm -4:00 pm SciNet Teaching Room

    Learn the basics of how to use the unix shell in two hours. Very useful for new users of SciNet that have little or no experience with unix or linux.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/430

  • COMPUTE ONTARIO SUMMER SCHOOL Summer 2019

    The Compute Ontario Summer School on Scientific and High Performance Computing is an annual educational event for graduate/undergraduate students, postdocs and researchers to learn and share knowledge and experience in high performance and technical computing on modern HPC platforms.

    As in previous years, the 2019 Summer School is expected to have three installments, “West”, “Central”, and “East”. Each will be a week-long event with multiple parallel streams.

    More details will be announced later in 2019.

  • SCALING TO PETASCALE INSTITUTE August 19-23, 2019

    This will be a free virtual advanced HPC summer school, organized by a number of the US XSEDE sites, also hosted at SciNet. Topics will likely include OpenMP, MPI, CUDA, and OpenACC.

    More details will be announced later in 2019.

SciNet News February 2019

February 4, 2019 in for_researchers, for_users, newsletter

Summary

  • Two additional ways to interact with the nearline/archive HPSS system: Globus and a Virtual File System.
  • Announcement of several techtalks at the SciNet User Group Meetings.

Details can be found below and are also available on the SciNet education website courses.scinet.utoronto.ca and the SciNet wiki docs.scinet.utoronto.ca.

System News

Events Coming Up

Unless stated otherwise, all events listed below are free of charge and take place at the SciNet Teaching Room or Boardroom at our offices on the eleventh floor of the MaRS West Tower, suite 1140A (661 University Avenue, Toronto ON M5G 1M1).

Most events are recorded and some are broadcast, but only some of the courses can be taken remotely for SciNet certificate credits.

Registration for SciNet courses should be done by logging into https://courses.scinet.utoronto.ca with your Compute Canada account (the same one that you use to log into Niagara).

  • INTRO TO SCINET AND NIAGARA Wednesday February 13, 10:00 am – 11:30 am SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    This is a class of approximately 60-90 minutes to introduce SciNet and the new supercomputer Niagara and teach you how to use Niagara.

    Participation counts towards the SciNet HPC Certificate.

    For more information and registration, go to https://courses.scinet.utoronto.ca/419

  • SCINET USER GROUP MEETING Wednesday February 13, 12:00 noon – 1:00 pm SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    Pizza, user discussion, and a tech talk by Mubdi Rahman, Ph.D, on

    “The Grammar and Tools of Data Visualization in the Era of Big Data”

    For sign up and more information, see https://courses.scinet.utoronto.ca/424

  • ADVANCED LINUX SHELL PROGRAMMING Wednesday February 13, 1:00 pm – 4:00 pm SciNet Teaching Room

    Learn how to write bash script, use environment variables, how to control process, and much more. Requires some linux basic command line experience.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/431

  • INTRO TO SCINET AND NIAGARA Wednesday March 13, 10:00 am – 11:30 am SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    This is a class of approximately 60-90 minutes to introduce SciNet and the new supercomputer Niagara and teach you how to use Niagara.

    Participation counts towards the SciNet HPC Certificate.

    For more information and registration, go to https://courses.scinet.utoronto.ca/420

  • SCINET USER GROUP MEETING Wednesday March 13, 12:00 noon – 1:00 pm SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    Pizza, user discussion, and a tech talk TBA.

    For sign up and more information, see https://courses.scinet.utoronto.ca/425

  • INTRODUCTION TO THE LINUX SHELL Wednesday March 13, 1:00 pm -4:00 pm SciNet Teaching Room

    Learn the basics of how to use the unix shell in two hours. Very useful for new users of SciNet that have little or no experience with unix or linux.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/429

  • INTRO TO SCINET AND NIAGARA Wednesday April 10, 10:00 am – 11:30 am SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    This is a class of approximately 60-90 minutes to introduce SciNet and the new supercomputer Niagara and teach you how to use Niagara.

    Participation counts towards the SciNet HPC Certificate.

    For more information and registration, go to https://courses.scinet.utoronto.ca/420

  • SCINET USER GROUP MEETING Wednesday April 10, 12:00 noon – 1:00 pm SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    Pizza, user discussion, and a tech talk by Anjali Silva on

    “Bayesian Model-Based Clustering Approaches for Discrete-Valued Gene Expression Data”

    For sign up and more information, see https://courses.scinet.utoronto.ca/425

  • ADVANCED LINUX COMMAND LINE Wednesday April 10, 1:00 pm – 4:00 pm SciNet Teaching Room

    Increase you Linux (bash) command line productivity. Requires some basic Linux command line experience.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/432

  • INTRODUCTION TO NEURAL NETWORK PROGRAMMING Starting April 23, 6 weeks, Tuesdays and Thursdays 11:00 am – 12:00 noon SciNet Teaching Room

    This six-week class will introduce neural network programming concepts, theory and techniques. The class material will begin at an introductory level, intended for those with no experience with neural networks, eventually covering intermediate-to-advanced concepts. The programming language will be Python 3.7; experience with Python programming will be assumed. The Keras neural network framework will be used for neural network programming; no experience with Keras will be expected.

    Participation counts towards the SciNet Data Science Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/417

  • RELATIONAL DATABASE BASICS Wednesday May 1, 1:00 pm – 4:00 pm SciNet Teaching Room

    Principles and uses of relational databases with practical examples using python and sqlite.

    Participation counts towards the SciNet Data Science Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/433

  • INTRO TO SCINET AND NIAGARA Wednesday May 8, 10:00 am – 11:30 am SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    This is a class of approximately 60-90 minutes to introduce SciNet and the new supercomputer Niagara and teach you how to use Niagara.

    Participation counts towards the SciNet HPC Certificate.

    For more information and registration, go to https://courses.scinet.utoronto.ca/422

  • SCINET USER GROUP MEETING Wednesday May 8, 12:00 noon – 1:00 pm SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    Pizza, user discussion, and a tech talk TBA.

    For sign up and more information, see https://courses.scinet.utoronto.ca/427

  • INTRODUCTION TO THE LINUX SHELL Wednesday May 8, 1:00 pm -4:00 pm SciNet Teaching Room

    Learn the basics of how to use the unix shell in two hours. Very useful for new users of SciNet that have little or no experience with unix or linux.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/430

  • COMPUTE ONTARIO SUMMER SCHOOL Summer 2019

    The Compute Ontario Summer School on Scientific and High Performance Computing is an annual educational event for graduate/undergraduate students, postdocs and researchers to learn and share knowledge and experience in high performance and technical computing on modern HPC platforms.

    As in previous years, the 2019 Summer School is expected to have three installments, “West”, “Central”, and “East”. Each will be a week-long event with multiple parallel streams.

    More details will be announced later in 2019.

  • SCALING TO PETASCALE INSTITUTE August 19-23, 2019

    This will be a free virtual advanced HPC summer school, organized by a number of the US XSEDE sites, also hosted at SciNet. Topics will likely include OpenMP, MPI, CUDA, and OpenACC.

    More details will be announced later in 2019.

Computational Science Education Publications by SciNet

January 16, 2019 in blog, blog-general, for_educators, for_press, for_researchers, frontpage, news, Uncategorized

This year starts with the publication of three papers by SciNet analysts in the Journal of Computational Science Education that are a reflection of SciNet’s ten years of training and educating academic researchers in the practical and scalable use of high performance computing.

  1. Bridging the Educational Gap between Emerging and Established Scientific Computing Disciplines
    Marcelo Ponce, Erik Spence, Ramses van Zon, and Daniel Gruner, Journal of Computational Science Education Vol 10 (1) 4-11 (2019)
    https://doi.org/10.22369/issn.2153-4136/10/1/1

    In this paper, we describe our experience in developing curriculum courses aimed at graduate students in emerging computational fields, including biology and medical science. We focus primarily on computational data analysis and statistical analysis, while at the same time teaching students best practices in coding and software development. Our approach combines a theoretical background and practical applications of concepts. The outcomes and feedback we have obtained so far have revealed several issues: students in these particular areas lack instruction like this although they would tremendously benefit from it; we have detected several weaknesses in the formation of students, in particular in the statistical foundations but also in analytical thinking skills. We present here the tools, techniques and methodology we employ while teaching and developing this type of courses. We also show several outcomes from this initiative, including potential pathways for fruitful multi- disciplinary collaboration.

  2. Scientific Computing, High-Performance Computing and Data Science in Higher Education
    Marcelo Ponce, Erik Spence, Ramses van Zon, and Daniel Gruner, Journal of Computational Science Education Vol 10 (1), 24-31 (2019)
    https://doi.org/10.22369/issn.2153-4136/10/1/5

    We present an overview of current academic curricula for Scientific Computing, High-Performance Computing and Data Science. After a survey of current academic and non-academic programs across the globe, we focus on Canadian programs and specifically on the education program of the SciNet HPC Consortium, using its detailed enrollment and course statistics for the past six to seven years. Not only do these data display a steady and rapid increase in the demand for research-computing instruction, they also show a clear shift from traditional (high performance) computing to data- oriented methods. It is argued that this growing demand warrants specialized research computing degrees.

  3. Trends in Scientific Computing Training Delivered by a High-Performance Computing Center
    Ramses van Zon, Marcelo Ponce, Erik Spence, and Daniel Gruner, Journal of Computational Science Education Vol 10 (1), 53-60 (2019)
    https://doi.org/10.22369/issn.2153-4136/10/1/9

    We analyze the changes in the training and educational efforts of the SciNet HPC Consortium, a Canadian academic High Performance Computing center, in the areas of Scientific Computing and High-Performance Computing, over the last six years. Initially, SciNet offered isolated training events on how to use HPC systems and write parallel code, but the training program now consists of a broad range of workshops and courses that users can take toward certificates in scientific computing, data science, or high-performance computing. Using data on enrollment, attendence, and certificate numbers from SciNet’s education website, used by almost 1800 users so far, we extract trends on the growth, demand, and breadth of SciNet’s training program. Among the results are a steady overall growth, a sharp and steady increase in the demand for data science training, and a wider participation of ‘non-traditional’ computing disciplines, which has motivated an increasingly broad spectrum of training offerings. Of interest is also that many of the training initiatives have evolved into courses that can be taken as part of the graduate curriculum at the University of Toronto.

We also recently gave a webinar on these topics.

Other publications by SciNet people can be found at here.

SciNet News January 2019

January 4, 2019 in for_researchers, for_users, newsletter

Summary

  • Full datacentre maintenance shutdown on January 15 and 16, 2019.
  • Winter training and education schedule open for registration.
  • Application for International HPC Summer School in Japan in July are still open until February 4th 2019.

Details can be found below and are also available on the SciNet education website courses.scinet.utoronto.ca and the SciNet wiki docs.scinet.utoronto.ca.

System News: Maintenance Shutdown

The SciNet datacentre will undergo a two-day maintenance shutdown on January 15th and 16th 2019, starting at 7 am EST on the 15th. There will be no access to any of the SciNet systems (Niagara, P7, P8, BGQ, SGC, HPSS, Teach cluster, or the filesystems) during this time.

This is in preparation for the upcoming installation of an emergency power generator and a larger UPS, which will result in increased resilience to power glitches and outages.

It is expected that the system will be available to users late on Wednesday January 16th, 2019

The status of the Niagara cluster can be checked on status.computecanada.ca. For up-to-date and more detailed information on the status of all the SciNet systems, check https://docs.scinet.utoronto.ca.

Events Coming Up

Unless stated otherwise, all events listed below take place at the SciNet Teaching Room or Boardroom at our offices on the eleventh floor of the MaRS West Tower, suite 1140A (661 University Avenue, Toronto ON M5G 1M1).

Most events are recorded and some are broadcast, but only some of the courses can be taken remotely for SciNet certificate credits.

Registration for SciNet courses should be done by logging into https://courses.scinet.utoronto.ca with your Compute Canada account (the same one that you use to log into Niagara).

  • INTRO TO SCINET AND NIAGARA Wednesday Jan 16, 2019, 10:00 am – 11:30 am SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    This is a class of approximately 60-90 minutes to introduce SciNet and the new supercomputer Niagara and teach you how to use Niagara.

    Participation counts towards the SciNet HPC Certificate.

    For more information and (free) registration, go to https://courses.scinet.utoronto.ca/418

    Further sessions of this Intro are planned for February 13, March 13, April 10 and May 8, 2019.

  • SCINET USER GROUP MEETING Wednesday Jan 16, 2019, 12:00 noon – 1:00 pm SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    Pizza, user discussion, and a tech talk TDB

    For sign up and more information, see https://courses.scinet.utoronto.ca/423

    Further SNUG sessions are planned for: February 13, March 13, April 10, and May 8, 2019.

  • INTRODUCTION TO THE LINUX SHELL Wednesday January 16, 10:00 am – 12:00 noon SciNet Teaching Room

    Learn the basics of how to use the unix shell in two hours. Very useful for new users of SciNet that have little or no experience with unix or linux.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/428

    Further “Intro to the Linux shell” sessions are planned for: March 13 and May 8, 2019.

  • APPLICATION DEADLINE FOR INTERNATIONAL HPC SUMMER SCHOOL Applications due: February 4, 2019 Event dates: July 7-12, 2019 Location: Kobe, Japan

    This summer school will familiarize the best students in computational sciences with major state-of-the-art aspects of HPC for a variety of scientific disciplines, catalyze the formation of networks, provide advanced mentoring, facilitate international exchange and open up further career options. Leading Canadian, European, Japanese and American computational scientists and HPC technologists will offer instruction in parallel sessions on a variety of topics as: HPC challenges in major scientific disciplines, HPC programming proficiencies, Performance analysis and profiling, Software engineering, Numerical libraries, Big data analysis and analytics, Machine learning, Scientific visualization, and Canadian, European, Japanese and US HPC infrastructure.

    SciNet invites students and early-career post-docs in science, engineering, and other fields at Canadian post-secondary institutions to apply for one of the eight spots allocated to Canada. Travel, lodging and meal expenses of the selected candidates will be covered.

    For more information and application see https://ss19.ihpcss.org.

  • ADVANCED LINUX SHELL PROGRAMMING Wednesday February 13, 1:00 pm – 4:00 pm SciNet Teaching Room

    Learn how to write bash script, use environment variables, how to control process, and much more. Requires some linux basic command line experience.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/431

    Another “Advanced Shell Programming” session is planned for April 10, 2019.

  • SCIENTIFIC COMPUTING FOR PHYSICISTS (PHY1610) Winter 2019, starting January 8 SciNet Teaching Room

    This course is aimed at reducing your struggle in getting started with computational projects, and make you a more efficient computational scientist. Topics include well-established best practices for developing software as it applies to scientific computations, common numerical techniques and packages, and aspects of high performance computing. While we will introduce the C++ language, in one language or another, students should already have some programming experience. Despite the title, this course is suitable for many physical scientists (chemists, astronomers, …).

    This course is part of the physics graduate program. Students willing to take the course as part of their graduate program have to enroll through Acorn/ROSI.

    For more information, see https://courses.scinet.utoronto.ca/398

  • QUANTITATIVE APPLICATIONS FOR DATA ANALYSIS Winter 2019, starting January 9 University of Toronto Scarborough Campus

    In this course data analysis techniques utilizing Python and R statistical language, will be discussed and introduced, as well as, the basics of programming and scientific computing. The goal of this course is to prepare graduate students to perform scientific data analysis. Successful students will learn how to use statistical inference tools to gain insight into large and small data sets, as well as be exposed to cutting-edge techniques and best practises to store, manage and analyze (large) data.

    Topics include: Python and R programming, version control, automation, modular programming and scientific visualization.

    Students willing to take the course as part of their graduate program have to enroll through Acorn/ROSI. This course is part of the EES graduate program and to be taught at the UTSc campus.

    For more information, see https://courses.scinet.utoronto.ca/403

  • INTRODUCTION TO NEURAL NETWORK PROGRAMMING Starting April 23, 2019, 6 weeks, Tuesdays and Thursdays 11:00 am – 12:00 noon SciNet Teaching Room

    This six-week class will introduce neural network programming concepts, theory and techniques. The class material will begin at an introductory level, intended for those with no experience with neural networks, eventually covering intermediate-to-advanced concepts. The programming language will be Python 3.7; experience with Python programming will be assumed. The Keras neural network framework will be used for neural network programming; no experience with Keras will be expected.

    Participation counts towards the SciNet Data Science Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/417

  • RELATIONAL DATABASE BASICS Wednesday May 1, 1:00 pm – 4:00 pm SciNet Teaching Room

    Principles and uses of relational databases with practical examples using python and sqlite.

    Participation counts towards the SciNet Data Science Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/433

  • COMPUTE ONTARIO SUMMER SCHOOL Summer 2019

    The Compute Ontario Summer School on Scientific and High Performance Computing is an annual educational event for graduate/undergraduate students, postdocs and researchers to learn and share knowledge and experience in high performance and technical computing on modern HPC platforms.

    As in previous years, the 2019 Summer School is expected to have three installments, “West”, “Central”, and “East”. Each will be a week-long event with multiple parallel streams.

    More details will be announced later in 2019.

  • SCALING TO PETASCALE INSTITUTE August 19-23, 2019

    This will be a free virtual advanced HPC summer school, organized by a number of the US XSEDE sites, also hosted at SciNet. Topics will likely include OpenMP, MPI, CUDA, and OpenACC.

    More details will be announced later in 2019.

2019 International Summer School on HPC Challenges in Computational Sciences, Japan, July 7-12

December 7, 2018 in blog, for_educators, for_press, for_researchers, for_users, frontpage, news

Graduate students and postdoctoral scholars from institutions in Canada, Europe, Japan and the United States are invited to apply for the tenth International Summer School on HPC Challenges in Computational Sciences, to be held from July 7 to 12, 2019, at the campus of the RIKEN Center for Computational Science (R-CCS) in Kobe, Japan.

Applications are due Feb 4, 2019. The summer school is organized by the Extreme Science and Engineering Discovery Environment (XSEDE), the Partnership for Advanced Computing in Europe (PRACE), the R-CCS, and the SciNet HPC Consortium.

Leading computational scientists and HPC technologists from the U.S., Japan, Europe and Canada will offer instructions on a variety of topics as well as dedicated mentoring. Topics include:

  • HPC challenges by discipline
  • HPC programming proficiency
  • Performance analysis & profiling
  • Scientific visualization
  • Big Data Analytics
  • Machine Learning
  • Canadian, EU, Japanese and U.S. HPC-infrastructures

The expense-paid program will benefit scholars from Canadian, European, Japanese and U.S. institutions who use advanced computing in their research. The ideal candidate will have many of the following qualities, however this list is not meant to be a “checklist” for applicants to meet all criteria:

  • Familiar with HPC, not necessarily an HPC expert, but rather a scholar who could benefit from including advanced computing tools and methods into their existing computational work
  • A graduate student with a strong research plan or a postdoctoral fellow in the early stages of their research careers
  • Regular practice with parallel programming (i.e., student utilizes parallel programming generally on a monthly basis or more)
  • May have a science or engineering background, however, applicants from other disciplines are welcome, provided their research activities include computational work.

Students from underrepresented groups in computing are highly encouraged to apply (i.e., women, racial/ethnic minorities, persons with disabilities, etc.). If you have any questions regarding your eligibility or how this program may benefit you or your research group, please do not hesitate to contact the individual associated with your region below.

Interested students should apply by February 4, 2019. School fees, meals and housing will be covered for all accepted applicants, as well as intercontinental flight costs.
Further information and application: http://ss19.ihpcss.org.

Contacts:

SciNet HPC Consortium
Ramses van Zon
SciNet, Univ. of Toronto, Canada
Email: rzon at scinet.utoronto.ca

PRACE:
Hermann Lederer
Max Planck Computing and Data Facility, Germany
Email: lederer at mpcdf.mpg.de

Simon Wong
ICHEC, Ireland
Email: simon.wong at ichec.ie

RIKEN:
Toshiyuki Imamura
CCS, RIKEN
Email: Imamura.toshiyuki at riken.jp

XSEDE:
Jay Alameda
NCSA, University of Illinois at Urbana-Champaign, United States
Email: alameda at illinois.edu

SciNet Receives HPCwire Award

December 6, 2018 in blog, for_press, for_researchers, for_users, frontpage, in_the_news, news, Road_to_Niagara, success_story

We are very proud that SciNet has received the 2018 HPCwire Editor’s Award for Best Use of HPC in Physical Sciences. The award was announced at the 2018 International Conference for High Performance Computing, Networking, Storage and Analysis (SC18), in Dallas, Texas.

SciNet used Lenovo and Mellanox technologies on the new Niagara cluster to create spatial resolution models of the Pacific Ocean, helping to validate ocean waves movement and to assist in global warming calculations. These calculations were performed by a team of scientists involving University of Toronto’s Prof. W. Richard Peltier, University of Michigan oceanographer Prof. Brian Arbic, and NASA JPL’s Dr. Dimitris Menemenlis. More on this calculation can be found here.

This calculation was part of the “early science” program of the Niagara supercomputer at the SciNet HPC Consortion. In this short period in March of 2018, a number of scientists were given the opportunity to perform “heroic” calculations. These large scale calculation were essential to test, to tune and to get Niagara ready for use as a Canada’s fastest national academic supercomputer.

SciNet News December 2018

December 4, 2018 in for_researchers, for_users, newsletter

Summary

  • The full datacentre shutdown that was originally scheduled for December 18, 2018, is postponed to January 15 and 16, 2019, and combined with annual maintenance.
  • While the University of Toronto will be closed from Dec 22 to Jan 6. SciNet support will be on a “best effort” basis.
  • Time limits of jobs on Niagara are now 24 hours for all users.
  • Winter training and education schedule announced.
  • Application for International HPC Summer School in Japan in July is open.

Details can be found below and are also available on the SciNet education website courses.scinet.utoronto.ca and the SciNet wiki docs.scinet.utoronto.ca.

System News

  • The SciNet datacentre will undergo a two-day maintenance shutdown on January 15th and 16th 2019, starting at 7 am EST on the 15th. There will be no access to any of the SciNet systems (Niagara, P7, P8, BGQ, SGC, HPSS, Teach cluster, or the filesystems) during this time.

    This is in preparation for the upcoming installation of an emergency power generator and a larger UPS, which will result in increased resilience to power glitches and outages.

    The status of the Niagara cluster can be checked on status.computecanada.ca. For up-to-date and more detailed information on the status of all the SciNet systems, you can always check https://docs.scinet.utoronto.ca.

  • The walltime limit of jobs on Niagara is now 24 hours for all users.
  • While the University of Toronto is closed from December 22, 2018 to January 6, 2019, the SciNet systems will remain available. However, support will be on a “best effort” basis during the break.

Events Coming Up

Unless stated otherwise, all events listed below take place at the SciNet Teaching Room at our offices on the eleventh floor of the MaRS West Tower, suite 1140A (661 University Avenue, Toronto ON M5G 1M1).

Most events will be recorded and some are broadcast, but only some of the courses can be taken remotely for SciNet certificate credits, as indicated below.

Registration for SciNet courses is done by logging into https://support.scinet.utoronto.ca/education with your Compute Canada account (the same one that you use to log into Niagara).

  • INTRO TO SCINET AND NIAGARA Wednesday Jan 16, 2019, 10:00 am – 11:30 am SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    This is a class of approximately 60-90 minutes to introduce SciNet and the new supercomputer Niagara and teach you how to use Niagara.

    Participation counts towards the SciNet HPC Certificate.

    For more information and (free) registration, go to https://courses.scinet.utoronto.ca/418

    Further sessions of this Intro are planned for February 13, March 13, April 10 and May 8, 2019.

  • SCINET USER GROUP MEETING Wednesday Jan 16, 2019, 12:00 noon – 1:00 pm SciNet Boardroom (suite 1140, 661 University Avenue, Toronto ON M5G 1M1).

    Pizza, user discussion, and a tech talk TDB

    For sign up and more information, see https://courses.scinet.utoronto.ca/423

    Further SNUG sessions are planned for: February 13, March 13, April 10, and May 8, 2019.

  • INTRODUCTION TO THE LINUX SHELL Wednesday January 16, 10:00 am – 12:00 noon SciNet Teaching Room

    Learn the basics of how to use the unix shell in two hours. Very useful for new users of SciNet that have little or no experience with unix or linux.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/428

    Further “Intro to the Linux shell” sessions are planned for: March 13 and May 8, 2019.

  • ADVANCED LINUX SHELL PROGRAMMING Wednesday February 13, 1:00 pm – 4:00 pm SciNet Teaching Room

    Learn how to write bash script, use environment variables, how to control process, and much more. Requires some linux basic command line experience.

    Participation counts towards the SciNet Scientific Computing Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/431

    Another “Advanced Shell Programming” session is planned for April 10, 2019.

  • SCIENTIFIC COMPUTING FOR PHYSICISTS (PHY1610) Winter 2019, starting January 8 SciNet Teaching Room

    This course is aimed at reducing your struggle in getting started with computational projects, and make you a more efficient computational scientist. Topics include well-established best practices for developing software as it applies to scientific computations, common numerical techniques and packages, and aspects of high performance computing. While we will introduce the C++ language, in one language or another, students should already have some programming experience. Despite the title, this course is suitable for many physical scientists (chemists, astronomers, …).

    This course is part of the physics graduate program. Students willing to take the course as part of their graduate program have to enroll through Acorn/ROSI.

    For more information, see https://courses.scinet.utoronto.ca/398

  • QUANTITATIVE APPLICATIONS FOR DATA ANALYSIS Winter 2019, starting January 9 University of Toronto Scarborough Campus

    In this course data analysis techniques utilizing Python and R statistical language, will be discussed and introduced, as well as, the basics of programming and scientific computing. The goal of this course is to prepare graduate students to perform scientific data analysis. Successful students will learn how to use statistical inference tools to gain insight into large and small data sets, as well as be exposed to cutting-edge techniques and best practises to store, manage and analyze (large) data.

    Topics include: Python and R programming, version control, automation, modular programming and scientific visualization.

    Students willing to take the course as part of their graduate program have to enroll through Acorn/ROSI. This course is part of the EES graduate program and to be taught at the UTSc campus.

    For more information, see https://courses.scinet.utoronto.ca/403

  • INTRODUCTION TO NEURAL NETWORK PROGRAMMING Starting April 23, 2019, 6 weeks, Tuesdays and Thursdays 11:00 am – 12:00 noon SciNet Teaching Room

    This six-week class will introduce neural network programming concepts, theory and techniques. The class material will begin at an introductory level, intended for those with no experience with neural networks, eventually covering intermediate-to-advanced concepts. The programming language will be Python 3.7; experience with Python programming will be assumed. The Keras neural network framework will be used for neural network programming; no experience with Keras will be expected.

    Participation counts towards the SciNet Data Science Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/417

  • RELATIONAL DATABASE BASICS Wednesday May 1, 1:00 pm – 4:00 pm SciNet Teaching Room

    Principles and uses of relational databases with practical examples using python and sqlite.

    Participation counts towards the SciNet Data Science Certificate.

    For sign up and more information, see https://courses.scinet.utoronto.ca/433

  • INTERNATIONAL HPC SUMMER SCHOOL July 7-12, 2019 Kobe, Japan

    This summer school will familiarize the best students in computational sciences with major state-of-the-art aspects of HPC for a variety of scientific disciplines, catalyze the formation of networks, provide advanced mentoring, facilitate international exchange and open up further career options. Leading Canadian, European, Japanese and American computational scientists and HPC technologists will offer instruction in parallel sessions on a variety of topics as: HPC challenges in major scientific disciplines, HPC programming proficiencies, Performance analysis and profiling, Software engineering, Numerical libraries, Big data analysis and analytics, Machine learning, Scientific visualization, and Canadian, European, Japanese and US HPC infrastructure.

    SciNet invites students and early-career post-docs in science, engineering, and other fields at Canadian post-secondary institutions to apply for one of the eight spots allocated to Canada. Travel, lodging and meal expenses of the selected candidates will be covered.

    Applications are due February 4, 2019.

    For more information and application see https://ss19.ihpcss.org.

  • COMPUTE ONTARIO SUMMER SCHOOL Summer 2019

    The Compute Ontario Summer School on Scientific and High Performance Computing is an annual educational event for graduate/undergraduate students, postdocs and researchers to learn and share knowledge and experience in high performance and technical computing on modern HPC platforms.

    As in previous years, the 2019 Summer School is expected to have three installments, “West”, “Central”, and “East”. Each will be a week-long event with multiple parallel streams.

    More details will be announced later in 2019.

  • SCALING TO PETASCALE INSTITUTE Summer 2019

    This will be a free virtual advanced HPC summer school, organized by a number of the US XSEDE sites, also hosted at SciNet. Topics will likely include OpenMP, MPI, CUDA, and OpenACC.

    More details will be announced later in 2019.