Niagara

March 5, 2018 in Systems

Niagara is a homogeneous cluster of 60,000 cores, owned by the University of Toronto and operated by SciNet, intended to enable large parallel jobs of 1024 cores and more. It is the most powerful supercomputer in Canada available for academic research. Compute allocations are handled through Compute Canada’s annual resource allocation competition. Niagara was designed to optimize throughput of a range of scientific codes running at scale, energy efficiency, and network and storage performance and capacity.

Niagara was officially launched on March 5, 2018.

The user experience on Niagara will be similar to that on the General Purpose clusters Graham (at U. of Waterloo) and Cedar (at Simon Fraser U.), but specific instructions on how to use the Niagara cluster are still in preparation, given that details of the setup are still in flux at present (ca. February 2018).

Specifics of the cluster:

  • Niagara consists of 1500 nodes.
  • Each node has 40 Intel Skylake cores at 2.4GHz, for a total of 60,000 cores.
  • There is 202 GB (188 GiB) of RAM per node.
  • EDR Infiniband network in a so-called ‘Dragonfly+’ topology.
  • 6PB of scratch, 3PB of project space (parallel filesystem: IBM Spectrum Scale, formerly known as GPFS).
  • 256 TB burst buffer (Excelero + IBM Spectrum Scale).
  • No local disks.
  • Rpeak of 4.61 PF.
  • Rmax of 3.0 PF.
  • 685 kW power consumption.