Cori is a Cray XC40 with a peak performance of about 30 petaflops. The system is named in honor of American biochemist Gerty Cori, the first American woman to win a Nobel Prize and the first woman to be awarded the prize in Physiology or Medicine. Cori is comprised of 2,388 Intel Xeon "Haswell" processor nodes, 9,688 Intel Xeon Phi "Knight's Landing" (KNL) nodes. The system also has a large Lustre scratch file system and a first-of-its kind NVRAM "burst buffer" storage device.
|System Partition||# of cabinets||# of nodes||Aggregate Theoretical Peak||Aggregate Memory|
|Haswell||14||2,388||2.81 PFlops||298.5 TB|
|KNL||54||9,688||29.5 PFlops||1.09 PB|
|Large Memory||-||20||30.7 TFlops||40 TB|
|System Partition||Processor||Clock Rate||Physical Cores Per Node||Threads/Core||Sockets Per Node||Memory Per Node|
|Login||Intel Xeon Processor E5-2698 v3||2.3 GHz||32||2||2||515 GB|
|Haswell||Intel Xeon Processor E5-2698 v3||2.3 GHz||32||2||2||128 GB|
|KNL||Intel Xeon Phi Processor 7250||1.4 GHz||68||4||1||96 GB (DDR4), 16 GB (MCDRAM)|
|Large Memory||AMD EPYC 7302||3.0 GHz||32||2||2||2 TB|
Each XC40 cabinet housing Haswell and KNL nodes has 3 chassis; each chassis has 16 compute blades with 4 nodes per blade. Login nodes and the Large Memory Nodes are in separate cabinets.
- Cori has 12 Login nodes (
cori[01-12]) open to public.
- 2 Large Memory Login nodes (
cori[22,23]) to submit to
bigmemqos. These nodes have 750GB of memory.
- 4 Jupyter nodes (
cori[13,14,16,19]]) access via Jupyter
- 2 Workflow nodes (
cori[20,21]) - requires approval before access to node
- Each node has two sockets, each socket is populated with a 2.3 GHz 16-core Haswell processor.
Users cannot occupy entire login node because we have enforced Control Groups (cgroups)
For more details on connecting to login nodes click here.
Haswell Compute Nodes¶
- Each node has two sockets, each socket is populated with a 2.3 GHz 16-core Haswell processor. Intel Xeon Processor E5-2698 v3.
- Each core supports 2 hyper-threads, and has two 256-bit-wide vector units
- 36.8 Gflops/core (theoretical peak)
- 1.2 TFlops/node (theoretical peak)
- 2.81 PFlops total (theoretical peak)
- Each node has 128 GB DDR4 2133 MHz memory (four 16 GB DIMMs per socket)
- 298.5 TB total aggregate memory
KNL Compute Nodes¶
- Each node is a single-socket Intel Xeon Phi Processor 7250 ("Knights Landing") processor with 68 cores per node @ 1.4 GHz
- Each core has two 512-bit-wide vector processing units
- Each core has 4 hardware threads (272 threads total)
- AVX-512 vector pipelines with a hardware vector length of 512 bits (eight double-precision elements).
- 44.8 GFlops/core (theoretical peak)
- 3 TFlops/node (theoretical peak)
- 29.5 PFlops total (theoretical peak)
- Each node has 96 GB DDR4 2400 MHz memory, six 16 GB DIMMs (102 GiB/s peak bandwidth)
- Total aggregate memory (combined with MCDRAM) is 1.09 PB.
- Each node has 16 GB MCDRAM (multi-channel DRAM), > 460 GB/s peak bandwidth
- Each core has its own L1 caches, with 64 KB (32 KiB instruction cache, 32 KB data)
- Each tile (2 cores) shares a 1MB L2 cache
- Processor cores connected in a 2D mesh network with 2 cores per tile, with a 1 MB cache-coherent L2 cache shared between 2 cores in a tile, with two vector processing units per core.
Large Memory Nodes (
For node specification details, please see here.
- Cori has 6 rows of cabinets.
- Row 0 has 12 Haswell cabinets.
- Rows 1 to 4 has 12 KNL cabinets each.
- Row 5 has 6 KNL cabinets and 2 Haswell cabinets.
- More detailed info in Cori Cabinet Layout Diagram.
Cray Aries with Dragonfly topology with >45 TB/s global peak bisection bandwidth. Cori employs the "Dragonfly" topology for the interconnection network.
More details about the interconnect for Dragonfly topology and cabinet coordinates
Compute Nodes Topology¶
Below are some compute nodes topology information for some of the Haswell and KNL node types. You may need to zoom in to view details: