Dense Memory Cluster (DMC)

The DMC at the Alabama Supercomputer Center has 2360 CPU cores and 14 terabytes of distributed memory. Each compute node has a local disk (up to 3.5 terabytes of which are accessible as /tmp). Also attached to the DMC is a high performance GPFS storage cluster, which has 93 terabytes of high performance storage accessible as /scratch from each node. Home directories as well as third party applications use a separate GPFS volume and share 184 terabytes of storage.

The machine is physically configured as a cluster of 20, 24, 36, or 192 CPU core SMP boards.  Thirty-nine nodes have 2.5 GHz Intel 10-core Xeon Ivy Bridge processors and 128 gigabytes of memory.  Twelve nodes have 2.1 GHz 18-core Broadwell processors and 128 GB of memory. One node has 2.1 GHz Skylake-SP processors and 6 TB of memory.  Twenty-four nodes have 2.7 GHz 18-core Skylake-SP processors and 96 GB of memory.  One node has a 2.3 GHz Intel 12-core Haswell processors and 128 gigabytes of memory.  One node has 12-core 2.2 GHz Broadwell processors and 128 gigabytes of memory.  One node has a 1.3 GHz Intel 64-core Knights Landing preproduction processor and 94 gigabytes of memory.   The three login nodes are an 8-core virtual machines emulating Ivy Bridge, but running on Haswell hardware.

The DMC has 18 NVIDIA GPU (Graphic Processing Unit) chips. These are a combination of: three DMC nodes configured with four Tesla K20m cards each, and one nodes with two Tesla P100 cards with 16 GB of memory, and one node with four Volta V100 cards with 32 GB of memory. These multicore GPU chips are similar to those in video cards, but are installed as math coprocessors. This can give significant performance advantages for software that has been adapted to use these processors.

The SpecFP benchmark (a per core measurement) and the SpecFP Rate benchmark (a per node measurement) correlate well with most types of simulations that use floating point mathematics.  The sum of the SpecFP Rate 2006 benchmark numbers for the DMC nodes is approximately 82,000.  Note that the SpecFP benchmark will not run on GPU math coprocessors, so those are exclude from this value.  The experimental Knights landing processor is included in this value.

Thus the processing capacity of the DMC cluster is: 

Expected conventional processing capacity - 82,000 SpecFP Rate 2006 aggregate (does not include GPUs)
Conventional processing theoretical peak capacity - 155 TFLOPs
Single precision theoretical peak GPU capacity - 119 TFLOPs 
Double precision theoretical peak GPU capacity - 53 TFLOPs

Total DMC capacity - 274 TFLOPs

The DMC has a 10 gbps connection to the internet via a firewall.  Within the cluster, message passing and access to shared file system servers send traffic over an FDR InfiniBand network.

Home directory storage, applications, and shared data is on a parallel, shared, file system (currently BeeGFS).  The home file system currently has 750 TB of usable disk space.  A high performance, shared, scratch file system has a 92 TB capacity (currently Spectrum Scale).