The cluster currently employs allows access to: -> High-Availability Cores: 72 ( Auxiliary Cores) -> 512 GB's Available RAM (+128 GB Aux RAM) -> 80.0 TB Available Pool Storage
Introduction to the Cluster
This page is a dedicated portal into the current state of the cluster, as well as the current statistics in size and speed, as well as uptime. Forthcoming graphs will represent a minute by minute window into the cluster but generalized stats are typically more useful since machines come up and down frequently due to the current development environment and this can impact apparent statistics.
The cluster is built as the skeletal framework to run all of the software developed for the final senior thesis project. This project seeks to encode and process raw 4K cinema files in realtime via HPC and Parallel Computing.
The cluster is built using all free and open source tools. CentOS is used as the OS, Ansible for provisioning, Zabbix for monitoring and SLURM for cluster operations. Current usable dependencies are PTS, dc.RAW, cimg and other packages as well.
Computationally, the cluster is comprised of a variety of enterprise grade servers from manufacturers like Dell, HP, Intel Development and SuperMicro.
Current work is relevant to faster copying and streaming of files from the headnode to each of the hosts. Current stages posted here. Current investigation is parallel compression, streaming and decompression.
Current investigated software includes NetCat (NC) as well as BBCP, Pigz and PopeViewer (PV). This allows node to node operations to operate faster since the information exchanged between them will take less time to copy.
# SOURCE: > tar -cf - /u02/databases/mydb/data_file-1.dbf | pigz | nc -l 8888 # TARGET: > nc
"Pigz is essentially gzip, but can use multiple parallel streams to compress/decompress the data. If we replace gzip with pigz, we can achieve fantastical speeds and cut our transfer time again by the factor of ~ 2-10, comparing to scp."
Project and code by @jordanwesthoff. Click on the profile for more information. RIT 2015. Senior thesis content.
This webpage is generated automatically from the GitHub Pages repo. The content is rsync'd to the host securely and updates are automatic. This page is only a satellite.
"You are not like Cerberus, three gentlemen at once, are you?" - Richard Brinsley Sheridan: Act IV, sc. ii.