The University at Buffalo (UB), part of the State University of New York (SUNY) system, is a major public research university with campuses in Buffalo and Amherst, New York. Its Center for Computational Research (CCR) has served the university’s research community since 1998 as a High-Performance Computing (HPC) and data analytics provider. CCR complements departmental IT, stepping in when researchers need large-scale computing resources beyond what individual departments can provide.
CCR’s primary portfolio is HPC. Its Slurm-managed Linux Beowulf clusters provide over 26,000 CPU cores and more than 3 PFlop/s of peak compute capacity, backed by 4.6PB of Vast Data storage. Researchers across Physics, Chemistry, Engineering, and Bioinformatics, and increasingly, driven by AI adoption, departments like Linguistics, Business Management, and Geology, rely on these clusters for batch-scheduled number crunching and data analytics.
But as demand grew, the limitations of HPC alone became clear. Not all research fits the batch-scheduling model. Researchers began requesting capabilities that HPC couldn’t deliver: self-service access to long-running virtual machines, flexible software environments, and on-demand infrastructure for workloads that need to stay online continuously.
To address this, CCR deployed an on-premises research cloud using OpenStack – branded internally as Lake Effect – to complement its HPC capabilities with the scalability, flexibility, and multi-tenancy of a private cloud model.
Lake Effect serves a fundamentally different purpose than CCR’s HPC clusters. Where HPC handles number crunching through batch job scheduling, Lake Effect supports long-running research infrastructure that needs to stay online continuously.In practice, this means websites fronting research projects, databases providing backend support for calculations running on HPC clusters, and portals where researchers submit data for processing. Real-world examples include:
Any research activity that is longer-term and not number crunching or data analytics runs in the cloud. HPC and OpenStack are completely separate environments serving completely different needs and together they give CCR’s researchers a comprehensive set of tools.
The initial deployment of OpenStack gave researchers what they needed: self-service access to virtual machines, better resource utilization, and support for workloads that didn’t fit neatly into traditional batch-scheduling models. The cloud also allowed the university to maintain the security and compliance posture required in an academic setting, with greater control over data access and governance.
However, managing and maintaining the OpenStack environment in-house quickly became a challenge. As Salvatore Guercio, Systems Administrator at UB’s Center for Computational Research, put it:
“For the first few years we managed and maintained the Research Cloud by ourselves, and it was an incredibly challenging task that required more staff time than we had available.”
The team lacked the depth of OpenStack expertise needed to enable advanced features, and critical operational tasks – particularly upgrades – were not even a consideration. Over time, the environment fell behind on releases and began showing signs of instability: slowdowns, minor but persistent issues, and a growing gap between the version in production and the current upstream releases. CCR had almost a full-time person dedicated solely to maintaining the OpenStack deployment.
Rather than abandon the investment in OpenStack, UB’s CCR sought out professional support from the OpenStack ecosystem. After evaluating providers, the university partnered with VEXXHOST, an OpenInfra Foundation member and active upstream contributor.The first priority was stabilization: bringing the environment up to date by upgrading from the Wallaby release to the Zed release of OpenStack. This was a non-trivial leap across multiple release cycles and required careful planning to minimize disruption to active research workloads.
The upgrade addressed key concerns around high availability and performance, while also unlocking features from several intermediate releases that the team had previously been unable to adopt. The environment was subsequently converted to run on Atmosphere, VEXXHOST’s open-source deployment and lifecycle management tooling. CCR currently runs Atmosphere 7.6.
Lake Effect’s physical deployment consists of 3 controller nodes, 24 compute hosts, and 18 Ceph OSD storage hosts – all Dell PowerEdge servers. Controllers are Dell PowerEdge R440s, each with a 20-core Intel Xeon Gold 6230 processor and 160GB RAM. The compute tier is a mix of configurations: the majority are Dell PowerEdge R650s with dual 56-core Intel Xeon Gold 6330 processors and 1TB RAM. Five Dell PowerEdge R7525 nodes provide GPU capability with AMD EPYC 7452 processors, 512GB RAM, and three NVIDIA A40 GPUs per host. Additional R740 nodes carry dual NVIDIA V100 GPUs.
Storage is backed by 216 Ceph OSDs distributed across three server racks on Dell PowerEdge R740xd hosts, each with 12x 8TB SATA drives and 4x 1.75TB SSDs for database journals. CRUSH maps are configured for full rack-level redundancy. The environment provides 1.7PB of Ceph block storage, backing both Cinder and Glance.
Deployed OpenStack services include Nova, Neutron, Cinder, Glance, Swift, Magnum, Octavia, and Manila.
Every host has bonded network connections, with each host connected to two separate network switches for full physical redundancy. A provider network tied to the campus external network allows provisioning of VMs with public-facing IP addresses, while a separate physical network handles cloud management traffic.
All hosts are connected at 10G with 40G uplinks to core switches. CCR encourages users to leverage private networks wherever possible and limit external-facing VMs to maintain a strong security posture.
A recurring theme in OpenStack operator discussions is the burden of day-2 operations –monitoring, patching, upgrading, and incident response. UB’s experience underscores this challenge.After the conversion to Atmosphere, the operational model changed dramatically. Host OS and firmware upgrades are now handled quarterly, live-migrating VMs during the process so that most users never notice. Atmosphere itself is upgraded every six months unless a security issue requires an earlier update. Thanks to this shift, upgrades are super easy now, often completed in just a few hours.
Monitoring is handled through Prometheus and Grafana, supplemented by custom scripts for overall status and failure checks. All physical servers are enrolled in Dell OpenManage Enterprise for hardware failure monitoring and firmware updates. User access is provided through the Horizon dashboard and OpenStack CLI, with single sign-on integration through the university’s identity management system via the OpenID protocol.
This experience echoes a broader lesson for the OpenInfra community: the sustainability of an OpenStack deployment depends not just on the initial setup, but on having a reliable, repeatable path for ongoing operations.
While CCR has Magnum deployed and available as a service within Lake Effect, Kubernetes adoption among the research community has not yet taken hold, primarily due to the learning curve for users more accustomed to traditional VM-based workflows. The capability is in place and, as the research community’s familiarity with cloud-native tooling grows, Magnum provides a ready path for Kubernetes cluster provisioning and lifecycle management without requiring additional infrastructure investment. This is a pattern increasingly relevant across the OpenInfra community, bridging OpenStack and cloud-native tooling using upstream-aligned projects.
An additional capability unlocked during the modernization was detailed usage reporting. For a university environment where cloud resources are shared across departments, research groups, and grant-funded projects, visibility into who is consuming what is essential. CCR operates Lake Effect on a cost-recovery basis, billing researchers by CPU hours, similar to a public cloud model, but only to recoup hardware costs. For researchers, the savings over bare-metal alternatives are significant: spinning up cloud infrastructure takes a day or two, compared to weeks or months for hardware procurement, and with physical server costs rising 3–4x in the past year alone, the cloud model has become even more compelling.
The most significant outcome of the transition is operational efficiency. Before the partnership, CCR had almost a full-time person dedicated to maintaining OpenStack. Today, Guerciomanages the entire cloud environment himself, spending only a fraction of his time on maintenance. The environment currently runs approximately 110 VMs on average, with some instances operating without major service interruption for years.
Lessons for the OpenInfra Community
With a stable, up-to-date OpenStack environment in place, CCR is positioned to continue expanding Lake Effect’s capabilities. The team plans to integrate its Vast storage platform as an additional Cinder provider when Atmosphere 8.x is released, further bridging the HPC and cloud storage environments.
Superuser Magazine is the Open Infrastructure Foundation's official online publication. It covers open infrastructure ecosystem news, case studies, event recaps, product updates and announcements, project releases, and more.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。