I was at a client, a healthcare facility, to replace some hard drives. The official line was a familiar one: no budget for significant upgrades. This meant we had to keep the current setup running, a system that was both outdated and, frankly, unreliable by modern standards. We were holding everything together with metaphorical duct tape and sheer willpower, but somehow, it was stable and functional. I had managed to set up a makeshift "cluster" using OpenNebula with GlusterFS for storage (which I later, through hard lessons, replaced with MooseFS and eventually Ceph), squeezing every last drop of performance from the available hardware.

The IT manager, a decent fellow, seemed to genuinely believe the purse strings were just exceptionally tight. However, another company, known to be friends of the general manager, were already providing support on two specific VMs. Their presence felt disproportionate to their limited scope, and there was a subtle, persistent pressure from the GM to involve them more, a context that would only become clear later.

We scheduled an intervention and meticulously notified everyone to disconnect and shut down their machines by 12:30. By 13:00, we had completed the necessary backups, aiming to start the core intervention by 14:00 and have the essential systems back up and running by 15:30. The goal was straightforward: update the underlying systems and check the health of the disks. We shut down all the VMs; everyone disconnected. It was lunchtime.

We updated the servers, rebooted them. And then, one of the disks started throwing errors. GlusterFS, for reasons I never fully pinned down but have strong suspicions about, decided in that critical moment to overwrite both the failing disk and its supposed replica with zeros. I hadn’t changed a single configuration file related to it. From that day forward, GlusterFS, for me, simply ceased to exist as a viable option.

Panic. Pure, cold panic – there were backups, of course, but on a USB 2.0 disk (these were old servers, no USB 3.0 in sight)! The restore time would be agonizingly slow. I immediately stopped everything, feeling like I was about to faint. The IT manager, seeing my face, didn’t immediately grasp what had happened, so I quickly explained the catastrophic data wipe on that specific storage volume. He composed himself and announced to the waiting staff that we would cancel the remainder of the planned intervention, restore from backup, and still aim to have the critical services back up by 15:30 as originally planned, prioritizing the most essential VMs.

Just as we were about to begin the painstaking restore, the "competing" company – the GM’s friends – reached out. They had, against all explicit instructions, powered on one of their VMs and started "doing their own interventions". And now, they were complaining that we had lost some of their data.

This was the moment the general manager and those guys went full "carpe diem". They immediately put me under accusation, loudly proclaiming I had undoubtedly made a critical error that led to this "lost data". It was then the penny truly dropped for the IT manager, and the GM's earlier insistence on involving his 'friends' company took on a sinister new light. Now I can say it: his primary goal hadn't just been about perceived cost-saving; it was to systematically give work to this company, to hand everything over to them, paving the way for them to take over the entire infrastructure, regardless of competence or genuine need.

I wrote a detailed technical report explaining exactly what I had observed, noting various unauthorized SSH logins from the IP addresses associated with those individuals during our clearly demarcated intervention window. Conveniently, the command history on the affected systems had been wiped clean. Of course, not by me.

They continued to harass me for a while. The last, most audacious thing they asked was for me to attend a meeting "to explain in person". They deliberately tried to schedule this meeting for the day before my wedding. And they knew it.

They threatened to demand unspecified (and undoubtedly high) financial compensation for "the lost data". What lost data, you ask? The data they claimed to have entered onto their VMs during our intervention window – the very window they'd been explicitly warned to stay out of and during which they had no business touching the systems.

Final result: no tangible problem for me. I hadn’t done anything wrong. The backups, though slow to restore, were intact and complete. They only calmed down when I proved (with network logs and firewall records in hand) that I wasn’t the only one connected to those machines during the critical period, and my witnesses (two colleagues present with me and the IT manager himself) had seen all my actions and could confirm I hadn’t deviated from standard, careful procedure.

In the end, I realized they would undoubtedly try something like this again. I made the decision to leave the client. Even the IT manager, who had initially tried to believe in the good faith of the leadership and had then witnessed firsthand the technical realities and the subsequent malicious accusations, decided he’d had enough and resigned to change jobs. He understood the financial and political interests at play were far beyond his control.

Unsurprisingly, the general manager managed to install the company he wanted, at astronomical figures. After just two months, true to form, they got their hands fully on the system I had painstakingly kept alive… and broke it. They actually had the gall to ask me for assistance, which I refused. At any price.

After a few years, I found out through the local press that the general manager had ended up in jail for corruption, bribes, and for systematically favoring his friends' companies across many sectors, not just IT. Don't ask me why, but I wasn't surprised at all.

That day, I celebrated. It was a stark reminder that while integrity might not always win the immediate skirmishes in such environments, the rot of corruption often consumes itself in the end. But the cost, for those caught in the interim, can be immense.