The organization’s server could not be reliably verified, so it was treated as if it were compromised: nothing was copied off it blindly and the system was not cleaned – a new, trustworthy infrastructure was built and the services moved onto it under control. This is how it went.
The old server
A single machine that had accumulated services for years:
- an outdated operating system;
- a MySQL database;
- web applications in PHP – some off-the-shelf, some customized;
- the full e-mail stack.
Its integrity could not be verified, so it was treated as untrusted. The goal was to get the operations onto a clean foundation with minimal downtime – the whole organization depended on it.
The new infrastructure as a KVM host
- One new server as a KVM host on a current Linux distribution, running the services as separate virtual machines.
- Split by purpose: separate virtual machines for the database, the mail server and a web server for the smaller projects; larger projects kept apart. A compromise of one service then does not spread to the others.
- Moving data, not the system: content and configuration were moved deliberately and verifiably, not as a bit copy of the suspect machine.
What caused trouble
- Stuck versions: the old PHP and MySQL, and the applications on top of them, could not be moved unchanged. Newer libraries changed behaviour and the customized applications had to be tuned piece by piece, not just copied.
- Customized glue scripts: the old server was held together by a set of scripts accreted over the years. They had to be untangled and replaced with a cleaner, maintainable solution.
Hardening
- Firewall – only what is needed is open.
- Remote access – separate accounts, keys instead of passwords, least privilege.
- Isolation of the individual virtual machines and separation of their networks.
- A proxy in front of the web services.
- Backups verified by restoring, and basic monitoring of availability and load.
- Later containers instead of virtual machines for the smaller projects – lighter and quicker to deploy.
Result
The operations moved onto infrastructure with verifiable integrity, isolated services and standard hardening. The suspect server was taken out of service, not cleaned – with an assumed compromise, a clean foundation is the only reliable path.