Whilst the main sites had been on Cloudflare for a couple of months, the server IP was easily findable on Google and we had experienced several DDoS attacks.
The server is hosted with Linode and so our plan was to request an extra IP, switch over the DNS and then kill the old one.
Note: I had to reboot the server to get it to recognise the new IP. Thinking about it, restarting
systemd-networkd may also work.
Our first attempt nearly worked. We migrated over to the new IP and it worked - great. We tested some of the domains in a Cloudflare IP resolver and found our new IP! Uh-oh. We quickly found out what was wrong. We had a wildcard DNS configured for one of the sites and on the free Cloudflare plan, this doesn't get routed through their protected network. So any subdomain would point straight to the server - eek. Nice upsell, Cloudflare.
The wildcard domain was actually a bit of a legacy thing, so we removed it. Now we had to do the whole process again to hide the leaked IP. We requested another IP from Linode but they wanted us to try swapping IP addresses with a temporary Nanode. This actually seemed easier for us and it was a straightforward process. Create a new Nanode, swap the IPs, reboot and simultaneously switch the IPs in DNS, and then finally kill the Nanode.
This method worked great. Cloudflare DNS changes can be pretty much instant so there was virtually no downtime.
It's still possible our IP can be leaked, but at least we're now protected against the most common method of finding out.