I did something very bad today. I’m ashamed but I’ll tell you because it was interesting.
I had a SSL certificate to renew. I just got the new my.server.crt, and I just had to replace the old one with this new one. Nothing complicated, but the files on the server were named badly, so I started putting everything back in order :
- my.server.csr for the Certificate Signing Request
- my.server.crt for the certificate itself
- my.server.key for the private key
During this basic operation of renaming files, I failed. My brain wasn’t there anymore for a moment, and I renamed 2 files to the same name.
$ mv badly.named.key my.server.key
$ mv badly.named.crt my.server.key
One second later I realized I had lost my private key.
Of course, no backup. I know, that’s veeery bad. I’m ashamed I already told you.
And, to make the story even more dramatic, that was for a client already very angry with me, not that I had done something wrong but the situation was bad and that’s the kind of client who are never satisfied with anything anyway.
So, the choices were : to make them even more angry, or try to recover that stupid private key.
How could I recover the file? Quickly thought of extundelete, but skeptically, as I never could recover anything useful that way.
I tried nevertheless, but nothing came out. It just told me that it hadn’t found my file. Then I tried foremost, but it didn’t seem to be able to recover text files. So I searched the web for an idea, found this post (awesome blog BTW, always interesting and useful), and tried :
$ grep -i -a -B30 -A50 ‘BEGIN RSA PRIVATE KEY’ /dev/sda2 > results.txt
It took several minutes (maybe half an hour or more), and I had 2 mbits of data in results.txt. I found 6 or 7 private keys in it (from openvpn amongst other things).
To check wether the results were what I was looking for, I tested every key with openssl :
[cc]openssl rsa -noout -modulus -in my.server.key | openssl md5[/cc]
And compared the output with the ones for the certificate and the CSR :
[cc]openssl x509 -noout -modulus -in my.server.crt | openssl md5
openssl req -noout -modulus -in my.server.csr | openssl md5[/cc]
But no luck with grep, none of the keys were the one I wanted, so I gave up recovering from the filesystem.
My second attempt was to recover the private key from memory. I hadn’t restarted Apache, and the website was still served in HTTPS as if the key was still there, so I thought Apache must had stored it somewhere in memory.
I tried to use lsof to see if apache was still using the file directly, and found this post describing the procedure, but no luck either. Nobody was still accessing the file directly.
I searched further, half because I knew the file was there somewhere, half because I didn’t want to get back to the client in such context…
I found this PDF explaining how to extract RSA private keys and certificates from process memory. But the user interface looked windowsish and the software is unavailable anyway, due to german laws.
Then I found this, written by 2 french guys, on the same topic as above. The explanation is also very interesting, and the tool “passe-partout” is available. I installed it, and one minute later it had extracted 4 private keys from Apache memory! I just had to check with openssl as explained above, and the third key was the one I wanted.
I copied the file to my.server.key, modified Apache config to match the new file names, restarted Apache, and the new certificate was in use, as if nothing had happened. And told the client that it was done! :p
So relieved now! Still ashamed about having no backups, but I found the damn key, and I learnt a lot in the meantime! 🙂