Hi!
Long story short:
Self-signed certificates in required .pfx-format are not working in MongoDB and Deadline.
Error - "Private key doesn't match certificate: [SSL] PEM lib (_ssl.c:3917)"
However, openssl is able to verify and authenticate the .pfx against the custom CA added to Mongo. Self-signed .pem-files work perfectly fine.
How can I make Deadline-compatible .pfx files?
The steps outlined in the documentation does not provide desired results.
Short story long:
I’m trying to create a docker-compose setup for Deadline, using separate containers for MongoDB, Deadline RCS and Web Service.
I’ve seen a few old repositories out there that claims to have a working setup, but they’re all non-functional for one reason or another.
And they all rely on Thinkbox’s python utility script for generating certificates, which has since been removed from Thinkbox’s GitHub and removed from newer documentation.
I want to use openssl for generating certificates with a custom CA, and avoid juggling certificates made by Deadline during installation.
The Docker setup works great. I’m using Mongo’s official image, and running RCS in an Ubuntu container.
The MongoDB server works fine. When TLS is set to “required” in Mongo’s config, I can still connect to it using the .crt/.pem certificates I’ve generated myself and self-signed, both from inside Docker and externally through Python.
I can also make the Deadline RCS container connect to MongoDB if I turn off TLS.
But where it all breaks down, is when I have to use pkcs12 .pfx-files, as required by Deadline. No matter what I do, the .pfx files do not authenticate.
I’ve tried using the .pfx-files in RCS (Docker), and when connecting directly to the Repository from Deadline Monitor. Neither works.
I get a generic "The remote certificate was rejected by the provided RemoteCertificateValidationCallback" error.
To diagnose the error, I’ve tried removing Deadline from the equation (and the minefield of potentially misconfigured .ini-files in Deadline), by connecting to the MongoDB directly through pymongo python library. There, the .pem-files correctly authenticate.
But the .pfx-files give a more interesting error: “Private key doesn’t match certificate: [SSL] PEM lib (_ssl.c:3917)”
If I extract a pem-file from the same non-functioning .pfx-file, it will connect.
I also compare the modulus of both the working .pem file and the .pfx file using openssl, and both match.
I’ve used openssl to verify and authenticate both files against their common CA key, and they both authenticate successfully.
So there’s nothing to suggest to me that the private key is wrong in the .pfx file.
However, when I extract a .pem-file from the .pfx-file to inspect the certificate trust chain, I see they’re different.
In the original .pem, the private key is at the top.
In the .pem extracted from .pfx, the private key is at the bottom.
(There are no intermediate CAs, so only two certs should be present in both)
Is it possible that Deadline and MongoDB gets confused over this, and simply try to authenticate using the first certificate it finds?
Is there some secret sauce or flag that prevents the certificate reordering?
I’ve tried following the documentation for self-signed certificates from Deadline’s documentation, and searched every nook and corner in the forums for a working recipe, but I’m completely lost.
I’ve attached the script I’m using to generate certificates (with dummy-passwords), along with the certificates generated.
debug_certificates.zip (25.7 KB)
