Use more clever certificate subject

[sbernauer]

Well, currently all certificates get the subject CN=generated certificate for pod.
This imposes real security problems as shown in the code links below.

We should change that, so that one can actually use the subject for authorization. Things that come to my mind:

1. OPA rules for Kafka using mTLS
2. NiFi OPA rules and config
3. @siegfriedweber mentioned the OpenSearch implementation also struggles with our current subject

[nightkr]

The question is.. what would a good subject DN be? Node name is too broad to be useful (and is gonna cause bleed between different services that may or may not have a trust relationship..), the service scope should generally die in a fire, and pod name is too specific (you don't really want to distinguish between different replicas of the same service). The listener scope suffers from all of these problems at the same time, in addition to not really having a "primary" identity at all (it's just a bag of addresses, which works fine for SANs that we can have multiple of, but not so much for having to pick one subject DN).

I think a reasonable way to do it would be to use the Pod's ServiceAccount (and Namespace), which gives the user a reasonable amount of control but is also already treated as the permission boundary for accessing the K8s API itself. In an ideal world, it might have been worth using SPIFFE (like Istio already does), but in practice (at least last I checked) tooling support for URI SANs is abysmal (and of course generally even worse for things that assume that the subject DN is the only meaningful part of the certificate, or that the subject DN is meaningful at all).

[NickLarsenNZ]

Do our products require that the Serving cert is also the one used for Client auth?

I've always used completely separate certificates for that (for one or both of ZooKeeper and Kafka). That way you can use a general CA for the server cert (eg: internal PKI), and for mTLS you can use a super restricted CA that is added to the client auth trust store for node-to-node comms.

I assume some of our products can't differentiate.

[lfrancke]

We decided to look at this for real now as it's needed for OpenSearch.

The next/first step is a refinement where we discuss/brainstorm what a solution could look like, what we want to do and a recommendation how to continue so we can do a final go/no-go decision.