Cephx

Intro

The protocol design looks a lot like kerberos. The authorizer “KDC”role is served by the monitor, who has a database of shared secretsfor each entity. Clients and non-monitor daemons all start byauthenticating with the monitor to obtain tickets, mostly referreed toin the code as authorizers. These tickets provide bothauthentication and authorization in that they include adescription of the capabilities for the entity, a concise structureddescription of what actions are allowed, that can be interpreted andenforced by the service daemons.

Other references

Terms

  • monitor(s): central authorization authority

  • service: the set of all daemons of a particular type (e.g., allOSDs, all MDSs)

  • client: an entity or principal that is accessing the service

  • entity name: the string identifier for a principal(e.g. client.admin, osd.123)

  • ticket: a bit of data that cryptographically asserts identify andauthorization

  • principal: a client or daemon, identified by a unique entity_name,that shares a secret with the monitor.

  • principal_secret: principal secret, a shared secret (16 bytes)known by the principal and the monitor

  • mon_secret: monitor secret, a shared secret known by all monitors

  • service_secret: a rotating secret known by all members of aservice class (e.g., all OSDs)

  • auth ticket: a ticket proving identity to the monitors

  • service ticket: a ticket proving identify and authorization to aservice

Terminology

{foo, bar}^secret denotes encryption by secret.

Context

The authentication messages described here are specific to the cephxauth implementation. The messages are transferred by the Messengerprotocol or by MAuth messages, depending on the version of themessenger protocol. See also msgr2 protocol.

An initial (messenger) handshake negotiates an authentication methodto be used (cephx vs none or krb or whatever) and an assertion of whatentity the client or daemon is attempting to authenticate as.

Phase I: obtaining auth ticket

The cephx exchange begins with the monitor knowing who the clientclaims to be, and an initial cephx message from the monitor to theclient/principal.:

  1. a->p :
  2.   CephxServerChallenge {
  3.     u64 server_challenge     # random (by server)
  4.   }

The client responds by adding its own challenge, and calculating avalue derived from both challenges and its shared keyprincipal_secret.:

  1. p->a :
  2.   CephxRequestHeader {
  3.     u16 CEPHX_GET_AUTH_SESSION_KEY
  4.   }
  5.   CephXAuthenticate {
  6.     u8 2                     # 2 means nautilus+
  7.     u64 client_challenge     # random (by client)
  8.     u64 key = {client_challenge ^ server_challenge}^principal_secret   # (roughly)
  9.     blob old_ticket          # old ticket, if we are reconnecting or renewing
  10.     u32 other_keys           # bit mask of service keys we want
  11.   }

Prior to nautilus,:

  1. CephXAuthenticate {
  2.   u8 1                     # 2 means nautilus+
  3.   u64 client_challenge     # random (by client)
  4.   u64 key = {client_challenge + server_challenge}^principal_secret   # (roughly)
  5.   blob old_ticket          # old ticket, if we are reconnecting or renewing
  6. }

The monitor looks up principal_secret in database, and verifies thekey is correct. If old_ticket is present, verify it is valid, and wecan reuse the same global_id. (Otherwise, a new global_id is assignedby the monitor.):

  1. a->p :
  2.   CephxReplyHeader {
  3.     u16 CEPHX_GET_AUTH_SESSION_KEY
  4.     s32 result (0)
  5.   }
  6.   u8 encoding_version = 1
  7.   u32 num_tickets ( = 1)
  8.   ticket_info           # (N = 1)

plus (for Nautilus and later):

  1. u32 connection_secret_len      # in bytes
  2. connection_secret^session_key
  3. u32 other_keys_len             # bytes of other keys (encoded)
  4. other_keys {
  5.   u8 encoding_version = 1
  6.   u32 num_tickets
  7.   service_ticket_info * N      # for each service ticket
  8. }

where:

  1. ticket_info {
  2.   u32 service_id       # CEPH_ENTITY_TYPE_AUTH
  3.   u8 msg_version (1)
  4.   {CephXServiceTicket service_ticket}^principal_secret
  5.   {CephxTicketBlob ticket_blob}^existing session_key   # if we are renewing a ticket,
  6.   CephxTicketBlob ticket_blob                          # otherwise
  7. }
  8.  
  9. service_ticket_info {
  10.   u32 service_id       # CEPH_ENTITY_TYPE_{OSD,MDS,MGR}
  11.   u8 msg_version (1)
  12.   {CephXServiceTicket service_ticket}^principal_secret
  13.   CephxTicketBlob ticket_blob
  14. }
  15.  
  16. CephxServiceTicket {
  17.   CryptoKey session_key      # freshly generated (even if old_ticket is present)
  18.   utime_t expiration         # now + auth_mon_ticket_ttl
  19. }
  20.  
  21. CephxTicketBlob {
  22.   u64 secret_id             # which service ticket encrypted this; -1 == monsecret, otherwise service's rotating key id
  23.   {CephXServiceTicketInfo ticket}^mon_secret
  24. }
  25.  
  26. CephxServiceTicketInfo {
  27.   CryptoKey session_key     # same session_key as above
  28.   AuthTicket ticket
  29. }
  30.  
  31. AuthTicket {
  32.   EntityName name           # client's identity, as proven by its possession of principal_secret
  33.   u64 global_id             # newly assigned, or from old_ticket
  34.   utime_t created, renew_after, expires
  35.   AuthCapsInfo       # what client is allowed to do
  36.   u32 flags = 0      # unused
  37. }

So: for each ticket, principal gets a part that it decrypts with itssecret to get the session_key (CephxServiceTicket). And theCephxTicketBlob is opaque (secured by the mon secret) but can be usedlater to prove who we are and what we can do (see CephxAuthorizerbelow).

For Nautilus+, we also include the service tickets.

The client can infer that the monitor is authentic because it candecrypt the service_ticket with its secret (i.e., the server has itssecret key).

Phase II: Obtaining service tickets (pre-nautilus)

Now the client needs the keys used to talk to non-monitors (osd, mds,mgr).:

  1. p->a :
  2.   CephxRequestHeader {
  3.     u16 CEPHX_GET_PRINCIPAL_SESSION_KEY
  4.   }
  5.   CephxAuthorizer authorizer
  6.   CephxServiceTicketRequest {
  7.     u32 keys    # bitmask of CEPH_ENTITY_TYPE_NAME (MGR, OSD, MDS, etc)
  8.   }

where:

  1. CephxAuthorizer {
  2.   u8 AUTH_MODE_AUTHORIZER (1)
  3.   u64 global_id
  4.   u32 service_id    # CEPH_ENTITY_TYPE_*
  5.   CephxTicketBlob auth_ticket
  6.   {CephxAuthorize msg}^session_key
  7. }
  8.  
  9. CephxAuthorize msg {
  10.   u8 2
  11.   u64 nonce                         # random from client
  12.   bool have_challenge = false       # not used here
  13.   u64 server_challenge_plus_one = 0 # not used here
  14. }

The monitor validates the authorizer by decrypting the auth_ticketwith mon_secret and confirming that it says this principal is whothey say they are in the CephxAuthorizer fields. Note that the noncerandom bytes aren’t used here (the field exists for Phase III below).

Assuming all is well, the authorizer can generate service ticketsbased on the CEPHENTITY_TYPE* bits in the keys bitmask.

The response looks like:

  1. CephxResponseHeader {
  2.   u16 CEPHX_GET_PRINCIPAL_SESSION_KEY
  3.   s32 result (= 0)
  4. }
  5. u8 encoding_version = 1
  6. u32 num_tickets
  7. ticket_info * N

Where, as above,:

  1. ticket_info {
  2.   u32 service_id      # CEPH_ENTITY_TYPE_{OSD,MGR,MDS}
  3.   u8 msg_version (1)
  4.   {CephXServiceTicket service_ticket}^principal_secret
  5.   CephxTicketBlob ticket_blob
  6. }
  7.  
  8. CephxServiceTicket {
  9.   CryptoKey session_key
  10.   utime_t expiration
  11. }
  12.  
  13. CephxTicketBlob {
  14.   u64 secret_id       # which version of the (rotating) service ticket encrypted this
  15.   {CephXServiceTicketInfo ticket}^rotating_service_secret
  16. }
  17.  
  18. CephxServiceTicketInfo {
  19.   CryptoKey session_key
  20.   AuthTicket ticket
  21. }
  22.  
  23. AuthTicket {
  24.   EntityName name
  25.   u64 global_id
  26.   utime_t created, renew_after, expires
  27.   AuthCapsInfo       # what you are allowed to do
  28.   u32 flags = 0      # unused
  29. }

This concludes the authentication exchange with the monitor. Theclient or daemon now has tickets to talk to the mon and all otherdaemons of interest.

Phase III: Opening a connection to a service

When a connection is opened, an “authorizer” payload is sent:

  1. p->s :
  2.   CephxAuthorizer {
  3.     u8 AUTH_MODE_AUTHORIZER (1)
  4.     u64 global_id
  5.     u32 service_id    # CEPH_ENTITY_TYPE_*
  6.     CephxTicketBlob auth_ticket
  7.     {CephxAuthorize msg}^session_key
  8.   }
  9.  
  10.   CephxAuthorize msg {
  11.     u8 2
  12.     u64 nonce               # random from client
  13.     bool have_challenge = false
  14.     u64 server_challenge_plus_one = 0
  15.   }

Note that prior to the Luminous v12.2.6 or Mimic v13.2.2 releases, theCephxAuthorize msg did not contain a challenge, and consisted onlyof:

  1. CephxAuthorize msg {
  2.   u8 1
  3.   u64 nonce               # random from client
  4. }

The server will inspect the auth_ticket CephxTicketBlob (by decryptingit with its current rotating service key). If it is a pre-v12.2.6 orpre-v13.2.2 client, the server immediately replies with:

  1. s->p :
  2.   {CephxAuthorizeReply reply}^session_key

where:

  1. CephxAuthorizeReply {
  2.   u64 nonce_plus_one
  3. }

Otherwise, the server will respond with a challenge (to prevent replayattacks):

  1. s->p :
  2.   {CephxAuthorizeChallenge challenge}^session_key

where:

  1. CephxAuthorizeChallenge {
  2.   u64 server_challenge        # random from server
  3. }

The client decrypts and updates its CephxAuthorize msg accordingly,resending most of the same information as before:

  1. p->s :
  2.   CephxAuthorizer {
  3.     u8 AUTH_MODE_AUTHORIZER (1)
  4.     u64 global_id
  5.     u32 service_id    # CEPH_ENTITY_TYPE_*
  6.     CephxTicketBlob auth_ticket
  7.     {CephxAuthorize msg}^session_key
  8.   }

where:

  1. CephxAuthorize msg {
  2.   u8 2
  3.   u64 nonce                        # (new) random from client
  4.   bool have_challenge = true
  5.   u64 server_challenge_plus_one    # server_challenge + 1
  6. }

The server validates the ticket as before, and then also verifies themsg nonce has it’s challenge + 1, confirming this is a liveauthentication attempt (not a replay).

Finally, the server responds with a reply that proves its authenticityto the client. It also includes some entropy to use for encryption ofthe session, if it is needed for the mode.:

  1. s->p :
  2.   {CephxAuthorizeReply reply}^session_key

where:

  1. CephxAuthorizeReply {
  2.   u64 nonce_plus_one
  3.   u32 connection_secret_length
  4.   connection secret
  5. }

Prior to nautilus, there is no connection secret:

  1. CephxAuthorizeReply {
  2.   u64 nonce_plus_one
  3. }

The client decrypts and confirms that the server incremented nonceproperly and that this is thus a live authentication request and not areplay.

Rotating service secrets

Daemons make use of a rotating secret for their tickets instead of afixed secret in order to limit the severity of a compromised daemon.If a daemon’s secret key is compromised by an attacker, that daemonand its key can be removed from the monitor’s database, but theattacker may also have obtained a copy of the service secret shared byall daemons. To mitigate this, service keys rotate periodically sothat after a period of time (auth_service_ticket_ttl) the key theattacker obtained will no longer be valid.:

  1. p->a :
  2.   CephxRequestHeader {
  3.     u16 CEPHX_GET_ROTATING_KEY
  4.   }
  5.  
  6. a->p :
  7.   CephxReplyHeader {
  8.     u16 CEPHX_GET_ROTATING_KEY
  9.     s32 result = 0
  10.   }
  11.   {CryptoKey service_key}^principal_secret

That is, the new rotating key is simply protected by the daemon’srotating secret.

Note that, as an implementation detail, the services keep the currentkey and the prior key on hand so that the can continue to validaterequests while the key is being rotated.