How does mutual Tls work

Transport Layer Security (TLS)

Cryptology, cryptography and cryptanalysis

The Transport Layer Security (Transport security, short TLS) is the Successor to the Secure Sockets Layer (SSL). Specifically, it is a hybrid encryption protocol for secure communication on the Internet. Most often you will come into contact with TLS via HTTPS, where the encryption method is used.

TLS protocol structure

Functionality and procedure

The general process with TLS begins with the establishment of a connection from the client to the server. At the same time he sends a list of supported cipher suites. The server then authenticates itself to the client with a certificate and the selected cipher suite. The client checks the certificate and, if necessary, authenticates itself to the server with its own certificate. Now either the client sends the server an encrypted random number, which is encrypted with the public key of the server, or both parties calculate a shared secret using the Diffie-Hellman key exchange method. With the cryptographic key derived from this, all messages on the connection are now encrypted with a selected symmetric encryption method. The following components are active at TLS:

TLS Record Protocol

The TLS Record Protocol is a layered protocol that takes the messages to be sent, divides the data into blocks (fragmented), optionally compresses them (not the default, otherwise the compression method is negotiated in the handshake), MAC applies them to them (for integrity protection ), encrypted (symmetrical encryption - stream ciphers, block ciphers or combined procedures) and the result is sent. Received data is decrypted, verified, decompressed and reassembled. Four protocols are based on the TLS Record Protocol:
  • Handshake Protocol
  • Alert Protocol
  • Change Cipher Spec Protocol
  • Application Data Protocol
The core tasks of the TLS Record Protocol are therefore the management of the TLS session, the fragmentation / compression of the application data and the cryptographic processing. This ensures the protection goals of confidentiality and integrity.

Calculation of the key material

As you will see a little further below, the premaster secret is created within the handshake protocol. This is the basis for the master secret, which in turn is generated within the record protocol. At this point one naturally wonders why one cannot use the Premaster Secret right away? The answer to the question is, on the one hand, that the master secret must be of sufficient length and, on the other hand, it increases security. The random client number may not be secure - both communication parties want to contribute to security - in addition, attackers could import the encrypted premaster secret without the random numbers. Repeated attacks are also prevented.

The master secret is now made up of the client random number and the server random number in addition to the premaster secret. Using a pseudo-random function (PRF), the master secret is now generated from these three components. As soon as the Master Secret has been calculated, the Pre Master Secret should be deleted from memory.

Calculation of the master secret:
master_secret = PRF (pre_master_secret, "master secret", ClientHello.random + ServerHello.random)
The master secret is always exactly 48 bytes long.

TLS handshake protocol

The TLS Handshake Protocol is responsible for negotiating a session (and thus for the security parameters used for it). The handshake messages are then delivered to the TLS Record Layer, where they are processed accordingly together with the other data. The TLS handshake protocol has the following tasks:
  • Negotiation of the used cryptographic algorithms (for encryption & integrity) â € “but also unencrypted transmission possible
  • Authentication of the communication partner (server and / or client - but mostly only server versus client)
  • Negotiating the key materials

Procedure (simplified)

The handshake protocol begins with a ClientHello from the client to the server. This contains a list of cipher suites, a 32-byte nonce from the client (28-byte random number + 7-byte current time from the client) and a session number for a later session resumption. In addition, a list of compression algorithms that the client supports is sent.
The server must respond to a ClientHello with a ServerHello. If this does not happen, a fatal error occurs and the connection fails. The ServerHello response contains the selected CipherSuite, the nonce of the server and a selected session number.
Server Certificate
This message is required if the selected key exchange method calls for a certificate. The Server Certificate message always follows the ServerHello message directly. Ultimately, the x509 certificate is sent, matching the selected CipherSuite.
The ServerKeyExchange message is only sent if the server certificate does not contain all parameters for the key calculation (the premaster secret). This is the case with:
The Certificate-Request message is also optional. With this message, the server can also request a certificate from the client if this is appropriate for the selected cipher suite. The Certificate Request message is sent immediately after the ServerKeyExchange message or, if it is not sent, after the message with the server's certificate.
Server Hello Done
The ServerHelloDone message is the completion of the server's authentication and key exchange. It indicates the end of the communication associated with the ServerHello. The server is now waiting for the client to answer.

Before the client starts sending messages, it should first verify whether the server's certificate is valid. The other parameters of the Server Hello messages should also be checked.
If there was a request from the server, the client sends its X509 certificate with the ClientCertificate message, matching the selected CipherSuite. If no suitable certificate is available, the client still has to send a certificate message. If it does not send this message, the server can break the connection or continue the handshake without client authentication.
Client Key Exchange
While the ServerKey-Exchange message is optional, the ClientKeyExchange message is always sent. It either follows the certificate message immediately, or it is the first message from the client that follows the ServerHelloDone message. This message is used to set the premaster secret, either directly by sending an RSA-encrypted secret or by transmitting the Diffie-Hellman parameters.
Certificate Verify
On the other hand, the CertificateVerify message is again optional. This results in an explicit verification of the client certificate, i.e. proof that the client also has the private key and is therefore who it claims to be. This message is only sent if the client's certificate can be signed at all.
Change Cipher Spec
The ChangeCipherSpec message indicates that communication will be carried out securely from now on. From this point on, the TLS Record Layer uses the negotiated cryptographic algorithms and keys. The ChangeCipherSpec message is thus also the first protected message.
The Finnished message confirms that the key exchange and the authentication process were successful. The recipient of the Finnished message has to verify that the content of the message is correct. For this purpose, a hash value is calculated and checked for all exchanged messages. Among other things, this prevents a downgrade attack if, for example, an attacker had deleted strong cipher suites from the ClientHello.

TLS alert protocol

As the name suggests, the Alert Protocol is responsible for alerts. This is divided into two levels: â € œWarningâ € and â € œFatalâ €. Fatal alert messages lead to an immediate disconnection. Like the other messages, the alert messages are encrypted and compressed.

The TLS alert protocol knows various alerts. One of the most important alerts is the CloseNotify message, as TLS requires an explicit disconnection. Otherwise it could lead to truncation attacks, for example. Each party can initiate the exchange of a closing message and thus signal to the remote station that the sender will no longer send the recipient any more messages via this connection. Any data that is subsequently received is ignored. The recipient in turn must respond with a close_notify alert, but the initiator does not have to wait for this response.

Datagram Transport Layer Security (DTLS)

TLS is found above a reliable transport protocol (e.g. in the TCP / IP model above the transport layer, e.g. via TCP). A reliable transport protocol such as TCP is required above all for the Handshake Protocol and the Record Protocol. The reliability of the handshake must be guaranteed, since otherwise packet loss will always lead to a new handshake. With the Record Protocol, on the other hand, the cryptographic processing of the packets plays a role, which is not possible independently of one another.

So that the use of TLS is also possible with unreliable transport protocols such as UDP, the Datagram Transport Layer Security (DTLS) invented. This became necessary because more and more applications are increasingly using UDP, for example Voice over IP (VoIP).

DTLS does not differ fundamentally from TLS, but has only been adapted where changes or the use of a non-reliable transport protocol were necessary. The handshake protocol was modified, sequence numbers were introduced in the record protocol, stream ciphers were deleted in the record layer and the handling of lost packets was revised.

Sources and References

  • Secure Network Communication - Fundamentals, Protocols, and Architectures; Bless, R., Mink, S., Blaß, E.-O., Conrad, M., Hof, H.-J., Kutzner, K., Schöller, M.

Article from 08/30/2016