[BlueOnyx:18074] Re: Sendmail TLS problem

[Michael Stauber]

Hi Carl,

> Nope - the STARTTLS=server indicates that his machine is the server,
> 199.175.188.52 is the client sending to him.

Correct. At the end of the day it doesn't matter, though, as it went
bust at the TLS negotiation stage. So it might work in neither direction
if the sending server would prefer TLS over sending non-encrypted.

> 199.175.188.52 probably has no encryption algorithm in common with
> Michael Aronoff's BlueOnyx server.

My OpenSSL query reported this cipher for the *.52:

DHE-RSA-AES256-SHA

Which is the OpenSSL name for TLS_DHE_RSA_WITH_AES_256_CBC_SHA (in the
RFC annotation), which we actually support for Sendmail on all BlueOnyx
versions. We also support TLSv1.0 as minimum on all BlueOnyx versions.
So ciphers and protocol? We should get an easy match there.

Which leaves Diffie-Hellman. So let's take a look:

#> openssl s_client -connect 199.175.188.52:25 -tls1 -cipher EDH
-starttls smtp -msg


<<< TLS 1.0 Handshake [length 020d], ServerKeyExchange
    0c 00 02 09 00 80 f8 d5 cc e8 7a 39 61 b5 f5 cb
    c8 34 40 c5 18 56 e0 e6 fa 6d 5a b2 83 10 78 c8
    67 62 1c a4 6c a8 7d 7f a3 b1 af 75 b8 34 3c 69
    93 74 d3 69 20 f2 e3 9a 65 3d e8 f0 72 5a a6 e2
    [...]

0C 00 02 09: message of type "ServerKeyExchange" (that's the "0c") of
length 0x000209 bytes.

First element is the DH modulus as a big integer, with a two-byte length
header. Here, the length is encoded as 00 80, meaning 128 decimal. It's
half the modulus we use for 2048 bit DH, so we can assume that it's 1024
bit DH at the most. But: The overall length of the ServerKeyExchange is
only 2/3rds of what it would be if we were using 2048 bit DH. So it
might as well be just 768 bit, which would explain the problem: DH too
short.

-- 
With best regards

Michael Stauber