Can you Use ‘Let’s Encrypt’ on Hybrid Exchange?

KB ID 0001770

Problem

A few months ago I wrote an article about getting free exchange certificates with Lets ‘Encrypt’. And that still works peachy, I implemented it on my test Exchange 2019 server (in Hybrid mode) and all appeared well. 

Some time later, (remember this is my test network so it’s not in production.) I noticed that mail was not getting forwarded from O365 to the on premises Exchange server. I’ve also had to fix similar problems like this before here.

These were to sort of errors I was seeing;

450 4.4.317 Cannot connect to remote server [Message=451 5.7.3 STARTTLS is required to send mail] [LastAttemptedServerName=mail.petenetlive.com] [LastAttemptedIP=x.x.x.x:25] [DB8EUR05FT046.eop-eur05.prod.protection.outlook.com]

450 4.4.317 Cannot connect to remote server [Message=451 4.4.0 TLS negotiation failed with error SocketError] [LastAttemptedServerName=mail.petenetlive.com] [LastAttemptedIP=x.x.x.x:25] [DB3EUR04FT033.eop-eur04.prod.protection.outlook.com]

So the problem is obviously something to do with the certificate on the on premises server, so either it’s misconfigured or Microsoft O365 does not like the ‘Lets Encrypt’ certificate.

Solution

Firstly I made sure the send and receive connecters were using the correct certificate.

 

[box]

Get-ReceiveConnector "PNL-MAIL-2019\Default Frontend PNL-MAIL-2019" | fl TlsCertificateName  
Get-SendConnector "Outbound to Office 365" | fl TlsCertificateName  

[/box]

Note: Above, change the server names and send connector name to match your own.

At this point I was pretty sure that the problem was with that cert so I bought myself a 12 month SSL cert (for the princely sum of $17) and used that instead.

How To Swap Hybrid Connector Certificates

Get the Thumbprint for the NEW publicly signed certificate (Get-ExchangeCertificate). Then use the following PowerShell.

[box]

$cert = Get-ExchangeCertificate -Thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
$tlscertificatename = "<i>$($cert.Issuer)<s>$($cert.Subject)"
Set-SendConnector "Outbound to Office 365" -TlsCertificateName $tlscertificatename
Set-ReceiveConnector "PNL-MAIL-2019\Default Frontend PNL-MAIL-2019" -TlsCertificateName $tlscertificatename
Restart-Service MSExchangeTransport

[/box]

Now recheck the connectors again;

 

Then attempt to re-validate the connector in Office 365, and it works straight away.

SO NO YOU CAN’T USE ‘LETS ENCRYPT’ FREE CERTIFICATES IF YOUR EXCHANGE SERVER IS IN HYBRID MODE

Related Articles, References, Credits, or External Links

Office 365: Migrating To Exchange Online

Free Exchange Certificate

No Mail Flow On-Premises To/From Office 365

Mail Routing Issue ‘451 5.7.3 Cannot achieve Exchange Server authentication’

KB ID 0000791

Problem

While putting in a New Exchange 2010 server today, I test moved a mailbox to this new site, and could not get mail to flow to the Exchange 2010 server at the clients main site.

451 4.4.0 Primary target IP address responded with: “451 5.7.3 Cannot achieve Exchange Server authentication.” Attempted failover to alternate host , but that did not succeed. Either there are no alternative hosts, or delivery failed to all alternative hosts.

Mail flowed from the main site to this new site, and internal mail at the new site was fine, but any mail destined for the main site, or going external (because the main site has the only server that can use the Exchange organization send connector) would fail with this error.

Solution

I did a lot of trawling to try and find the answer to this, and discovered lots of reasons for this to happen, so rather than just posting what fixed mine, from the most popular to the most obscure try these in order, and attempt to send mail after each step.

Note: Any change on an Exchange Server’s Receive Connectors should be followed by you restarting the Microsoft Exchange Transport Service (on the server you made the change on) before you try again.

1. On the server you are trying to send TO, check the properties of the Default receive connector and ensure ‘Exchange Server authentication’ is selected.

2. On the server you are trying to send TO, If you have a connector configured to ‘relay’ mail, make sure that the server(s) or network specified DOES NOT include the IP address of the server you cannot send FROM. Also Make sure on the authentication tab ‘Exchange Server authentication’ is NOT selected.

3. If you have Cisco PIX Firewalls between these two mail servers (running version 6 or earlier) make sure smtp fixup is disabled.

[box]

Petes-PIX>
Petes-PIX> enable
Password: *******
Petes-PIX# configure terminal
Petes-PIX(config)# no fixup protocol smtp 25
Petes-PIX(config)# write mem

Building configuration...
Cryptochecksum: f59a9bd3 3129b8bc 474b2415 52f2db0f

1049 bytes copied in 0.430 secs
[OK]

[/box]

4. If you have Cisco ASA Firewalls between these two mail servers, then remove esmtp from the default inspection map.

Cisco ASA Disable ESMTP Inspection

At this point I admitted defeat and picked up the phone and called Microsoft. One of their support engineers looked at the connectors and settings, and tested the DNS, had me create a new connector, still mail refused to flow. He did however get me pointed in the right direction. When attempting to Telnet to the Exchange server on the main site this is what we saw;

This is what we should be seeing;

Eureka!

I put all the firewalls in, so I know how they are configured, and I know this client has a CSC module in the ASA 5510 at the mail site, I managed to get the output above by rebooting that module, as soon as it was back online we reverted to the short list again. Also while the CSC was rebooting all the mails stuck on the outbound queue cleared.

Enabling CSC Bypass for a Remote Mail Server

Note: Your class-maps, and access-lists may have different names but this should point you in the right direction.

1. Connect to the ASA, view the policy-maps in use.

[box]

Petes-ASA# show run policy-map
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect icmp
inspect pptp
inspect ip-options
class global-class <<<<< Here we go
csc fail-open <<<< This is the one we are looking for
!
Petes-ASA#

[/box]

2. We can see the class-map the CSC is using is called ‘global-class’, let’s take look at that.

[box]

Petes-ASA# show run class-map global-class
!
class-map global-class
match access-list global_mpc <<<< Here's how its being applied
!

[/box]

3. Now we know that’s being applied with an access-list called global_mpc, let’s see what that’s doing.

[box]

Petes-ASA# show run access-list global_mpc
access-list global_mpc extended deny ip host 10.1.0.253 any
access-list global_mpc extended permit tcp any any object-group DM_INLINE_TCP_1
Petes-ASA#

[/box]

Note: Remember permit means inspect and deny means don’t inspect, you can see mine’s set not to scan the CSC update traffic because that’s good practice;

Cisco CSC Module – Stop it scanning its own update traffic

4. So I just need to add in the IP of the Exchange server I cannot send from to make its traffic bypass the CSC Module. Remember to put it at the top so it gets processed before the permit or it will get ignored.

[box]

Petes-ASA# configure terminal
Petes-ASA(config)# access-list global_mpc line 1 extended deny ip host 10.3.0.2 any

[/box]

5. Finally save the changes.

[box]

Petes-ASA(config)# write memory
Building configuration...
Cryptochecksum: aab5e5a2 c707770d f7350728 d9ac34de
[OK]
Petes-ASA(config)#

[/box]

 

Related Articles, References, Credits, or External Links

Thanks to Jeebu Mathew at Microsoft for his assistance, and pointing me in the right direction.