A few months ago I wrote an article about getting free exchange certificates with Lets ‘Encrypt’. And that still works peachy, I implemented it on my test Exchange 2019 server (in Hybrid mode) and all appeared well.
Some time later, (remember this is my test network so it’s not in production.) I noticed that mail was not getting forwarded from O365 to the on premises Exchange server. I’ve also had to fix similar problems like this before here.
These were to sort of errors I was seeing;
450 4.4.317 Cannot connect to remote server [Message=451 5.7.3 STARTTLS is required to send mail] [LastAttemptedServerName=mail.petenetlive.com] [LastAttemptedIP=x.x.x.x:25] [DB8EUR05FT046.eop-eur05.prod.protection.outlook.com]
450 4.4.317 Cannot connect to remote server [Message=451 4.4.0 TLS negotiation failed with error SocketError] [LastAttemptedServerName=mail.petenetlive.com] [LastAttemptedIP=x.x.x.x:25] [DB3EUR04FT033.eop-eur04.prod.protection.outlook.com]
So the problem is obviously something to do with the certificate on the on premises server, so either it’s misconfigured or Microsoft O365 does not like the ‘Lets Encrypt’ certificate.
Solution
Firstly I made sure the send and receive connecters were using the correct certificate.
Note: Above, change the server names and send connector name to match your own.
At this point I was pretty sure that the problem was with that cert so I bought myself a 12 month SSL cert (for the princely sum of $17) and used that instead.
How To Swap Hybrid Connector Certificates
Get the Thumbprint for the NEW publicly signed certificate (Get-ExchangeCertificate). Then use the following PowerShell.
While putting in a New Exchange 2010 server today, I test moved a mailbox to this new site, and could not get mail to flow to the Exchange 2010 server at the clients main site.
451 4.4.0 Primary target IP address responded with: “451 5.7.3 Cannot achieve Exchange Server authentication.” Attempted failover to alternate host , but that did not succeed. Either there are no alternative hosts, or delivery failed to all alternative hosts.
Mail flowed from the main site to this new site, and internal mail at the new site was fine, but any mail destined for the main site, or going external (because the main site has the only server that can use the Exchange organization send connector) would fail with this error.
Solution
I did a lot of trawling to try and find the answer to this, and discovered lots of reasons for this to happen, so rather than just posting what fixed mine, from the most popular to the most obscure try these in order, and attempt to send mail after each step.
Note: Any change on an Exchange Server’s Receive Connectors should be followed by you restarting the Microsoft Exchange Transport Service (on the server you made the change on) before you try again.
1. On the server you are trying to send TO, check the properties of the Default receive connector and ensure ‘Exchange Server authentication’ is selected.
2. On the server you are trying to send TO, If you have a connector configured to ‘relay’ mail, make sure that the server(s) or network specified DOES NOT include the IP address of the server you cannot send FROM. Also Make sure on the authentication tab ‘Exchange Server authentication’ is NOT selected.
3. If you have Cisco PIX Firewalls between these two mail servers (running version 6 or earlier) make sure smtp fixup is disabled.
[box]
Petes-PIX>
Petes-PIX> enable
Password: *******
Petes-PIX# configure terminal
Petes-PIX(config)# no fixup protocol smtp 25
Petes-PIX(config)# write mem
Building configuration...
Cryptochecksum: f59a9bd3 3129b8bc 474b2415 52f2db0f
1049 bytes copied in 0.430 secs
[OK]
[/box]
4. If you have Cisco ASA Firewalls between these two mail servers, then remove esmtp from the default inspection map.
At this point I admitted defeat and picked up the phone and called Microsoft. One of their support engineers looked at the connectors and settings, and tested the DNS, had me create a new connector, still mail refused to flow. He did however get me pointed in the right direction. When attempting to Telnet to the Exchange server on the main site this is what we saw;
This is what we should be seeing;
Eureka!
I put all the firewalls in, so I know how they are configured, and I know this client has a CSC module in the ASA 5510 at the mail site, I managed to get the output above by rebooting that module, as soon as it was back online we reverted to the short list again. Also while the CSC was rebooting all the mails stuck on the outbound queue cleared.
Enabling CSC Bypass for a Remote Mail Server
Note: Your class-maps, and access-lists may have different names but this should point you in the right direction.
1. Connect to the ASA, view the policy-maps in use.
[box]
Petes-ASA# show run policy-map
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect icmp
inspect pptp
inspect ip-options
class global-class <<<<< Here we go
csc fail-open <<<< This is the one we are looking for
!
Petes-ASA#
[/box]
2. We can see the class-map the CSC is using is called ‘global-class’, let’s take look at that.
[box]
Petes-ASA# show run class-map global-class
!
class-map global-class
match access-list global_mpc <<<<Here's how its being applied
!
[/box]
3. Now we know that’s being applied with an access-list called global_mpc, let’s see what that’s doing.
[box]
Petes-ASA# show run access-list global_mpc
access-list global_mpc extended deny ip host 10.1.0.253 any
access-list global_mpc extended permit tcp any any object-group DM_INLINE_TCP_1
Petes-ASA#
[/box]
Note: Remember permit means inspect and deny means don’t inspect, you can see mine’s set not to scan the CSC update traffic because that’s good practice;
4. So I just need to add in the IP of the Exchange server I cannot send from to make its traffic bypass the CSC Module. Remember to put it at the top so it gets processed before the permit or it will get ignored.
[box]
Petes-ASA# configure terminal
Petes-ASA(config)# access-list global_mpc line 1 extended deny ip host 10.3.0.2 any