Perfect Draft Exclamation Mark

Perfect Draft Exclamation Mark KB ID 0001807

Problem

On the bottom of the display you will see a minus sign and an exclamation mark, like so.

Perfect Draft Exclamation Mark Solution

Theres two reasons that you will see this, firstly your keg has been in the machine more than 30 days, or you have powered off your Perfect Draft Machine, and changed kegs.

Remember always leave your machine ON when changing kegs!.

To fix this, you can either use the Plus & Minus keys to change the figure back to 30, of if you’ve changed kegs, leave the machine powered on, and remove  / replace the keg.

Related Articles, References, Credits, or External Links

NA

Can you Use ‘Let’s Encrypt’ on Hybrid Exchange?

KB ID 0001770

Problem

A few months ago I wrote an article about getting free exchange certificates with Lets ‘Encrypt’. And that still works peachy, I implemented it on my test Exchange 2019 server (in Hybrid mode) and all appeared well. 

Some time later, (remember this is my test network so it’s not in production.) I noticed that mail was not getting forwarded from O365 to the on premises Exchange server. I’ve also had to fix similar problems like this before here.

These were to sort of errors I was seeing;

450 4.4.317 Cannot connect to remote server [Message=451 5.7.3 STARTTLS is required to send mail] [LastAttemptedServerName=mail.petenetlive.com] [LastAttemptedIP=x.x.x.x:25] [DB8EUR05FT046.eop-eur05.prod.protection.outlook.com]

450 4.4.317 Cannot connect to remote server [Message=451 4.4.0 TLS negotiation failed with error SocketError] [LastAttemptedServerName=mail.petenetlive.com] [LastAttemptedIP=x.x.x.x:25] [DB3EUR04FT033.eop-eur04.prod.protection.outlook.com]

So the problem is obviously something to do with the certificate on the on premises server, so either it’s misconfigured or Microsoft O365 does not like the ‘Lets Encrypt’ certificate.

Solution

Firstly I made sure the send and receive connecters were using the correct certificate.

 

[box]

Get-ReceiveConnector "PNL-MAIL-2019\Default Frontend PNL-MAIL-2019" | fl TlsCertificateName  
Get-SendConnector "Outbound to Office 365" | fl TlsCertificateName  

[/box]

Note: Above, change the server names and send connector name to match your own.

At this point I was pretty sure that the problem was with that cert so I bought myself a 12 month SSL cert (for the princely sum of $17) and used that instead.

How To Swap Hybrid Connector Certificates

Get the Thumbprint for the NEW publicly signed certificate (Get-ExchangeCertificate). Then use the following PowerShell.

[box]

$cert = Get-ExchangeCertificate -Thumbprint XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
$tlscertificatename = "<i>$($cert.Issuer)<s>$($cert.Subject)"
Set-SendConnector "Outbound to Office 365" -TlsCertificateName $tlscertificatename
Set-ReceiveConnector "PNL-MAIL-2019\Default Frontend PNL-MAIL-2019" -TlsCertificateName $tlscertificatename
Restart-Service MSExchangeTransport

[/box]

Now recheck the connectors again;

 

Then attempt to re-validate the connector in Office 365, and it works straight away.

SO NO YOU CAN’T USE ‘LETS ENCRYPT’ FREE CERTIFICATES IF YOUR EXCHANGE SERVER IS IN HYBRID MODE

Related Articles, References, Credits, or External Links

Office 365: Migrating To Exchange Online

Free Exchange Certificate

No Mail Flow On-Premises To/From Office 365

Exchange Stopped Working?

KB ID 0001749

Problem

I see various posts in online forums, and the process is held up because a lot of techs can’t do basic troubleshooting on their Exchange deployments. So I thought I’d put together some basic ‘first steps’ for you to do when your  ‘Exchange Stopped Working

This way you can at least get an error code to search for or a better idea of what’s going on.

Exchange Stopped Working – First Steps

Well it is Microsoft, reboot it, (yes turn it off and turn it on again.) I myself have lost time troubleshooting a problem that would have been cured with a simple reboot!

Disk Space: Look at you drives, if you are low on space, databases will fail to mount and bad things will happen, If you’re not backing up and logs are not getting flushed, then drives will fill up. It takes two seconds to check.

Services:  Next culprit, look for all the Exchange services that are set to ‘Automatic‘ and manually start them (some may start and stop and that’s OK), but if you get an error, then screenshot that error, look in event viewer and get a copy of the error/warning then you are armed with good internet searching ammunition!

Use The Tool

For a long time, Microsoft have supplied a Remote Connectivity Analyzer. Run though the tests on here.

Exchange Stopped Working – Are you Accepting SMTP?

The simplest test is to see if Exchange is actually providing main services (on TCP Port 25 (SMTP)). You can do this by (on the Exchange server).

[box]

telnet localhost 25

[/box]

And you should get a response (see below). If that fails, then the usual culprit, (if all the services are running), is AV software blocking port 25.

Exchange Stopped Working  – Test Email from CLI (Telnet)

I consider this a basic Exchange Skill, (the ability to send an email from command line). You first need to Telnet to the server (as above) you can then send an email. WARNING if you make a mistake, or have a typo (even if you correct that typo, before hitting enter) it will error! If in doubt write all your commands in notepad and paste them in!

Commands to use;

ehlo {Enter}

This is an ‘Extended’ helo (yes that’s spelled correctly) it opens communications and the server should respond with a list of the services it offers.

Tech Note: Also a good way to see if TLS is supported (it should respond with 250-STARTTLS)

mail from: {email-address} {Enter}

This can be ‘any’ source email address, even one that does not exist.

rcpt to:{email-address} {Enter}

Needs to be an email address that this Exchange has a mailbox for (on a domain that it’s authoritative for).

data {Enter}

Then type some text, you can continue typing as much text as you like.

.{Enter}

That’s a full stop, (period for the US), followed by Enter.

Then, (assuming it was successful and said 250 2.6.0 mail queued for delivery). If not you have an error code to Google. Check the mail was received.

Exchange Stopped Working – Test ‘Inbound’ Mail flow.

If you got this far then internally things are looking healthy, you need to see if you can accept mail from ‘Outside’. Repeat the test you carried out above but for a machine on the public internet, telnet in to the public DNS name of your Exchange and send an email via command line.

This proves your DNS is correct*, your firewall is setup correctly, and your Exchange is receiving mail. 

*Note: Assuming you connected to the DNS name, (if you connected to the public IP then your problem may be DNS!)

Exchange Stopped Working – Test ‘Outbound’ Mail flow.

This one is easy to troubleshoot, send an outbound mail.

Locate the Exchange Toolbox > Queue Viewer.

Look for mail ‘stuck‘ on the outbound queue, in production there will probably be hundreds of them, but what you are looking for is an error message. In the example below that’s 451 4.4.397 (Which I forced to happen by blocking outbound traffic from the Exchange server on the firewall). If you see this make sure the Exchange has TCP port 25 open outbound. 

Now you have enough information to either get on the forums and search, or open a new question if your stuck (AND TRY SEARCHING THIS WEBSITE FOR THE ERROR (TOP RIGHT)). I’ve fixed a few!

Related Articles, References, Credits, or External Links

NA

Software is Preventing Firefox From Safely Connecting to this Site

KB ID 0001727

Problem

I was setting up some HTTPS/SSL inspection this week and while testing it, I ran into this problem;

Firefox Certificate Settings

So the machine I’m using DOES trust the CA that issued that certificate, (it’s a FortiGate firewall) But the BROWSER does not. (Firefox maintains its own list of certificates, and more importantly which CA certificates it will trust). Essentially the browser is trying to protect you from a MITM attack.

Browse to about:prefernces#privacy > Certificates  > View Certificates.

Import.

Navigate to the CA certificate for the authority that signed the certificate(s) you are having a problem with, and import it > Select ‘Trust this CA to identify websites” > OK

Related Articles, References, Credits, or External Links

NA

Cisco ASA: Received a DELETE PFKey message from IKE

KB ID 0001720

Problem

I was debugging a VPN tunnel today. (From a Fortigate to a Cisco ASAv). I was messing around with the encryption and hashing, when the tunnel fell over. Phase 1 was establishing fine but not Phase 2 (IPSEC). 

I’ve got better skills on the ASA, so that’s where I was debugging;

[box]

IPSEC: Received a PFKey message from IKE
IPSEC: Parsing PFKey GETSPI message
IPSEC: Creating IPsec SA
IPSEC: Getting the inbound SPI
IPSEC DEBUG: Inbound SA (SPI 0x00000000) state change from inactive to embryonic
IPSEC: New embryonic SA created @ 0x00007fc98613ea60,
    SCB: 0x85567700,
    Direction: inbound
    SPI      : 0x3B5A332E
    Session ID: 0x00004000
    VPIF num  : 0x00000002
    Tunnel type: l2l
    Protocol   : esp
    Lifetime   : 240 seconds
IPSEC: Received a PFKey message from IKE
IPSEC DEBUG: Received a DELETE PFKey message from IKE for an inbound SA (SPI 0x3B5A332E)
IPSEC DEBUG: Inbound SA (SPI 0x3B5A332E) destroy started, state embryonic
IPSEC: Destroy current inbound SPI: 0x3B5A332E
IPSEC DEBUG: Inbound SA (SPI 0x3B5A332E) free started, state embryonic
IPSEC DEBUG: Inbound SA (SPI 0x3B5A332E) state change from embryonic to dead
IPSEC DEBUG: Inbound SA (SPI 0x3B5A332E) free completed
IPSEC DEBUG: Inbound SA (SPI 0x3B5A332E) destroy completed

[/box]

Solution

Google that error and you get some posts about NAT, that we’re  not applicable to me. I took a look on the Fortigate and the only clue there was;

[box]

Forti-FW # diagnose vpn tunnel list
list all ipsec tunnel in vd 0
------------------------------------------------------
name=Tunnel-To-SiteB ver=2 serial=1 192.168.100.100:0->192.168.100.111:0 dst_mtu=1500
bound_if=4 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/512 options[0200]=frag-rfc  run_state=0 accept_traffic=0 overlay_id=0

proxyid_num=1 child_num=0 refcnt=14 ilast=1 olast=782 ad=/0
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=0
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=Tunnel-To-SiteB proto=0 sa=0 ref=1 serial=2
  src: 0:192.168.1.0/255.255.255.0:0
  dst: 0:172.16.1.0/255.255.255.0:0
run_tally=1

[/box]

There’s not much I can discern from that either; 

sa=0 There is a mismatch between selectors (or no traffic is being initiated).
sa=1 IPsec SA is matching and there is traffic between the selectors.
sa=2 Only seen during IPsec SA rekey

So I went back to basics and checked the Phase 2 on BOTH, firstly the Fortigate;

For the uninitiated: GCM Protocols DON’T require a hashing algorithm, (that’s why you can’t see SHA or MD5 on there), they disappear when a GCM protocol is selected.

Then on the Cisco ASA;

[box]

Cisco-ASA(config-ipsec-proposal)# show run crypto ipsec
crypto ipsec ikev2 ipsec-proposal FORTIGATE
 protocol esp encryption aes-gmac-256
 protocol esp integrity null <--Note: This can say anything it gets ignored!

[/box]

Or if you prefer the ASDM;

THE ANSWER IS STARING YOU/ME IN THE FACE. I just didn’t realise yet, I changed the phase 2 protocols to DES/MD5 and the tunnel came up, I walked up through the protocols and options and discovered what I’d done wrong.

Root Cause: The ASA is set to use AES-GMAC-256 that’s a DIFFERENT PROTOCOL to the AES256GCM configured on the Fortigate! The ASA should be set to AES-GCM-256! (So the Phase 2 proposals didn’t match).

[box]

Cisco-ASA(config)# crypto ipsec ikev2 ipsec-proposal FORTIGATE
Cisco-ASA(config-ipsec-proposal)# protocol esp encryption aes-gcm-256
WARNING: GCM\GMAC are authenticated encryption algorithms.esp integrity config is ignored

[/box]

Or, via ASDM (from the same location as above);

Problem solved!

Related Articles, References, Credits, or External Links

NA

AnyConnect Error: Unable To Verify IP Forwarding Table Modifications

KB ID 0001646

Problem

While attempting to connect to a clients AnyConnect, this happened;

The VPN client was unable to successfully verify the IP forwarding table modifications. A VPN connection will not be established.

Or on older clients, you may see;

The VPN client was unable to modify the IP forwarding table. A VPN connection will not be established. Please restart your computer or device, then try again.

Solution

I was trying to connect from my house, I’d used this connection before from work and it was fine. I worked my way round the problem got my work finished, then re-looked at it the next time I was working from home.

The problem is actually quite simple, take a look at the IP I was using in my house.

Then take a look at the VPN Pool addresses that get allocated to the remote VPN clients (they overlap);

[box]

show run | incl pool

[/box]

Note: This assumes you are using an ‘IP Pool’, If you are using an external DHCP server at the ‘Head end’ then you will need to check/change the scope there.

AnyConnect – Using a Windows DHCP Server to Lease IP Addresses to the Remote Clients

I fixed the problem by simply changing the ‘pool’ so it didn’t overlap.

WARNING: If you have any routing going on behind your firewall (i.e you have layer 3 switches internally, routing between networks or VLANS) you may need to change them to route the ‘new’ AnyConnect subnet back to the firewall.

Update: Solution Windows 10

If you are experiencing this problem on Windows 10, and the above solution is not applicable, consider deleting the following two files;

C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\routechangesv4.bin
C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\routechangesv6.bin

Related Articles, References, Credits, or External Links

NA

Veeam: No Backup Proxy is Able to Process this VM

KB ID 0001644

Problem

While setting up some new backups, all the virtual machine failed like so;

Unable to allocate processing resources. Error: No backup proxy is able to process this VM due to proxy processing mode restrictions.

Solution

On the face of it this looks like a simple problem, either you backup proxy does not have the capacity, you have too many jobs running at one, or you need to deploy some more backup proxies.

However, I tested this by creating a new job with one ‘problem’ VM in it and made sure no other jobs were running. It failed with the same error. So it’s got nothing to do with lack of resources!

The Actual Problem: Was due to the fact that the backup proxy (a physical machine) had direct fiber access to the storage array, and its backup mode was set to ‘Direct Storage Access’. These new VMs were in a different Datastore/LUN that the Veeam server could not see! So you can either tick ‘Failover to network mode if primary mode fails or is unavailable’

Or a better option would be to present the correct storage LUNS to the Veeam Backup server.

Related Articles, References, Credits, or External Links

Veeam Backup and Recovery Download

Veeam Availability Suite Download

Veeam Backup For Office 365 Download

Veeam Backup For Azure Download

Veeam Backup for AWS Download

Certificate Services 0xc8000202 Error

KB ID 0001639

Problem

You will see this error if you are migrating a Certificate Services Server from Server 2008, (NOT Server 2008 R2) to Windows Server 2016, (or newer).

Version of log file is not compatible with the Jet version 0xc8000202 (ESE: 514 Jet_errBadLogVersion)

You will also see the following events logged;

Event ID 17

Log Name: Application
Source: Microsoft-Windows-CertificationAuthority
Date: xx/xx/xxxx xx:xx:xx
Event ID: 17
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: 2019-CA.migrate.com
Description:
Active Directory Certificate Services did not start: Unable to initialize the database connection for MIGRATE-CA. Version of log file is not compatible with Jet version 0xc8000202 (ESE: -514 JET_errBadLogVersion).

Event ID 454

Log Name: Application
Source: ESENT
Date: 1xx/xx/xxxx xx:xx:xx
Event ID: 454
Task Category: Logging/Recovery
Level: Error
Keywords: Classic
User: N/A
Computer: 2019-CA.migrate.com
Description:
certsrv.exe (1268,P,98) Restore0001: Database recovery/restore failed with unexpected error -514.

Event ID 640

Log Name: Application
Source: ESENT
Date: xx/xx/xxxx xx:xx:xx
Event ID: 640
Task Category: General
Level: Warning
Keywords: Classic
User: N/A
Computer: 2019-CA.migrate.com
Description:
certsrv.exe (1268,P,98) Restore0001: Error -1919 validating header page on flush map file “C:\Windows\system32\CertLog\{CA-Name}.jfm”. The flush map file will be invalidated.
Additional information: [SignDbHdrFromDb:Create time:00/00/1900 00:00:00.000 Rand:0 Computer:] [SignFmHdrFromDb:Create time:00/00/1900 00:00:00.000 Rand:0 Computer:] [SignDbHdrFromFm:Create time:01/17/2020 22:30:48.514 Rand:248810345 Computer:] [SignFmHdrFromFm:Create time:01/17/2020 22:30:48.529 Rand:4091580707 Computer:]

Solution

OK, if you followed a good CA migration guide like mine here, then you already have a copy of the the Database, CA certs, Private keys, and Registry settings. So you are good, don’t panic.

This has happened because the source Jet Database that Certificate Services used on the old 2008 Server, (Note: not 2008 R2) is simply too old to be upgraded straight to the one on Server 2016 or newer.

You need to spin up a 2012 R2 server, migrate Certificate Services, onto that, then migrate to Server 2016 (or 2019) from there.

Related Articles, References, Credits, or External Links

NA

Connections From Machines That Don’t Map to Sites?

KB ID 0001635

Problem

I was troubleshooting some replication issues for a client, and carried out a dcdiag on one of their domain controllers, and saw this;

 Starting test: SystemLog
A warning event occurred. EventID: 0x000016AF
Time Generated: xx/xx/xxxx xx:xx:xx
Event String:
During the past 4.21 hours there have been {xxx} connections to this Domain Controller from client machines whose IP addresses don’t map to any of the existing sites in the enterprise. Those clients, therefore, have undefined sites and may connect to any Domain Controller including those that are in far distant locations from the clients. A client’s site is determined by the mapping of its subnet to one of the existing sites. To move the above clients to one of the sites, please consider creating subnet object(s) covering the above IP addresses with mapping to one of the existing sites. The names and IP addresses of the clients in question have been logged on this computer in the following log file ‘%SystemRoot%\debug\netlogon.log’ and, potentially, in the log file ‘%SystemRoot%\debug\netlogon.bak’ created if the former log becomes full. The log(s) may contain additional unrelated debugging information. To filter out the needed information, please search for lines which contain text ‘NO_CLIENT_SITE:’. The first word after this string is the client name and the second word is the client IP address. The maximum size of the log(s) is controlled by the following registry DWORD value ‘HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters\LogFileMaxSize’; the default is 20000000 bytes. The current maximum size is 20000000 bytes. To set a different maximum size, create the above registry value and set the desired maximum size in bytes.

Solution

On the DC in question, Windows Key+R > %Systemroot%\debug\netlogon.log > OK > There’s you missing subnet!

Go to Active Directory Sites and Services, add the missing subnet, and allocate it to the correct site.

Related Articles, References, Credits, or External Links

NA