FortiGate LDAPS Authentication Failure

Posted on 29/01/2021 by PeteLong

KB ID 0001733

Problem

Here’s a brief one that tripped me up a couple of weeks ago, I was deploying FortiGate LDAPS authentication for some FortiClient SSL VPN connections into a FortiGate firewall like so;

Despite my best efforts I was getting authentication failures? If I tested the username and password in the GUI web management portal, that worked fine?

Testing FortiGate LDAPS

First step is to test authentication at command line, like so;

[box]

Forti-FW # diag test auth ldap My-DC test.user Password123
authenticate 'test.user' against 'My-DC' failed!

[/box]

Note: My-DC is the domain controller, test, user is the username, and Password123 is the password for my AD user. (The fact I need to explain that is depressing, but c’est la vie).

So despite what the GUI is telling me, authentication is actually failing, remember I’m using LDAPS, so the FortiGate needs to have the CA certificate, (that issued the Kerberos certificates on my domain controller(s)), in its trusted CA list! And TCP port 636 needs to be open between the firewall and the domain controllers.

Debugging FortiGate LDAPS

So now we need to debug what’s going on;

[box]

Forti-FW # diagnose debug enable
Forti-FW # diagnose debug application fnbamd 255
Debug messages will be on for 30 minutes.

[/box]

Then simply attempt to authenticate via FortiClient, or recall the ‘diag test’ command from above.

[box]

Forti-FW # diag test auth ldap My-DC test.user Password123
[1932] handle_req-Rcvd auth req 1296531457 for test.user in My-DC opt=0000001b prot=0
[424] __compose_group_list_from_req-Group 'My-DC', type 1
[617] fnbamd_pop3_start-test.user
[970] __fnbamd_cfg_get_ldap_list_by_server-
[976] __fnbamd_cfg_get_ldap_list_by_server-Loaded LDAP server 'My-DC'
[1131] fnbamd_cfg_get_ldap_list-Total ldap servers to try: 1
[1713] fnbamd_ldap_init-search filter is: sAMAccountName=test.user
[1722] fnbamd_ldap_init-search base is: dc=testbench,dc=co,dc=uk
[1146] __fnbamd_ldap_dns_cb-Resolved My-DC:192.168.1.122 to 192.168.1.122, cur stack size:1
[919] __fnbamd_ldap_get_next_addr-
[1152] __fnbamd_ldap_dns_cb-Connection starts My-DC:192.168.1.122, addr 192.168.1.122 over SSL
[874] __fnbamd_ldap_start_conn-Still connecting 192.168.1.122.
[591] create_auth_session-Total 1 server(s) to try
[1097] __ldap_connect-tcps_connect(192.168.1.122) failed: ssl_connect() failed: 337047686 (error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed).
[930] __ldap_error-My-DC:192.168.1.122, addr 192.168.1.122
[725] __ldap_stop-Conn with 192.168.1.122 destroyed.
[919] __fnbamd_ldap_get_next_addr-
[902] __ldap_try_next_server-No more server to try for 'My-DC'.
[785] __ldap_done-svr 'My-DC'
[755] __ldap_destroy-
[2870] fnbamd_ldap_result-Error (3) for req 1296531457
[217] fnbamd_comm_send_result-Sending result 3 (nid 0) for req 1296531457, len=2044
authenticate 'test.user' against 'My-DC' failed!
Forti-FW # [747] destroy_auth_session-delete session 1296531457
[755] __ldap_destroy-
[1764] fnbamd_ldap_auth_ctx_free-Freeing 'My-DC' ctx
[2099] fnbamd_ldap_free-Freeing 'My-DC'

[/box]

OK so it’s SSL related? For SSL to work you need the following;

To trust the CA that issued the certificate
To be able to resolve (via DNS) the common name (or Subject Alternative Name) on the certificate
If you’ve specified the LDAP server by IP address the IP address of the server needs to be on the certificate as a Subject Alternative Name (SAN).
Your firewall and the AD/LDAP server need to have compatible SSL ciphers.

So I had number 1 covered, and the chance of it being number 4 are rare, (server and firewall are fully updated).

So my problem was initially number 2 I’d specified the LDAPS server via its internal IP. I needed to use its FQDN, then of course the firewall needed to be able to resolve that IP with a DNS lookup (try execute ping server-name.doman-name if you’re unsure!)

[box]

Forti-FW # execute ping win-server.testbench.co.uk
Unable to resolve hostname.  <---OOPS THAT'S NOT GOOD!

Forti-FW # execute ping 192.168.1.122 <---CONNECTIVITY IS OK!
PING 192.168.1.122 (192.168.1.122): 56 data bytes
64 bytes from 192.168.1.122: icmp_seq=0 ttl=128 time=5.4 ms
64 bytes from 192.168.1.122: icmp_seq=1 ttl=128 time=2.0 ms
64 bytes from 192.168.1.122: icmp_seq=2 ttl=128 time=1.9 ms
^C
--- 192.168.1.122 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.9/3.1/5.4 ms

[/box]

Once DNS was setup correctly;

[box]

Forti-FW # execute ping win-server.testbench.co.uk
PING win-server.testbench.co.uk (192.168.1.122): 56 data bytes
64 bytes from 192.168.1.122: icmp_seq=0 ttl=128 time=1.9 ms
64 bytes from 192.168.1.122: icmp_seq=1 ttl=128 time=2.3 ms
64 bytes from 192.168.1.122: icmp_seq=2 ttl=128 time=2.1 ms
^C
--- win-server.testbench.co.uk ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss <---BOOM THAT'S BETTER
round-trip min/avg/max = 1.9/2.1/2.3 ms

[/box]

Then retest.

[box]

Forti-FW # diag test auth ldap My-DC test.user Password123
authenticate 'test.user' against 'My-DC' succeeded!
Group membership(s) - CN=GS-VPN-Users,OU=Securty-Groups,DC=testbench,DC=co,DC=uk
                      CN=Domain Users,CN=Users,DC=testbench,DC=co,DC=uk

[/box]

Setup Cant Continue PowerShell Has Open Files

Posted on 06/01/2020 by PeteLong

KB ID 0001633

Problem

While attempting to uninstall Microsoft Exchange server;

Setup can’t continue with the uninstall because the powershell (PID) has open files. Close the process, and then restart setup.

Solution

Seems to be a common error, and is usually caused because someone has the Exchange Web Management page open, (probably in another user session);

In some cases you may need to reboot, but in my case I was simply being a doofus, look at the window I’m running the command from! Open an administrative command window and try again 😉

Exchange ContentIndexState ‘Failed’

Posted on 04/09/2019 by PeteLong

KB ID 0001591

Problem

A client from a recent Exchange migration I had done, emailed me to say that his clients were having trouble searching their mailboxes, I jumped on remotely and saw that the mailbox Database content index status was showing ‘Failed’.

ContentIndexState: Failed

Solution

I’ve had problems before with indexing here, I replicated the fix to remove the index and generate a new one. The process for this is, you need to stop the Fast Search and Host Controller Services, then find out where the Mailbox Database is located;

[box]

Stop-Service MSExchangeFastSearch
Stop-Search HostControllerService
Get-MailboxDatabase Mailbox-Database-Name | Select EdbFilePath

[/box]

Go to that location, as well as the .edb file you will see the ‘index folder‘, it’s name will just be a long GUID. DELETE that folder.

Then start those services again.

[box]

Start-Service MSExchangeFastSearch
Start-Search HostControllerService

[/box]

In a few minutes another index folder will ‘appear’, you will now have to wait for the index to rebuild, if you check the status now it will say ‘ContentIndexState Crawling‘. It will say this for quite some time, (hours) while the new index is rebuilt.

How To Monitor ‘Exchange Indexing’ Crawling Progress

Launch ‘Perfmon‘ and remove all the counters, then add in a new one for ‘Local Computer’ > ‘MSExchange Search Index ‘> ‘Crawler: Mailboxes Remaining’ > Select your database > Add > OK.

Here you can watch the amount of mailboxes remaining decreasing.

When complete, the ContentIndexState should say Healthy.

Remote Desktop Services: Can’t Remove Dead Server

Posted on 22/03/2018 by PeteLong

KB ID 0001415

Problem

I was doing some RDS work for a client today, and it would seem that at some time in the past their RDS Licensing server had died, it had been replaced, and everything was working OK. But when I was adding roles to the new servers, this kept popping up;

The following server in this deployment are not part of the server pool
1. Server-Name
The servers must be added to the server pool.

I could have ignored the error and finished the job, but things like this remaining ‘unfinished’ really wind me up. So I thought I’d sort it out.

Solution

At first I thought I could just dive into either ADSIEdit or ‘AD Sites And Services’, make a quick change and everything would be fixed. That revealed that the site licence server was set to a server that also didn’t exist! (So I fixed that, still the problem remained).

So if all else fails then use PowerShell right?

[box]Remove-RDServer {Server-FQDN} {ROLE}[/box]

Error: Object Reference not set to an instance of an object

After some research I discovered that the RDS servers are stored in a database, (Windows Internal Database) on the connection broker(s). So you need to download the ‘SQL Management Studio’ software on your connection broker(s). Then ‘Run As’ administrator.

Connect to, “\\.\pipe\MICROSOFT##WID\tsql\query“

Under Databases you will find a database called RDCms >Expand that, and drill down to the tables. Locate rds.server. Press the ‘Query’ button > Right click the rds.server table > List top 1000 rows.

Locate your ‘Dead’ server here you can see mine has an ‘Id’ of 3. Look in the following tables and make sure there are no references to Id 3. (I didn’t have any, my only reference was in the rds.server table.)

rds.RoleRdcb (Connection Broker)
rds.RoleRdls (License Server)
rds.RoleRdsh (Session Host)
rds.RoleRdvh (Virtualisation Host)
rds.RoleRdwa (Web Access Host)

In the bottom of the Query Section enter the following, (as applicable, i.e your column might be ServerId, and your server might be number 123)

[box]use RDCms

delete from rds.server where Id=3[/box]

Press ‘Execute’, Close the SQL Manager, repeat on any remaining ‘Session Brokers’. Have a coffee, then try again, the problem should be resolved.

Mailbox Move Error ‘Active Directory property ‎’homeMDB‎‘ is not writeable’

Posted on 11/07/2017 by PeteLong

KB ID 0001326

Problem

I was migrating some mailboxes form Exchange 2007 to Exchange 2013 today, when once again this happened;

Error: MigrationPermanentException: Active Directory property ‎’homeMDB‎’ is not writeable on recipient ‎’petenetlive.com/Migration/Users/Pete.Long’. –> Active Directory property ‎’homeMDB‎‘ is not writeable on recipient

Solution

I can’t believe there’s no fix for this yet! This is being caused by exactly the same problem as we had back with Exchange 2007 and Exchange 2010 (see link below)

Exchange Mailbox Move Error – (INSUFF_ACCESS_RIGHTS)

So to fix, open ‘Active Directory Users and Computers’ > View > Advanced Features.

Security tab (if you can’t see it, advanced features is not on) > Advanced.

Enable inheritance > Apply.

Yes > OK > OK.

Now retry your mailbox move.

To Find out who else will be affected: Run the following command;

[box]Get-ADuser -Filter {admincount -gt 0} -Properties AdminCount | select name[/box]

VMware – This Virtual Machine Appears To Be In Use

Posted on 17/11/2015 by Migrated

KB ID 0000959

Problem

I wanted to give a copy of a VM to a colleague, so I removed all the snapshots, and cloned one of my test VM’s. When I went to power on the original this happened;

The virtual machine appears to be in use.

If this virtual machine is not in use press the “Take Ownership” button to obtain ownership of it. Otherwise press the “Cancel” button to avoid damaging it.

Configuration file: {path-to-vmx-file}

And when I attempted to ‘Take Ownership’ of the machine, that failed also;

Could not open virtual machine: {path-to-vmx-file}
Taking ownership of this virtual machine failed.
The virtual machine is in use by an application on your host computer.
Configuration file: {path-to-vmx-file}

Solution

I knew it was not in use, as there was only me using my laptop. So I figured VMware had some files ‘locked’. Navigate to the folder that holds the VM’s files, (Note: The path is on the error message above).

Locate any folders that have a .lck extension appended to their name (as above), and move them to another folder. Then attempt to power on your VM.

Windows Update Error – 80243004

Posted on 17/11/2015 by Migrated

KB ID 0000643

Problem

Seen on Server 2008, when attempting to run Windows update.

Code 80243004 Windows Update encountered an unknown error

Solution

Without a doubt the strangest fix I’ve ever seen!

1. Right click the Taskbar > Properties > Taskbar Tab > Customize.

2. Tick the box that says “Always show all icons and notifications on the taksbar” > OK.

3. Retry your Windows updates.

XCOPY – Insufficient Memory

Posted on 17/11/2015 by Migrated

KB ID 0000810

Problem

If I’m migrating client data, I use Xcopy a lot, especially if I want to preserve the permissions. One of the questions I usually ask is “Do any of your users have file names that are very long, i.e. longer than 256 characters?” Because if you are moving a lot of data and it’s been running for a few hours, then suddenly fails saying ‘Insufficient Memory‘, then that’s probably what the problem is.

Solution

1. At the point of failure, you will have successfully moved some data, so you want a solution that just moves the remaining data, Robocopy will do that for you.

Note: For Server 2008, Server 2008 R2, Server 2012, and Windows Vista/7/8, you will already have Robocopy installed, for older clients you will need to install the 2003 Resource Kit.

[box]

Syntax

robocopy "source" "destination" "options"

robocopy S: D:Shared /MIR /SECFIX /SEC

Note:Robocopy by default will retry 1000000 times, and wait 30 seconds each time (if it has a problem). You might want to add /R:3 /W:1 as switches to stop that.

[/box]

/MIR – Mirror the two locations, this will copy the difference IN BOTH DIRECTIONS! (Simply use /E if you don’t want this).

/SECFIX – Checks the Permissions on ALL FILES as it goes through (to make sure)

/SEC – Copies the data with its security ACL’s intact.

Use Robocopy to copy only newer files/folders

Thankfully this is the default behaviour, simply run the same command again.

SQL Sever 2012 Setup Error – Rule ‘Setup Account Privileges’ Failed

Posted on 17/11/2015 by Migrated

KB ID 0000976

Problem

When trying to install SQL Server 2012 (Standard c/w SP2). This popped up;

Rule Check Result
Rule "Setup account privileges" failed.
The account that is running SQL Server Setup does not have one or all of the following rights: the right to back up files and directories, the right to manage auditing and the security log and the right to debug programs. To continue, use an account with both of these rights. For more information, see
http://msdn.microsoft.com/en-us/library/ms813696.aspx,
http://msdn.microsoft.com/en-us/library/ms813959.aspx and
http://msdn.microsoft.com/en-us/library/ms813847.aspx.

As I was logged in as the domain administrator, (yeah I know, but it’s my test network). I was confused.

Solution

1. Well, it turns out I don’t have those rights after all so lets grant them, Windows Key+R > Control admintools {Enter}.

2. Launch ‘Local Security Policy’.

3. Navigate to Security Settings >Local Policies > User Rights Assignment > Back up files and directories > Add in the user that’s being denied > Apply > OK.

4. In the same location locate ‘Manage auditing and security log’ > Add in the user that’s being denied > Apply > OK.

5. In the same location locate ‘Debug programs’ > Add in the user that’s being denied > Apply > OK.

6. Now try again, (remember if you added a group, you will need to log off and back on again.)

SBS – Alert – ‘The following disk has low idle time’

Posted on 17/11/2015 by Migrated

KB ID 0000583

Problem

I got this alert forwarded to me, from a client that I’d put in new hard drives for a few week ago.

Alert:

The following disk has low idle time, which may cause slow response time when reading or writing files to the disk. Disk: {Number} {Drive Letter}: Review the Disk Transfers/sec and % Idle Time counters for the PhysicalDisk performance object. If the Disk Transfers/sec counter is consistently below 150 while the % Idle Time counter remains very low (close to 0), there may be a problem with the disk driver or hardware. If the review shows that the disk is functioning properly, use Task Manager to determine which processes are causing the majority of the disk activity. You can attempt to correct the problem by stopping and then restarting those processes. You can disable this alert or change its threshold by using the Change Alert Notifications task in the Server Management Monitoring and Reporting tasked.

Solution

1. It’s telling me review some counters (Start > Run >Perfmon {Enter}). I added in the counters that it asked me to, and sure enough this disk was getting thrashed with a very high disk latency.

2. While discussing it in the office, a colleague suggested I check the BBWC on the RAID card. Sure enough a quick look at the System Management Homepage shows;

4. The battery has failed on the internal E200i RAID card. The server in question was an HP ML350 (G5). So my first thought was to update the firmware for the RAID card, (If for no other reason than it’s the first thing HP would ask me to do, if I logged a call). This did not resolve the problem, so I logged the call for a replacement (The server is under care pack).

5. After fitting, I left it 24 hours for the battery to charge, and checked it again.

Note: Latency has dropped from 1100 to 70).

Problem

Testing FortiGate LDAPS

Debugging FortiGate LDAPS

Related Articles, References, Credits, or External Links

Problem

Solution

Related Articles, References, Credits, or External Links

Problem

Solution

How To Monitor ‘Exchange Indexing’ Crawling Progress

Related Articles, References, Credits, or External Links

Problem

Solution

Related Articles, References, Credits, or External Links

Problem

Solution

Related Articles, References, Credits, or External Links

Problem

Solution

Related Articles, References, Credits, or External Links

Problem

Solution

Related Articles, References, Credits, or External Links

Problem

Solution

Use Robocopy to copy only newer files/folders

Related Articles, References, Credits, or External Links

Problem

Solution

Related Articles, References, Credits, or External Links

Problem

Solution

Related Articles, References, Credits, or External Links