VMware ‘Disable DelayedAck’ Does Not Work?

KB ID 0001525

Problem

I’ve got a client that’s been having some performance issues with their VMs. Their storage vendor, (EMC) said that as a result of finding this in the logs;

[box]

B       02/28/19 09:50:53.953 scsitarg          117000e [INFO] System: iSCSI Logout Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x01200801
B       02/28/19 09:50:53.969 scsitarg          117000e [INFO] System: iSCSI Logout Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x01200801
B       02/28/19 09:51:16.413 Health              608fe [WARN] User: Host ESXi-01.petenetlive.com does not have any initiators logged into the storage system.
A       02/28/19 10:04:25.968 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x00000000 [Target]
B       02/28/19 10:04:26.034 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.200.161 Name=...-ec-21 Target Data: Port=2 Flags=0x00002002 Info=0x00000000
A       02/28/19 10:04:31.996 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x00000000 [Target]
B       02/28/19 10:04:32.055 scsitarg          117000d [INFO] System: iSCSI Login Initiator Data: IP=192.168.201.161 Name=...-ec-21 Target Data: Port=3 Flags=0x00002002 Info=0x00000000
B       02/28/19 10:04:57.438 Health              608fc [INFO] User: Host ESXi-01.petenetlive.com is operating normally.
Host Host ESXi-01.petenetlive.com is accessing lun Datastore_3 as HLU 3, After the initiators for this host start logging in/logging,  unit attention update events will be logged as the paths to the luns have changed this is expected
2019/02/28-09:50:41.607527 ~~~~     7F3C92369703      std:TCD:   Unit Attention update from 0000001A to 0001030D for LUN 0x3.
2019/02/28-10:02:55.860669 ~~~~     7FE476E61702      std:TCD:   Unit Attention update from 00010149 to 00010157 for LUN 0x3.

[/box]

We should disable DelayedAck and they kindly gave me the VMware KB that outlined the procedure.

Solution

The procedure outlined (for VMware 6.x) is to put the host in maintenance mode, then edit the properties of the iSCSI controller(s), untick the DelayedAck options, reboot the Host, and everything will be peachy. However, even though (post reboot) everything looks good in the the vSphere Web console. If you look on the host you may find something like this;

[box]

vmkiscsid --dump-db | grep Delayed

[/box]

DelayedAck = ‘1’ means ENABLED, DelayedAck = ‘0’ means DISABLED

So half my iSCSI entries in the iSCSI database still have DelayedAck ENABLED?

Some Internet searching told me this was quite common, and that the best way to ‘fix‘ it was to, disable the iSCSI initiator, remove the iSCSI database, reboot and then setup iSCSI again;

[box]

cd /etc/vmware/vmkiscsid
esxcfg-swiscsi -d
rm -f vmkiscsid.db
reboot

[/box]

Which is fine IF YOU ARE USING A SOFTWARE iSCSI INITIATOR, I however was not, I had 2x dedicated hardware iSCSI HBAs on each host!

After many hours of messing about and trial and error, it became clear, I had to do things in a certain order, or DelayedAck would simply just be enabled whether I liked it or not. 🙁

Disable DelayedAck With Hardware iSCSI NICs / HBAs

MAKE SURE THE HOST IS IN MAINTENANCE MODE FIRST

Then take a note of your iSCSI setup, Port Groups, VMKernel Ports, and Physical NICs, you are going to delete the iSCSI database in a minute, and you will need to ‘rebind’ the VMKernel Ports and add the iSCSI targets back in again.

Manually remove your iSCSI target(s) for ALL the iSCSI NIC/HBA’s

Below if you re-run the command, vmkiscsid –dump-db | grep Delayed you will see there’s still some entries in the database with DelayedAck enabled! So unlike above (see example for software iSCSI) we are going to remove the iSCSI database, only here we don’t need to disable the software iSCSI initiator (because we are not using one!) Finally reboot the host.

[box]

cd /etc/vmware/vmkiscsid
rm -f vmkiscsid.db
reboot

[/box]

When the host is back online ADD in the Network Port Binding for the appropriate VMkernel adaptor.

Like so;

DON’T RESCAN THE CONTROLLER AS PROMPTED TO DO SO!

On the Advanced Settings of EACH hardware iSCSI NIC/HBA > Edit > UNTICK ‘DelayedAck’.

Double check they are both still unticked (I’ve seen them re-tick themselves for no discernible reason!) Then rescan the controller(s).

Target > Add.

Re-add the iSCSI target back in, (that you took note of above).

Select the Target > Advanced > Untick the DelayedAck option (Note: This time it’s not inherited). Repeat for any additional iSCSI targets.

When they are all added, rescan the storage controllers again.

Finally recheck all the database entries are set to DISABLED.

[box]

vmkiscsid --dump-db | grep Delayed

[/box]

Related Articles, References, Credits, or External Links

Thanks to Russell and Iain for their patience while I worked all that out!

GFI MailArchiver

Deploying GFI MailArchiver with Exchange 2010, Using the Outlook Connector, Importing and Exporting Data.

KB ID 0000666

Problem

I’ve been involved with Exchange for a long time, either being directly responsible for it, or supporting others that do. And like all systems it would run a lot smoother if you kept users away from it! But unfortunately you can’t, so your system is probably full of users “Deleted Items” and “Sent Items” and a copy of every email that some users have ever received “Because I might need them”. And for years I’ve been saying “It’s a messaging system not a file storage system” and lecturing users about tidying up their inbox. Yes you can setup retention policies but most people don’t.

Even if you do have conscientious users, some of them click “Yes” when Outlook asks them “Would you like to Auto-Archive your old messages now?” Then you have PST files all over the place getting corrupted and not getting backed up.

In addition, just as techs like me are shouting at users to delete things, businesses are now finding that they need to keep ALL their digital messaging for things like Sarbanes-Oxley and Freedom of Information enquiries. Yes I’m sure you back up your Exchange server but what if you needed to produce a message thread about a particular project that was two years ago in the middle of a month?

Archiving

Is the process of taking a copy of mail messages as they pass through Exchange and copying them off to an external location, in this case a database. The advantage of doing this is that database can be on cheaper midline storage and (potentially) keeps the Exchange database sizes down. With Exchange this is done by setting up Journaling (that’s the process of sending a copy of all [or specified] mail to a journal users mailbox). MailArchiver then takes this mail and put it into its database.

Advantages

1. Performance: Getting all the ‘Old’ Data out of the exchange databases makes them smaller, and more efficient.

2. Elimination of PST files: There’s a big list of reasons why I don’t like PST files. If you’ve ever had a user lose email because of them, then as far as they’re concerned its YOUR fault. There are tools to import PST file into the database, (and to export as well).

3. Disaster Recovery: How nice would it be, the next time users email message ‘disappears’ (Users never delete emails they just disappear), they could restore it themselves, from within Outlook. the Outlook Mail Connector software will do this. Also your backup window for Exchange will be a lot smaller.

4. Compliance: Saving a copy of all messages in a database has the advantage, that it’s searchable, and you can export the data you find in a format that you can import straight back into Exchange or send to someone.

5. Investigation: MailArchiver provides some powerful search and reporting tools. If your HR department needed to see who said what to who about projectXYZ then using conventional methods would be a nightmare.

Solution

GFI MailArchiver 2012 Installation and Configuration

Enabling Journaling in Exchange 2010

Note: GFI Mail Archive setup can do this for you but I prefer to do things manually.

1. Launch the Exchange System Management Console > Recipient Configuration > Mailbox > New Mailbox.

2. User Mailbox > Give it a name and set the password > Don’t create an archive > New > Finish.

3. Option 1: Standard Journaling can be enabled on a mailbox store > Organisational Configuration > Mailbox > Locate the store> Properties.

4. Maintenance tab > Tick Journal recipient then browse for the user you created > Apply OK.

5. Option 2: (Note: Requires an Exchange Enterprise CAL) This is set up using a Journaling rule > Organizational Configuration > Hub Transport > Journal Rules > New Journal Rule.

6. Give the rule a name > Browse for the user you created earlier > Set the scope (in most cases you will want global) > New > Finish.

7. To test it’s working send and email then log on as your ‘Journal’ user and make sure you have a copy in the inbox.

Installing GFI MailArchiver 2012

8. Before installing the GFI MailArchiver server needs the Exchange MAPI Client and Collaboration Data Objects 1.2.1 (Unless you are installing it on the Exchanges server).

9. Also (though not essential) I prefer to disable IE Enhanced Security Configuration, so IE wont get upset with the management console.

10. You can add the URL to trusted sites if you want, but I’m not a fan of IE ESC anyway so I simply disable it.

11. Now run the GFI MailArchiver installer.

12. Now this I DO LIKE, why can’t Exchange do this! These are the server pre-requisites, rather than just falling over and giving you an error (I’m looking at you Windows Exchange Developers!) It offers to do the hard work for you. It takes a while though, best go for a coffee at this point > Next > Select whether you want to check for a newer version > Accept the EULA > Set the destination folder > Next.

13. Here the default setting, WARNING if you already have web services on this server (or even UPS software using port 80), have a common sense check.

Note: If you are not sure, the following command will tell you if port 80 (http) is in use;

[box]netstat -aon | find “:80″[/box]

14. Select Install > Finish.

Configuring MailArchiver 2012

15. The GFI MailArchiver management console will launch > Configure.

16. Next.

17. Enter your licence key and select verify licence key > Next.

18. Next.

19. In this example I’m using the Firebird database, in a production environment you should be using SQL server > Next.

20. Change the paths if required > Enter some domain credentials> Next.

21. Next

22. Next

23. Next

24. I’m selecting Auto > Next

25. As I set the user up myself I’m choosing Manual > Next.

26. Exchange Web Services seems to be less problematic > Enter your Journal user account > Next.

27. Next.

28. Finish.

29. We are now up and configured.

GFI MailArchiver – Outlook Connector, Import and Export

GFI MailArchiver – Using the Outlook Connector

30. Firstly you need to enable ‘Mailbox Folder Structure Retrieval’ on the GFI server > Launch the MailArchiver Management console.

31. Mailbox Folder Structure Retrieval.

32. Change Settings.

33. Enter a user account to connect to Exchange Web Services with, here I’m using my domain admin, whichever account you use needs to have administrative access on the Exchange and the GFI server. It also (domain admin included) needs the following Powershell commands running on the Exchange server before it will work, (change the user name to match your own);

[box]New-ManagementScope -name “MAUMPolling” -recipientrestrictionfilter {recipienttype -eq “UserMailbox”}</p> <p>New-ManagementRoleAssignment -name “MAUMPollingRA” -role:applicationimpersonation -user “administrator@petenetlive.net” -customrecipientwritescope “MAUMpolling”[/box]

34. Next.

35. Finish.

Installing the Outlook Connector on a Client

36. I’ve already got a client PC setup with Outlook 2010.

37. I’m just connecting to the GFI servers management URL from the client and downloading the Outlook Connector.

Note: The versions are for Outlook 32 bit and Outlook 64 bit, even if your Windows client is 64 bit you may still be running 32 bit Outlook/Office. If you get it wrong it will tell you your version of Office is not supported.

38. Install the client software, accept the defaults, all you need to specify is the URL of the GFI MailArchiver server.

39. Now when your user opens Outlook, they get an additional mailbox called “GFI MailArchiver Mailbox” that carries a copy of ALL the users mail (Note: Not the mail from before the product was installed, to import that see the import section below, and choose ‘Import for Exchange mailbox’. In addition each user now has a MailArchiver tool bar from which they can search for their mails.

Note: If a user ‘loses’ a mail they can simply drag a copy from their GFI mailbox to their live inbox.

GFI MailArchiver – Importing Data from PST Files

Note: To do this the machine needs to have Outlook installed on it (and NOT Outlook 64 bit!)

40. Launch the GFI MailArchiver Import and Export Tool.

41. Import from .pst files.

42. Add PST file > Browse to your PST file > Open.

43. Select the folder(s) required > Next

44. Select a date range > Select the user that will own the imported data > Next.

Note: I’d rather have an “Import Everything” option!

45. When complete > Finish.

46. Now that user will have the imported mail as well (Note: The default view is “Emails in last 30 days” so don’t panic if you don’t see it all).

GFI MailArchiver – Exporting Data to PST File

47. Launch the GFI MailArchiver Import and Export Tool.

48. Enter the URL of the GFI server > Verify > Ensure it says OK > Next.

49. Select what you want to export (I’m going to search for email containing particular words) > Next.

50. Type in your search text > Find > Next.

51. Here you can restore the mail to a mailbox, export it to .msg or .eml format, or my old nemesis .pst files, I’ll choose the latter > Next.

52. And there’s my .pst file, ready to be sent out to satisfy my freedom of Information enquiry.

 

Related Articles, References, Credits, or External Links

NA

Why is my Computer So Slow?

KB ID 0000290 

Problem

“Why is my computer running slowly” “My computer is not as fast as it was” “What’s making it run so slow?” “It just Crawls” “I have to turn in on, then go and make a cup of coffee”

These are questions every techy hates “Why is my PC running so slowly?”, my reply is always “How long have you had it?” If its a business machine most users inherit them from someone else, who got it from someone else. If its a home machine its probably had months/years of software being installed and uninstalled.

The truth is, even a machine that’s being looked after and updated will slow down over time, the PC that ran like a rocket on Christmas morning may well run like a dog the following Christmas.

So what can you do? The best option (If its older than 9 months) is to back up your data and reinstall Windows, If it’s quite young, or the thought of installing Windows scares the hell out of you, there are a few steps you can take to recapture your machines “Just installed” speed.

Solution

Step 1 What’s Starting and Running in the Background?

A lot of software developers write their programs so they respond better by having them ticking over in the background, that’s great for that application, but over time more and more programs are taking valuable RAM and processor time.

Lets get that lot turned off. Log in as an administrator and do the following.

1. Start > Run > msconfig {enter}

2. The “System Configuration” window will open > Select the Startup Tab.

3. Untick EVERYTHING except your anti virus software (and possibly any anti-spy anti-malware programs). > Apply > OK > Reboot > Post reboot you will get a pop up message to say you have changed the settings.

Step 2 Do You Have a Virus?

Having decent AV software is a must! And then, just as importantly you need to keep it updated. Ask 100 Techies what’s the best Anti virus and you will get about 50 different answers. The following get my vote,

  • McAfee
  • Kaspersky
  • ESET/Nod32
  • Trend

All those are NOT Free, If you are wanting a free solution I’d recommend,

  • Microsoft
  • Security Essentials
  • AVG

1. Install some AV software > Update it and perform a full system scan.

 

Step 3 Is your PC full of Adware/Spyware/Malware

1. I’ve already written on this subject here. Use both the tools I’ve outlined in that article (Malwarebytes and SuperAntispyware) and make sure both packages report clean before proceeding.

2. If you are especially paranoid and you may have run some software that is not from the most “legal” of sources, I’d suggest running a rootkit scanner as well this one is free.

Step 4 Remove any old unwanted Programs

1. Click Start > Run > appwiz.cpl {enter}

2. Select any unused programs and uninstall them.

 

Step 5 Update your machine with the latest fixes and updates

Many times I’ve heard people tell me they don’t enable updates in case Microsoft bring out an update that breaks something, If we are talking about a production server that your business depends on, then fair enough, you need need a strategy for testing updates beforehand. But for client PC’s I’ve seen more problems because people have NOT got fixes installed for known problems that I’ve ever seen updates break something. I can count on my two hands how many times an update has caused a problem, and I’ve worked on thousands of systems.

To enable “Microsoft Update” (Ref)

1. Go here

2. Click Start Now > On the Review the license agreement page > Continue > Install.

Note: You may not be prompted to install the Microsoft Update software if this software is already available on the computer. After you successfully install the software, a Microsoft Update shortcut is added to the All Programs menu on the Start menu.

 

Step 6 Check Your system Drivers

Most pieces of hardware on your PC have a small piece of software (Driver) that lets them talk to the operating system, fist make sure non of yours are missing or failing,

1. Click Start > Run > devmgmt.msc {enter}

2. When it opens you will see and problem devices are usually marked with an exclamation mark.

Device Manager Errors in Windows XP Device Manager Errors in Windows 7

3. Locate the drivers for the hardware in question and install it.

4. Then use DriverMax to see if there are any new drivers available for your hardware. Install and run the software > Select Driver Downloads and Info > Check Online for Driver updates > You will need to create an account with DriverMax > Follow the instructions.

 

Step 7 Sort Out Your Hard Drive

1. If you open “My Computer” it should show you how much free space there is on your drive, If you are running low delete some data or move it elsewhere (Burn it to DVD or put on external storage).

2. Then to make sure all the files on your drive have been stored sensible by the operating system download and run De-Fraggler.

 

Step 8 Clean the Registry

1. There a loads of tools that you can use for this, In the past I’ve used Regseeker, but the most user friendly and easiest to use is CCleaner. Install and run it, then select the Registry tab > Scan for issues.

2. When it’s finished its run through, click “Fix Selected Issues” Answer Yes to backup the registry first > Save the backup file where you can find it. (If anything breaks afterwards simply double click this file and answer Yes to add/merge it into the registry).

<p<3. Then click “Fix All Selected Issues” > Close.

(Note: At this point you can select the “Cleaner” tab and perform some other routine cleanup tasks).

Step 9 Tidy Your Desktop

Specifically shortcuts, every shortcut on your desktop is a small piece of code that points somewhere else, every time your desktop is “Re-Drawn” those links are checked to see if they are OK. If you must have 100 shortcuts, then put all your shortcuts in one folder then have just one shortcut to that folder.

Shortcuts are usually indicated with an arrow

 

Step 10 Turn Down Windows Settings

1. All the fancy Windows shadows and aero settings take processor Power and RAM > Click Start > run > Sysdm.cpl {enter}

2. Select the Advanced Tab> In the Performance Section Click Settings > Select “Adjust for Best Performance” > Apply > OK.

Step 11 (Windows XP Only!)

1. Turn off Indexing (The indexing service speeds up searches) > Start >My Computer > Right Click C: > Properties > Untick “Allow Indexing Service to index this disk for fast file searching” > OK. (Repeat for any further hard drives you have).

Step 12 Install more Physical RAM in your Machine

If you have done all the above and it’s still running slow then simply consider adding more RAM, go to Crucial.com to see what memory upgrade your system will support.

Note: If anyone has anything to add, or thinks I’ve made a glaring omission, click here.

Related Articles, References, Credits, or External Links

NA