I see this get asked in forums A LOT, typically the poster has another problem they are trying to fix, someone has asked them to debug the problem and they cant see any debug output.
Solution
Firstly you need to understand what logging is, and how debugging fits within it. (Bear with me, this is good knowledge to have).
The firewall saves logs in syslog format, and there are 8 Levels of logs, the one with the MOST information is called ‘debugging’ (or severity 7 in Syslog world)
0=Emergencies
1=Alert
2=Critical
3=Errors
4=Warnings
5=Notifications
6=Informational
7=Debugging
So if you are debugging, then all you are doing is looking at syslog output thats severity 7. The ASA can send these logs to an internal memory buffer, and external Syslog server, or to the screen, either the console (via rollover cable) or the monitor (SSH/Telnet session, or what router types, call the virtual terminal lines).
Fine but I cant see anything doofus, that’s why I’m here!
OK, now you understand how it all works, you should understand when you see the commands, why it wasn’t working!
Issue a ‘show log’ command;
What does this tell us? Well mose importantly it tells us logging in ON.
[box]Syslog logging: enabled[/box]
If it were disabled then you turn it on with;
[box]logging on[/box]
The next piece of pertinent information is.
[box]Timestamp Logging: Disabled[/box]
While not critical, logs are much easier to interpret when they are stamped with the correct time! I’m in the UK so this is the command I would use (Note: I’m enabling NTP Time sync, this can take a while to synchronise);
[box]
clock timezone GMT 0
clock summer-time BST recurring last Sun Mar 1:00 last Sun Oct 2:00
!
ntp server 130.88.203.12 source outside
!
logging timestamp
[/box]
Sending Debug Output to the Screen
As mentioned above, you can send output to the console or the monitor;
Send Debug to SSH/Telnet Session
[box]logging monitor debugging
terminal monitor[/box]
Note: To disable, the command is ‘terminal no monitor‘ NOT ‘no terminal monitor’ (Thanks Cisco!)
Sending Debug Output to the Console (Serial Connection)
Send Debug to SSH/Telnet Session
[box]logging console debugging[/box]
Note: To stop it, set it back to ‘warnings’ (the default).
[box]logging console warnings[/box]
Sending Debug Output to the Internal Log (Buffer)
This is easier, as you can filter the results for particular IP addresses/ports/usernames etc, which is handy if there are pages and pages to look though, and they are not scrolling past you yes, faster than you can read them!
And all was well, then a week later I got an email…
One of our teachers is doing a project with MATHS and ICT involving bitcoin. Basically, he has something called BITCOIN CORE WALLET installed and it used to work with the old Firewall.
I’ve installed it on my work laptop and taken it home on my Internet connection & it works fine.
BUT, when I bring it back into school, its failing. When I bypass the Firewall, it also works – so I guess IPS/AMP is blocking something.
The software seems to start and then download/sync “stuff” for bitcoin.
In school it tries and then says “NO block source available“
Google seems to hint towards network issues.
If definitely did work, as the teacher has screen grabs of it working.
Any ideas what could be blocking this ?
Now Bitcoin uses a series of ledgers that update each other around the world, (to make it resilient). So if the FirePOWER was the culprit, then it was either identifying it as a bot, or I had a rule specifically blocking Bitcoin?
Solution
Note: Bitcoin does need TCP port 8883 open, but that didn’t seem to be the problem.
Thankfully on the monitoring tab, as soon as I logged in, the answer was staring me in the face, (I had to change the time frame to last 30 days first).
Not only does it confirm the FirePOWER IDS blocked it, but it also told me which ‘Rule’ it had matched, (PUA-OTHER Bitcoin outbound request attempt). PUA stands for Probably Unwanted Application, in case you were wondering. Edit your IDS policy, and search for ‘Bitcoin’.
I’ll leave the Malware rules alone, but I’ll allow both the PUA-Bitcoin rules, (i.e. set them to ‘Disable’).
Then don’t forget, you need to deploy the new FirePOWER policy and ensure that your access control policy says it’s up to date on all devices, before you test again. If you’re unsure how to do that, see the link I posted above.
Related Articles, References, Credits, or External Links
This was driving me nuts on my Windows 7 x64 Laptop.
Log Name: System
Source: Schannel
Event ID: 36888
Task Category: None
Level: Error
User: SYSTEM
Description:
The following fatal alert was generated: 10. The internal error state is 10.
I was getting a dozen of these an hour!
Solution
This error is caused (from what I can gather) by an error in certificate negotiation, your machine is trying to initiate communications with another machine/server using a certificate and TLS and the process is producing this error TLS1_ALERT_UNEXPECTED_MESSAGE (10).
1. If your browser is the cause of the problem, then simply open Internet Options > Advanced > Untick all the TLS options > Apply.
2. However this DID NOT WORK for me, so something is programmatically chatting from my laptop using TLS. The bottom line is, this problem is probably not even on your machine, so I’m simply going to disable SCHANNEL logging.
Note: If your Error does NOT say “The following fatal alert was generated: 10. The internal error state is 10“. then I would suggest NOT doing this.
3. In the search run box type regedit and navigate to the following key;
[box]
HKEY_LOCAL_MACHINE > System > CurrentControlSet > Control > SecurityProviders > SCHANNEL
[/box]
Change the EventLogging value from 1 to 0 (that’s a zero).
Related Articles, References, Credits, or External Links
c. On the Edit menu, click Add Value, and then add the following registry value:
Value name: ExtensionDebugLevel Data type: DWORD Value data: 2
d. Quit Registry Editor.
2. Refresh the policy settings to reproduce the failure. To refresh the policy settings, type the following at the command prompt, and then press ENTER:
System Access configuration was completed successfully.
Audit/Log configuration was completed successfully.
Kerberos Policy configuration was completed successfully.
Configure machinesoftwaremicrosoftdriver signingpolicy. Undo value for the undefined group policy setting <machinesoftwaremicrosoftdriver signingpolicy> wasn’t reset successfully (1627). Undo value was not removed. Error 1627: Function failed during execution. Error configuring machinesoftwaremicrosoftdriver signingpolicy. Configure machinesystemcurrentcontrolsetcontrollsalmcompatibilitylevel. There is already an undo value for group policy setting <machinesystemcurrentcontrolsetcontrollsalmcompatibilitylevel>. Configure machinesystemcurrentcontrolsetserviceslanmanserverparametersenablesecuritysignature. There is already an undo value for group policy setting <machinesystemcurrentcontrolsetserviceslanmanserverparametersenablesecuritysignature>. Configure machinesystemcurrentcontrolsetserviceslanmanserverparametersrequiresecuritysignature. There is already an undo value for group policy setting <machinesystemcurrentcontrolsetserviceslanmanserverparametersrequiresecuritysignature>. Configure machinesystemcurrentcontrolsetservicesnetlogonparametersrequiresignorseal. There is already an undo value for group policy setting <machinesystemcurrentcontrolsetservicesnetlogonparametersrequiresignorseal>. Configure machinesystemcurrentcontrolsetservicesntdsparametersldapserverintegrity. There is already an undo value for group policy setting <machinesystemcurrentcontrolsetservicesntdsparametersldapserverintegrity>.
Configuration of Registry Values was completed with one or more errors.
Before SP1 you would have to install a copy of Outlook on the Exchange server and use a PowerShell command that looks like this (once you had granted Import/Export rights);
[box]
BEFORE Exchange 2010 SP1
Get-Mailbox | Import-Mailbox –PSTFolderPath C:Folder_Containing_PST_Files
[/box]
However try that after SP1 and you will get an error message, saying that Import-Mailbox is not a commandlet. That’s because now you no longer use this command you use “New-MailboxImportRequest”, and you also no longer need Outlook installing on the server.
“This mailbox exceeded the maximum number of large items that were specified for this request. (Fatal error TooManyLargeItemsPermanentException has occurred.)”
To fix that error you can use the ‘-LargeItemLimit 200 -AcceptLargeDataLoss’ flags (which sounds alarming, but I have not seen it break anything in the last ten years).
Importing PST Files From the Exchange Admin Center
You can import PST files directly in the management GUI > Recipients > Mailboxes > Select the target mailbox > {Ellipses} > Import PST.
Enter the path to the .PST file > Next.
Select the target mailbox > Next.
Optional: Select a user to be emailed an export report.
Note: To view progress and troubleshoot failures, you will have to revert to PowerShell.
How To BULK Import .PST Files
Note: To BULK Import successfully, the .PST file MUST have the same name as the alias of the target mailbox.
Commands Required
1. Once you have created a “Universal Security Group” in this example called “Mailbox_Import” then assign the mailbox import/export roles with the following command;
FailureType : TooManyBadItemsPermanentException Message : Error: This mailbox exceeded the maximum number of corrupted items that were specified for this move request.
This happens when it sees items in the mailbox it does not like, or considers corrupt. To get round this problem, import the .pst file on its own with the following command;
When looking at a router, switch or firewall running config, it will usually display a page at a time, you can page down with the space bar, or line down with the Enter/Return key.
Normally that’s fine, but what if you want to capture (take a quick backup,) of the config?
If you do that, and page down you get a copy of the config that looks like this;
–More–
Yes, you can delete them, but in a big config that can take time, how about making the config scroll right to the end without the breaks/pauses.
Solution
Cisco ASA Disable Paging
On a firewall that’s done with a pager command, normally a firewall config will display 25 lines at a time, to get it to scroll straight to the end set the pager length to zero.
[box]
Type help or '?' for a list of available commands.
Petes-ASA> enable
Password:*********
Petes-ASA# configure terminal
Petes-ASA(config)# pager 0
Petes-ASA(config)#
[/box]
Tip: If you want to take a copy of a firewall config it will blank, (replace with asterisks) the VPN shared secrets and failover keys, you can suppress that from happening, and show the hidden values with the following command;
[box]
Petes-ASA(config)# more system:running-config
[/box]
To return it back to pausing every 25 lines and giving the <— More —> prompt again.
[box]
Petes-ASA(config)# pager 25
[/box]
Cisco Router / Switch IOS Terminal Length
On IOS the default is 24 lines at a time (show terminal will tell you). You can change this by changing the terminal length. Note: This is NOT a global configuration command.
[box]
Petes-Router#terminal length 0
[/box]
To reset it, and get the –More– prompt back again;
[box]
Petes-Router#terminal length 24
[/box]
Related Articles, References, Credits, or External Links
Cisco NetFlow lets you export information about traffic flow, it was originally written for the router IOS, but is now available for Cisco ASA, which uses NSEL (Note ASA uses NetFlow version 9 {newest at time of writing})
Note: NetFlow can not give you “Live” data, but it can show you what has happened over a period of time, and remember like any other “Logging” this will have an adverse affect on the firewall (depending on how busy it is).
Setting this up is a two step process, the firewall is configured as the NetFlow “Exporter”, then you install an application that accepts and collates that information, that is the NetFlow “Collector”.
Solution
1. Log into your firewall via CLI and enter enable mode, then enter configure terminal mode.
[box]
User Access Verification
Password:********
Type help or '?' for a list of available commands.
PetesASA> enable
Password: ********
PetesASA# conf t
PetesASA(config)#
[/box]
1. We haven’t set one up yet, but we need to let the firewall know the IP address that the NetFlow “Collector” will be running on, in this case I’m going to use 10.254.254.253. (Note: the port number on the end is unimportant).
4. I’m going to apply this with the default global-policy, because most of you will have one, (Though I notice every 8.2(1) 5505 I’ve put in recently does NOT have one so check).
[box]
PetesASA(config)# policy-map global_policy
PetesASA(config-pmap)# class class-default
PetesASA(config-pmap-c)# flow-export event-type all destination 10.254.254.234
PetesASA(config-pmap-c)# exit
[/box]
6. If you haven’t got a global policy, this will not apply until you have applied the global_policy globally, this is done with a service-policy command, check to see if you already have this command in your config, or simply execute the command and the firewall and will tell you, like so….
Note: If it does not error then it was NOT applied 🙂
[box]
PetesASA(config)#
PetesASA(config)# service-policy global_policy global
WARNING: Policy map global_policy is already configured as a service policy
PetesASA(config)#
[/box]
7. Don’t forget the save the config with a “write mem” command.
8. Now go to the machine you want to install your NetFlow collector software on, I prefer Plixer Scrutinizer because its free and its easy to set up. Connect to it via the built in web site (username admin password admin) > Click Status > Expand Ungrouped > Expand the firewall > Flow templates > Pick one.
9. There’s your throughput 🙂
Related Articles, References, Credits, or External Links
If you look after a firewall, sooner or later something will fail, and the blame (rightly or wrongly), will be leveled at the firewall. I came back from holiday this week to find a client had got a problem with secure POP email. The problem had been fixed (temporarily) by dropping the affected users into a group, and opening all ports. As this had fixed the problem then it’s fair to say that the ASA was the root cause of the problem.
So I was asked to take a look and open the correct ports and lock the firewall back down again.
Solution
Step 1 – Setting up logging on the ASA
I’m going to do some real time testing, so the internal buffer on the ASA will hold enough logs for me, if you have an intermittent problem you might want to setup an external syslog server. I’m going to set the log buffer size, and the logging level, and finally turn logging on.
[box]User Access Verification
Password:
Type help or ‘?’ for a list of available commands.
PetesASA> enable
Password: *******
PetesASA# conf t
PetesASA(config) logg buffer-size 4096
PetesASA(config)# logg buffered 7
PetesASA(config)# logg on[/box]
Step 2 – Attempt communication
At this point I got the client to attempt connection to the secure POP server, then had a look at the logs. I could view the whole log with ‘show logg’, but I filtered it down just to include traffic to and from this client (192.168.1.2).
Note: The ports being used are highlighted in red, (YES I know that these are the ports required for secure POP, but your application could be using anything!)
[box]PetesASA(config)# show logg | inc 192.168.1.2
There are a few ways of doing this. I just created some network objects, then if any other hosts need secure POP, I can simply add them to the object group.
WARNING: This assumes you DON’T have an outbound traffic access list. If you DO replace the word ‘outbound’ with the name of yours. Also remember as soon as you allow traffic like this all other traffic gets blocked!
Step 4 – Disable Logging
Simply prefix your earlier command with the word ‘no’.
[box]PetesASA(config)# no logg on[/box]
Related Articles, References, Credits, or External Links
Yeah, it’s funny because it’s true! The article title might not sound like the most professional approach, but when the ‘Well it’s not working now’ finger gets pointed at the ‘firewall guy/girl’, they need to ascertain two things;
1. Is the problem actually the firewall, if not then help the frustrated party track down the actual problem.
2. If your problem IS the firewall, fix it!
I’m just coming out of a major network greenfield site build, all the individual technologies that have been getting planned and designed are now starting to come online and require comms though the firewall solution that I’ve been working on. So my days are pretty much filled with conversations like this;
Consultant/Engineer: Pete I need some ports opening on the firewall.
Me: OK let me know the IP addresses, host-names, ports, protocols etc, and I’ll open them for you. I then open the requested ports/protocols. Consultant/Engineer: You know those ports you opened? They don’t work.
At this point one of the following has occurred;
1. I’ve made an error, (it happens I’m human), I might have entered the wrong information, or not applied an ACL, or put the rule on the wrong firewall. Always assume you have done something wrong, until you are 100% sure that’s not the case.
2. The person who asked for traffic to be allowed, asked for the wrong thing, either they didn’t RTFM, or someone has given them the wrong IP addresses, or because they are human too, they’ve made a mistake.
3. The traffics not even getting to the firewall, because either it’s getting blocked before it gets to you, or there is a routing problem stopping the traffic hitting the firewall. (Remember routing works by Unicorns and Magic, so routing people are not to be trusted!)
4. The traffic needs some kind of special inspection to work through the firewall i.e. ICMP, FTP, or PPTP etc.
5. Some annoying bug in the ASA code is stopping you, which either requires a lot of Internet and forum searching or a call to TAC to confirm.
If I’ve forgotten another reason – feel free to contact me. (Link at the bottom of the page).
Solution
Step 1: Make sure you are not blocking the Traffic
Packet tracer is your friend! Use it to simulate traffic going though the firewall, and the firewall will tell you what it will do with that traffic. I prefer to use command line, but you can also run packet tracer graphically in the ASDM.
Xml: (Optional) Displays the trace capture in XML format.
Example
Below I’m checking that an internal host (10.254.254.5) can get access to a public web server (123.123.123.123) via http (TCP port 80). Note: As mentioned above I just picked a random source port (1024).
OK, so if packet-tracer shows the firewall is not blocking the traffic. Then either there’s other ports we don’t know about that may need opening, or the traffic is not getting to the firewall. Normally at this point I’d test to see if the traffic is getting to the firewall. To do that I would do a packet capture.
To demonstrate, below someone has requested that we open https from Server A on our LAN, to an Internet server Server B.
Above the traffic is not getting to the firewall as there’s a problem between Server A and the Firewall, either something is blocking the traffic downstream, or Server A cannot route traffic to the firewall.
Below we can see traffic hitting the firewall, in fact 10.0.0.1 sends out three packets on TCP port 443 (https). What we CANNOT SEE is any traffic coming back, in this case Server B is not replying to us, either its down or it cannot route traffic back to us.
Now the port(s) we want to allow, we can see are actually working, so if theres still a problem, theres probably another port / protocol that’s being blocked. To find out we need to enable logging and see if any packets are being denied.
Try the connection again, then view the log, (here I’m filtering it on 10.0.0.1, as the log can be quite sizable);
[box]Petes-ASA(config)# show logg | incl 10.0.0.1
%ASA-7-609001: Built local-host inside:10.0.0.1
%ASA-6-302013: Built outbound TCP connection 15 for outside:123.123.123.123/443 (123.123.123.123/443) to inside:10.0.0.1/1070 (10.0.0.1/1070) %ASA-4-106023: Deny tcp src inside:10.0.0.1/1073 dst outside:123.123.123.123/21 by access-group “outbound” [0x0, 0x0]
%ASA-4-106023: Deny tcp src inside:10.0.0.1/1073 dst outside:123.123.123.123/21 by access-group “outbound” [0x0, 0x0]
%ASA-4-106023: Deny tcp src inside:10.0.0.1/1073 dst outside:123.123.123.123/21 by access-group “outbound” [0x0, 0x0]
%ASA-6-302014: Teardown TCP connection 15 for outside:123.123.123.123/443 to inside:10.0.0.1/1070 duration 0:00:30 bytes 1420 TCP FINs
%ASA-7-609002: Teardown local-host inside:10.0.0.1 duration 0:00:30
Petes-ASA(config)# [/box]
As we can see traffic is being denied and it’s on TCP port 21 (That’s FTP if your interested). So let’s open that port, and try again;
[box]Petes-ASA(config)# show logg | incl 10.0.0.1
%ASA-5-111008: User ‘enable_15’ executed the ‘access-list outbound extended permit tcp host 10.0.0.1 host 123.123.123.123 eq 21’ command.
%ASA-5-111010: User ‘enable_15’, running ‘CLI’ from IP 0.0.0.0, executed ‘access-list outbound extended permit tcp host 10.0.0.1 host 123.123.123.123 eq 21’
%ASA-7-609001: Built local-host inside:10.0.0.1
%ASA-6-302013: Built outbound TCP connection 16 for outside:123.123.123.123/443 (123.123.123.123/443) to inside:10.0.0.1/1077 (10.0.0.1/1077)
%ASA-6-302013: Built outbound TCP connection 17 for outside:123.123.123.123/21 (123.123.123.123/21) to inside:10.0.0.1/1080 (10.0.0.1/1080)
%ASA-6-302014: Teardown TCP connection 16 for outside:123.123.123.123/443 to inside:10.0.0.1/1077 duration 0:00:30 bytes 1420 TCP FINs
Petes-ASA(config)# [/box]
And we are working!
If we have got this far and you are still not working, then check the traffic you are trying to send does not need any special inspection enabling. Or the port number you are using may have been reserved for a particular type of traffic (like this).
Failing that, upgrade the ASA, then open a TAC call.
Related Articles, References, Credits, or External Links
With NTP, there will be two things you want to do, 1) Allow a device behind the ASA to take its time from a public NTP server, and 2) Set the ASA to take its system time from a public NTP sever (for accurate date stanps on the logs, and for time critical things like Kerberos authentication.)
Solution
Allow internal host(s) to get system time though the firewall.
1. Connect to the ASA, go to “enable mode”, then to “Configure terminal mode”
[box]
User Access Verification
Password:
Type help or '?' for a list of available commands.
PetesASA> enable
Password: ********
PetesASA# configure Terminal
PetesASA(config)#
[/box]
2. To rules are being applied to traffic going OUT through the firewall, run a “show run access-group” command.
[box]
PetesASA(config)# show run access-group
Sample Output
access-group outbound in interface inside
access-group inbound in interface outside
[/box]
Note: If it returns nothing then outbound traffic is NOT being filtered, and NTP should work anyway, but in the example above I can see the traffic that is going IN the inside interface (That’s traffic going out if you think about it!) Is being filtered by an access list called ‘outbound’ (Because I give the ACL’s sensible names, yours could be called anything!)
3. To allow ALL hosts use the word any, for a specific host use the keyword host.
[box]
Allow all hosts access to NTP
PetesASA(config)# access-list outbound permit udp any any eq 123
Allow one host (192.168.1.1)
to NTP
PetesASA(config)# access-list outbound permit udp host 10.254.254.1 any eq 123
[/box]
4. Finally save the updated config.
[box]
PetesASA# write mem
Building configuration...
Cryptochecksum: 79745c0a 509726e5 b2c66028 021fdc7d
7424 bytes copied in 1.710 secs (7424 bytes/sec)
[OK]
PetesASA#
[/box]
Set the ASA to get its System Time from an External NTP Source
1. Connect to the ASA, go to “enable mode”, then to “Configure terminal mode”
[box]
User Access Verification
Password:
Type help or '?' for a list of available commands.
PetesASA> enable
Password: ********
PetesASA# configure Terminal
PetesASA(config)#
[/box]
2. The IP address I’m using is in the UK if you want one more local look here.
[box]
PetesASA(config)# ntp server 130.88.212.143 source outside
[/box]
3. To check on its status, simply execute a “show ntp status” command. BUT it will take a few minutes to synchronise, until it does you will see;
[box]
PetesASA(config)# show ntp status
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 99.9984 Hz, actual freq is 99.9984 Hz, precision is 2**6
reference time is d36a01de.60ad92ea (13:04:30.377 UTC Fri May 25 2012)
clock offset is 3414265.0854 msec, root delay is 26.09 msec
root dispersion is 3430186.81 msec, peer dispersion is 16000.00 msec
PetesASA(config)#
[/box]
When it is finally synchronised it will say;
[box]
PetesASA(config)# show ntp status
Clock is synchronized, stratum 3, reference is 130.88.212.143
nominal freq is 99.9984 Hz, actual freq is 99.9984 Hz, precision is 2**6
reference time is d36a0f74.a34d5dde (14:02:28.637 UTC Fri May 25 2012)
clock offset is -9.1688 msec, root delay is 25.91 msec
root dispersion is 15915.95 msec, peer dispersion is 15890.63 msec
PetesASA(config)#
[/box]
4. Finally save the updated config.
[box]
PetesASA# write mem
Building configuration...
Cryptochecksum: 79745c0a 509726e5 b2c66028 021fdc7d
7424 bytes copied in 1.710 secs (7424 bytes/sec)
[OK]
PetesASA#
[/box]
Related Articles, References, Credits, or External Links
