I am in the process of installing a number of Juniper EX2200, EX3200 and EX4200 switches for a client and as part of the setup need to be able to monitor the switches for any alarms  (eg Switch Management interface down or Switch booted from Backup Partition) and have them dealt with accordingly.

Having a look at the SNMP OID tree for the EX switches I came across the following useful table

http://www.oidview.com/mibs/2636/JUNIPER-ALARM-MIB.html

Object Name Object Identifier
jnxAlarms jnxAlarms 1.3.6.1.4.1.2636.3.4
jnxCraftAlarms jnxCraftAlarms 1.3.6.1.4.1.2636.3.4.2
jnxAlarmRelayMode jnxAlarmRelayMode 1.3.6.1.4.1.2636.3.4.2.1
jnxYellowAlarms jnxYellowAlarms 1.3.6.1.4.1.2636.3.4.2.2
jnxYellowAlarmState jnxYellowAlarmState 1.3.6.1.4.1.2636.3.4.2.2.1
jnxYellowAlarmCount jnxYellowAlarmCount 1.3.6.1.4.1.2636.3.4.2.2.2
jnxYellowAlarmLastChange jnxYellowAlarmLastChange 1.3.6.1.4.1.2636.3.4.2.2.3
jnxRedAlarms jnxRedAlarms 1.3.6.1.4.1.2636.3.4.2.3
jnxRedAlarmState jnxRedAlarmState 1.3.6.1.4.1.2636.3.4.2.3.1
jnxRedAlarmCount jnxRedAlarmCount 1.3.6.1.4.1.2636.3.4.2.3.2
jnxRedAlarmLastChange jnxRedAlarmLastChange 1.3.6.1.4.1.2636.3.4.2.3.3

I have used the jnxRedAlarmCount and jnxYellowAlarmCount oid values as basic Opsview SNMP Service Checks to give me an initial overview but in the long term will be looking to combine this into a full service check script that can be used to check a number of different things.

The setup of the Service Check in Opsview is fairly simple and below are screenshots of the config that I have for each service check.

All you need to configure on your hosts is the SNMP community string and you can apply these checks individually or via a Host Template.

Once I performed a reload I could see the following in Opsview for one of my switches:

A bit of inspection showed that the Red Alarm was for the Management Interface being down (but wasnt being used on this switch) and the Yellow alarm was due to not setting a rescue configuration. I cleared the alarms by isuing the following commands

Now when I refresh the checks in Opsview I get an OK state for both checks

Just stated to deploy my first SEP 12.1 implementation for a new client and came across a bug whereby the disk space on the system drive where SEPM had been installed was decreasing rapidly.  Investigation showed that the Endpoint Protection Manager is not configured by default to backup or truncate the log files for its database.

For more information from Symantec on the configuration of the truncate and index rebuild options please review the following KB article: http://www.symantec.com/business/support/index?page=content&id=TECH166658

One other thing that was pointed out by a colleague was that Backup Exec is unable to backup the Database files and you will need to configure SEPM to backup and export the data if you would like to recover the current SEPM configuration in the event of having to restore the server from backup.

To fix this is straightforward but led me to ask the question why Symantec wouldn’t think this needs to be enabled by default for the product.

 

I have spent the last few days trying to understand why a successful one time backup hadn’t flushed the transaction logs on my client’s Exchange 2010 server. We spent a lot of time troubleshooting message queues and looking for a transaction that hadn’t completed as the Backup job had reported successful. Digging a bit deeper into some of the job logs I can see that the one-time backup was doing a COPY – Full database and logs and not a FULL – database and flush committed logs.

Googling this came up with the following technote: http://www.symantec.com/business/support/index?page=content&id=TECH187838 and there is no way that you can change the option to do a full log flush in the one time backup.

I can’t fathom why this wouldn’t be a useful feature of the software to at least have the flush committed logs as a tickbox in the job options for the one-time backup.

CCNA Certification Obtained

Posted by mattywhi | IT

Passed my CCNA yesterday with a score of 894. Next stop should be training for the CCNP and maybe some JNCIS stuff possibly.

When making changes to Juniper EX switches yesterday I wanted to check the changes that I had made to my configuration before committing them. A quick look in the reference manual gave me the following command:

This will show the edited candidate config and pipe that into the compare function and look at the changes to the specified version (rollback 0). I could look at the changes compared to a previous config by replacing 0 with another number in the rollback sequence.

 

Today I sat and passed my JNCIA-FWV to re-certify myself in the eyes of Juniper for another two years. I was in and out in 30 minutes and achieved a 93% pass mark. Work have also booked my CCNA exam for the end of May so hopefully I will soon have that to add to my list of accreditations.

Last month I installed a new Cisco ASA 5510 for a client and came across an issue where traffic was hitting the “inside” interface of the firewall before travelling back out the same interface and into another router on the internal LAN – an issue as reported in this article Cisco ASA Deny TCP (no connection)

The diagram below demonstrates the network setup with PC1 trying to communicate with PC2. When the traffic leaves the MPLS router (RED line) it does not traverse the ASA and the next packet will follow the original route (GREEN then ORANGE lines) to get to PC2

Representing the traffic flow

Long term the resolution is to place the extra routers into their own DMZ networks on the perimeter network but as this didn’t exist at the time I needed to disable the TCP SYN checking for the traffic being routed to the MPLS routers – a process described in this article by Cisco – Configuring TCP State Bypass

First thing we do is create an ACL for all the items we want to bypass the SYN check

Now we create a class map to match the ACL

Then apply this to a policy map

Finally we assign that policy to the inside interface on the firewall

Traffic that hits the inside interface of the firewall that matches the rules on the ACL will not be checked for their tcp state and traffic should now flow.

In the long term it is recommended that this isnt the adopted approach and the firewall is configured to have the traffic traverse through from the inside to a DMZ interface to prevent the issues with the TCP SYN issue

I was playing around with the check_route plugin and noticed a few issues with it not running. In order to get it to work on my Opsview boxes I had to install a new package, change some settings on the traceroute program and then make a patch in the script itself.

First thing you need to do is download the traceroute package if its not already installed

Once installed you will find that the plugin will fail and show the following error:

Googling the first line I found that you have to setuid root for the traceroute binary

Trying the plugin again you get the following error

To get around this you need the plugin to ignore the first line of the output from the traceroute which can be done with the following patch

http://snipt.net/mattywhi/opsview-check_route-diff/

Now the script runs as expected and you get the following output

 

I have been working with the Netscreen, and then Juniper firewall products for the past five years and am still learning new and interesting features they offer. One thing that I have been configuring more and more recently are secondary Internet connections and fail-over between them for clients. This post runs through the steps required to configure an SSG firewall to use track-IP to monitor IP addresses on the Internet and then automatically fail-over and fail-back an Internet connection.

The first thing we need to do is move the interfaces that will contain the Internet connections so each is in their own virtual router. This will allow us to have an active default route for each connection and they can behave independently of each other.

For this example I am using the 192.0.2.0/24 address range for my WAN connections – this was defined by the IETF as a subnet to be used for testing and documentation in RFC 5735. As these interfaces are both public facing I am also going to restrict the management to secure protocols only

Now we need to setup the default routes out of each virtual router so that each connection can communicate with the rest of the Internet

We need to ensure that our internal users are able to route to both the untrust-vr and adsl-vr. This can be done by exporting the default static route from the untrust-vr and adsl-vr

This will import both default routes to the trust-vr and set maintain the preference of the export from the untrust-vr at 20 whilst setting the metric of the adsl-vr export to 140.

Now that our users can connect to the Internet we need to make sure that should there be an issue with the primary internet circuit the backup circuit can be used for Internet access. This is achieved by using track-ip to monitor a number of hosts on the Internet and should they become unreachable shut the interface down.

In this example we are using the IP address of some of the root DNS servers as the addresses the firewall will use to check for a valid Internet connection but they could be any IP addresses that you expect to remain online and will respond to PING requests

This will PING the three addresses every second and will consider the address to have failed when the test has failed 25 times consecutively. Summing these three failures together will hit the weight and threshold limits of 75 needed to shut down the interface.

UPDATE: Since this was written Juniper released newer firmware that allowed you to specify the interface threshold for failure in addition to the Track-IP threshold. This would mean that track-ip would fail at 75 but the interface default was set to 255 for failover, the config above has been amended accordingly to reflect this change in behaviour.

If you want to test the status of the track-ip monitoring you can issue the following commands

and you will be able to see the failure statistics as well as whether the interface is failed or not.

When the interface is shut down the default route no longer becomes valid in the untrust-vr and will be deleted in the trust-vr leaving the export from the adsl-vr active and Internet traffic will continue to function as normal. In the background, the management address on the primary connection will continue to poll the IP addresses configured and when they become available the weight and threshold will be below the failure values, the interface comes back up and the untrust-vr route export re-appaers in the trust-vr.

The only other thing to consider here is inbound services on the backup line such as MX records to permit mail delivery to a MIP or VIP on the secondary circuit

If this is all configured correctly the only things the user should notice is that any websites/services that login and use session data (eg online banking) will need to login after fail-over or fail-back as their existing session will no longer be valid.

The only remaining task is to commit the changes you have made to flash

 

Opsview Labs

Posted by mattywhi | IT, Monitoring

I had an unexpected, but much welcomed, tweet today from the team at Opsview who would like to make use of some of my writing about Opsview on their own Labs blog. I can honestly say that I wasn’t expecting the blog to be read and picked up in this way but I am pleased that I can hopefully reach a few more people with the data appearing on the Opsview Labs blog as well.