FWSM software upgrade


I had a challenge in having to upgrade a pair of resilient production firewalls which were housed in the same chassis with minimal downtime. The Cisco recommended solution is potentially the best to follow. However, I went slightly off piste and came up with a solution that resulted in a sub 15 second loss of service. The FWSM has a multi partitioned flash for allowing an easy means to access the underlying maintenance code or booting different software versions. The key partitions in this scenario were cf:4 and cf:5, these are the standard operating partitions which hold the ASDM image the FOS software image (generally refered to as FLASH:). I wont go into all of the other partitions on this post. The scenario is that we have two FWSM’s located in slot 6 and 7 on a 6509E. The FWSM’s are operating in Active\Passive failover using code version 3.2. Here’s what I did:

  • TFTP live config off for backup:
  • copy run tftp:///configs/live-startup-config
  • Shutdown the secondary FWSM Module:
  • no power enable mod 7
  • Create a new FWSM vlan group and associate vlan 100:
  • firewall vlan-group 9 100
  • Assign the new vlan group to mod 7:
  • firewall module 7 vlan-group 9
  • Configure the switch to boot module 7 into cf:5 rather than cf:4:
  • boot device module 7 cf:5
  • power on the secondary FWSM module:
  • power enable mod 7
  • Access the FWSM from the switch console:
  • sess slot 7 proc 1
  • Clear FWSM configuration in cf:5 and reboot for a factory reset:
  • write erase and then reload without saving.
  • Configure FWSM with basic settings:
  • int vlan 100
  • ip address
  • namif INSIDE
  • no shut
  • icmp permit any inside
The FWSM only has an active interface in vlan 100 and the configured vlans on the 6509E is currently only vlan 100. In this state it is not affecting the live FWSM and or any operational changes are only to that device. It would be worthwhile noting that vlan 100 is not already an active vlan used anywhere in the configuration previously. Setup a TFTP server in VLAN 100 on the 6509E on a spare ip address in the range. You should be able to ping from the FWSM to the TFTP server and from the TFTP server to the FWSM.
  • TFTP the previously backed up live config onto the FWSM:
  • x
  • TFTP the required software image onto the FWSM:
  • x
  • TFTP the required ASDM version onto the FWSM:
  • x
  • reboot the FWSM
  • x
  • Remotely connect to the FWSM and very that the configuration and software has been installed:
  • sess slot 7 proc 1
  • show version
  • Save the configuration (this is required often when code versions are updated)
  • write me
  • Remotely connect to the FWSM and very that the configuration and software has been installed:
  • sess slot 7 proc 1

Once verified the FWSM is up then issuing a show interface will show that all of the logical vlan’s are down down. This is where I altered my plan and shutdown the primary FWSM module in slot 6. I then allocated the normal production firewall vlan-group to module 7. During this whole period I had a ping running against one of the servers in the data centre. The loss of 13 pings was not to dramatic as module 7 came in to full operation and module 6 was taken offline. I will add that make no mistake in my method above, this will clear all active sessions and connections and will require the communication between two nodes to be re-initiated. All said and done 13 seconds was far less than the 3 minutes awaiting for both FWSM’s to reboot.

The same steps were carried out on the primary FWSM and booting into the alternative flash and then carrying out an upgrade. Once completed the FWSM was brought up and before re-allocating the firewall vlan-group back to module 6 the configuration was updated to identify it as the failover secondary unit. The configuration was saved and the FWSM in slot 6 shutdown. The vlan-group’s were applied to module 6 and the FWSM was brought online.

Job Done…

Netflow locally on a router


To enable Netflow locally on a router use in global configuration:

  • # ip flow-top-talkers
  • # top 25
  • # sort-by bytes

On the interface

  • # ip flow ingress
  • # ip flow egress

To see the statistics from the above use:

  • # show ip flow top-talkers

KRON to backup config files


This was a new feature that came in with the later IOS versions where you can use KRON a’la CRON to run tasks on a scheduled basis. In smaller environments it negates the need for things such as Solarwinds NCM and or RANCID.

Define the Kron policy

# kron policy-list Backup
# cli show run | redirect tftp://<TFTP Server IP>/2801/config

Define the Kron Occurrence

# Kron occurrence Backup at 17:00 Wed recurring
# policy-list Backup

RANCID Ubuntu Install Stage2



Using the CVS Web Interface we can enhance Rancid with a pretty neat interface. To do this we need to edit with vi: /etc/cvsweb/cvsweb.conf go to line 59 to make a few changes. Here is an example from the configuration file. You will need to change the second row to match your group name (BACKUPS line) and point this to where rancid store the CVS. After that is completed you need to save the file (:wq!).

@CVSrepositories = (
‘local’ => [‘Local Repository’, ‘/var/lib/cvs’],
‘BACKUPS’ => [‘AUTOMATED BACKUPS’,’/var/lib/rancid/CVS’]

In addition to the above change to secure the main CVSROOT you should also select the 1 variable on the hide CVSROOT option in the same file. At this point there is the option to tidy up the HTML to suit your purpose. The next step is to get the CVS icons etc into the /var/www folder we create a symbolic link for it.
o #sudo ln -s /usr/share/cvsweb /var/www/cvsweb

After that you should be able to access your Rancid-CVS on

http://”IP of the Rancid Machine”/cgi-bin/cvsweb/


We only then need to create a simple cron job to run rancid-run frequently such as every day or 12 hours depending on the size of your network or requirements
o #crontab -e -u rancid

run ranid-run script every day at 00:30 and removed the old logs the first day of every month at 00:15

30 00 * * * /home/rancid/bin/rancid-run

The crontab command will update the /var/spool/cron/crontabs/rancid file.


Finally to ensure that all packages are upto date at the point of install issue the following commands.

o #sudo apt-get update
o #sudo apt-get upgrade

RANCID Ubuntu Install Stage1


This is a brain dump of the RANCID install process that I’ve put together and fully tested using Ubuntu 11.04


o Ubuntu-Server 11.04 “ubuntu-11.04-server-i386.iso”
o username: notwork
o password: notworking


text preceeded by # is a command to be issued

text enclosed in a box ~~~~~~~~~~~# is the editing of a file

I personally use vi to edit text/config files remembering that “ESC” then “:wq!” will save the file (write,quit and force) and “i” will allow you to insert text.



Use apt-get to install the programs

Install Ubuntu following a standard build and at package selection only choose ssh server. Once the build has completed login as user notwork with a password of notworking (change username/password as you see fit). Once logged int the console enter the following:

o #sudo apt-get install apache2
o #sudo apt-get install expect
o #sudo apt-get install cvs
o #sudo apt-get install cvsweb
o #sudo apt-get install checkinstall
o #sudo apt-get install rancid-core rancid-util build-essential

This will have installed Rancid in /etc/rancid.


We need to configure the /etc/rancid/rancid.conf file to create groups of devices. At least one group needs to be configured and adding multiple groups means that the names must be separated with a space. The example below will create the groups where all the device configurations will be stored:





The next step is to setup the CVS configuration with the following command:

o #sudo su -c /var/lib/rancid/bin/rancid-cvs -s /bin/bash -l rancid

There should be a new folder added in the /var/lib/rancid directory, this has been named after the group(s) you created earlier in 2). Navigate into this folder and open up the router.db with vi. Here you will specify what router/switches you want to add to your CVS. For example:





After this has been completed we then need to tell rancid how to access the devices. This is done by creating a .cloginrc script file in the /var/lib/rancid folder with the following commands:

#sudo touch .cloginrc
#sudo vi .cloginrc

Use the following example which will attempt to telnet to all devices with a username of user and a password of password. This file can be updated to meet many of your standard requirements or use multiple methods for varying examples.

add method * telnet
add user * user
add password * password


We need to secure the .cloginrc file by changing the owner and file permission.

#sudo chown rancid:rancid /var/lib/rancid/.cloginrc
#sudo chmod 640 /var/lib/rancid/.cloginrc


Lastly we should be done and should see if rancid is working correctly by running the process as the rancid user with the following command

#sudo -u rancid -H /usr/bin/rancid-run

You can check if the command was successful by checking the logs in /var/log/rancid/switches. It should say in the log message “All routers successfully completed.”.

See the Stage 2 post to install/configure the CVS web interface and Stage 2.5 to use rancid to update device configurations either in bulk or singularly.

RANCID Ubuntu clearing SSH keys


One for Mr Grumpy…

In the likely event that you upgrade a FWSM or possibly some other similar hardware. It seems that during the upgrade process it kills your SSH configuration and you need to regenerate the keys. This can be achieved with a simple “crypto key generate rsa”. However, if you run a linux management system then you may need to delete and reset the key on the linux host for the remote node your attempting to access..

sudo ssh-keygen -f “/var/lib/<user>/.ssh/known_hosts” -R <ip-address>

where <user> is the user such as root on the local linux host and <ip-address> is the address of the host your connecting to. An example to remove the key from the local user is:

ssh-keygen -f “/home/network/.ssh/known_hosts” -R






Say What! “really awesome new Cisco confIg differ”


RANCID is one of those things that once you have it up and running you wonder how you ever survived without it. In years of implementation and management of network infrastructure the means to easily identify adhoc changes that magically occur are every network engineers nightmare. Mix in a little bit of TACACS+ and you have an ideal solution for offering accountability and management of your infrastructure. The great thing that RANCID does is to chuck the output of numerous commands into a sub-version process and highlights the changes which occur. So, I’ve put together a step by step process of installing RANCID onto a certain Linux distribution, as well as dropping some brain dump elements and some tit-bits identifed by Mr Grumpy …