berk@imt.ru
Last modified May 8 2007
Linux on a modern PC hardware is a very capable router and can achieve the same routing performance as Cisco 72**. But it never gained the same popularity as a router as it did as a server.
One reason is probably administrative: in a big IT department different teams are attending to connectivity and to services. This is natural work division but it has its drawbacks. The networking team would consider Linux routers as a project for the servers team (after all, it is Linux), and the servers team would rightfully think otherwise (it is, after all, networking). Usually both stay polite and mind their own business.
With smaller installations, where a systems administrator and a network administrator are the same person, there is no such problem, and in the SOHO world Linux PC router is a natural choice. This led to the wrong impression that Linux is only good for low-end routers. The examples of Uppsala University or regional network of Pereslavl, Russia, prove the contrary [1],[2].
The second reason is more technical. Dedicated routers boot from flash and work from memory and know nothing of hard disk crashes or filesystem corruption due to unexpected loss of power. This greatly improves reliability. The problem is that all mainstream Linux distributions are targeted for hard disk installations. Specialized mini-distributions for diskless clients or SOHO routers are numerous, but they all diverge too far from mainstream distributions and cannot fully benefit from mainstream development.
My goal was to take a major Linux distribution (I prefer Debian) and make it boot from flash and work from filesystem in memory, but to do it very carefully to change as little as possible. What comes out is not a debian-based distribution but rather a boot wrapper for Debian proper. It is work in progress - use at your own risk.
Have fun, feeback is welcome.
64 to 128MB flash memory, 128MB RAM or more if you run BGP. It can run it with 64MB RAM if you configure /usr to be mounted read-only from flash, see Tuning.
To install flash memory to PC you can use:
As a side note: if you need a Linux distribution for a more space-restricted device, you have other choices. This solution buys you painless upgrades and uniform environment between your servers and your routers, but this is a trade-off. Fortunately affordable RAM and flash are getting bigger faster than debian package management.
Linux can run from 2 MB of flash memory and 8 MB of RAM. Try buildroot or buy a Linksys WRT54G(S) and use OpenWrt or Sveasoft firmware.
Download and unpack this tarball.
Put your CF card or DOM in your second IDE controller (or insert your key in a USB socket) and make two partitions on it, one for /boot and one for the rest of the system. With a 128MB CF card I reserved 20 MB for the boot partition. Make ext2 filesystems on both.
Edit Config, all configurable parameters are documented in this file.
Run "make". You will need wget, debootstrap, gawk, libuclibc-dev and uclibc-toolchain installed.
If you need a different kernel configuration, cd to the kernel build directory (linux/linux-2.6.*), run "make config|menuconfig|xconfig", then rerun "make" at top-level directory. Default kernel configuration is stored in linux/linux.config.
Run "make install".
To test, set the first boot device in your BIOS to your flash disk (or to USB-HDD if you used a USB key) and try booting. Apt-get additional packages, for instance quagga, if you need dynamic routing. Then install your flash device to a production system.
I tested the installation on Debian sarge and on Ubuntu dapper and edgy.
Extensive documentation for Debian is available from www.debian.org.
Only one command is specific to debian router, it was called
wr to make Cisco people feel at home. When you change
a configuration file, all changes happen in memory and will be lost on reboot.
To make the changes permanent you can run wr. This script
uses rsync to write only the files that differ in memory
and on flash. wr -h displays help.
The fact that changes first happen in memory can be turned to advantage. For instance when you reconfigure ssh on a remote router, you can schedule a reboot in 10 minutes, reconfigure ssh and if nothing breaks, cancel the reboot. Else the router will reboot and return to the previous state.
With a default installation the running system will reside in memory, and only the boot partition will be mounted read-only from flash media. You can save about 40 MB of RAM if you pass "usr=flash" to the kernel as command line parameter, then /usr will be mounted read-only from flash instead of being copied to RAM. To test it, press "e" when you see the grub menu and edit the command line. To make it permanent, edit the kernel command line in the file /boot/grub/menu.lst after you have booted.
Regular packages can be installed and updated as usual. Updates will happen
in memory and will be lost on reboot, unless you run wr -a
to commit the changes to flash. You can reconfigure apt to auto-commit the changes,
it is especially recommended if you mounted /usr from flash. Edit /etc/apt.conf and
replace the line:
Post-Invoke {"/usr/local/sbin/dpkg-post-invoke";};
with
Post-Invoke {"/usr/local/sbin/dpkg-post-invoke -w";};
Kernel updates are different. Don't try to install kernel packages from the main Debian distribution, it won't work. You should prepare your own debian kernel package for your router using the same tarball that you used to install the system: unpack it to your workstation, edit Config to reflect the new kernel version, run "make linux", scp the resulting debian package linux/linux-2.6.*_2.6.*_i386.deb to your router and install it there with dpkg.
Now reboot and start pinging the router. You know the thrill if your router is at remote location. If you are at the console, the new kernel will appear as "linux" in the grub menu and the previous kernel will become "linux.old". If grub could not load the new kernel or if new kernel would panic, the system will restart with the old kernel.
This fallback mechanism is mostly implemented as in grub documentation. Grub sets the old kernel as the default target after the next reboot and starts loading the new kernel. Then the new kernel starts the normal boot sequence and then one of the init scripts (/etc/init.d/grub-set-default) checks, what kernel was loaded. If it is indeed the new kernel, this script resets grub default target back to the new kernel.
If you are very paranoid or your router is really far away, save somewhere this script and schedule it to run ten minutes after rebooting with the new kernel:
#!/bin/sh
# reboot with the old kernel
mount /boot -o remount,rw
grub-set-default 1
mount /boot -o remount,ro
reboot
This will save you if the new kernel boots, but you cannot access the system.