Tuesday, March 15, 2011

Debian Diskless Setup and Configuration

Here we are going setup a server that serves over network to pxe clients a diskless debian. We will be using the following:
  • DHCP server: dh1 (see how to install here)
  • TFTP server: tftp1 (IP: 192.168.10.35)
  • NFS server: nfs1 (IP: 192.168.10.30)
  • PXE image location on NFS server: /srv/diskless/c1

DHCP Server

  1. Allow issue ip addresses to pxe clients:
    subnet 192.168.10.0 netmask 255.255.255.0 {
      pool {
        range 192.168.10.101 192.168.10.199;
      }
      option subnet-mask 255.255.255.0;
      option broadcast-address 192.168.10.255;
      option routers gw1.dev.local;
      
      allow bootp;
      allow booting;
      next-server tftp1.dev.local;
      use-host-decl-names on;
      if substring (option vendor-class-identifier, 0, 9) = "PXEClient" {
        filename "pxelinux.0";
      }
    }
    
  2. If you wish dhcp server to serve same ip address to the clients, here are few lines to add. Note that if you need the client to be accessable via its name (c1) you need to add a record to dns server (since the ip address served by dhcp is the same it is fine to add it do dns).
    host c1 { 
      # this is client MAC address
      hardware ethernet 00:1D:77:7B:F8:19; 
      fixed-address 192.168.10.101;
    }
    

TFTP Server

  1. Install the following packages (it is fine to use lxc container if any):
    apt-get -y install syslinux tftpd-hpa
    
  2. TFTP default directory location:
    root@tftp1:~# cat /etc/default/tftpd-hpa | grep TFTP_DIRECTORY
    TFTP_DIRECTORY="/srv/tftp"
    
  3. PXE configuration:
    # Copy files for pxe boot
    cp -r /usr/lib/syslinux/pxelinux.0 /srv/tftp
    cp -r /usr/lib/syslinux/menu.c32 /srv/tftp
    # create configuration folder
    mkdir /srv/tftp/pxelinux.cfg
    
  4. Prepare initramfs to boot over nfs (this step and one that follows needs to be run on machine that has a kernel you are going to serve to your clients, e.g. lxc container doesn't have this so you need to run it on host)
    cp -r /etc/initramfs-tools /etc/initramfs-pxe
    cat /etc/initramfs-tools/initramfs.conf | \
        sed "s/BOOT=local/BOOT=nfs/" > \
        /etc/initramfs-pxe/initramfs.conf
    
  5. Copy kernel and make nfs specific changes to initrd image:
    cp /boot/vmlinuz-`uname -r` /srv/tftp
    mkinitramfs -d /etc/initramfs-pxe -o \
        /srv/tftp/initrd.img-`uname -r` `uname -r`
    
    The above changes gave me following:
    tftp1:~# ls -hl /srv/tftp/
    initrd.img-2.6.32-5-amd64  pxelinux.0  
    pxelinux.cfg  vmlinuz-2.6.32-5-amd64
    
  6. Now it is time configure default pxe configuration (file /srv/tftp/pxelinux.cfg/default):
    DEFAULT menu.c32
    
    PROMPT 0
    TIMEOUT 100 
    NOESCAPE 1
    ALLOWOPTIONS 0
    
    MENU TITLE PXE Network Boot Menu
    
    LABEL c1
        MENU LABEL Debian GNU/Linux, with Linux 2.6.32-5-amd64
        KERNEL vmlinuz-2.6.32-5-amd64
        APPEND root=/dev/nfs quiet panic=0 initrd=initrd.img-2.6.32-5-amd64 nfsroot=192.168.10.30:/srv/diskless/c1 ip=dhcp rw
    
    LABEL local
        MENU LABEL Boot Local Hard Drive
        LOCALBOOT 0
    
    At this point PXE client should be cable to load and boot the kernel image.

NFS Server

NFS server doesn't work in lxc container, so choose host machine.
  1. Install necessary packages:
    apt-get -y install nfs-kernel-server nfs-common
    
  2. Create a root mount point for pxe clients:
    mkdir /srv/diskless/c1
    
  3. Let NFS server know about it (file /etc/exports ):
    /srv/diskless/c1  192.168.10.0/255.255.255.0(rw,sync,no_root_squash,no_subtree_check)
    
  4. Issue the following command so it exported:
    exportfs -a
    
  5. It is recommended to tune the following kernel parameters (file /etc/sysctl.conf):
    # Default OS receive buffer size for all types of
    # connections.
    # net.core.rmem_default = 229376
    net.core.rmem_default = 1048576
    
    # Max OS receive buffer size for all types of 
    # connections.
    # net.core.rmem_max = 131071
    net.core.rmem_max = 1048576
    #
    # Default OS send buffer size for all types of 
    # connections.
    # net.core.wmem_default = 229376
    net.core.wmem_default = 262144
    
    # Max OS send buffer size for all types of 
    # connections.
    # net.core.wmem_max = 131071
    net.core.wmem_max = 262144
    
    Configure kernel parameters at runtime:
    sysctl -f
    

PXE Client

  1. This is run on NFS server since it will serve root fs ro PXE clients:
    apt-get -y install debootstrap
    
  2. Create debian minimal installation:
    debootstrap --arch amd64 testing /srv/diskless/c1
    
  3. Change mount options for the client (file /srv/diskless/c1/etc/fstab):
    proc            /proc         proc    defaults 0 0
    /dev/nfs        /             nfs     defaults 0 0
    none            /tmp          tmpfs   defaults 0 0
    #none            /var/run      tmpfs   defaults 0 0
    #none            /var/lock     tmpfs   defaults 0 0
    none            /var/tmp      tmpfs   defaults 0 0
    none            /media        tmpfs   defaults 0 0
    none            /var/log      tmpfs   defaults 0 0
    
  4. Configure network interface (file /srv/diskless/c1/etc/network/interfaces):
    auto lo
    iface lo inet loopback
    iface eth0 inet dhcp
    
  5. Copy timezone files and kernel modules from host:
    cp /etc/timezone /srv/diskless/c1/etc/timezone
    cp /etc/localtime /srv/diskless/c1/etc/localtime
    cp -r /lib/modules/2.6.32-5-amd64/ /srv/diskless/c1/lib/modules/
    
  6. Here we install few packges (not we chroot to client root file system).
    chroot /srv/diskless/c1
    apt-get update
    apt-get -y install ifupdown locales libui-dialog-perl \
        dialog isc-dhcp-client netbase net-tools iproute vim \
        apt-utils
    
  7. Configure locale
    echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
    locale-gen en_US.UTF-8
    update-locale LANG=en_US.UTF-8
    dpkg-reconfigure -u tzdata
    
  8. If the machine name is served by dhcp server than /etc/hostname must be empty:
    echo "" > /etc/hostname
    
    otherwise give machine name:
    echo "c1" > /etc/hostname
    
  9. Set root user password:
    echo "root:root" | chpasswd
    
  10. Exit client chroot environment.
You should be able now load the minimal debian over network.

Gnome Desktop

That is pretty easy (chroot to /srv/diskless/c1):
tasksel install gnome-desktop --new-install

Troubleshooting

It is strongly recommended remove tracker package since it just kills NFS server IO. Check if you have it installed:
dpkg --list | grep tracker
... and get rid of it:
apt-get remove --purge tracker

9 comments :

  1. hi all i am daniel have some doubts i do every step but my system have a raid 0 installe and i change the path of the node in your case c1 , but in me case it appear a error that the raid dosent mount and this error that say that /dev/nfs dont exist, well for that i ask if you know how i resolve this problem. well i for your anwser thanks :)

    ReplyDelete
  2. Since you changed path to c1, ensure the nfs export is valid, as well as path in pxelinux.cfg/default.

    ReplyDelete
  3. inside dev i dont have a file or path call nfs i dont know why and other thing is why if i have a raid 0 instaled, the c1 not recognize the raid 0. i mount it in the fstab of c1 but this dont configure the raid, i will change the configuration as the same way that you show to test an mount the raid has a sigle path

    ReplyDelete
  4. hi again and sorry for bother you, i already resolved the problem but now I get others errors on the part of the system mounted diskless, says he can not mount sys, proc or the scripts/init-bottom that does not get /dev since it is not mount :s, well i wait here for your help thanks for your time :)

    ReplyDelete
  5. this instructions help me a lot !!

    many Thanks Andriy.

    from Barcelona.(Spain)

    ReplyDelete
  6. Hello Andriy.

    You've got my admiration for this great post! I tried to follow it in a test environment because I am going to boot all my servers from NAS some time in the future but I can't get pass the boot menu.
    In syslog I've got in.tftpd[4777]: tftp: client does not accept options and I don't know what to do. I double checked everything and can't find my mistake.
    I will really appreciate it if you give me some guidance.

    Best Regards,
    Stilyan

    ReplyDelete
  7. Please discrepant my previous post. I've done done it correctly following your instructions, I just have to fix minor errors.
    After all I couldn't boot because of a typo..

    Anyway, thank you very much man! I appreciate your work a lot!

    ReplyDelete
  8. Hi
    thank u for your tutorial ! Everything is ok for me (my nodes boot up!) !
    using deboostrap with the option--include=ssh i am facing this problem : i can not connect to nodes with ssh because it asks for a password which I do not know... And consequently, I can not use openmpi. Any idea to bypass this (small) problem.
    Thanks in advance
    joe

    ReplyDelete