This is how to setup a linux deployment system using pxe and saltstack. These directions are for a sandboxed virtual environment but are directly applicable to a production environment.

Router

I create a virtual machine with two interfaces so that our environment is sandboxed, mostly because of the dhcp server we need to run.

Memory 512 MB
CPUs 1 core
Hard disk 1 8 GB
Network adapter 1 VM Network
Network adapter 2 Internal

After the install, update and upgrade

apt-get update
apt-get upgrade

Make the inside interface static and give it an address

vi /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface (outside)
auto ens160
iface ens160 inet dhcp

# The secondary network interface (inside)
auto ens192
iface ens192 inet static
address 10.20.30.1
netmask 255.255.0.0

Reboot

reboot

Configure kernel to forward packets

vi /etc/sysctl.conf

# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

Add iptable rules to forward packets

iptables -t nat -A POSTROUTING -o ens160 -j MASQUERADE
iptables -A FORWARD -i ens192 -o ens160 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i ens160 -o ens192 -j ACCEPT

Make the ip tables rules persistent across reboots

apt-get install iptables-persistent

Now to save the iptable rules

netfilter-persistent save
netfilter-persistent reload

Other

sysctl -p
apt-get install netstat-nat

Main Server

I’m going to combine the deployment services onto one system. It’s a basic Ubuntu 16.04 Server install and I’m calling it “server”.

Memory 512 MB
CPUs 1 core
Hard disk 1 8 GB
Network adapter 1 Internal Net

DNS

Install BIND

Let’s install bind9 to have a nameserver running

apt-get install bind9

Edit /etc/bind/named.conf.options to get caching to work and add this

forwarders {
   8.8.8.8;
   8.8.4.4;
};

Restart bind

systemctl restart bind

Install dnsutils

apt-get install dnsutils

Test the caching

nslookup
server localhost
google.com

Edit /etc/network/interfaces and change the nameserver to localhost

# The primary network interface
auto ens160
iface ens160 inet static
       address 10.20.30.10
       netmask 255.255.255.0
       network 10.20.30.0
       broadcast 10.20.30.255
       gateway 10.20.30.1
       # dns-* options are implemented by the resolvconf package, if installed
       dns-nameservers 127.0.0.1

Reboot

reboot

Back on the Router

I like to ssh to my router and then ssh to the clients on the inside network instead of working on the vSphere console so I’d like internal clients resolve on the router. In order to do this we need to override the nameserver that the dhcp client gets, this is done in the file /etc/resolvconf/resolv.conf.d/head

vi /etc/resolvconf/resolv.conf.d/head

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN 

nameserver 10.20.30.10
search thunderhouse.com

The warning in the file is talking about /etc/resolv.conf not this file.

Restart networking

systemctl restart networking

Note: another good command to know is

systemctl status networking

Test

root@router:~# nslookup google.com
Server:         10.20.30.10
Address:        10.20.30.10#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.4.110

Setup Internal DNS

Let’s set up the primary and reverse zones back on our server for the internal network. The computers will name themselves based on the reverse lookup. To add a DNS Forward and Reverse resolution to bind9, edit /etc/bind/named.conf.local

vi /etc/bind/named.conf.local

zone "thunderhouse.com" {
       type master;
       file "/etc/bind/db.thunderhouse.com";
}

zone "0.30.20.10.in-addr.arpa" {
       type master;
       notify no;
       file "/etc/bind/db.10";
}

Now the file /etc/bind/db.thunderhouse.com will have the details for resolving hostname to IP address for this domain/zone, and the file /etc/bind/db.10 will have the details for resolving IP address to hostname. Now we will add the details which is necessary for forward resolution into /etc/bind/db.thunderhouse.com

First, copy /etc/bind/db.local to /etc/bind/db.thunderhouse.com

cp /etc/bind/db.local /etc/bind/db.thunderhouse.com

Next, edit the /etc/bind/db.thunderhouse.com and replace the following.

In the line which has SOA: localhost. – This is the FQDN of the server in charge for this domain. I’ve installed bind9 in 10.20.30.10, whose hostname is “server”. So replace the “localhost.” with “server.thunderhouse.com.”. Make sure it end’s with a dot(.).

In the line which has SOA: root.localhost. – This is the E-Mail address of the person who is responsible for this server. Use dot(.) instead of @. I’ve replaced with tom.localhost.

In the line which has NS: localhost. - This is defining the Name server for the domain (NS). We have to change this to the fully qualified domain name of the name server. Change it to “server.thunderhouse.com.”. Make sure you have a “.” at the end.

Once the changes are done, the /etc/bind/db.thunderhouse.com file will look like the following:

;
; BIND data file
;
$TTL    604800
@       IN      SOA     server.thunderhouse.com. tom.localhost. (
                              5         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      server.thunderhouse.com.
thunderhouse.com        IN      MX      10      mail.thunderhouse.com.
router  IN      A       10.20.30.1
server  IN      A       10.20.30.10
mail    IN      A       10.20.30.10
ns      IN      CNAME   10.20.30.10
ldap    IN      A       10.20.30.11
www     IN      A       10.20.30.12
print   IN      A       10.20.30.15
ad01    IN      A       10.20.30.21
joomla  IN      A       10.20.30.22
kali    IN      A       10.20.30.23
workstation01   IN      A       10.20.30.50

And the reverse zone will look like this:
;
; BIND reverse data file
;
$TTL    604800
@       IN      SOA     server.thunderhouse.com. tom.localhost. (
                              7         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      server.thunderhosue.com.
1       IN      PTR     router.thunderhouse.com.
10      IN      PTR     server.thunderhouse.com.
11      IN      PTR     ldap.thunderhouse.com.
12      IN      PTR     www.thunderhouse.com.
15      IN      PTR     print.thunderhouse.com.
21      IN      PTR     ad01.thunderhouse.com.
22      IN      PTR     joomla.thunderhouse.com.
23      IN      PTR     kali.thunderhouse.com.
50      IN      PTR     workstation01.thunderhouse.com.

Reload dns and check out systemctl status and /var/log/syslog for any errors

systemctl restart bind9
systemctl status bind9
tail -100 /var/log/syslog

DHCP

Install a dhcp server

We’re going to install the isc dhcp server on the internal network we’ve created.

apt-get install isc-dhcp-server

Edit dhcp server config

vi /etc/dhcp/dhcpd.conf

subnet 10.20.30.0 netmask 255.255.255.0 {
  range 10.20.30.40 10.20.30.100;
  option routers 10.20.30.1;
}

Restart the dhcp server

systemctl restart isc-dhcp-server

Check it with

systemctl status isc-dhcp-server

This is just a simple configuration to get us going.

iPXE configuration in DHCP

Let’s add this to the dhcp configuration, above the subnet declaration.

# option definitions common to all supported networks...
option domain-name "thunderhouse.com";
option domain-name-servers 10.20.30.10;

# iPXE poop
option space ipxe;
option ipxe-encap-opts code 175 = encapsulate ipxe;
option ipxe.priority code 1 = signed integer 8;
option ipxe.keep-san code 8 = unsigned integer 8;
option ipxe.skip-san-boot code 9 = unsigned integer 8;
option ipxe.syslogs code 85 = string;
option ipxe.cert code 91 = string;
option ipxe.privkey code 92 = string;
option ipxe.crosscert code 93 = string;
option ipxe.no-pxedhcp code 176 = unsigned integer 8;
option ipxe.bus-id code 177 = string;
option ipxe.bios-drive code 189 = unsigned integer 8;
option ipxe.username code 190 = string;
option ipxe.password code 191 = string;
option ipxe.reverse-username code 192 = string;
option ipxe.reverse-password code 193 = string;
option ipxe.version code 235 = string;
option iscsi-initiator-iqn code 203 = string;
# Feature indicators
option ipxe.pxeext code 16 = unsigned integer 8;
option ipxe.iscsi code 17 = unsigned integer 8;
option ipxe.aoe code 18 = unsigned integer 8;
option ipxe.http code 19 = unsigned integer 8;
option ipxe.https code 20 = unsigned integer 8;
option ipxe.tftp code 21 = unsigned integer 8;
option ipxe.ftp code 22 = unsigned integer 8;
option ipxe.dns code 23 = unsigned integer 8;
option ipxe.bzimage code 24 = unsigned integer 8;
option ipxe.multiboot code 25 = unsigned integer 8;
option ipxe.slam code 26 = unsigned integer 8;
option ipxe.srp code 27 = unsigned integer 8;
option ipxe.nbi code 32 = unsigned integer 8;
option ipxe.pxe code 33 = unsigned integer 8;
option ipxe.elf code 34 = unsigned integer 8;
option ipxe.comboot code 35 = unsigned integer 8;
option ipxe.efi code 36 = unsigned integer 8;
option ipxe.fcoe code 37 = unsigned integer 8;
option ipxe.vlan code 38 = unsigned integer 8;
option ipxe.menu code 39 = unsigned integer 8;
option ipxe.sdi code 40 = unsigned integer 8;
option ipxe.nfs code 41 = unsigned integer 8;

# pxelinux poop
option space pxelinux;
option pxelinux.magic code 208 = string;
option pxelinux.configfile code 209 = text;
option pxelinux.pathprefix code 210 = text;
option pxelinux.reboottime code 211 = unsigned integer 32;

if exists user-class and option user-class = "iPXE" {
       filename "http://server.thunderhouse.com/ipxe/boot.ipxe";
} else {
       filename "undionly.kpxe";
}

This if statement will cause the DHCP server to first tell clients to download iPXE. Once iPXE starts up and does another DHCP request, it will be told the actual location of the configuration file to download. Without this, we would end up with a continuous loop of iPXE downloading itself.

Create the undionly.kpxe image

The undionly.kpxe image is a PXE image that keeps UNDI loaded and unloads PXE. This is for clients which don’t natively support iPXE, which is pretty much everyone.

We need to build the undionly.kpxe image. First, install the dependencies.

apt-get install build-essential
apt-get install liblzma-dev

clone the image source

git clone git://git.ipxe.org/ipxe.git

Let’s make the image

cd ipxe/src
make

tftp

We need a tftp server to host the iPXE image we just created

Install tftpd-hpa

apt-get install tftpd-hpa

Now move the iPXE image into place

cp ~/ipxe/src/bin/undionly.kpxe /var/lib/tftpboot

Check the status

systemctl status tftpd-hpa

Web

Install Apache

We need a web server to serve the ipxe files, the preseed file and the install service.

apt-get install apache2
systemctl status apache2

Salt

Install the salt-master

We’ll install the salt master from apt.

apt-get install salt-master
systemctl status salt-master

Create the salt directories

mkdir -p /srv/salt
mkdir -p /srv/formulas

Create the autosign.conf file

vi /etc/salt/autosign.conf

Make it look like this:

www.thunderhouse.com
print.thunderhouse.com
ldap.thunderhouse.com
www.thunderhouse.com
WORKSTATION[0-9]{2}.THUNDERHOUSE.COM
workstation[0-9]{2}.thunderhouse.com

Files

iPXE Files

We’ll use apache’s default location for serving web files, /var/www/html

boot.ipxe

#!ipxe

# Global variables used by all other iPXE scripts
chain --autofree boot.ipxe.cfg ||

# Boot <boot-url>/menu.ipxe script if all other options have been exhausted
chain --replace --autofree ${menu-url} ||

boot.ipxe.cfg

#!ipxe

## OPTIONAL: Base URL used to resolve most other resources
## Should always end with a slash
set boot-url http://server.thunderhouse.com

# REQUIRED: Absolute URL to the menu script, used by boot.ipxe
# and commonly used at the end of simple override scripts
# in ${boot-dir}.
set menu-url ${boot-url}/menu.ipxe

# where we put our configs
#set config-dir ${boot-url}/configs
set config-dir ${boot-url}

# fedora bits
set fedora-mirror http://mirror.mit.edu/fedora/linux/releases
set fedora-release 23
set fedora-next 24

#Ubuntu bits 
set ubuntu-mirror http://mirrors.mit.edu/ubuntu-releases/
set ubuntu-release 16.04 

# memtest bits
# note: the plus (+) doesn't work in the url
#set memtest-latest ${config-dir}/memtest/memtest86plus-5.01.iso

# Some menu defaults
set menu-timeout 5000
set submenu-timeout ${menu-timeout}
isset ${menu-default} || set menu-default exit

# Figure out if client is 64-bit capable
cpuid --ext 29 && set arch x64 || set arch x86
cpuid --ext 29 && set archl amd64 || set archl i386

menu.ipxe

#!ipxe

# Variables are specified in boot.ipxe.cfg

###################### MAIN MENU ####################################

:start
menu iPXE boot menu
item --key u ubuntu             Boot Ubuntu ${ubuntu-release} Installer
item
# item --key d menu-diag          Diagnostics tools...
item
item reboot                     Reboot computer
item --key x exit               Exit iPXE and continue BIOS boot
choose --timeout ${menu-timeout} --default ${menu-default} selected || goto cancel
set menu-timeout 0
goto ${selected}

:shell
echo Type 'exit' to get the back to the menu
shell
set menu-timeout 0
set submenu-timeout 0
goto start

:cancel
echo You cancelled the menu, dropping you to a shell
goto shell

:failed
echo Booting failed, dropping to shell
goto shell

:reboot
reboot

:exit
exit

:config
config
goto start

:back
set submenu-timeout 0
clear submenu-default
goto start


############ MAIN MENU ITEMS ############

:ubuntu 
#chain --autofree --replace ${config-dir}/ubuntu/${ubuntu-release}/install.ipxe || goto failed
chain --autofree --replace ubuntu.ipxe || goto failed

# :menu-diag
# chain --autofree --replace ${boot-url}/menu.diag.ipxe

ubuntu.ipxe

#!ipxe
dhcp

# Variables are specified in boot.ipxe.cfg
set base-url http://mirrors.mit.edu/ubuntu/dists/xenial/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/
kernel ${base-url}/linux
initrd ${base-url}/initrd.gz 
imgargs linux auto=true url=http://server.thunderhouse.com/preseed.cfg hostname=${hostname} domain=thunderhouse.com
boot

Debian Preseed

The example preseed file is located here: wget https://help.ubuntu.com/16.04/installation-guide/example-preseed.txt However you should use mine, the example doesn’t have everything, do a diff if you want to see the difference.

To generate the password hash, install the mkpasswd command which is in the whois package:

apt-get install whois
mkpasswd -m sha-512

preseed.cfg

The hashed password is Passsword1
d-i debian-installer/locale string en_US
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/xkb-keymap select us
d-i keyboard-configuration/layoutcode string us
d-i netcfg/choose_interface select auto
d-i netcfg/get_hostname string unassigned-hostname
d-i netcfg/get_domain string unassigned-domain
d-i netcfg/wireless_wep string
d-i mirror/country string manual
d-i mirror/http/hostname string archive.ubuntu.com
d-i mirror/http/directory string /ubuntu
d-i mirror/http/proxy string
d-i passwd/root-login boolean true
d-i passwd/make-user boolean false
d-i passwd/root-password-crypted password $6$46V.E/.7e2hpmE$4JQSRGhVrrb/HthkQ27WWUlAROz/1Sm9iDfRwbh2V24xYG7OsxlgWnpTqitPxzn67Sa1KtiGOoUKkU6M/NvQ70
d-i user-setup/encrypt-home boolean false
d-i clock-setup/utc boolean true
d-i time/zone string US/Eastern
d-i clock-setup/ntp boolean true
d-i partman-auto/method string regular
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto/choose_recipe select atomic
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-md/confirm boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
tasksel tasksel/first multiselect ubuntu-server
d-i pkgsel/include string wget
d-i pkgsel/update-policy select none
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
d-i finish-install/reboot_in_progress note
d-i preseed/late_command string sed -i 's/^GRUB_HIDDEN_TIMEOUT=0/GRUB_HIDDEN_TIMEOUT=5/' /etc/default/grub; \
        in-target sed -i 's/^GRUB_HIDDEN_TIMEOUT_QUIET=true/GRUB_HIDDEN_TIMEOUT_QUIET=false/' /etc/default/grub; \
        in-target update-grub; \
        in-target mkdir /usr/share/mitmath; \
        in-target wget -O /usr/share/mitmath/ubuntu-install http://server.thunderhouse.com/ubuntu-install; \
        in-target chmod 755 /usr/share/mitmath/ubuntu-install; \
        in-target wget -O /lib/systemd/system/ubuntu-install.service http://server.thunderhouse.com/ubuntu-install.service; \
        in-target systemctl enable ubuntu-install.service; \
        in-target systemctl start ubuntu-install.service; \
        in-target sed -i '1 i\Please wait while system finishes configuration...' /etc/issue

Ubuntu Install Service

ubuntu-install

#!/bin/bash

salt_master="server.thunderhouse.com"

# show a message on the splash screen with our progress
# restart splash screen if the process is gone
message () {
  pgrep plymouthd || plymouthd && plymouth show-splash
  plymouth message --text="This workstation is being configured. Please wait, and do not reboot... ${1}..."
}

#sed '1 i\ Please wait while system finishes configuration...' /etc/issue

# can't do anything here without network
count=0
until [ $(ping -c 1 server.thunderhouse.com > /dev/null 2>&1 ; echo $?) = "0" ] ||
      [ $count = "30" ]
do
        # message "waiting for network"
        echo "waiting for network" 1> /dev/tty1
        sleep 2
        let count=$count+1
done

if [ $count = "10" ]; then
        echo "where's my network?  dropping out."
        exit 1
fi

# start SSH for debugging
systemctl start ssh

# make sure system time is correct
# message "setting correct system time"
echo "setting correct system time" 1> /dev/tty1
systemctl start chronyd
chronyc waitsync

# install salt
#message "installing salt"
echo "installing salt" 1> /dev/tty1
#yum -qy install salt-minion
apt-get --yes -q  install python-software-properties
apt-add-repository  ppa:saltstack/salt -y 
apt-get --yes -q update
apt-get install --yes -q  salt-minion

mkdir /etc/salt/minion.d

echo "master: ${salt_master}" > /etc/salt/minion.d/server.thunderhouse.com.conf

systemctl enable salt-minion

salt_return=1

until [ $salt_return -eq 0 ]; do

        # message "Waiting for the Salt Master to accept our key..."
        echo "Waiting for the Salt Master to accept our key..." 1> /dev/tty
        #salt-call test.ping && salt_return=0 || salt_return=1; sleep 30
        systemctl start salt-minion.service
        systemctl status salt-minion.service && salt_return=0 || salt_return=1; sleep 30
done

#message "Running system salt configurations..."
echo "Running system salt configurations..." 1> /dev/tty1
salt-call --log-level=quiet --out-file=/tmp/salt state.highstate

# salt exits clean with a status of "0"
if [ $? -eq 0 ]; then
        mail -s "$(hostname) install complete." tmullaly@gmail.com </tmp/salt state.highstate
        systemctl disable ubuntu-install.service
        echo "Configuration success!!! Rebooting..." 1> /dev/tty1
        sleep 5;
        sed -i '1d' /etc/issue
        reboot
else
        # message "Something went wrong :(  SSH in to check."
        echo "Something went wrong :(  SSH in to check." 1> /dev/tty1
fi

ubuntu-install.service

[Unit]
Description=Ubuntu Installer
After=network.target
Before=display-manager.service
Conflicts=display-manager.service

[Service]
Type=oneshot
ExecStart=/usr/share/installer/ubuntu-install

[Install]
WantedBy=network.target

Salt States

Create the directory /srv/salt

mkdir -p /srv/salt

top.sls

base:
  '*':
    - vm-tools
  'print*':
    - test
  'workstation*':
    - scratch

There are three salt states here, vm-tools gets applied to all hosts. test gets applied to the print servers and scratch gets applied to the workstations.

vm-tools

{% if grains['os_family'] == 'Debian' %}
installing_vm-tools:
  pkg.installed:
    - name: open-vm-tools
{% endif %}

{% if grains['os_family'] == 'RedHat' %}
installing_vm-tools:
  pkg.installed:
    - name: open-vm-tools
{% endif %}

test.sls

/test:
  file:
    - managed
    - user: root
    - group: root

scratch

/scratch:
  file.directory:
    - user: root
    - group: root
    - mode: 1777

Web Server

Create virtual machine

Create a new vm on the internal network.

Memory 512
CPUs 1 core
Hard disk 1 8 GB
Network adapter 1 Internal
remove the cdrom and remove the floppy

When it boots, hit f2 and make the network interface the first boot device.

Add mac address to dhcp

Shut it down and get the mac address it created from the vSphere interface.

Edit /etc/dhcp/dhcpd.conf and add it to the bottom.

host www {
  hardware ethernet 00:0c:29:2e:fe:ae;
  fixed-address www.thunderhouse.com;
}