Thomas Mullaly

DevOps, Security and IT Leadership

Linux Auto Deployment Using iPXE and SaltStack

This is how to setup a linux deployment system using pxe and saltstack. These directions are for a sandboxed virtual environment but are directly applicable to a production environment.


I create a virtual machine with two interfaces so that our environment is sandboxed, mostly because of the dhcp server we need to run.

  • Memory 512 MB
  • CPUs 1 core
  • Hard disk 1 8 GB
  • Network adapter 1 VM Network
  • Network adapter 2 Internal

After the install, update and upgrade

apt-get update
apt-get upgrade

Make the inside interface static and give it an address

vi /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface (outside)
auto ens160
iface ens160 inet dhcp

# The secondary network interface (inside)
auto ens192
iface ens192 inet static



Configure kernel to forward packets

vi /etc/sysctl.conf
# Uncomment the next line to enable packet forwarding for IPv4

Add iptable rules to forward packets

iptables -t nat -A POSTROUTING -o ens160 -j MASQUERADE
iptables -A FORWARD -i ens192 -o ens160 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i ens160 -o ens192 -j ACCEPT

Make the ip tables rules persistent across reboots

apt-get install iptables-persistent

Now to save the iptable rules

netfilter-persistent save
netfilter-persistent reload


sysctl -p
apt-get install netstat-nat

Main Server

I’m going to combine the deployment services onto one system. It’s a basic Ubuntu 16.04 Server install and I’m calling it “server”.

  • Memory 512 MB
  • CPUs 1 core
  • Hard disk 1 8 GB
  • Network adapter 1 Internal Net


Install BIND

Let’s install bind9 to have a nameserver running

apt-get install bind9

Edit /etc/bind/named.conf.options to get caching to work and add this

forwarders {;;

Restart bind

systemctl restart bind

Install dnsutils

apt-get install dnsutils

Test the caching

server localhost

Edit /etc/network/interfaces and change the nameserver to localhost

# The primary network interface
auto ens160
iface ens160 inet static
       # dns-* options are implemented by the resolvconf package, if installed



Back on the Router

I like to ssh to my router and then ssh to the clients on the inside network instead of working on the vSphere console so I’d like internal clients resolve on the router. In order to do this we need to override the nameserver that the dhcp client gets, this is done in the file /etc/resolvconf/resolv.conf.d/head

vi /etc/resolvconf/resolv.conf.d/head
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)


The warning in the file is talking about /etc/resolv.conf not this file.

Restart networking

systemctl restart networking

Note: another good command to know is

systemctl status networking


root@router:~# nslookup

Non-authoritative answer:

Setup Internal DNS

Let’s set up the primary and reverse zones back on our server for the internal network. The computers will name themselves based on the reverse lookup. To add a DNS Forward and Reverse resolution to bind9, edit /etc/bind/named.conf.local

vi /etc/bind/named.conf.local
zone "" {
       type master;
       file "/etc/bind/";

zone "" {
       type master;
       notify no;
       file "/etc/bind/db.10";

Now the file /etc/bind/ will have the details for resolving hostname to IP address for this domain/zone, and the file /etc/bind/db.10 will have the details for resolving IP address to hostname. Now we will add the details which is necessary for forward resolution into /etc/bind/

First, copy /etc/bind/db.local to /etc/bind/

cp /etc/bind/db.local /etc/bind/

Next, edit the /etc/bind/ and replace the following.

In the line which has SOA: localhost. – This is the FQDN of the server in charge for this domain. I’ve installed bind9 in, whose hostname is “server”. So replace the “localhost.” with “”. Make sure it end’s with a dot(.).

In the line which has SOA: root.localhost. – This is the E-Mail address of the person who is responsible for this server. Use dot(.) instead of @. I’ve replaced with tom.localhost.

In the line which has NS: localhost. - This is defining the Name server for the domain (NS). We have to change this to the fully qualified domain name of the name server. Change it to “”. Make sure you have a “.” at the end.

Once the changes are done, the /etc/bind/ file will look like the following:

; BIND data file
$TTL    604800
@       IN      SOA tom.localhost. (
                              5         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
@       IN      NS        IN      MX      10
router  IN      A
server  IN      A
mail    IN      A
ns      IN      CNAME
ldap    IN      A
www     IN      A
print   IN      A
ad01    IN      A
joomla  IN      A
kali    IN      A
workstation01   IN      A
And the reverse zone will look like this:
; BIND reverse data file
$TTL    604800
@       IN      SOA tom.localhost. (
                              7         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
@       IN      NS
1       IN      PTR
10      IN      PTR
11      IN      PTR
12      IN      PTR
15      IN      PTR
21      IN      PTR
22      IN      PTR
23      IN      PTR
50      IN      PTR

Reload dns and check out systemctl status and /var/log/syslog for any errors

systemctl restart bind9
systemctl status bind9
tail -100 /var/log/syslog


Install a dhcp server

We’re going to install the isc dhcp server on the internal network we’ve created.

apt-get install isc-dhcp-server

Edit dhcp server config

vi /etc/dhcp/dhcpd.conf
subnet netmask {
  option routers;

Restart the dhcp server

systemctl restart isc-dhcp-server

Check it with

systemctl status isc-dhcp-server

This is just a simple configuration to get us going.

iPXE configuration in DHCP

Let’s add this to the dhcp configuration, above the subnet declaration.

# option definitions common to all supported networks...
option domain-name "";
option domain-name-servers;

# iPXE poop
option space ipxe;
option ipxe-encap-opts code 175 = encapsulate ipxe;
option ipxe.priority code 1 = signed integer 8;
option ipxe.keep-san code 8 = unsigned integer 8;
option ipxe.skip-san-boot code 9 = unsigned integer 8;
option ipxe.syslogs code 85 = string;
option ipxe.cert code 91 = string;
option ipxe.privkey code 92 = string;
option ipxe.crosscert code 93 = string;
option code 176 = unsigned integer 8;
option ipxe.bus-id code 177 = string;
option ipxe.bios-drive code 189 = unsigned integer 8;
option ipxe.username code 190 = string;
option ipxe.password code 191 = string;
option ipxe.reverse-username code 192 = string;
option ipxe.reverse-password code 193 = string;
option ipxe.version code 235 = string;
option iscsi-initiator-iqn code 203 = string;
# Feature indicators
option ipxe.pxeext code 16 = unsigned integer 8;
option ipxe.iscsi code 17 = unsigned integer 8;
option ipxe.aoe code 18 = unsigned integer 8;
option ipxe.http code 19 = unsigned integer 8;
option ipxe.https code 20 = unsigned integer 8;
option ipxe.tftp code 21 = unsigned integer 8;
option ipxe.ftp code 22 = unsigned integer 8;
option ipxe.dns code 23 = unsigned integer 8;
option ipxe.bzimage code 24 = unsigned integer 8;
option ipxe.multiboot code 25 = unsigned integer 8;
option ipxe.slam code 26 = unsigned integer 8;
option ipxe.srp code 27 = unsigned integer 8;
option ipxe.nbi code 32 = unsigned integer 8;
option ipxe.pxe code 33 = unsigned integer 8;
option ipxe.elf code 34 = unsigned integer 8;
option ipxe.comboot code 35 = unsigned integer 8;
option ipxe.efi code 36 = unsigned integer 8;
option ipxe.fcoe code 37 = unsigned integer 8;
option ipxe.vlan code 38 = unsigned integer 8;
option code 39 = unsigned integer 8;
option ipxe.sdi code 40 = unsigned integer 8;
option ipxe.nfs code 41 = unsigned integer 8;

# pxelinux poop
option space pxelinux;
option pxelinux.magic code 208 = string;
option pxelinux.configfile code 209 = text;
option pxelinux.pathprefix code 210 = text;
option pxelinux.reboottime code 211 = unsigned integer 32;

if exists user-class and option user-class = "iPXE" {
       filename "";
} else {
       filename "undionly.kpxe";

This if statement will cause the DHCP server to first tell clients to download iPXE. Once iPXE starts up and does another DHCP request, it will be told the actual location of the configuration file to download. Without this, we would end up with a continuous loop of iPXE downloading itself.

Create the undionly.kpxe image

The undionly.kpxe image is a PXE image that keeps UNDI loaded and unloads PXE. This is for clients which don’t natively support iPXE, which is pretty much everyone.

We need to build the undionly.kpxe image. First, install the dependencies.

apt-get install build-essential
apt-get install liblzma-dev

clone the image source

git clone git://

Let’s make the image

cd ipxe/src


We need a tftp server to host the iPXE image we just created

Install tftpd-hpa

apt-get install tftpd-hpa

Now move the iPXE image into place

cp ~/ipxe/src/bin/undionly.kpxe /var/lib/tftpboot

Check the status

systemctl status tftpd-hpa


Install Apache

We need a web server to serve the ipxe files, the preseed file and the install service.

apt-get install apache2
systemctl status apache2


Install the salt-master

We’ll install the salt master from apt.

apt-get install salt-master
systemctl status salt-master

Create the salt directories

mkdir -p /srv/salt
mkdir -p /srv/formulas

Create the autosign.conf file

vi /etc/salt/autosign.conf

Make it look like this:


iPXE Files

We’ll use apache’s default location for serving web files, /var/www/html



# Global variables used by all other iPXE scripts
chain --autofree boot.ipxe.cfg ||

# Boot <boot-url>/menu.ipxe script if all other options have been exhausted
chain --replace --autofree ${menu-url} ||



## OPTIONAL: Base URL used to resolve most other resources
## Should always end with a slash
set boot-url

# REQUIRED: Absolute URL to the menu script, used by boot.ipxe
# and commonly used at the end of simple override scripts
# in ${boot-dir}.
set menu-url ${boot-url}/menu.ipxe

# where we put our configs
#set config-dir ${boot-url}/configs
set config-dir ${boot-url}

# fedora bits
set fedora-mirror
set fedora-release 23
set fedora-next 24

#Ubuntu bits 
set ubuntu-mirror
set ubuntu-release 16.04 

# memtest bits
# note: the plus (+) doesn't work in the url
#set memtest-latest ${config-dir}/memtest/memtest86plus-5.01.iso

# Some menu defaults
set menu-timeout 5000
set submenu-timeout ${menu-timeout}
isset ${menu-default} || set menu-default exit

# Figure out if client is 64-bit capable
cpuid --ext 29 && set arch x64 || set arch x86
cpuid --ext 29 && set archl amd64 || set archl i386

# Variables are specified in boot.ipxe.cfg

###################### MAIN MENU ####################################

menu iPXE boot menu
item --key u ubuntu             Boot Ubuntu ${ubuntu-release} Installer
# item --key d menu-diag          Diagnostics tools...
item reboot                     Reboot computer
item --key x exit               Exit iPXE and continue BIOS boot
choose --timeout ${menu-timeout} --default ${menu-default} selected || goto cancel
set menu-timeout 0
goto ${selected}

echo Type 'exit' to get the back to the menu
set menu-timeout 0
set submenu-timeout 0
goto start

echo You cancelled the menu, dropping you to a shell
goto shell

echo Booting failed, dropping to shell
goto shell



goto start

set submenu-timeout 0
clear submenu-default
goto start

############ MAIN MENU ITEMS ############

#chain --autofree --replace ${config-dir}/ubuntu/${ubuntu-release}/install.ipxe || goto failed
chain --autofree --replace ubuntu.ipxe || goto failed

# :menu-diag
# chain --autofree --replace ${boot-url}/menu.diag.ipxe



# Variables are specified in boot.ipxe.cfg
set base-url
kernel ${base-url}/linux
initrd ${base-url}/initrd.gz 
imgargs linux auto=true url= hostname=${hostname}

Debian Preseed

The example preseed file is located here: wget However you should use mine, the example doesn’t have everything, do a diff if you want to see the difference.

To generate the password hash, install the mkpasswd command which is in the whois package:

apt-get install whois
mkpasswd -m sha-512


The hashed password is Passsword1
d-i debian-installer/locale string en_US
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/xkb-keymap select us
d-i keyboard-configuration/layoutcode string us
d-i netcfg/choose_interface select auto
d-i netcfg/get_hostname string unassigned-hostname
d-i netcfg/get_domain string unassigned-domain
d-i netcfg/wireless_wep string
d-i mirror/country string manual
d-i mirror/http/hostname string
d-i mirror/http/directory string /ubuntu
d-i mirror/http/proxy string
d-i passwd/root-login boolean true
d-i passwd/make-user boolean false
d-i passwd/root-password-crypted password $6$46V.E/.7e2hpmE$4JQSRGhVrrb/HthkQ27WWUlAROz/1Sm9iDfRwbh2V24xYG7OsxlgWnpTqitPxzn67Sa1KtiGOoUKkU6M/NvQ70
d-i user-setup/encrypt-home boolean false
d-i clock-setup/utc boolean true
d-i time/zone string US/Eastern
d-i clock-setup/ntp boolean true
d-i partman-auto/method string regular
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto/choose_recipe select atomic
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-md/confirm boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
tasksel tasksel/first multiselect ubuntu-server
d-i pkgsel/include string wget
d-i pkgsel/update-policy select none
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
d-i finish-install/reboot_in_progress note
d-i preseed/late_command string sed -i 's/^GRUB_HIDDEN_TIMEOUT=0/GRUB_HIDDEN_TIMEOUT=5/' /etc/default/grub; \
        in-target sed -i 's/^GRUB_HIDDEN_TIMEOUT_QUIET=true/GRUB_HIDDEN_TIMEOUT_QUIET=false/' /etc/default/grub; \
        in-target update-grub; \
        in-target mkdir /usr/share/mitmath; \
        in-target wget -O /usr/share/mitmath/ubuntu-install; \
        in-target chmod 755 /usr/share/mitmath/ubuntu-install; \
        in-target wget -O /lib/systemd/system/ubuntu-install.service; \
        in-target systemctl enable ubuntu-install.service; \
        in-target systemctl start ubuntu-install.service; \
        in-target sed -i '1 i\Please wait while system finishes configuration...' /etc/issue

Ubuntu Install Service




# show a message on the splash screen with our progress
# restart splash screen if the process is gone
message () {
  pgrep plymouthd || plymouthd && plymouth show-splash
  plymouth message --text="This workstation is being configured. Please wait, and do not reboot... ${1}..."

#sed '1 i\ Please wait while system finishes configuration...' /etc/issue

# can't do anything here without network
until [ $(ping -c 1 > /dev/null 2>&1 ; echo $?) = "0" ] ||
      [ $count = "30" ]
        # message "waiting for network"
        echo "waiting for network" 1> /dev/tty1
        sleep 2
        let count=$count+1

if [ $count = "10" ]; then
        echo "where's my network?  dropping out."
        exit 1

# start SSH for debugging
systemctl start ssh

# make sure system time is correct
# message "setting correct system time"
echo "setting correct system time" 1> /dev/tty1
systemctl start chronyd
chronyc waitsync

# install salt
#message "installing salt"
echo "installing salt" 1> /dev/tty1
#yum -qy install salt-minion
apt-get --yes -q  install python-software-properties
apt-add-repository  ppa:saltstack/salt -y 
apt-get --yes -q update
apt-get install --yes -q  salt-minion

mkdir /etc/salt/minion.d

echo "master: ${salt_master}" > /etc/salt/minion.d/

systemctl enable salt-minion


until [ $salt_return -eq 0 ]; do

        # message "Waiting for the Salt Master to accept our key..."
        echo "Waiting for the Salt Master to accept our key..." 1> /dev/tty
        #salt-call && salt_return=0 || salt_return=1; sleep 30
        systemctl start salt-minion.service
        systemctl status salt-minion.service && salt_return=0 || salt_return=1; sleep 30

#message "Running system salt configurations..."
echo "Running system salt configurations..." 1> /dev/tty1
salt-call --log-level=quiet --out-file=/tmp/salt state.highstate

# salt exits clean with a status of "0"
if [ $? -eq 0 ]; then
        mail -s "$(hostname) install complete." </tmp/salt state.highstate
        systemctl disable ubuntu-install.service
        echo "Configuration success!!! Rebooting..." 1> /dev/tty1
        sleep 5;
        sed -i '1d' /etc/issue
        # message "Something went wrong :(  SSH in to check."
        echo "Something went wrong :(  SSH in to check." 1> /dev/tty1


Description=Ubuntu Installer



Salt States

Create the directory /srv/salt

mkdir -p /srv/salt


    - vm-tools
    - test
    - scratch

There are three salt states here, vm-tools gets applied to all hosts. test gets applied to the print servers and scratch gets applied to the workstations.


{% if grains['os_family'] == 'Debian' %}
    - name: open-vm-tools
{% endif %}

{% if grains['os_family'] == 'RedHat' %}
    - name: open-vm-tools
{% endif %}


    - managed
    - user: root
    - group: root


    - user: root
    - group: root
    - mode: 1777

Web Server

Create virtual machine

Create a new vm on the internal network.

  • Memory 512
  • CPUs 1 core
  • Hard disk 1 8 GB
  • Network adapter 1 Internal
  • remove the cdrom and remove the floppy

When it boots, hit f2 and make the network interface the first boot device.

Add mac address to dhcp

Shut it down and get the mac address it created from the vSphere interface.

Edit /etc/dhcp/dhcpd.conf and add it to the bottom.

host www {
  hardware ethernet 00:0c:29:2e:fe:ae;

Restart the dhcp server

systemctl restart isc-dhcp-server

Add the salt formulas

We’ll use the apache and the php formulas from github

cd /srv/formulas
git clone
git clone

Add formula location to the salt master file

Edit the master file on the salt master server

vi /etc/salt/master

The file_roots section will look like this.

    - /srv/salt
    - /srv/formulas/openldap-formula
    - /srv/formulas/apache-formula
    - /srv/formulas/ntp-formula
    - /srv/formulas/openssh-formula

Save it ans restart the salt master service.

systemctl restart salt-master
systemctl status salt-master

Edit top.sls

Add www to /srv/salt/top.sls

vi /srv/salt/top.sls

It’ll look something like this:

    - vm-tools
    - apache
    - php
    - test
    - apache
    - openldap.server
    - ntp
    - scratch
    - ubuntu-desktop

Ubuntu Desktop