This is how to setup a linux deployment system using pxe and saltstack. These directions are for a sandboxed virtual environment but are directly applicable to a production environment.
Router
I create a virtual machine with two interfaces so that our environment is sandboxed, mostly because of the dhcp server we need to run.
- Memory 512 MB
- CPUs 1 core
- Hard disk 1 8 GB
- Network adapter 1 VM Network
- Network adapter 2 Internal
After the install, update and upgrade
apt-get update
apt-get upgrade
Make the inside interface static and give it an address
vi /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface (outside)
auto ens160
iface ens160 inet dhcp
# The secondary network interface (inside)
auto ens192
iface ens192 inet static
address 10.20.30.1
netmask 255.255.0.0
Reboot
reboot
Configure kernel to forward packets
vi /etc/sysctl.conf
# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1
Add iptable rules to forward packets
iptables -t nat -A POSTROUTING -o ens160 -j MASQUERADE
iptables -A FORWARD -i ens192 -o ens160 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i ens160 -o ens192 -j ACCEPT
Make the ip tables rules persistent across reboots
apt-get install iptables-persistent
Now to save the iptable rules
netfilter-persistent save
netfilter-persistent reload
Other
sysctl -p
apt-get install netstat-nat
Main Server
I’m going to combine the deployment services onto one system. It’s a basic Ubuntu 16.04 Server install and I’m calling it “server”.
- Memory 512 MB
- CPUs 1 core
- Hard disk 1 8 GB
- Network adapter 1 Internal Net
DNS
Install BIND
Let’s install bind9 to have a nameserver running
apt-get install bind9
Edit /etc/bind/named.conf.options to get caching to work and add this
forwarders {
8.8.8.8;
8.8.4.4;
};
Restart bind
systemctl restart bind
Install dnsutils
apt-get install dnsutils
Test the caching
nslookup
server localhost
google.com
Edit /etc/network/interfaces and change the nameserver to localhost
# The primary network interface
auto ens160
iface ens160 inet static
address 10.20.30.10
netmask 255.255.255.0
network 10.20.30.0
broadcast 10.20.30.255
gateway 10.20.30.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 127.0.0.1
Reboot
reboot
Back on the Router
I like to ssh to my router and then ssh to the clients on the inside network instead of working on the vSphere console so I’d like internal clients resolve on the router. In order to do this we need to override the nameserver that the dhcp client gets, this is done in the file /etc/resolvconf/resolv.conf.d/head
vi /etc/resolvconf/resolv.conf.d/head
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.20.30.10
search thunderhouse.com
The warning in the file is talking about /etc/resolv.conf not this file.
Restart networking
systemctl restart networking
Note: another good command to know is
systemctl status networking
Test
root@router:~# nslookup google.com
Server: 10.20.30.10
Address: 10.20.30.10#53
Non-authoritative answer:
Name: google.com
Address: 172.217.4.110
Setup Internal DNS
Let’s set up the primary and reverse zones back on our server for the internal network. The computers will name themselves based on the reverse lookup. To add a DNS Forward and Reverse resolution to bind9, edit /etc/bind/named.conf.local
vi /etc/bind/named.conf.local
zone "thunderhouse.com" {
type master;
file "/etc/bind/db.thunderhouse.com";
}
zone "0.30.20.10.in-addr.arpa" {
type master;
notify no;
file "/etc/bind/db.10";
}
Now the file /etc/bind/db.thunderhouse.com will have the details for resolving hostname to IP address for this domain/zone, and the file /etc/bind/db.10 will have the details for resolving IP address to hostname. Now we will add the details which is necessary for forward resolution into /etc/bind/db.thunderhouse.com
First, copy /etc/bind/db.local to /etc/bind/db.thunderhouse.com
cp /etc/bind/db.local /etc/bind/db.thunderhouse.com
Next, edit the /etc/bind/db.thunderhouse.com and replace the following.
In the line which has SOA: localhost. – This is the FQDN of the server in charge for this domain. I’ve installed bind9 in 10.20.30.10, whose hostname is “server”. So replace the “localhost.” with “server.thunderhouse.com.”. Make sure it end’s with a dot(.).
In the line which has SOA: root.localhost. – This is the E-Mail address of the person who is responsible for this server. Use dot(.) instead of @. I’ve replaced with tom.localhost.
In the line which has NS: localhost. - This is defining the Name server for the domain (NS). We have to change this to the fully qualified domain name of the name server. Change it to “server.thunderhouse.com.”. Make sure you have a “.” at the end.
Once the changes are done, the /etc/bind/db.thunderhouse.com file will look like the following:
;
; BIND data file
;
$TTL 604800
@ IN SOA server.thunderhouse.com. tom.localhost. (
5 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
@ IN NS server.thunderhouse.com.
thunderhouse.com IN MX 10 mail.thunderhouse.com.
router IN A 10.20.30.1
server IN A 10.20.30.10
mail IN A 10.20.30.10
ns IN CNAME 10.20.30.10
ldap IN A 10.20.30.11
www IN A 10.20.30.12
print IN A 10.20.30.15
ad01 IN A 10.20.30.21
joomla IN A 10.20.30.22
kali IN A 10.20.30.23
workstation01 IN A 10.20.30.50
And the reverse zone will look like this:
;
; BIND reverse data file
;
$TTL 604800
@ IN SOA server.thunderhouse.com. tom.localhost. (
7 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
@ IN NS server.thunderhosue.com.
1 IN PTR router.thunderhouse.com.
10 IN PTR server.thunderhouse.com.
11 IN PTR ldap.thunderhouse.com.
12 IN PTR www.thunderhouse.com.
15 IN PTR print.thunderhouse.com.
21 IN PTR ad01.thunderhouse.com.
22 IN PTR joomla.thunderhouse.com.
23 IN PTR kali.thunderhouse.com.
50 IN PTR workstation01.thunderhouse.com.
Reload dns and check out systemctl status and /var/log/syslog for any errors
systemctl restart bind9
systemctl status bind9
tail -100 /var/log/syslog
DHCP
Install a dhcp server
We’re going to install the isc dhcp server on the internal network we’ve created.
apt-get install isc-dhcp-server
Edit dhcp server config
vi /etc/dhcp/dhcpd.conf
subnet 10.20.30.0 netmask 255.255.255.0 {
range 10.20.30.40 10.20.30.100;
option routers 10.20.30.1;
}
Restart the dhcp server
systemctl restart isc-dhcp-server
Check it with
systemctl status isc-dhcp-server
This is just a simple configuration to get us going.
iPXE configuration in DHCP
Let’s add this to the dhcp configuration, above the subnet declaration.
# option definitions common to all supported networks...
option domain-name "thunderhouse.com";
option domain-name-servers 10.20.30.10;
# iPXE poop
option space ipxe;
option ipxe-encap-opts code 175 = encapsulate ipxe;
option ipxe.priority code 1 = signed integer 8;
option ipxe.keep-san code 8 = unsigned integer 8;
option ipxe.skip-san-boot code 9 = unsigned integer 8;
option ipxe.syslogs code 85 = string;
option ipxe.cert code 91 = string;
option ipxe.privkey code 92 = string;
option ipxe.crosscert code 93 = string;
option ipxe.no-pxedhcp code 176 = unsigned integer 8;
option ipxe.bus-id code 177 = string;
option ipxe.bios-drive code 189 = unsigned integer 8;
option ipxe.username code 190 = string;
option ipxe.password code 191 = string;
option ipxe.reverse-username code 192 = string;
option ipxe.reverse-password code 193 = string;
option ipxe.version code 235 = string;
option iscsi-initiator-iqn code 203 = string;
# Feature indicators
option ipxe.pxeext code 16 = unsigned integer 8;
option ipxe.iscsi code 17 = unsigned integer 8;
option ipxe.aoe code 18 = unsigned integer 8;
option ipxe.http code 19 = unsigned integer 8;
option ipxe.https code 20 = unsigned integer 8;
option ipxe.tftp code 21 = unsigned integer 8;
option ipxe.ftp code 22 = unsigned integer 8;
option ipxe.dns code 23 = unsigned integer 8;
option ipxe.bzimage code 24 = unsigned integer 8;
option ipxe.multiboot code 25 = unsigned integer 8;
option ipxe.slam code 26 = unsigned integer 8;
option ipxe.srp code 27 = unsigned integer 8;
option ipxe.nbi code 32 = unsigned integer 8;
option ipxe.pxe code 33 = unsigned integer 8;
option ipxe.elf code 34 = unsigned integer 8;
option ipxe.comboot code 35 = unsigned integer 8;
option ipxe.efi code 36 = unsigned integer 8;
option ipxe.fcoe code 37 = unsigned integer 8;
option ipxe.vlan code 38 = unsigned integer 8;
option ipxe.menu code 39 = unsigned integer 8;
option ipxe.sdi code 40 = unsigned integer 8;
option ipxe.nfs code 41 = unsigned integer 8;
# pxelinux poop
option space pxelinux;
option pxelinux.magic code 208 = string;
option pxelinux.configfile code 209 = text;
option pxelinux.pathprefix code 210 = text;
option pxelinux.reboottime code 211 = unsigned integer 32;
if exists user-class and option user-class = "iPXE" {
filename "http://server.thunderhouse.com/ipxe/boot.ipxe";
} else {
filename "undionly.kpxe";
}
This if statement will cause the DHCP server to first tell clients to download iPXE. Once iPXE starts up and does another DHCP request, it will be told the actual location of the configuration file to download. Without this, we would end up with a continuous loop of iPXE downloading itself.
Create the undionly.kpxe image
The undionly.kpxe image is a PXE image that keeps UNDI loaded and unloads PXE. This is for clients which don’t natively support iPXE, which is pretty much everyone.
We need to build the undionly.kpxe image. First, install the dependencies.
apt-get install build-essential
apt-get install liblzma-dev
clone the image source
git clone git://git.ipxe.org/ipxe.git
Let’s make the image
cd ipxe/src
make
tftp
We need a tftp server to host the iPXE image we just created
Install tftpd-hpa
apt-get install tftpd-hpa
Now move the iPXE image into place
cp ~/ipxe/src/bin/undionly.kpxe /var/lib/tftpboot
Check the status
systemctl status tftpd-hpa
Web
Install Apache
We need a web server to serve the ipxe files, the preseed file and the install service.
apt-get install apache2
systemctl status apache2
Salt
Install the salt-master
We’ll install the salt master from apt.
apt-get install salt-master
systemctl status salt-master
Create the salt directories
mkdir -p /srv/salt
mkdir -p /srv/formulas
Create the autosign.conf file
vi /etc/salt/autosign.conf
Make it look like this:
www.thunderhouse.com
print.thunderhouse.com
ldap.thunderhouse.com
www.thunderhouse.com
WORKSTATION[0-9]{2}.THUNDERHOUSE.COM
workstation[0-9]{2}.thunderhouse.com
Files
iPXE Files
We’ll use apache’s default location for serving web files, /var/www/html
boot.ipxe
#!ipxe
# Global variables used by all other iPXE scripts
chain --autofree boot.ipxe.cfg ||
# Boot <boot-url>/menu.ipxe script if all other options have been exhausted
chain --replace --autofree ${menu-url} ||
boot.ipxe.cfg
#!ipxe
## OPTIONAL: Base URL used to resolve most other resources
## Should always end with a slash
set boot-url http://server.thunderhouse.com
# REQUIRED: Absolute URL to the menu script, used by boot.ipxe
# and commonly used at the end of simple override scripts
# in ${boot-dir}.
set menu-url ${boot-url}/menu.ipxe
# where we put our configs
#set config-dir ${boot-url}/configs
set config-dir ${boot-url}
# fedora bits
set fedora-mirror http://mirror.mit.edu/fedora/linux/releases
set fedora-release 23
set fedora-next 24
#Ubuntu bits
set ubuntu-mirror http://mirrors.mit.edu/ubuntu-releases/
set ubuntu-release 16.04
# memtest bits
# note: the plus (+) doesn't work in the url
#set memtest-latest ${config-dir}/memtest/memtest86plus-5.01.iso
# Some menu defaults
set menu-timeout 5000
set submenu-timeout ${menu-timeout}
isset ${menu-default} || set menu-default exit
# Figure out if client is 64-bit capable
cpuid --ext 29 && set arch x64 || set arch x86
cpuid --ext 29 && set archl amd64 || set archl i386
menu.ipxe
#!ipxe
# Variables are specified in boot.ipxe.cfg
###################### MAIN MENU ####################################
:start
menu iPXE boot menu
item --key u ubuntu Boot Ubuntu ${ubuntu-release} Installer
item
# item --key d menu-diag Diagnostics tools...
item
item reboot Reboot computer
item --key x exit Exit iPXE and continue BIOS boot
choose --timeout ${menu-timeout} --default ${menu-default} selected || goto cancel
set menu-timeout 0
goto ${selected}
:shell
echo Type 'exit' to get the back to the menu
shell
set menu-timeout 0
set submenu-timeout 0
goto start
:cancel
echo You cancelled the menu, dropping you to a shell
goto shell
:failed
echo Booting failed, dropping to shell
goto shell
:reboot
reboot
:exit
exit
:config
config
goto start
:back
set submenu-timeout 0
clear submenu-default
goto start
############ MAIN MENU ITEMS ############
:ubuntu
#chain --autofree --replace ${config-dir}/ubuntu/${ubuntu-release}/install.ipxe || goto failed
chain --autofree --replace ubuntu.ipxe || goto failed
# :menu-diag
# chain --autofree --replace ${boot-url}/menu.diag.ipxe
ubuntu.ipxe
#!ipxe
dhcp
# Variables are specified in boot.ipxe.cfg
set base-url http://mirrors.mit.edu/ubuntu/dists/xenial/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/
kernel ${base-url}/linux
initrd ${base-url}/initrd.gz
imgargs linux auto=true url=http://server.thunderhouse.com/preseed.cfg hostname=${hostname} domain=thunderhouse.com
boot
Debian Preseed
The example preseed file is located here: wget https://help.ubuntu.com/16.04/installation-guide/example-preseed.txt However you should use mine, the example doesn’t have everything, do a diff if you want to see the difference.
To generate the password hash, install the mkpasswd command which is in the whois package:
apt-get install whois
mkpasswd -m sha-512
preseed.cfg
The hashed password is Passsword1
d-i debian-installer/locale string en_US
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/xkb-keymap select us
d-i keyboard-configuration/layoutcode string us
d-i netcfg/choose_interface select auto
d-i netcfg/get_hostname string unassigned-hostname
d-i netcfg/get_domain string unassigned-domain
d-i netcfg/wireless_wep string
d-i mirror/country string manual
d-i mirror/http/hostname string archive.ubuntu.com
d-i mirror/http/directory string /ubuntu
d-i mirror/http/proxy string
d-i passwd/root-login boolean true
d-i passwd/make-user boolean false
d-i passwd/root-password-crypted password $6$46V.E/.7e2hpmE$4JQSRGhVrrb/HthkQ27WWUlAROz/1Sm9iDfRwbh2V24xYG7OsxlgWnpTqitPxzn67Sa1KtiGOoUKkU6M/NvQ70
d-i user-setup/encrypt-home boolean false
d-i clock-setup/utc boolean true
d-i time/zone string US/Eastern
d-i clock-setup/ntp boolean true
d-i partman-auto/method string regular
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto/choose_recipe select atomic
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-md/confirm boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
tasksel tasksel/first multiselect ubuntu-server
d-i pkgsel/include string wget
d-i pkgsel/update-policy select none
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
d-i finish-install/reboot_in_progress note
d-i preseed/late_command string sed -i 's/^GRUB_HIDDEN_TIMEOUT=0/GRUB_HIDDEN_TIMEOUT=5/' /etc/default/grub; \
in-target sed -i 's/^GRUB_HIDDEN_TIMEOUT_QUIET=true/GRUB_HIDDEN_TIMEOUT_QUIET=false/' /etc/default/grub; \
in-target update-grub; \
in-target mkdir /usr/share/mitmath; \
in-target wget -O /usr/share/mitmath/ubuntu-install http://server.thunderhouse.com/ubuntu-install; \
in-target chmod 755 /usr/share/mitmath/ubuntu-install; \
in-target wget -O /lib/systemd/system/ubuntu-install.service http://server.thunderhouse.com/ubuntu-install.service; \
in-target systemctl enable ubuntu-install.service; \
in-target systemctl start ubuntu-install.service; \
in-target sed -i '1 i\Please wait while system finishes configuration...' /etc/issue
Ubuntu Install Service
ubuntu-install
#!/bin/bash
salt_master="server.thunderhouse.com"
# show a message on the splash screen with our progress
# restart splash screen if the process is gone
message () {
pgrep plymouthd || plymouthd && plymouth show-splash
plymouth message --text="This workstation is being configured. Please wait, and do not reboot... ${1}..."
}
#sed '1 i\ Please wait while system finishes configuration...' /etc/issue
# can't do anything here without network
count=0
until [ $(ping -c 1 server.thunderhouse.com > /dev/null 2>&1 ; echo $?) = "0" ] ||
[ $count = "30" ]
do
# message "waiting for network"
echo "waiting for network" 1> /dev/tty1
sleep 2
let count=$count+1
done
if [ $count = "10" ]; then
echo "where's my network? dropping out."
exit 1
fi
# start SSH for debugging
systemctl start ssh
# make sure system time is correct
# message "setting correct system time"
echo "setting correct system time" 1> /dev/tty1
systemctl start chronyd
chronyc waitsync
# install salt
#message "installing salt"
echo "installing salt" 1> /dev/tty1
#yum -qy install salt-minion
apt-get --yes -q install python-software-properties
apt-add-repository ppa:saltstack/salt -y
apt-get --yes -q update
apt-get install --yes -q salt-minion
mkdir /etc/salt/minion.d
echo "master: ${salt_master}" > /etc/salt/minion.d/server.thunderhouse.com.conf
systemctl enable salt-minion
salt_return=1
until [ $salt_return -eq 0 ]; do
# message "Waiting for the Salt Master to accept our key..."
echo "Waiting for the Salt Master to accept our key..." 1> /dev/tty
#salt-call test.ping && salt_return=0 || salt_return=1; sleep 30
systemctl start salt-minion.service
systemctl status salt-minion.service && salt_return=0 || salt_return=1; sleep 30
done
#message "Running system salt configurations..."
echo "Running system salt configurations..." 1> /dev/tty1
salt-call --log-level=quiet --out-file=/tmp/salt state.highstate
# salt exits clean with a status of "0"
if [ $? -eq 0 ]; then
mail -s "$(hostname) install complete." tmullaly@gmail.com </tmp/salt state.highstate
systemctl disable ubuntu-install.service
echo "Configuration success!!! Rebooting..." 1> /dev/tty1
sleep 5;
sed -i '1d' /etc/issue
reboot
else
# message "Something went wrong :( SSH in to check."
echo "Something went wrong :( SSH in to check." 1> /dev/tty1
fi
ubuntu-install.service
[Unit]
Description=Ubuntu Installer
After=network.target
Before=display-manager.service
Conflicts=display-manager.service
[Service]
Type=oneshot
ExecStart=/usr/share/installer/ubuntu-install
[Install]
WantedBy=network.target
Salt States
Create the directory /srv/salt
mkdir -p /srv/salt
top.sls
base:
'*':
- vm-tools
'print*':
- test
'workstation*':
- scratch
There are three salt states here, vm-tools gets applied to all hosts. test gets applied to the print servers and scratch gets applied to the workstations.
vm-tools
{% if grains['os_family'] == 'Debian' %}
installing_vm-tools:
pkg.installed:
- name: open-vm-tools
{% endif %}
{% if grains['os_family'] == 'RedHat' %}
installing_vm-tools:
pkg.installed:
- name: open-vm-tools
{% endif %}
test.sls
/test:
file:
- managed
- user: root
- group: root
scratch
/scratch:
file.directory:
- user: root
- group: root
- mode: 1777
Web Server
Create virtual machine
Create a new vm on the internal network.
- Memory 512
- CPUs 1 core
- Hard disk 1 8 GB
- Network adapter 1 Internal
- remove the cdrom and remove the floppy
When it boots, hit f2 and make the network interface the first boot device.
Add mac address to dhcp
Shut it down and get the mac address it created from the vSphere interface.
Edit /etc/dhcp/dhcpd.conf and add it to the bottom.
host www {
hardware ethernet 00:0c:29:2e:fe:ae;
fixed-address www.thunderhouse.com;
}
Restart the dhcp server
systemctl restart isc-dhcp-server
Add the salt formulas
We’ll use the apache and the php formulas from github
cd /srv/formulas
git clone https://github.com/saltstack-formulas/php-formula
git clone https://github.com/saltstack-formulas/apache-formula
Add formula location to the salt master file
Edit the master file on the salt master server
vi /etc/salt/master
The file_roots section will look like this.
file_roots:
base:
- /srv/salt
- /srv/formulas/openldap-formula
- /srv/formulas/apache-formula
- /srv/formulas/ntp-formula
- /srv/formulas/openssh-formula
Save it ans restart the salt master service.
systemctl restart salt-master
systemctl status salt-master
Edit top.sls
Add www to /srv/salt/top.sls
vi /srv/salt/top.sls
It’ll look something like this:
base:
'*':
- vm-tools
'www*':
- apache
- php
'print*':
- test
'ldap*':
- apache
- openldap.server
- ntp
'workstation*':
- scratch
- ubuntu-desktop