The goal of this new few series of posts is to be able to setup virtual machines to simulate a real-world openstack deployment using maas and juju. This goes through setting up a maas-server in a VM as well as setting up maas-nodes in VMs and getting them enlisted/commissioned into the maas-server. Next juju is configured to use the maas cluster. Finally, openstack is deployed using juju.
Overview
Requirements
Ideally, a large server with 16 cores, 32G memory, 500G disk. Obviously you can tweak this setup to work with less; but be prepared to lock up lesser machines. In addition your host machine needs to be able to support nested virtualization.
Topology
Here is the basics of what will be setup for our virtual maas cluster. Each red box is a virtual machine with two interfaces. The eth0 interface in the VM connects to the NATed maas-internet network, while the VM’s eth1 interface connects to the isolated maas-management network. The number of maas-nodes should match what is required for the deployment; however it is simple enough to enlist more nodes later. I choose to use a public/private network in order to be more flexible later in how openstack networking is set up.
Setup Host Machine
Install Requirements
First install all required programs on the host machine.
sudo apt-get install libvirt-bin qemu-kvm cpu-checker virtinst uvtool
Next, check if kvm is working correctly.
Ensure nested KVM is enabled. (replace intel with amd if necessary)
cat /sys/module/kvm_intel/parameters/nested
This should output Y, if it doesn’t do the following:
sudo modprobe -r kvm_intel
sudo modprobe kvm_intel nested=1
Ensure $USER is added to libvirtd group.
Ensure host machine has SSH keys generated and setup. (Be careful, don’t overwrite your existing keys)
[ -d ~/.ssh ] || ssh-keygen -t rsa
Virtual Network Setup
This step can be done via virt-manager, but also done via command line using virsh.
Setup a virtual network which uses NAT to communicate with the host machine with the following parameters:
Network Name: maas_internet
Network: 192.168.100.0/24
Do _not_ Enable DHCP.
Forwarding to physical network; Any physical device; NAT
And setup an isolated virtual network the following parameters:
Network Name: maas_management
Network: 10.10.10.0/24
Do _not_ Enable DHCP.
Isolated network;
Install the MAAS Server
Download and Start the Install
Ensure you have virt-manager connected to the hypervisor.
While there are many ways we can create virtual machines, I chose the tool uvtool because it works well in Trusty and quickly creates VM based on the Ubuntu cloud image.
Sync the latest cloud trusty cloud image:
uvt-simplestreams-libvirt sync release=trusty arch=amd64
Create a maas-server VM:
uvt-kvm create maas-server release=trusty arch=amd64 --disk 20 --memory 2048 --password ubuntu
After it boots, shut it down and edit the VM’s machine configuration.
Make the two network interfaces connect to maas_internet and maas_management respectively.
Now edit /etc/network/interfaces to have the following:
auto eth0
iface eth0 inet static
address 192.168.100.10
netmask 255.255.255.0
gateway 192.168.100.1
dns-nameservers 10.10.10.10 192.168.100.1
auto eth1
iface eth1 inet static
address 10.10.10.10
netmask 255.255.255.0
dns-nameservers 10.10.10.10 192.168.100.1
And follow the instructions here:
http://maas.ubuntu.com/docs/install.html#pkg-install
Which is essentially:
sudo apt-get install maas maas-dhcp maas-dns
MAAS Server Post Install Tasks
http://maas.ubuntu.com/docs/install.html#post-install
First let’s check if the webpage is working correctly. Depending on your installation, you may need to proxy into a remote host hypervisor before accessing the webpage. If you’re working locally you should be able to access this address directly (as the libvirt maas_internet network is already connected to your local machine).
If you need to access it indirectly (and 192.168.100.0 is a non-conflicting subnet):
sshuttle -D -r <hypervisor IP> 192.168.100.0/24
Access the following:
http://192.168.100.10/MAAS
It should remind you that post installation tasks need to be completed.
Let’s create the admin user from the hypervisor machine:
ssh ubuntu@192.168.100.10
sudo maas-region-admin createadmin --username=root --email="user@host.com" --password=ubuntu
If you want to limit the types of boot images that can be created you need to edit
sudo vim /etc/maas/bootresources.yaml
Import boot images, using the new root user you created to log in:
http://192.168.100.10/MAAS/clusters/
Now click 'import boot images' and be patient as it will take some time before these images are imported.
Add a key for the host machine here:
http://192.168.100.10/MAAS/account/prefs/sshkey/add/
Configure the MAAS Cluster
Follow instructions here to setup cluster:
http://maas.ubuntu.com/docs/cluster-configuration.html
http://192.168.100.10/MAAS/clusters/
Click on ‘Cluster master’
Click on edit interface eth1.
Interface: eth1
Management: DHCP and DNS
IP: 10.10.10.10
Subnet mask: 255.255.255.0
Broadcast IP: 10.10.10.255
Router IP: 10.10.10.10
IP Range Low: 10.10.10.100
IP Range High: 10.10.10.200
Click Save Interface
Ensure Nodes Auto-Enlist
Create a MAAS key and use that to log in:
http://192.168.100.10/MAAS/account/prefs/
Click on ‘+ Generate MAAS key’ and copy that down.
Log into the maas-server, and then log into maas using the MAAS key:
maas login maas-server http://192.168.100.10/MAAS
Now set all nodes to auto accept:
maas maas-server nodes accept-all
Setup keys on the maas-server so it can access the virtual machine host
sudo mkdir -p ~maas
sudo chown maas:maas ~maas
sudo -u maas ssh-keygen
Add the pubkey in ~maas/.ssh/id_rsa.pub to the virsh servers authorized_keys and to the maas SSH keys (
http://192.168.100.10/MAAS/account/prefs/sshkey/add/)
sudo cat /home/maas/.ssh/id_rsa.pub
Now install virsh to test a connection and allow the maas-server to control maas-nodes.
sudo apt-get install libvirt-bin
Test the connection to the hypervisor (replace ubuntu with hypervisor host user)
sudo -u maas virsh -c qemu+ssh://ubuntu@192.168.100.1/system list --all
Confirm Maas-Server Networking
Ensure we can reach important address via maas-server:
host streams.canonical.com
host store.juju.ubuntu.com
host archive.ubuntu.com
And that we can download charms if needed:
wget https://store.juju.ubuntu.com/charm-info
Setup Traffic Forwarding
Setup maas-server to forward traffic from eth1 to eth0:
You can type the following out manually or add it as an upstart script to ensure forwarding is setup properly each time add this file to
/etc/init/ovs-routing.conf (thanks to Juan Negron):
description "Setup NAT rules for ovs bridge"
start on runlevel [2345]
env EXTIF="eth0"
env BRIDGE="eth1"
task
script
echo "Configuring modules"
modprobe ip_tables || :
modprobe nf_conntrack || :
modprobe nf_conntrack_ftp || :
modprobe nf_conntrack_irc || :
modprobe iptable_nat || :
modprobe nf_nat_ftp || :
echo "Configuring forwarding and dynaddr"
echo "1" > /proc/sys/net/ipv4/ip_forward
echo "1" > /proc/sys/net/ipv4/ip_dynaddr
echo "Configuring iptables rules"
iptables-restore <<-EOM
*nat
-A POSTROUTING -o ${EXTIF} -j MASQUERADE
COMMIT
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A FORWARD -i ${BRIDGE} -o ${EXTIF} -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i ${EXTIF} -o ${BRIDGE} -j ACCEPT
-A FORWARD -j LOG
COMMIT
EOM
end script
Then start the service:
sudo service ovs-routing start
Setup Squid Proxy
Ensure squid proxy can access cloud images:
echo "cloud-images.ubuntu.com" | sudo tee /etc/squid-deb-proxy/mirror-dstdomain.acl.d/98-cloud-images
sudo service squid-deb-proxy restart
Install MAAS Nodes
Now we can virt-install each maas-node on the hypervisor such that it automatically pxe boots and auto-enlists into MAAS. You can adjust the script below to create as many nodes as required. I’ve also simplified things by creating everything with dual nics and ample memory and hard drive space, but of course you could use custom machines per service. Compute-nodes need more compute power, ceph nodes will need more storage, and quantum-gateway will need dual nics. In addition you could specify raw disks instead of qcow2, or use storage pools; but in this case I wanted something simple that didn’t automatically use all the space it needed.
for i in {0..19}; do
virt-install \
--name=maas-node-${i} \
--connect=qemu:///system --ram=4096 --vcpus=1 --hvm --virt-type=kvm \
--pxe --boot network,hd \
--os-variant=ubuntutrusty --graphics vnc --noautoconsole --os-type=linux --accelerate \
--disk=/var/lib/libvirt/images/maas-node-${i}.qcow2,bus=virtio,format=qcow2,cache=none,sparse=true,size=32 \
--network=network=maas_internet,model=virtio \
--network=network=maas_management,model=virtio
done
Now each node needs to be manually enlisted with the proper power configuration.
http://maas.ubuntu.com/docs/nodes.html#virtual-machine-nodes
Host Name: maas-node-${i}.vmaas
Power Type: virsh
Power Address: qemu+ssh://ubuntu@192.168.100.1/system
Power ID: maas-node-${i}
Here we need to match the machines to the mac address and update the power requirements. You can get the mac addresses of each node by using the following on the hypervisor:
virsh dumpxml maas-node-${i} | grep "mac addr"
Here is a script that helps automate some of this process, it can be run from the maas-server (replace USER ubuntu with the appropriate value) this matches mac address from virsh to the ones in maas and then sets up the power accordingly:
#!/usr/bin/python
import sys, os, libvirt
from xml.dom.minidom import parseString
os.environ['DJANGO_SETTINGS_MODULE'] = 'maas.settings'
sys.path.append("/usr/share/maas")
from maasserver.models import Node, Tag
hhost = 'qemu+ssh://ubuntu@192.168.100.1/system'
conn = libvirt.open(hhost)
nodes_dict = {}
domains = conn.listDefinedDomains()
for node_name in domains:
node = conn.lookupByName(node_name)
node_xml = parseString(node.XMLDesc(0))
node_mac1 = node_xml.getElementsByTagName('interface')[0].getElementsByTagName('mac')[0].getAttribute('address')
nodes_dict[node_mac1] = node_name
maas_nodes = Node.objects.all()
for node in maas_nodes:
try:
system_id = node.system_id
mac = node.get_primary_mac()
node_name = nodes_dict[str(mac)]
node.hostname = node_name
node.power_type = 'virsh'
node.power_parameters = { 'power_address':hhost, 'power_id':node_name }
node.save()
except: pass
Note you will need python-libvirt and run the above command with something like the following:
sudo -u maas ./setup-nodes.py
Setup Fastpath and Commission Nodes
You most likely want to use fast-path installer on nodes to speed up installation times. Set all nodes to use fastpath installer using another bulk action on the nodes.
After you have all this done, click bulk action commission.
You should see all your machines starting up if you set things up properly, give this some time. You should have all the nodes in the 'Ready' state in maas now!
http://192.168.100.10/MAAS/nodes/
Confirm DNS setup
One point of trouble can be ensuring DNS is setup correctly. We can test this by starting a maas node and inside of that trying the following:
dig streams.canonical.com
dig store.juju.ubuntu.com
If we can’t hit those, we’ll need to ensure the maas server is setup correctly.
Go to:
http://192.168.100.10/MAAS/settings/
Enter the host machines upstream DNS if necessary here, it should setup the bind configuration file and restart that service. After this re-test.
In addition I had to disable dnssec-validation for bind. Edit the following file:
sudo vim /etc/bind/named.conf.options
And change the following value:
dnssec-validation no;
And restart the service:
sudo service bind9 restart
Now you have a working virtual maas setup using the latest Ubuntu LTS!
