Mark's IT Blog

After session 3 we have a running OpenStack deployment. Now to deploy a VM.

First off – after starting the OpenStack node I am getting connection refused when trying to connect to Horizon. To check OpenStack services I will follow: https://docs.openstack.org/fuel-docs/latest/userdocs/fuel-user-guide/troubleshooting/service-status.html. These instructions don’t really work for devstack on CentOS but they are a good starting point.

Horizon is dependent on apache so systemctl status httpd revealing apache not running was the first issue. After starting apache I receive an error “cannot import name cinder” when trying to load http://devstack/dashboard. So I need to check the status of the other OpenStack services. As this is a DevStack deployed OpenStack, the service names are not the same as the doc suggests:

[root@devstack ~]# systemctl list-units | grep stack
 devstack@c-api.service    loaded active running
 devstack@c-sch.service    loaded active running
 devstack@c-vol.service    loaded active running
 devstack@dstat.service    loaded active running
 devstack@etcd.service     loaded active running
 devstack@g-api.service    loaded active running
 devstack@g-reg.service    loaded active running
 devstack@keystone.service loaded active running
 devstack@n-api-meta.service loaded active running
 devstack@n-api.service    loaded active running
 devstack@n-cauth.service  loaded active running
 devstack@n-cond-cell1.service loaded active running
 devstack@n-novnc.service  loaded active running
 devstack@n-sch.service    loaded active running
 devstack@n-super-cond.service loaded active running
 devstack@placement-api.service loaded active running
● devstack@q-agt.service   loaded failed failed 
 devstack@q-dhcp.service loaded active running
 devstack@q-l3.service   loaded active running
 devstack@q-meta.service loaded active running
 devstack@q-svc.service  loaded active running
 system-devstack.slice  loaded active active

So I can see that q-agt.service is not running. This is a critical component of Neutron so lets continue troubleshooting by trying to start that service. The service started after running systemctl start devstack@q-agt.service but failed again within a minute or so.

journalctl -u devstack@q-agt.service revealed:

CRITICAL neutron [-] Unhandled error: Exception: Could not retrieve schema from tcp:127.0.0.1:6640: Connection refused
...
ovs|00002|db_ctl_base|ERR|transaction error: {"details":"Transaction causes multiple rows in \"Manager\" table to have identical values (\"ptcp:6640:127.0.0.1\") for index on column \"target\". First ro

A quick google search lead me to: https://ask.openstack.org/en/question/109750/systemd1-devstackq-agtservice-failed/

SELinux… just to confirm I ran a setenforce 0 and start the service again – all is fine. In a proper environment I would not be satisfied with just leaving SELinux disabled… but for the lab I will move on with it disabled. With devstack@q-agt.service running now – Horizon is loading as expected.

So back to the coursework, the objectics of session 4:

• Describe the purpose and use of tenants, users, and roles.
• Differentiate between administrative scopes in Horizon.
• Discuss the different components that are required for deploying instances from Horizon.
• Deploy an instance from Horizon.

Logging in as as admin we look at the admin interface in Horizon and discuss the separation of tenants via projects, the view of infrastructure and instances. Creating a tenant (project) and a user is then completed… pretty straight forward. Interesting note is that under projects/tenants a ‘service’ project is created by default for the OpenStack services. I can see that cinder, placement, glance, nova and neutron users have been created and added to the service project.

Project Quotas are discussed as a method for limiting the amount of resources a tenant can consume. Creating a user to add to the project is then conducts – providing them with a role, ‘User’ enables them to create VMs networks etc.

What is need to deploy an instance in an OpenStack environment:

• Compute node (nova)
• Networking – at least private network (neutron)
• VM Image (glance)
• Security – Security Groups (nova)
• Storage – Cinder

Creating an Instance via Horizon:

• Configure networking (create a SDN, private + generally attaching floating IPs)
• Assign floating IP addresses
• Define a security group in the cloud
• Create an SSH key pair
• Create a Glance image
• Choose a flavor
• The instance can be booted.

The session runs through these steps in more details. Anyone who has used AWS will be familiar with each step. The only one that  really takes some consideration with this lab environment is the software defined networks. This issue then spilled into my nova service being in accessible thus preventing VMs from being launched. Suffice to say at this point, altering the physical network underlying the whole stack is likely to end badly! I need to get a fuller understanding of how Neutron works with underlying hardware devices and how to reconfigure nova without redeploying the whole devstack.

Session 3 looks at deploying OpenStack via manual, scripted and large scale methods.

• Manual component deployments – see https://docs.openstack.org
• Scripted – PackStack and DevStack are the primary options for scripted deployments of OpenStack
• Large scale deployment – can be achieved with more advanced solutions such as TripleO, Director and others

In a typical OpenStack deployment there will be a number of node roles. For example:

• Controller node – Typically the node controlling services (Keystone, Message queue, MariaDB, time servers etc)
• Network controller node – Providing network services (routing -internal and external, software defined networking)
• Compute nodes – Hypervisors with Nova agents
• Storage nodes – Swift / Ceph etc

Of course for demo environment all of these roles may be fulfilled by 1 server.

DevStack is a scripted deployment tool that is ideal for testing and local machine lab environments. The course material is in reference to the Mitaka release of OpenStack which has already been EOLed and is for Ubuntu based servers which will provide less relevance for my CentOS/RHEL work environment. So, I will deviate from the course slightly by using CentOS 7 and the Pike release of OpenStack.

There are some Docker images of DevStack which were tempting but for the purposes of learning decided to stick with a VM. To install DevStack on CentOS 7 i completed the following:

1. Create a VM on Hyper-V (or whatever) with CentOS 7 minimal (I choose to provide 6GB RAM (4GB aluminum), 2 vCPU, 60GB storage
   1. I also created a Hyper V virtual internal network which enabled static internal IP addresses and an external Hyper V network for internet connectivity

2. ssh to the VM and download devstack and install via (depending on your internet connection this can take more than 1 hour):

   git clone https://git.openstack.org/openstack-dev/devstack
   cd devstack
   # create local.conf with the following contents:
   #	[[local|localrc]]
   #	ADMIN_PASSWORD=secret
   #	DATABASE_PASSWORD=$ADMIN_PASSWORD
   #	RABBIT_PASSWORD=$ADMIN_PASSWORD
   #	SERVICE_PASSWORD=$ADMIN_PASSWORD
   ./stash.sh

3. The Horizon interface should now be waiting for you when hitting the VMs IP / Hostname via a browser

Installing an all-on-one VM deployment of OpenStack on CentOS 7 using DevStack took all of about 30 minutes. This was pretty simple and seamless. Later I will need to try installing RDO via the scripted method – PackStack.

Back the course material we take a look at the node roles:

• Controller node – Keystone, message queue, MariaDB and other critical services. May be 1, 1+n redundancy, n+n were high load is expected.
• Network node – Providing the software defined networking (Neutron)
• Compute node – Run the instances (Nova agent + Hypervisor)
• Storage node – Swift/Ceph

There is a couple of slides now talking specifically about RedHat and CentOS. RDO is the OpenSource version of RedHats OpenStack platform and then a was through of deploying OpenStack on CentOS 7 with PackStack

After a pretty basic Session 1, looking forward to focusing in more on OpenStack. We start off with some history and a look at the OpenStack Foundation.

OpenStack started in 2010 as a joint project between RackSpace (providing Swift) and NASA (providing Nova). The role of the OpenStack Foundation is described as:

to promote the global development, distribution, and adoption of the cloud operating system. It provides shared resources to grow the OpenStack cloud. It also enables technology vendors and developers to assist in the production of cloud software.

That’s a bit to abstract for me to understand… but anyway.. also mentioned is information on how to contribute and get help with OpenStack. I think https://ask.openstack.org/en/questions/ will come in very handy. As OpenStack is a community project hopefully I can find something to contribute here – https://wiki.openstack.org/wiki/How_To_Contribute.

We know start looking at the OpenStack Projects. Being aware of these projects and their maturity status is critical for operating an OpenStack deployment effectively.

Core OpenStack Projects

• Nova Compute (interface to the hypervisor)
• Neutron Networking (software defined networking)
• Swift Object Storage (scattered binary object storage)
• Glance Image (Image service, dont install VMs, deploy images)
• Cinder Block Storage (provide persistent storage to VMs)
• Keystone Identity (users, roles, tenants, services)

There are some other project that have high adoptions rates (>50% of OpenStack deployments):

• Heat – Orchestration of Cloud Services via code (text definitions) and also provides auto-scaling ala AWS CloudFormation
• Horizon – OpenStack’s dashboard with web interface
• Ceilometer – Metering and data collection service enabling metering, billing, monitoring and data driven operations

Other projects introduced in this session:

• Trove – Database as a Service (ie: AWS RDS)
• Sahara – Hadoop as a Service
• Ironic – Bare metal provisioning (very good name!)
• Zaqar – Messaging service with multi tenant queues, high availability, scalability, REST API and web-socket API
• Manila – Shared File System service – Like running samba in the cloud
• Designate – DNS as a Service (backed by either Bind or PowerDNS) – also integrates with Nova and Neutron for auto-generation of DNS record
• Barbican – Secret and Key management
• Magnum – Aims to enable the usage of Swarm, Kubernetes more seamlessly in OpenStack
• Murano – Application catalogue
• Congress – Policy as a Service

After introducing these core services the session delves into a little more detail on the key.

Nove Compute is arguably the most important components. It manages the lifecycle (spawning, scheduling and decommissioning) of all VMs on the platform. Nova is not the hypervisor, it interfaces to the hypervisor you are  using (Xen, KVM, VMware, vSphere) via an agent that is installed on the hypervisor. Nova should be deployed in a distributed fashion where some agents run at local work points and some server processes run on the management servers.

Neutron Networking allows users to define their own networking between VMs they have deployed. Two instances may be deployed on 2 separate physical clusters but the user wants the on the same subnet and broadcast network. Though this can’t be done at the physical level, Neutron’s software defined network enable a logical network to be define which transparently configures the underlying network infrastructure to provide that experience to the user. Neutron uses a pluggable architecture meaning most vendors will enable Neutrons SDNs. Neutron has an API that allows networks to be defined and configured.

Swift Object Storage provides highly scalable storage. It is analgous to AWS’s S3 service. Applications running on OpenStack can talk to a swift proxy which stores the data provided to them on multiple storage nodes. This makes it very fault tolerant. The swift proxy is able to make many parallel requests to storage nodes making scalability quite easy. The swift services can be interfaced with via a RESTful api.

Glance Image provides the ability to store virtual disk images. Glance should use Swift/Ceph as a scalable backend for storing the images. A list of ready to download images can be found here: https://docs.openstack.org/image-guide/obtain-images.html – Windows images are available (supported with Hyper-V and KVM hypervisors). An example of deploying an image to Glance (when using KVM):

gunzip -cd windows_server_2012_r2_standard_eval_kvm_20170321.qcow2.gz | 
glance image-create --property hypervisor_type=QEMU --name "Windows Server 2012 R2 Std Eval" 
--container-format bare --disk-format qcow2 --property os_type=windows

Cinder Block Storage is, in essence, the same as AWS Elastic Block Storage [EBS] whereby persistent volumes can be attached to VMs. Cinder can use Swift/Ceph (or linux LVM) as a backend for storage. Instance storage, without Cinder Block Storage is ephemeral.

Keystone Identity provides authentic and authorization services for OpenStack services. Keystone also provides the central repository for available services and their end points. Keystone also enables definition of Users, Roles that can be assigned to Projects (Tenants). Keystone uses MariaDB by default but can use LDAP (not sure if a DB backend is still required in that case).

Behind the core OpenStack services above – there are some other critical services (dependencies):

• Time synchronization – OpenStack services depend on this for communication, in particular Keystone issues access tickets that are tied to timestamps
• Database – MariaDB (by default) for Keystone is a critical services
• Message Queue – Enable message passing between services – which given the RESTful communications is again critical

Following on from the brief overview of key components of OpenStack we look at the RESTful api – basically just stating that HTTP with JSON is prevalent. If one wanted to basically all OpenStack operations could be complete with cURL.

Horizon is the introduced as a web-based GUI alternative to using the RESTful APIs or the command line client. The command line client  can be configure to point to Keystone from which it will discover all the other available services (Nova, Neutron, Swift, Glance etc). The Horizon Dashboard distinguished between Administrators and Tenants but based on our initial testing.

That’s a wrap for, Session 3 we will start deploying OpenStack!

In looking for an online, at your own pace course for getting a foundation understanding of OpenStack I came across edx.org’s OpenStack course (LFS152x). The full syllabus can be downloaded here.

Out of this course I hope to get an understanding of:

• The key components of OpenStack
• Hands on experience via some practical work
• A local lab environment for further learning
• Some resources that I can go back to in the future (ie: best forums)
• The history and future of OpenStack
• The next steps for building expertise with OpenStack

The course kicks off in Session 1 with a bunch of introductory information (including a page or so on The Linux Foundation who run more project I use than I was aware).

After the introductory items we go over the evolution from physicals servers to virtualization to cloud and why each step has been take… which really boils down to efficiency and cost savings.

• Physical servers suck because they take up space and power and are difficult to properly utilize (physical hosts alone generally operate at < 10% capacity)
• Virtualization lacks self-service
• Virtualization has limited scalability as it is manual
• Virtualization is heavy -> every VM has its own kernel
• Containers are better than VMs by visualizing the operating system (many OS to 1 kernel)
• Containers are also good because they remove a number of challenges along the deployment/development pipeline

Interestingly this introductory seems to focus in on containerization, describing what container images as the Application, User Space dependencies and Libraries required to run. Every running container has 3 components:

1. Namespaces (network, mounts, PIDs) – provide isolation for processes in the container
2. CGroups – reserve and allocate resources to containers
3. Union file system – merge different filesystems into one, virtual filesystem (ie: overlayfs)

Some pros and cons of containers are discussed –  I am not sure about the security pros – versus VMs but I think the value provided by containerization has been well established.

Next up is some discussion on Cloud Computing. Though a lot of this stuff is fairly basic, its nice to review every now and then. The definition provided for Cloud Computing:

Cloud computing is an Internet-based computing that provides shared processing resources and data to computers and other devices on demand. It enables on-demand access to a shared pool of computing resources, such as networks, servers, storage, applications and services, which typically are hosted in third-party data centers.

The differences between IaaS, PaaS and SaaS are covered, a decent diagram to spot the differences (with Application representing the Software as a Service category):

A great point mention is that “If you do not need scalability and self-service, you might be better off using virtualization.” – which in my experience is very true. For some clients the added complexity that comes with enabling self service and dynamic scalability are not used and the stability and relative simplicity of static virtual machines is a better solution.

We then run through an example of deploying a VM on AWS… with the conclusion that OpenStack is about the same and has a more developed API (not sure about that yet!).

Will move on to Session 2 and hopefully start digging into OpenStack more specifically!