Nova seeks to provide a framework for the large-scale provisioning and management of virtual compute instances. Similar in functionality and scope to Amazon’s EC2 service, it allows you to create, manage, and destroy virtual servers based on your own system images through a programmable API.
Nova is architected as a distributed application with many components, but the majority of these are custom-written Python daemons of two varieties:
Web Server Gateway Interface (WSGI) applications to receive and mediate API calls
Worker daemons to carry out orchestration tasks
However, there are two essential pieces of the architecture that are neither custom written nor Python-based: the messaging queue and the database. These two components facilitate the asynchronous orchestration of complex tasks through message passing and information sharing. Piecing this all together we get a picture like Figure 4-1.
This complicated diagram can be summed up in three sentences:
End users who want to use Nova to create compute instances call nova-api with OpenStack API or EC2 API requests.
Nova daemons exchange information through the queue (actions) and database (information) to carry out these API requests.
Glance is a completely separate service that Nova interfaces through the Glance API to provide virtual disk imaging services.
Now that we’ve seen the overview of the processes and their interactions, let’s take a closer look at each component.
The nova-api daemon is the heart of Nova. You may see it illustrated on many pictures of Nova as API and “Cloud Controller.” While this is partly true, cloud controller is really just a class (specifically the CloudController in nova/api/ec2/cloud.py) within the nova-api daemon. Its primary purpose is to accept and fulfill incoming API requests.
To accept and fulfill API requests, nova-api provides an endpoint for all API queries (accepting requests using either the OpenStack API or the Amazon EC2 API), initiates most of the orchestration activities (such as running an instance), and also enforces some policy (mostly quota checks). For some requests, it will fulfill the entire request itself by querying the database and then returning the answer. For more complicated requests, it will pass messages to other daemons through a combination of writing information to the database and adding messages to the queue.
By default, nova-api listens on port 8773 for the EC2 API and 8774 for the OpenStack API.
The nova-scheduler process is conceptually the simplest piece of code in Nova: it takes a virtual machine instance request from the queue and determines where it should run (specifically, which compute server host it should run on). In practice, however, this will grow to be the most complex piece, as it needs to factor in the current state of the entire cloud infrastructure and apply complicated algorithms to ensure efficient usage. To that end, nova-scheduler implements a pluggable architecture that lets you choose (or write) your own algorithm for scheduling. Table 4-1 details the current scheduler choices.
Table 4-1. Nova Schedulers
|Simple||Attempts to find least loaded host.|
|Chance||Chooses random available host from service table. This is the default scheduler.|
|Zone||Picks random host from within an availability zone.|
To illustrate how simple nova-scheduler can be, here is the relevant code from the chance scheduler class in nova/schedule/chance.py:
class ChanceScheduler(driver.Scheduler): """Implements Scheduler as a random node selector.""" def schedule(self, context, topic, *_args, **_kwargs): """Picks a host that is up at random.""" hosts = self.hosts_up(context, topic) if not hosts: raise driver.NoValidHost(_("Scheduler was unable to locate a host" " for this request. Is the appropriate" " service running?")) return hosts[int(random.random() * len(hosts))]
As you can see from above code sample, the schedule method simply chooses a random host from the array of hosts that are currently known to be “up.”
The nova-compute process is
primarily a worker daemon that creates and terminates virtual machine
instances. The process by which it does so is fairly complex, but the
basics are simple: accept actions from the queue and then perform one or
a series of virtual machine API calls to carry them out while updating
state in the database. An example of this would be nova-compute accepting a message from the
queue to create a new instance and then using the
libvirt library to start a new KVM
There are a variety of ways that nova-compute manages virtual machines. The
most common is through a software package called
libvirt. This is a toolkit (API, daemon, and
utilities) created by Red Hat to interact with the capabilities of a
wide range of Linux virtualization technologies. While
libvirt may be the most common, nova-compute also uses the Xen API, vSphere
API, Windows Management Interface, and others to support other
One of strengths of Nova is its wide support for virtualization technologies. The virtualization technologies supported in the current release version of Nova are detailed in Table 4-2.
Table 4-2. Virtualization Support in Nova
|Virtualization Product||Supported||Interface||Support Notes|
|Kernel Virtual Machine (KVM)||Yes||libvirt|
Most popular technology for small scale deployments. Arguably the easiest to deploy and configure. Supports advanced operations such as live migration and resize.
Most popular (along with XCP/XenServer) technology for larger scale and production deployments.
Citrix’s commercial version of Xen-based virtualization product. Supports advanced features.
|Xen Cloud Platform (XCP)||Yes||XenAPI|
The open source version of Citrix XenServer available under the LGPL, GPL, Q Public License v1. Supports subset of XenServer features.
|VMware ESX / ESXi / vSphere||Yes||vSphere API|
Most popular enterprise virtualization platform. See the OpenStack VMware documentation for full information and restrictions when using this option.
|User Mode Linux||Yes||libvirt|
Generally considered a lower performance virtualization option, UML runs each guest as a regular process in user space.
|Microsoft Hyper-V||Yes||Windows Management Instrumentation (WMI)|
Hyper-V is Microsoft’s hypervisor-based virtualization technology.
Provides the basis for most Linux-based virtualization technologies (such as KVM and Virtualbox).
|Linux Containers (LXC)||Yes||libvirt|
LXC is an operating system-level partitioning technology that allows for running multiple isolated servers (containers) in a single kernel. LXC does not actually virtualize the server. Instead, it provides a virtual environment with its own process space. While this doesn’t provide the same level of isolation (as every partition shares the common kernel), it may provide some advantages in I/O performance.
|Oracle VM VirtualBox||No|
Virtualization Technology describes some important considerations that should be taken into account when choosing your virtualization technology.
As you can gather by the name, nova-volume manages the creation, attaching, and detaching of persistent volumes to compute instances (similar in functionality to Amazon’s Elastic Block Storage). It can use volumes from a variety of providers such as iSCSI or AoE. Table 4-3 shows the current volume provider options.
Table 4-3. Nova Volume Provider Options
|AoE||High performance layer 2 Ethernet technology that encapsulates SATA commands in Ethernet frames. Supported on Linux through the AoE Tools package, specifically the vblade program.|
|iSCSI||A commonly used IP-based encapsulation of SCSI commands. This is supported by most modern operating systems, but the Nova implementation only currently supports Linux through with this implementation. This driver does support CHAP for authentication.|
|Solaris iSCSI||Supports Solaris-hosted iSCSI volumes and uses ZFS commands. Solaris server must be prepared by following the instructions in the nova/volume/san.py file.|
|Sheepdog||An open-source, distributed storage system specifically designed for QEMU/KVM installations that is developed by NTT Laboratories. More information is available at http://www.osrg.net/sheepdog/.|
|RBD||RADOS block device (RBD) driver to interact with Ceph, a distributed file system based on a reliable and scalable distributed object store. As stated on http://ceph.newdream.net/ the wiki, “Ceph is under heavy development, and is not yet suitable for any uses other than benchmarking and review.”|
|LeftHand||A driver for interacting with HP Lefthand SAN solutions (as of now known as “HP P4000 SAN Solutions”). Unlike other providers mentioned above, this provider does not run directly on the SAN hardware. Instead, it accesses it via SSH commands.|
The nova-network worker daemon is very similar to nova-compute and nova-volume. It accepts networking tasks from the queue and then performs system commands to manipulate the network (such as setting up bridging interfaces or changing iptables rules).
Nova defines two different types of IP addresses for an instance: Fixed IPs and Floating IPs. These can be broadly thought of as private IPs (fixed) and public IPs (floating). Fixed IPs are assigned on instance startup and remain the same during their entire lifetimes. Floating IPs are dynamically allocated and associated to a domain to allow outside connectivity.
To support the assignment and connectivity of fixed IPs, Nova supports three networking managers:
Flat is the most basic network manager. Each new instance is assigned a fixed IP address and attached to a common bridge (which must be created by the administrator). IP configuration information must be “injected” (written into the new instance virtual disk image) to configure the instance.
FlatDHCP builds upon the Flat manager by providing DHCP services to handle instance addressing and creation of bridges.
VLAN supports that most features. In this mode, nova-network creates a VLAN, a subnet, and a separate bridge for each project. Each project also receives a range of IP only accessible within the VLAN.
Of these three network managers, VLAN is the most featured, Flat is the most barebones (but flexible), and FlatDHCP strikes a nice balance between the two.
The queue provides a central hub for passing messages between daemons. This is currently implemented with RabbitMQ today, but theoretically could be any AMPQ message queue supported by the Python ampqlib and carrot libraries.
Nova creates several types of message queues to facilitate communication between the various daemons. These include: topics queues, fanout queues, and host queues. Topics queues allow messages to be broadcast to the number of particular class of worker daemons. For example, Nova uses these to pass messages to all (or any) of the compute or volume daemons. This allows Nova to use the first available worker to process the message. Host queues allow Nova to send messages to specific services on specific hosts. For example, Nova often needs to send a message to a specific host’s compute worker to take action on a particular instance. Fanout queues are only currently used for the advertising of the service capabilities to nova-scheduler workers.
Here is an example of the queues created in RabbitMQ for a simple all-in-one node installation:
$ sudo rabbitmqctl list_queues Listing queues ... scheduler_fanout_15b1731c5ac34aae8970369911f04542 0 volume 0 volume_fanout_e42438faedb84ab8aad8d85e29916424 0 compute_fanout_38a37d3dc7564b66a5a540a1e222b12b 0 compute.cactus 0 volume_fanout_d62eb016a76341f4899c91d5a8fbb0a9 0 volume_fanout_dcaebd5edb3045ff8b86636040d34071 0 volume.cactus 0 network_fanout_64b9cb80b2c34c7a8da983219c787147 0 compute 0 network_fanout_362393151e7c465a8e3ed003ac6dbc1b 0 compute_fanout_74165ee38c9d4c1ea1003ccd88a91c22 0 scheduler 0 network.cactus 0 network 0 scheduler_fanout_9444b4c8d5d6497b9b5e2df4eca33b0d 0 scheduler.cactus 0
As you can see from the example, topic, fanout, and host queues have been created for each service (nova-scheduler, nova-compute, nova-volume, nova-network).
The database stores most of the configuration and run-time state for a cloud infrastructure. This includes the instance types that are available for use, instances in use, networks available, and projects. Table 4-4 details all the tables in the current Nova database scheme.
Table 4-4. Nova Database Schema
Stores current version of the database schema as well as other migration-related info. Only used internally and by developers during upgrades.
Used for running host-to-host migration.
Maps Authorization tokens (for all API transactions) to actual users (via the user id field).
Mappings for user, projects, and x509 certificates files
Information pertaining to networks defined in Nova. Includes IP addressing, VLAN, and VPN information.
Capabilities (vcpus, memory, etc.) and state (vcpus used, memory used, etc.) of each compute node.
Information about projects, including project manager.
Pool of consoles on the same physical node.
Quota overrides for particular projects. This is discussed further in Quotas.
Console session for an instance.
Shelf and blade information used primarily with the AoE volume driver.
|security_group_rules, security_groups and security_group_instance_association|
Represent security groups and their associated rules.
|fixed_ips and floating_ips|
Associates the fixed and floating IP addresses to instances.
Listing off registered services (nova-scheduler, nova-compute and so on) and their current state. The updated_at field is used to determine if a given service is considered healthy or not.
Lists guest VM’s actions and results.
|user_project_association, user_project_role_association and user_role_association|
Maintains relationship among users, projects, and roles.
Metadata key/value pairs for an instance that is used during instance startup.
Specifications (vCPUs, RAM, etc.) of flavors or instances types that users can use in this cloud. Described in much greater detail in Instance Types and Flavors.
Representation of virtual machine instances.
Representations of users.
Mapping of iSCSI targets, hosts, and volumes.
Representation of volumes in the cloud.
Public key pairs for SSH.
Represents a child zone of this zone. Only used in advanced configurations.
 WSGI is a Python standard specification that defines the communications between web and application servers.