About the Mintpress Infrastructure Framework

The Mintpress Infrastructure Framework provides the ability to access and optionally build/manage infrastructure in a target-agnostic fashion - the goal being that, with only some configuration changes, you can use the same code to build infrastructure across multiple cloud and non-cloud platforms (for example doing dev/test in AWS, but production on VMWare). In this way, it is similar to systems such as terraform however it is less singularly focused on cloud deployments.

To achieve this, we seperate the system into types of service and providers of that service - for example: * A Host is something that can be connected to via a transport, such as a VM, a Container, or a physical machine. * A VMHost is a virtual machine “somewhere”, whereas AmazonEC2 is a service which can provide virtual machines.
* A DnsEntry is an entry in DNS somewhere, whereas PowerDNS and AmazonRoute53 are services which can provide this * A BootstrapProvider is a way to bootstrap systems, whereas ChefBootstrapper is a service which can provide this

Additionally, we have Platform providers, which will provide an entire suite of services from a platform - examples of this include Amazon AWS and Oracle Cloud. These generally have an “all_platform_services” option, which if set to true will expose all providers - otherwise they can still be used for centralising items such as platform credentials.

Using Host Objects

The simplest use of Host objects is to provide an SSH-like endpoint (often with SSH). Most other resources take these as a parameter, in order to know where to target their items. In this form, they take a number of connection details, as you’d pass to a normal SSH session. For example:

# Require the infrastructure gem
require 'mintpress-infrastructure'
# Some shorthand
include MintPress::Infrastructure

# Define a host that we connect with 'mintpress', then switch to 'oracle', using a dedicated SSH key
my_ssh_host = Host.new(name: "testosi1.mintpress.io",
                       connect_user: "mintpress",
                       final_user: "oracle",
                       keys: "~/.ssh/mintpress-key.pem")

# Execute a command on this host using las-rpc-utils
# For more details on what you can do with transport, consult the las-rpc-utils documentation
puts my_ssh_host.transport.execute("uptime").stdout

Hosts can also have admin users - root on unix like systems, although this can be something else, and export an admin_transport which will connect with these credentials

require 'mintpress-infrastructure'
require 'mintpress-resources'
include MintPress::Infrastructure
include MintPress::Resources

# Define a host that we connect with 'mintpress', then switch to 'oracle', using a dedicated SSH key
my_ssh_host = Host.new(name: "testosi1.mintpress.io",
                       admin_connect_user: "srv-admin",
                       admin_final_user: "root",
                       admin_keys: "~/.ssh/root-key.pem")

# Show the contents of shadow via las-rpc-utils
puts my_ssh_host.admin_transport.content("/etc/shadow")

# Create a hosts file entry - see the mintpress-resources documentation for more details about this resource
HostsfileEntry.new(host: my_ssh_host,
                   name: "foo-host",
                   ip: "192.168.1.100").create

This is also true in devops tools - the chef equivalent of the above is:

infrastructure_host "testosi1.mintpress.io" do
  admin_connect_user "srv-admin"
  admin_final_user "root"
  keys "~/.ssh/root-key.pem"
end

os_hostsfile_entry "add foo-host entry" do
  host "infrastructure_host[testosi1.mintpress.io]"
  name "foo-host" # note, this is the name parameter for this resource and could be set above
  ip "192.168.1.100"
  action :create
end

By default, this will look up the name via DNS - see the reference documentation for other parameters which can be applied.

Accessing Cloud Hosts Directly

Of course, if we’re talking to a system with more intelligence at the API point, we may wish to look up hosts via a different mechanism - we’ll talk more about cloud providers in the following sections, however if you target a specific provider derivitive of Host, it will use the API to find the connect details rather than them having to be specified. A simple example for amazon EC2 looks like:

require 'mintpress-infrastructure-aws'
include MintPress::InfrastructureAws
my_ssh_host = EC2Host.new(name: "ec2-host.mintpress.io", keys: "~/.ssh/aws-key.pem", 'platform.region': 'us-west-2')
puts my_ssh_host.admin_transport.content("/etc/shadow")

Doing a similar thing in chef would look like:

infrastructure_aws_ec2_host "ec2-host.mintpress.io" do
  keys "~/.ssh/aws-key.pem"
  # note that, in chef, dots are always replaced with underscores
  platform_region 'us-west-2'
end

# Why are we doing a cat here instead of grabbing the content, as in ruby?
# That is because chef does not support resources returning values, so we can't just
# have a resource to "grab" the content into a thing.  But we can display the output
# by having the resource cat it instead, so that's what we'll do.
os_execute "cat /etc/shadow" do
  # the infrastructure_aws_ec2_host[] part is not required as long as your names are unique
  host "ec2-host.mintpress.io"
end

This works because the EC2Host object knows that the default connect user is ec2-user, and that the default admin user is root - so the system is able to form the connection details without help from the user.

Transport Factories

When dealing with many hosts with the same credentials, it is often helpful to externalize that configuration - especially outside of ruby, where you may not have the same looping constructs you have within ruby. To assist with this, we created the TransportFactory object. These act as a set of defaults for generic hosts (if you’re working with a cloud provider, it may make sense to use the APIs of the cloud instead - see the next section about providers for details on how to work in that way!). A simple example is:

TransportFactory.new(  admin_connect_user: "srv-admin",
                       admin_final_user: "root",
                       admin_keys: "~/.ssh/root-key.pem",
                       connect_user: "mintpress",
                       final_user: "oracle")
5.times { |i|
  Host.new("testosi#{i}.mintpress.io").transport.execute("uptime")
}
+
5.times { |i|
  Host.new("testosi#{i}.mintpress.io").admin_transport.execute("tail /var/log/messages")
}

More usefully, in most devops integration - such as chef - this allows host objects to be automaticaly created on demand. For example, to install java on 5 hosts:

host_list = [ "peter.mintpress.io", "paul.mintpress.io", "mary.mintpress.io", "simon.mintpress.io", "garfunkel.mintpress.io"]

infrastructure_transport_factory 'factory' do
  admin_final_user 'root'
  connect_user 'root'
  final_user 'oracle'
  keys ['~/.ssh/id_dsa']
  connect_port 22
end

host_list.each do |my_host|
  oracle_java_installation "java_#{my_host}" do
    version "1.8.0_172"
    software_stage "/oracle/stage/jdk/jdk-8u172-linux-x64.tar.gz"
    java_home "/oracle/app/orchtest/product/jdk"
    host my_host
    action :install
  end
end

Getting Started with providers

Providers are differentated from other resource types via the “Using” verb - the example we’ll use here is for Amazon EC2. Like other resources within the system, these are available via both native Ruby classes, as well as devops systems such as Chef and Ansible. By declaring the use of a provider, two things are provided to the system:

a) a set of default implementations - for example, “VMHost” will build machines on the particular cloud specified. b) a set of default parameters - any parameters passed to the provider will be automatically passed to any created instances, allowing you to only specify the things which are different between machines. In the examples below, these are the image to build from and the SSH keys to use for such, however any parameters which can be passed to a child of a provider can be passed to the provider itself.

A simple example using chef:

# Set up our platform
using_aws_platform "aws_platform" do
  all_platform_services true
  image 'ami-074e2d6769f445be5'
  key_name 'jj-keypair'
  connect_user 'centos'
  admin_connect_user 'centos'
end

# As per all resources, dots are replaced with underscores in chef resource names
infrastructure_vm_host "jjtest-small.limepoint-training.uk" do
  specs_cpu_count 4
  specs_ram_gb 32
  action :create
end

The same example using ruby:

# Require the AWS gem
require 'mintpress-infrastructure-aws'
# Some shorthand
include MintPress::InfrastructureAws
include MintPress::Infrastructure

# declare our platform 
UsingAwsPlatform.new(all_platform_services: true,
                     image: 'ami-074e2d6769f445be5',
                     key_name: 'jj-keypair',
                     region: 'us-west-1', 
                     connect_user: 'centos',
                     admin_connect_user: 'centos')

# Now do something with it.... lets just create a host (this will be explained further later)
my_host = VMHost.new(name: "jjtest-small.limepoint-training.uk",
                           'specs.cpu_count': 4,
                           'specs.ram_gb': 32)
my_host.create

Behind the scenes, the provider will turn the VMHost class into a native class for that type - in the AWS example above, the VMHost.new will actually instanstiate a EC2Host class - if you wish to pass in parameters which are specific to this type, or otherwise call the type directly rather than going through a provider, you are free to do that. For example:

# Specifically build an EC2 host in us-west-2, even though our general platform may well
# be somewhere else.
my_host = EC2Host.new(name: "jjtest-small.limepoint-training.uk",
                      region: 'us-west-2')
my_host.create

This distinction becomes particularly important outside of ruby, where parameter lists are more stringintly enforced (in ruby, the class transformation happens before the parameters are passed, therefore it is possible to pass provider-specific parameters to generic classes like VMHost, however this is not possible in systems such as Chef or Ansible).

Creating complex items via composite resources

Some infrastructure items, Hosts being the simplest example, are built of composite resources - that is to say, build from pieces added with references - as with other parts of the system. These can either be specified as Hashes, or as resources with back references. So the following two items in cheffish syntax

infrastructure_vm_host "testbox1.mintpress.io" do
  action :create
end

infrastructure_network_interface "mgt-nic" do
  host "testbox1.mintpress.io"
  postfix "-mgt"
end

infrastructure_network_interface "data-nic" do
  host "testbox1.mintpress.io"
  postfix "-data"
end

and

infrastructure_host "testbox1.mintpress.io" do
  network_interfaces { "mgt-nic" => { "postfix" => "-mgt" }, "data-nic" => { "postfix" => "-data" } }
  action :create
end

should yield an identical configuration. In ruby syntax, there are also add_xxx functions, as per all resources (for more details on this, see the Property System reference in mintpress-utils)

host = VMHost.new(name: "testbox1.mintpress.io")
host.add_network_interface(name: 'data', postfix: "-data", public_postfix: "")
host.add_network_interface(name: 'mgt', postfix: "-mgt")

Defining Networking

We saw in the previous section that you can add network interfaces, but often you’ll also need to provide some other configuration - perhaps static IP configuration, but also such as adding routes to your wider network. This is provided for via the ip, netmask, gateway, routes, and static_routes attributes.

The semantic difference between routes and static_routes, is that routes provide network routes which go through the interfaces gateway - this is almost always what you actually want to define. static_routes allows you to configure routes which traverse alternate gateways accessable through the same network interface - for example, if you have a second router on a different IP which can access other parts of your network.

# Access 192.168.1.0/24 through the MGT interface
host.add_network_interface(name: 'mgt', postfix: '-mgt', routes: [IPAddr.new('192.168.1.0/24')])

# Access 10.0.0.0/8 through a dedicated router - this is rare, and you would almost never do this
# in a usual network configuration, but of course not all network configurations are usual
host.add_network_interface(name: 'mgt', postfix: '-mgt', static_routes: {'192.168.1.4' => '10.0.0.0/8'})

In order to configure a static network ip, you need to set the static_ip property of a network interface to true - also note that not all cloud providers support this configuration.

# Define our network interface statically, and make it the default route
host.add_network_interface(name: 'primary', postfix: '', static_ip: true, ip: '192.168.1.44', netmask: '255.255.255.0', gateway: '192.168.1.1', routes: IPAddr.new('0.0.0.0/0'))

Mounting Disks

Addon Providers

An addon provider is a provider of a service to an existing provider - for example, EBS Block storage is provided to EC2 virtual machines, since it generally doesn’t make sense to use these in isolation (it is still possible to do so, but is not usually useful). We differentiate these by the word “Extra”, for example:

# Add a mintpress disk to every VM we create
UsingExtraEbsStorage(name: "mintpress-disk",
                     size_mb: 10000,
                     mount_point: "/limepoint",
                     device_node: "sdf")

# This will create a host with a 10gbyte disk mounted as /limepoint
my_host = VMHost.new(name: "jjtest-small.limepoint-training.uk")
my_host.create
# ... adn another!
my_host2 = VMHost.new(name: "jjtest-medium.limepoint-training.uk")
my_host2.create

Note that you could have, instead, passed a hash to the block_devices section of your UsingAwsPlatform or UsingEC2Host sections, however it’s generally considered that this is more pleasing to read and write

# This will work, but it is ugly and we don't recommend it
# Note the shortcut of using the hash key for the name though - this works everywhere in the system!
UsingAwsPlatform.new(all_platform_services: true,
                     image: 'ami-074e2d6769f445be5',
                     key_name: 'jj-keypair',
                     region: 'us-west-1', 
                     connect_user: 'centos',
                     admin_connect_user: 'centos',
                     block_devices: {'mintpress_disk' => { :size_mb => 10000, :mount_point => "/limepoint", :device_node => "sdf" }})

The other common usage for these is attaching a default set of network interfaces

# Always add a public and a management interface to our syste,s
UsingExtraEC2NetworkInterface.new(name: "default", postfix: "", interface_type: "public", subnet: 'subnet-0e5e4f55')
UsingExtraEC2NetworkInterface.new(name: "mgt", postfix: "-mgt", interface_type: "management", subnet: 'subnet-0e5e4f55')

# Create a host with these already attached
my_host = VMHost.new(name: "jjtest.limepoint-training.uk")

# This will show the two network interfaces
pp my_host.network_interfaces

# Actually do the creation
my_host.create

Actions on Hosts

Most providers will implement at least the following actions:

• create - ensure the host is created

• destroy - destroy the host

• start - startup the host

• stop - shut down the host

• restart - this will call stop then start

• bootstrap - run any bootstrappers

• unbootstrap - remove the host from any bootstrapped system

Actions on DNS Entries

DNS Entry providers will implement at least the following actions: * create - ensure the record exists DnsEntry.new(name: 'foo.bar', type: 'A', values: '192.168.1.4').create * remove - ensure the record does not exist DnsEntry.new(name: 'foo.bar').remove

Using multiple providers

Sometimes, you may wish to define multiple providers of a particular service - DNS is a reasonably common case, where you may have an internal DNS and an external DNS. To facilitate this, all items within the provider framework take a ‘provider’ parameter, to specify which provider to use. For example:

include MintPress::Infrastructure
include MintPress::InfrastructureAws

# Instantiate two providers
UsingPowerDnsEntry(name: 'pdns', api_key: 'abcd')
UsingRoute53DnsEntry(name: 'r53dns', hosted_zone_name: 'mintpress.io')

# Create an entry in powerdns
DnsEntry.new(name: 'foo.bar', type: 'A', values: '192.168.1.4', provider: 'pdns').create
# Create an entry in route53
DnsEntry.new(name: 'foo.bar', type: 'A', values: '192.168.1.4', provider: 'r53dns').create

# Create a DNS entry from a host - this would normally be donme in simple bootstrapper, but you can do it manually, for
# example if you need to split accross multple providers
DnsEntry.new(name: 'my_host_alias', type: 'A', values: host.public_ip, provider: 'r53dns').create
DnsEntry.new(name: 'my_host_alias-mgt', type: 'A', values: host.network_interfaces['mgt'].public_ip, provider: 'pdns').create

A more complex example involves building accross multiple cloud providers:

using_vsphere_host "vmware_provider" do
  username "jj@limepoint.com"
  password "secret1"
  datacenter "limepoint-internal"
  vm_folder "test-machines"
end

using_aws_platform "aws_platform" do
  all_platform_services true
  image "ami-074e2d6769f445be5"
  key_name "limepoint-test-key"
  region "us-west-2"
  connect_user "centos"
  final_user "mintpress"
  admin_connect_user "centos"
  admin_final_user "root"
  # Lets have a default amount oc CPU/RAM as well
  specs_cpu_count 2
  specs_ram_gb 8
end

host "aws_box" do
  provider "aws_platform"
end

host "vmware_box" do
  provider "vmware_provider"
end

Host Instance Types

Since we are attempting to provide a cloud-agnostic experience out of the box, the instance type is generally by specifying the CPU and RAM required, at which point the system will query the cloud provider to find the best matching instance type. If you wish to override this behaviour, you can specify the native_instance_type attribute with an appropriate instance type for your cloud platform. Of course, if you do this, you will lose the ability to build across multiple cloud providers - however, in many organisations, this may be a reasonable tradeoff in order to control exactly which instance types you recieve.

To control the matching, the following attributes are used, which are all a part of the InstanceType class:

• cpu_count represents the ideal number of CPUs to allocate. By default, this will be an exact match due to softare licensing reasons, but see the below properties for tweaking this. If min_cpu_count/max_cpu_count are specified, this can be omitted and they will be used instead.

• min_cpu_count represents the minimum number of CPUs to allocate. This defaults to being exactly cpu_count

• max_cpu_count represents the maximum number of CPUs to allocate. This defaults to being exactly cpu_count

• ram_gb represents the ideal amount of RAM to allocate

• tolerence_percent represents how far we are allowed to deviate from our requested amount of RAM. The default is 25%, which should be sufficient to make, for example, a 31G amazon instance match a 32G system

• min_ram_gb represents the minimum amount of ram to allocate. By default, this is ram_gb - tolerance_percent

• max_ram_gb represents the maximum amount of ram to allocate. By default, this is ram_gb + tolerance_percent

• dedicated_only tells the system to ONLY return dedicated instances

• is_gpu tells the system to require a GPU capable instance

To use AWS as an example, if you request: * {min_cpu_count: 1, max_cpu_count: 3, min_ram_gb: 0.5} -> t3.nano * {min_cpu_count: 4, max_cpu_count: 12, min_ram_gb: 30.5, max_ram_gb: 32, dedicated_only: true} -> m5.2xlarge * {min_cpu_count: 1, max_cpu_count: 3, min_ram_gb: 3.6} -> c4.large * {min_cpu_count: 1, max_cpu_count: 3, min_ram_gb: 0.75, max_ram_gb: 1} -> t3.micro

Host Bootstrapping

Mintpress provides an internal host bootstrapper, which is called SimpleBootstrapper, which by default will always be run (this can be disabled by setting the always_use_mintpress_bootstrap attribute set to false). This will perform the following basic new host tasks:

1) Set the system hostname 2) Bringup any attached networking interfaces, and configure them for persistence 3) set /etc/hosts and dns entries for these interfaces (DNS can be suppressed via the bootstrap_with_dns attribute on a host) 4) prepare any block devices 5) mount any other mounts specified

If you have a “mostly ready” system image, this may well be enough - especially in the cloud AMI type of case, this should be sufficient to get you moving. For more advanced needs, however, we can also plug in additional bootstrap providers - the inbuilt one is a Chef bootstrapper, however it is also possible to write your own by extending the Bootstrapper class (see the developers guide for more details on this). To register new hosts with a chef server, on a host where you already have knife (and/or chefdk) installed and configured, you can add:

UsingChefBootstrapper.new(chef_environment: 'jj', run_list: ['os-common::default'])

or equivalently:

using_chef_bootstrapper "bootstrap" do
  chef_environment "jj"
  run_list ["os-common::default"]
end

Note that the chef bootstrapper will run in two passes - it will initially run with no run list, to ensure the node is registered correctly, and then will run a second time with the correct run list.

Currently Implemented Providers

• VMHost - Amazon AWS, Oracle Cloud Infrastructure, Oracle Exalogic

• Block Storage - Amazon EBS, Oracle Cloud Infrastructure

• NasStorage - Amazon EFS, Oracle ZFS Appliance

• DnsEntry - Amazon Route 53, Active Directory, Power DNS

• VaultEntry - Amazon KMS

Using sub-providers to add default structure to your hosts

Of course, if you want to build multiple systems, you’re absolutely not going to want to do this for every single host. You could add them to your provider with hashes, like so:

using_aws_platform "aws_platform" do
  all_platform_services true
  image "ami-074e2d6769f445be5"
  key_name "limepoint-test-key"
  region "us-west-2"
  connect_user "centos"
  final_user "mintpress"
  admin_connect_user "centos"
  admin_final_user "root"
  # Lets have a default amount oc CPU/RAM as well
  specs_cpu_count 2
  specs_ram_gb 8
  # And a default blockdev setup
  block_devices { "data_disk" => { :size_mb => 20000, :mount_point => "/data", :device_node => "sdf" } }
end

however, it can be a difficult format to work with, and as such we have introduced sub-providers via the using_extra verb set - these are specific to a provider, and hence are covered in each providers documentation, however an example for AWS would be:

using_aws_platform "aws_platform" do
  all_platform_services true
  image "ami-074e2d6769f445be5"
  key_name "limepoint-test-key"
  region "us-west-2"
  connect_user "centos"
  final_user "mintpress"
  admin_connect_user "centos"
  admin_final_user "root"
  # Lets have a default amount oc CPU/RAM as well
  specs_cpu_count 2
  specs_ram_gb 8
end

using_extra_ebs_block_storage "data_disk" do
  size_mb 20000
  mount_point "/data"
  device_node "sdf"
  # This is optional if there is only a single provider - it's included here to be
  # explicit
  parent_provider "aws_platform"
end

using_extra_ebs_block_storage "log_disk" do
  size_mb 20000
  mount_point "/data"
  device_node "sdf"
  # This is optional if there is only a single provider - it's included here to be
  # explicit
  parent_provider "aws_platform"
end

host "testbox1.mintpress.io" do
  action :create
end

About Bootstrap Providers

Bootstrap providers are responsible for configuring the system, as well as registering them with any services such as DNS and configuration management systems. By default, we currently ship with two options - SimpleBootstrapper, which is the default, and ChefBootstrapper, whcih registers the node with a chef server and configures it. These can be stacked, and SimpleBootstrapper will always be run unless you explicitly exclude it. By default, SimpleBootstrapper does the following:

• Set the hostname

• Configure any defined network interfaces

• Format any empty disks, as well as configuring into LVM if desired

• Mount any disks

• Mount any NAS shares

• Register all hostnames defined into DNS

It is generally strongly recommended to not disable this, however it is allowed if you are confident that everything you need is baked into either your template image, or your alternative bootstrap option. Below is an example of using the chef bootstrapper instead of the default.

# If you have an existing knife.rb, and chef DK installer, all of the required configuration should be
# picked up from there
using_chef_bootstrapper "chef_bootstrap" do
  chef_environment "dev"
  run_list "limepoint-config::build-database-server"
end