Exalogic Support

Required Reading

This will only really make sense if you have read the general Mintpress Infastructure Framework documentation

What’s Implemented

The following infrasturcture types are implemented:

• Host/VMHost -> via the ExalogicHost provider, and the UsingExalogicHost resource class

• Networking -> via the UsingExtraExalogicNetworkInterface sub provider

• Block storage -> via the UsingExtraExalogicBlockDevice sub provider, as well as the UsingExalogicBlockDevice for allocations unattached to a host (this is almost never done!)

• ZFS appliance support is provided via the mintpress-infrastructure-zfs-appliance module, but is usually used with this provider - it is accessed via the UsingZfsPlatform resource, and attached via the UsingExtraZfsFilesystem. It can also be attached directly via UsingZfsFilesystem, which provides NasShare resources which attach to VMHosts.

Simple example

require 'mintpress-infrastructure-oracle-exalogic'
require 'mintpress-infrastructure-zfs-appliance'


# Use our exalogic host
MintPress::InfrastructureExalogic::UsingExalogicHost.new(api_url: "http://localhost:8080",
                                                         exalogic_user: "jj",
                                                         access_key: "exa_access_key.txt",
                                                         vserver_template: "EXABASE-OEL58x86-20611-64V3",
                                                         exalogic_key_name: "exakey")

# Use the ZFS appliance platform built into the exa
MintPress::Infrastructure::UsingZfsPlatform.new(api_url: 'https://172.18.117.225:215',
                                                username: 'root',
                                                password: 'welcome1',
                                                verify_ssl: false)

# Add our default network interfaces
MintPress::InfrastructureExalogic::UsingExtraExalogicNetworkInterface.new(name: "default", postfix: "", interface_type: "public", vnet: 'IpoIB_Private_32')
MintPress::InfrastructureExalogic::UsingExtraExalogicNetworkInterface.new(name: "mgt", postfix: "-mgt", interface_type: "management", vnet: 'EoIB-mgmt-32[pvid=2061]')
MintPress::InfrastructureExalogic::UsingExtraExalogicNetworkInterface.new(name: "storage", postfix: "-s", interface_type: "storage", vnet: 'IPoIB-default')
# Add a OracleVM block device for logs
MintPress::InfrastructureExalogic::UsingExtraExalogicBlockDevice.new(name: "bd1", size_mb: 40000, mount_point: "/logs")

# Add a ZFS-A nas share on foo-project, which is our /oracle disk, where we store our actual data
MintPress::Infrastructure::UsingExtraZfsFilesystem.new(name: "jjtest15-fs",
                                                  pool: 'foo',
                                                  project: 'foo-project', 
                                                  size_mb: 100000,
                                                      mount_point: "/oracle")


# Create a host
my_host = MintPress::Infrastructure::VMHost.new(name: "jjtest15.limepoint-training.uk", 
                                                'specs.cpu_count': 2,
                                                'specs.ram_gb': 4)

my_host.create

Simple Chef example

# metadata.rb
depends 'environmint-common'
depends 'mintpress-controllers-infrastructure'
depends 'mintpress-controllers-infrastructure-exalogic'
depends 'mintpress-controllers-infrastructure-zfs-appliance'


# recipe
# Use our exalogic host
using_exalogic_host 'exalogic provisioning' do
  api_url 'http://localhost:8080'
  exalogic_user 'jj'
  access_key 'exa_access_key.txt'
  vserver_template 'EXABASE-OEL58x86-20611-64V3'
  exalogic_key_name 'exakey'
end

# use the ZFS appliance platform built into the exa
using_zfs_platform 'zfs-exalogic' do
  api_url 'https://172.18.117.225:215'
  username 'root'
  password Mint::Secret.new('Welcome1')
  verify_ssl false
end

# set zfs pool
infrastructure_zfs_appliance_zfs_pool 'foo'

# default node attributes for chef
node_attrs = {
  useful_node_attribute: true
}

# define vm host -- don't call create yet -- we'll do that later
infrastructure_vm_host "jjtest15.limepoint-training.uk" do
  native_instance_type '20_CPU_2TB_MEM'
  keys ssh_keyfile
  admin_keys ssh_keyfile
  protocol 'ssh'
  action :update_model
end

# Add our default network interfaces
infrastructure_exalogic_exalogic_network_interface "default" do
  host "infrastructure_vm_host[jjtest15.limepoint-training.uk]"
  postfix ''
  interface_type 'public'
  vnet 'IpoIB_Private_32'
end

infrastructure_exalogic_exalogic_network_interface "mgt" do
  host "infrastructure_vm_host[jjtest15.limepoint-training.uk]"
  postfix '-mgt'
  interface_type 'management'
  vnet 'EoIB-mgmt-32[pvid=2061]'
end

infrastructure_exalogic_exalogic_network_interface 'storage' do
  host "infrastructure_vm_host[jjtest15.limepoint-training.uk]"
  postfix '-s'
  interface_type 'storage'
  vnet 'IpoIB-default'
end

# storage
infrastructure_zfs_appliance_zfs_filesystem "jjtest15-fs" do
  host "infrastructure_vm_host[jjtest15.limepoint-training.uk]"
  platform_api_url 'https://172.18.117.225:215'
  platform_verify_ssl false
  name 'jjtest15-fs'
  pool "infrastructure_zfs_appliance_zfs_pool[foo]"
  project 'foo-project'
  size_mb 100000
  mount_point '/oracle'
  action :create
end


# set up bootstrapper for node
infrastructure_chef_bootstrapper "jjtest15.limepoint-training.uk" do
  chef_environment 'my_env'
  node_attributes node_attrs
  knife_config_file '/path/to/non-default/knife/config.rb'
  run_list ['cookbook::recipe', 'other_cookbook::recipe']
  action :nothing
end

# create vm
infrastructure_vm_host "jjtest15.limepoint-training.uk" do
  bootstrapper "infrastructure_chef_bootstrapper[jjtest15.limepoint-training.uk]"
  action :create
end

Advanced Chef example

# metadata.rb
depends 'environmint-common'
depends 'mintpress-controllers-infrastructure'
depends 'mintpress-controllers-infrastructure-exalogic'
depends 'mintpress-controllers-infrastructure-zfs-appliance'


# recipe

# example list of vms - just one in this case
# this will usually be passed in from another source, eg JSON file with -j chef-client option or defined in another recipe.
# defined here for demonstrative purposes.

list_of_vms = [
  {
    "distribution_group": "abctst_osi_dg",
    "hostname": "abctstosi01.subdomain.domain.tld",
    "networks": {
      "mgt": {
        "name": "abctst-net1",
        "routes": "1.2.3.0/23"
      }
    },
    "storage": [
      {
        "device": "xvdm",
        "mount_point": "/zeros",
        "options": "rw",
        "size": 1,
        "virtual_name": "zeros"
      },
      {
        "device": "xvdn",
        "mount_point": "/ones",
        "options": "rw,bg",
        "size": 1,
        "virtual_name": "ones"
      }
    ]
  }
]

# set some variables here
ssh_keyfile = ::File.expand_path('/path/to/file/or/a/node/attribute/etc')

# sanity checking 
raise 'ssh_keyfile not specified!' if ssh_keyfile.nil? || ssh_keyfile.empty?

# exalogic config
using_exalogic_host 'exalogic provisioning' do
  api_url '1.2.3.4:9443'
  exalogic_user 'emoc_user_1'
  access_key '/path/to/emoc/access/key/file'
  vserver_template 'TEMPLATE_IMAGE_7'
  exalogic_key_name 'exalogic_keypair_name'
  use_provided_dns true
end

# dns config
using_active_directory_dns_entry 'dns' do
  account 'dns_user'
  password Mint::Secret.new("dns_password")
  ad_domain "ad.domain.tld"
  domain_server '9.10.11.12'
  zone 'not.always.the.same.as.ad_domain.tld'
end

# zfs platform config
using_zfs_platform 'exalogic' do
  api_url '1.2.3.4:9443'
  username 'zfs_user_1'
  mount_ip '13.14.15.16'
  password Mint::Secret.new('zfs_password')
  verify_ssl false
end

# set zfs pool
infrastructure_zfs_appliance_zfs_pool 'my_pool'

# default zfs properties
project_props = {
  'acls' => 'yes, but not too many pls',
}

# define zfs project
infrastructure_zfs_appliance_zfs_project 'zfs_project' do
  pool "infrastructure_zfs_appliance_zfs_pool[my_pool]"
  properties project_props
end

# loop over the assets, prepare then build them.
list_of_vms.each do |vm|
  mount_map = []

  # default node attributes for chef
  node_attrs = {
    skip_legacy_provisioning: true,
    useful_node_attribute: true
  }

  # node runlist
  runlist = ['cookbook::recipe', 'other_cookbook::recipe']

  # set up bootstrapper for node
  infrastructure_chef_bootstrapper (vm['name']).to_s do
    chef_environment 'my_env'
    node_attributes node_attrs
    knife_config_file '/path/to/non-default/knife/config.rb'
    run_list runlist
    action :nothing
  end

  # define vm host -- don't call create yet -- we'll do that later
  infrastructure_vm_host (vm['name']).to_s do
    native_instance_type '20_CPU_2TB_MEM'
    keys ssh_keyfile
    admin_keys ssh_keyfile
    protocol 'ssh'
    action :update_model
  end

  # storage
  vm['storage'].each do |vol|

    # create volume
    infrastructure_zfs_appliance_zfs_filesystem vol['name'].to_s do
      host "infrastructure_vm_host[#{vm['name']}]"
      platform_api_url '5.6.7.8'
      platform_verify_ssl false
      name vol['name']
      pool "infrastructure_zfs_appliance_zfs_pool[#{'my_pool'}]"
      project 'zfs_project'
      size_mb (vol['size'].to_i * 1000)
      mount_point vol['mount_point']
      options vol['options'].split(',')
      action :create
    end

    # add to mount_map for bootstrapping
    mount_map << "infrastructure_zfs_appliance_zfs_filesystem[#{vol['name']}]"

  end

  # networking
  vm['networks'].each do |net_name, net_data|
    
    # create network interface for each network defined
    infrastructure_exalogic_exalogic_network_interface "#{vm['name']}-#{net_name}" do
      host "infrastructure_vm_host[#{vm['name']}]"
      postfix net_data['postfix']
      interface_type net_name
      routes net_data['routes']
      vnet net_data['name']
    end
  end

  # create vm
  infrastructure_vm_host (vm['name']).to_s do
    bootstrapper "infrastructure_chef_bootstrapper[#{vm['name']}]"
    mounts mount_map
    action :create
  end
end

About IP Allocation

• Warning: Please read the entire section before using this functionality, particularly the final paragraphs about the risks involved in mixing dynamic and non-dynamic IP configurations

The default behaviour of this controller is to allow the exalogic EMOC to allocate IPs. This is the safest and simplest option, since it means that that system has a complete view of which IPs are used and which are available. If you would prefer, however, you can control the IP allocation yourself directly via the ips_are_dynamic property - if set to false, then this will use the ip property of the network interface resources - so in one of the above options, you’d have something that looks like this:

infrastructure_vm_host (vm['name']).to_s do
  bootstrapper "infrastructure_chef_bootstrapper[#{vm['name']}]"
  mounts mount_map
  ips_are_dynamic false
  action :create
end
    
infrastructure_exalogic_exalogic_network_interface "#{vm['name']}-#{net_name}" do
  host "infrastructure_vm_host[#{vm['name']}]"
  ips "192.168.24.42"
  postfix net_data['postfix']
  interface_type net_name
  routes net_data['routes']
  vnet net_data['name']
end

bviously, you wouldn’t usually hardcode the IP in your recipe, but you could do so if you wanted to have very specific recipes. Note that it is very important that you either (a) never use dynamic IPs on your exalogic, or (b) arrange for your IPs to be reserved on the exalogic - this can be done via the iaas-allocate-ip-addresses, documented at docs.oracle.com/cd/E18476_01/doc.220/e25258/GUID-EC9DF55D-B21A-4607-BB5A-44C2EC835EC9.htm#ELCLD77070 if you wish to do it manually. This can also be done via the mintpress SDK:

require 'mintpress-infrastructure-oracle-exalogic'

# Obtain an exalogic client
# If you've already defined a ExalogicHost object, you can also obtain
# if via:
#
# client = exalogic_host.client
#
api_url = "FIXME"
exalogic_user = "FIXME"
access_key = "FIXME"
vnet = "MY_VNET"
client = OracleExalogic::Client.new(api_url: api_url, username: exalogic_user, access_key: access_key)

# Allocate the IP
ip = client.allocate_ip_addresses(vnet: my_vnet)
puts "Allocated ip #{ip}"
# Release the IP back into the pool
client.release_ip_addresses(vnet: my_vnet, ip_addresses: ip)

It is not recommended to do this as a part of your build pipeline, becuase there is no metadata associated with the IP addresses, and hence no way to tell if an allocated IP address is acutally used, or by whom (you could, of course, list all vservers, however this would not cover IPs which are floating for use by things such as OTD and haproxy). Additionally, there is no way to allocate a specific IP address - we have succesfully used a script that allocates IPs until you get the one you want, and then releases every IP it allocated, however this carries the risk is that if the script is somehow interrupted, again you need to work out what happened and undo it. If you require assistance with this, please contact limepoint support, as it is a high risk operation if it were to go wrong.

In particular, be aware of the following failure mode:

1. You create a box with the static ip 192.168.10.4

2. Another user create a new box, which gets the dynamic ip 192.168.10.3

3. User creates a third new box, which gets the next IP in the pool, the unallocated 192.168.10.4

4. Both your box and the users second box are kicked from the IB gateway

This is not just theoretical - it has happened to multiple clients of ours, and in one case required oracle support intervention to resolve. So it is vitally important that you either never mix these two modes of operation, or always ensure that your IPs are allocated in the exalogic before using them.