mintpress-infrastructure-oci

Repository to provider helper functions for Oracle OCI infrastructure

Required Reading

If you haven’t read the introductory documentation for the Mintpress Infrastructure Framework, a lot of this won’t make a whole lot of sense. [link]

What Services Are Provided

  • OciPlatform - the Platform provider

  • OciHost - Compute Hosts

  • OciStorage - iSCSI block storage

What other resources are provided?

  • OciSecurityGroup

  • OciSecurityGroupRule

  • OciSecurityListRule

Limitations

  • Disks MUST be 50gbytes of greater - this is a platform limitations. Practically, this means you can NOT create, for example, a 4gb swap disk and expect that to work. (In the legacy mintpress platform, disks smaller than this will automatically be grown to this size for compatibility reasons).

Getting Started

To use the OCI controller, you will require:

  • OCI Credentials. This consist of a user, a keyfile, a fingerprint, a tennancy, and a compartment

  • A public/private keypair for the VMs you wish to create

  • A region and availability domain within that region

Most resources work better if you declare your platform first - this is done in the usual way (see the mintpress-infrastructure documentation for details about Using resources)

require 'mintpress-infrastructure-oci'
include MintPress::InfrastructureOci

# Setup the provider - this provider does not uses dnslabels provided by OCI
# This provider will attach a public IP to the VMs along with a private IP
# You can also do this without specifying the name property of the provider, if you do that, the VMHost initialization should remove the provider property as well
UsingOciPlatform.new(name: 'public_subnet',
                     all_platform_services: true,
                     fingerprint: 'redacted'
                     key_file: '/home/mintpress/.oci/oci_api_key.pem',
                     region: 'ap-sydney-1',
                     tenancy: 'ocid1.tenancy.oc1..redacted',
                     compartment: 'ocid1.tenancy.oc1..redacted',
                     availability_domain: 'BTaG:AP-SYDNEY-1-AD-1',
                     user: 'ocid1.user.oc1..redacted',
                     subnet_id: 'ocid1.subnet.oc1.ap-sydney-1.redacted',
                     assign_public_ip: true,
                     preserve_hostinfo: 2,
                     admin_connect_user: 'opc',
                     admin_final_user: 'root',
                     keys: ['/home/mintpress/oci-ssh-key'],
                     public_key: 'ssh-rsa redacted')

Flexable Shapes

The provider now supports flexable shapes - these will maek the VM match cpu_count and ram_gb. To use this: * Set the use_flex property to true * Set your flexable shape type in the flex_shape property - this defaults to VM.Standard.E4.Flex.

Image Selection

OCI implements a facility for searching for images by OS type and revision, rather than just via direct image IDs. This is controlled by the operating_system and operating_system_version parameters on either of OciHost(VMHost) or OciPlatform

For example:

# As always, we use OCIHost directly if we're going to use OCI specific parameters - this could be VMHost.new if you were not!
info 'Creating Latest Oracle Linux Version 6'
host_oel6latest = OCIHost.new(name: 'oel6latest.mintpress.io', native_instance_type: 'VM.Standard.E2.4', operating_system: 'Oracle Linux', operating_system_version: 6)
host_oel6latest.create

Will find an Oracle Linux 6 image to use. Note that, on the OCI platform, images are tied to shapes, which is the OCI term for compute sizes - hence for the general case, it makes sense to specify semantically if at all possible.

The search semantics are: * If you specified the name of the image, e.g. Oracle-Linux-7.7-2019.09.25-0 we search directly against that image, if the image is not found, we raise error * Otherwise, we select the newest image based on shape, operating_system, and operating_system_version

For the OS search, you can specify either a major version only (i.e 7), or a major/minor pair (i.e. 7.5)

Block Storage

From the end user perspective, block storage works the same as it does on any other platform - i.e by using add_block_device on the ost, by attaching the OCIStorage resource to the hosts block_devices property, or by using the UsingExtraOCIStorage resources.

# One way to create a vm with a 50gbyte /opt
UsingExtraOCIStorage(name: "opt_disk", mount_point: "/opt", size_mb: 50000)
VMHost.new(name: 'oel7latest.mintpress.io', native_instance_type: 'VM.Standard.E2.4', operating_system: 'Oracle Linux', operating_system_version: 7).create

# Another way - add it directly
h = VMHost.new(name: 'oel7latest.mintpress.io', native_instance_type: 'VM.Standard.E2.4', operating_system: 'Oracle Linux', operating_system_version: 7)
h.add_block_device(name: "opt_disk", mount_point: "/opt", size_mb: 50000)

# Or in chef - lets use the native OCI resources so we can specify OCI specific things
infrastructure_oci_oci_host 'oel7latest.mintpress.io' do
  native_instance_type "VM.Standard.E2.4"
  operating_system "Oracle Linux"
  operating_system_version 7
  action :create
end

# Note the shorthand here for the host link - see the main Infrastructure Framework for details on that.  The 
# usual chef syntax of "infrastructure_oci_oci_host[oel7latest.mintpress.io]" will also work!
infrastructure_oci_oci_storage 'opt_disk' do
  mount_point "/opt"
  size_mb 50000
  host 'oel7latest.mintpress.io'
end

Internally, all OCI block devices are iSCSI luns - behind the scenes, the mintpress bootstrapper will correctly prepare these for use, but be careful not to adjust the fstab entries, particulary the _netdev argument - if you remove this, the systems will not return after a reboot.

OCI Hostinfo

The OCI boot process involves resetting hostnames and such - this is controlled by the preserve_hostinfo attribute in mintpress.
* A setting of 0 will reset the hostname, /etc/hosts, and /etc/resolv.conf on every reboot.
* A setting of 1 will preserve the hostname, but nothing more * A setting of 2 will preserve all of your changes to hostname, /etc/hosts, and /etc/resolv.conf. This is the default. * A setting of 3 will preserve the hostname and /etc/hosts, but not /etc/resolv.conf

Note that changing this setting will require a reboot - mintpress will handle that reboot on a new create, howeever

Updating live system configuration

OCI implements the update action on hosts, which will reconfigure the network security groups of the system. It will also ensure that any block devices are configured and attached, however at this time it will not remove block devices which have been removed from the configuration.

Additionally, OCI provides the attach_storage action, to ensure all storage is attached, and the harvest action, to retrieve the current state of storage and security groups

Security Groups

We support both adding rules to existing Security Lists and Security Groups, as well as creating new security groups. When attaching to hosts, security groups can be specified as either string names (i.e. “testgroup” or OciSecurityGroup objects.

Security Group Rules can be attached via either the group name, group object, or group OCID. The VCN can also be specified, however it is only required if the subnet is not available in the platform definition, or a host that the security group is attached to.

Meaning: This will work:

sec_group = OCINetworkSecurityGroup.new(display_name: "jjtestgroup")
host = OCIHost.new(name: "testhost.mintpress.io", native_instance_type: 'VM.Standard.E2.8', operating_system: 'Oracle Linux', operating_system_version: 7,subnet: my_subnet, network_security_groups: sec_group)
# This works because host and sec_group are linked
sec_group.create
host.create

but this will not, assuming youj don’t have a subnet_id specified in your platform:

sec_group = OCINetworkSecurityGroup.new(display_name: "jjtestgroup")
# This will error because it does not know which VCN to associate the security group with
sec_group.create

See the resources OCINetworkSecurityGroup, OCINetworkSecurityGroupRule, and OCISecurityListRule for full parameter details

infrastructure_oci_security_list_rule "port26,27" do
  security_list_id "ocidxxx"
  destination_type "CIDR_BLOCK"
  destination "0.0.0.0/0"
  source "0.0.0.0/0"
  direction "INGRESS"
  tcp_dport_range_low 26
  tcp_dport_range_high 27
  protcol 6
end

# Create teh group first
infrastructure_oci_network_security_group "jjgroup" do
  display_name "jjgroup"
  action :create
end

infrastructure_oci_security_group_rule "port26,27" do
  # This can just be OCID as well
  network_security_group "infrastructure_oci_network_security_group[jjgroup]"
  description "jjrule"
  destination_type "CIDR_BLOCK"
  destination "0.0.0.0/0"
  source "0.0.0.0/0"
  direction "INGRESS"
  tcp_dport_range_low 26
  tcp_dport_range_high 27
  protcol 6
  action :create
end

# a host can use the groups by resource too, or by ocid
# Create the VM
infrastructure_vm_host osi_fqdn do
  network_security_groups "infrastructure_oci_network_security_group[jjgroup]"
  bootstrapper "infrastructure_chef_bootstrapper[#{osi_fqdn}]"
  specs_cpu_count 4
  specs_ram_gb 16
  action :create
  keys keys
  admin_keys keys
  admin_connect_user connect_user
  admin_final_user 'root'
  connect_user connect_user
  final_user 'oracle'
  action :create
end

Ruby Helper functions for Network Security Groups

We added some ruby helper functions for adding/removing network security groups to/from the list. The update action is also available from other devops tools (such as chef), however the update function will always replace the entire group list, in order to ensure that items can be removed.

# find a VM
testhost = MintPress::Infrastructure::VMHost.new(provider: 'public_subnet', name: 'ol7latest.mintpress.io')

# get the list of existing attached groups
testhost.harvest

# and remove jjtestgroup specifically
testhost.remove_network_security_group_by_display_name('testgroup')
# update
testhost.update


# add a group by name specifically
testhost.add_network_security_group_by_display_name('testgroup')
testhost.update

Examples

# Set up the provider
    using_oci_platform "oci_platform" do
      all_platform_services true
      fingerprint fingerprint
      key_file key_file
      tenancy tenancy
      user user
      region region
      subnet_id subnet_id
      admin_connect_user connect_user
      admin_final_user 'root'
      connect_user connect_user
      final_user 'oracle'
      keys keys
      admin_keys keys
      public_key public_key
      public_postfix ext_postfix
      private_postfix postfix
      
      assign_public_ip true
      display_name display_name
      operating_system operating_system
      operating_system_version operating_system_version
      availability_domain availability_domain
      compartment compartment
      preserve_hostinfo preserve_hostinfo
      image image
      subnet_id subnet_id
    end

    # Use chef to bootstrap this
    using_chef_bootstrapper osi_fqdn do
      chef_environment "new_env"
      run_list run_list
      node_attributes local_attrs
    end

    # Create the VM - this is using the generic infrastructrure_vm_host stanza, which does 
    # NOT allow for OCI specific attributes - if you want to specify OCI specific attributes, you'll
    # need to use the infrastructure_oci_oci_host resource instead - the usage is the same as below, however
    # it can also take OCI parameters directly.
    infrastructure_vm_host osi_fqdn do
      bootstrapper "infrastructure_chef_bootstrapper[#{osi_fqdn}]"
      specs_cpu_count 4
      specs_ram_gb 16
      action :create
      keys keys
      admin_keys keys
      admin_connect_user connect_user
      admin_final_user 'root'
      connect_user connect_user
      final_user 'oracle'
    end