Ansible, SSH jump hosts and multiple private keys

Today I ran into a situation where I was using a bastion to access instances in a private OpenStack tenancy. I wanted to use Ansible to manage these hosts, but I also wanted to do the development work on my laptop, rather than the bastion.

The problem I had was this: I wanted Ansible to use my private key on my laptop (host A) to access the jump host (host B), but then I wanted to use a private key of a user on the jump host to access the target.

The solution came from this Github post:

There are two pre-requisites for the jump host:

  1. The nmap-ncat package must be installed.
  2. The AllowAgentForwarding option in /etc/ssh/sshd_config must be “Yes”

Once I had nc installed on host B, I then set the following variable in my group_vars for the hosts I wanted to manage:

ansible_ssh_common_args: '-o ProxyCommand="ssh -o ''ForwardAgent yes'' centos@ ''ssh-add && nc %h %p''"'

(For more information about jump hosts and Ansible, see the FAQ).

This setup performs the following steps:

  1. Exposes the SSH agent running on host A (my laptop) to host B through the ForwardAgent yes option. The SSH agent is responsible for keeping your private keyfile in memory so you don’t need to enter your password all the time.
  2. We execute ssh-add on host B which adds the private keys for the centos user on host B to the forwarded SSH agent. At this time the keys for the centos user on host B are now available on my laptop in my SSH agent.
  3. nc then establishes a connection to SSH on the target (host C), and the SSH client on my laptop can now negotiate the final connection as the key from host B is available in the agent.

We now have a connection to host C, and:

[agoossen@agoossen cf-ansible]$ ansible -i hosts/cloudforms all -m ping -f 1 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
} | SUCCESS => {
    "changed": false, 
    "ping": "pong"



Configuration Domains with CloudForms/ManageIQ

The CloudForms/ManageIQ automation engine gives us great power to create classes and methods that we can execute in response to various actions (e.g. button presses, events in the environment, REST API calls, etc).

Using Ruby and Ansible we can execute custom code to customise and enhance the CloudForms experience for our customers and users – we can add buttons to interfaces, override provisioning workflows, override approval workflows…etc.

Normally, we’ll have at least two environments – a Quality Assurance environment, and a Production environment. We don’t want those two to mix, and quite often key settings will be different between the two environments. For example,  the FQDNs and credentials used to access remote services (e.g. Single Sign On, corporate DNS, etc) may differ between QA and Production. But the underlying logic in the code remains the same – it’s just environment-specific configuration that changes.

To get to a full Continuous Integration/Continuous Deployment model we need to be able to promote Automate code between environments cleanly. Having to execute find/replace across our automate code to replace QA settings with Production settings is not clean.

So how do we keep our common automation logic from intermixing with ou environment-specific settings and configuration?

The answer is a configuration domain. Read more “Configuration Domains with CloudForms/ManageIQ”