Ansible best practices: using project-local collections and roles

Note for Tower/AWX users: Currently, Tower requires role and collection requirements to be split out into different files; see Tower: Ansible Galaxy Support. Hopefully Tower will be able to support the requirements layout I outline in this post soon!

Since collections will be a major new part of every Ansible user's experience in the coming months, I thought I'd write a little about what I consider an Ansible best practice: that is, always using project-relative collection and role paths, so you can have multiple independent Ansible projects that track their own dependencies according to the needs of the project.

Early on in my Ansible usage, I would use a global roles path, and install all the roles I used (whether private or on Ansible Galaxy) into that path, and I would rarely have a playbook or project-specific role or use a different playbook-local version of the role.

Over time, I found that many playbooks benefitted from having all role dependencies managed independently. That meant adding a requirements.yml file for that playbook, (usually, but not always) defining each role's version or Git commit hash, and adding an ansible.cfg file in the playbook project's root directory so Ansible would know to only load roles from that playbook's roles directory.

With collections, I'm expanding that best practice, and generally recommend against installing collections into a global location (like the default, which is ~/.ansible/collections or /usr/share/ansible/collections).

For any new project I start, I add a ansible.cfg with the following (at minimum):

[defaults]

# Chick-Fil-A would like a word...
nocows = True

# Installs collections into [current dir]/ansible_collections/namespace/collection_name
collections_paths = ./

# Installs roles into [current dir]/roles/namespace.rolename
roles_path = ./roles

Then I manage all my playbook's dependencies in a requirements.yml file like this one:

---
roles:
  - name: geerlingguy.java
    version: 1.9.7

collections:
  - name: community.kubernetes
    version: 0.9.0

Assuming this PR is merged prior to Ansible 2.10's release, you can then install all the playbook project's dependencies with:

ansible-galaxy install -r requirements.yml

Then your playbooks will use the correct requirements, and won't affect any other playbooks and Ansible projects.

This also optimizes things because instead of Ansible having to scan through the likely hundreds (thousands?) of role and collection directories that I would need to have in global paths to support the hundreds of projects I work with, it only needs to scan the few that are required for the given project I'm working on.

A final major benefit is you can then commit the roles and collections to your project code repository, and then the dependencies are part of the deployable project, which gives two major benefits:

  1. Other people (e.g. coworkers, or even CI systems) which work with the project would necessarily have the same revision as everyone else as part of the project VCS, and updates to any dependencies would be checked in via source control.
  2. You could still run the playbook if GitHub, Ansible Galaxy, or any other system required to download the dependencies were offline, since you wouldn't need to run ansible-galaxy at all (except to update dependencies when you want to do so).

Further reading

ansible
ansible galaxy
collections
roles
best practices

Comments

Hey Jeff,

I just wanted to clarify for anyone using Ansible Tower:

Tower supports project-specific Ansible collections in job runs. If you specify a collections requirements file in the SCM at collections/requirements.yml, Tower will install collections in that file in the implicit project sync before a job run.

It works the same way for roles, you would need to specify a roles requirements file in the SCM at roles/requirements.yml.

https://docs.ansible.com/ansible-tower/latest/html/userguide/projects.html#ansible-galaxy-support

Thanks for the clarification; I didn't realize when writing this post that Tower doesn't support mixed roles and collections in one requirements file. This is a limitation in the current version of Tower, and not Ansible's core itself, so if you don't rely on Tower/AWX the original best practice still applies.

The most important part, however, is making sure you install collections and roles in a project-specific path—and it looks like Tower does that, which is nice. Otherwise you could run one playbook, and get version xyz of one dependency, then run another project, and it overwrites that to version abc, and breaks the first playbook!

I understand your points. But I have to disagree with some of your stated benefits. In my opinion it is ugly or borderline unacceptable to pull outside dependencies into a repository.

It is ugly and unacceptable, and I never do this in any of my normal software projects (PHP, Python, etc.)—however, since the ansible-galaxy CLI is severely lacking in ability to do dependency management basics (like equivalents to composer install (install all the versions in a lockfile / freeze file—galaxy won't overwrite an old version if it differs from the requirements.yml file unless you use --force which is more fragile since it will reinstall everything), composer update (or npm, gem, etc.)), it is a lot easier to track the deps locally so you can verify things after you have to run a ansible-galaxy install --force to update dependencies.

AWX seems to do this nicely (thanks for the previous comment!), so that is enough for me. I can avoid polluting my repositories ;)

I wish that ansible-galaxy could do dependencies properly. This will be extremely important in the future with the changes in Ansible 2.10 and I am very disappointed that the choice was done before ansible-galaxy has features to properly manage all those collections.

I am afraid it will be a horrible mess - a mess that the users might have to deal with for years to come.

Are there tickets open for getting `ansible-galaxy` to start doing that kind of thing, being more like Composer? If so, can you provide links?

Sure!

I may be missing a few others. The main one that would help with not having to ship dependencies with a repo is the 'lock file' and 'upgrade' issues.

Thanks for your article Jeff. Based on this structure, how would you recommend people use multiple collections that may require a reference to the same role in each collection? Would the role be cloned twice, into each collection? It's something I was hoping to avoid, but maybe it shouldn't be done. Thanks for any pointers. I've just been getting into the use of collections over the last few days.

Another benefit to a local, project-relative roles path: you can hack on the roles in this path to make local changes for faster development cycle than the normal commit-push-test cycle if using ansible galaxy.

Hey Jeff

Will i be able to use multiple roles from same repository? i have seen examples only with one role from one repository and also tried it. But failed . I have a repository 1 with multiple roles . I just need two from there. How can i update the requirements.yml? eg:

[root@oc1546875774 roles]# cat requirements.yml 
---
# test role for date and time for repository1
- name: dateandtime_role2
  src: "https://github.com/Ansible-Self-Study/repository1.git/dateandtime_role2"
  scmndtime_role2: 'git'
  version: master
  ignore_errors: true
# test role for hostname for repository1
- name: hostname_role1
  src: "https://github.com/Ansible-Self-Study/repository1.git/hostname_role1"
  scm: 'git'
  version: master

Samna, I'm sure by now you have figured it out, but it looks like you have typo in your code for first role. Rather than "scm" you have typed "scmndtime_role2". Hope this helps.

Hi Jeff,

I may be stupid in asking this question. But need to know is there a way we can upload collections to our private bitbucket repo and install them using requirements file.
Although I am able to install all the collections under one namespace using the below command:
ansible-galaxy collection install git+https://bitbucket.org.io/scm/~rohit.kainth/ansible_collections.git#name…

But is there a way to replicate same using requirements.yml for collections?

I am onto the same thing. My first success looks like this:
I have my collection in the main directory of the repository.
The galaxy.yml file states namespace and collection name .

Here is what I do:

laminar@ci:/tmp$ ansible-galaxy collection install -r requiments.yml
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Installing 'fkuep.deployers:1.0.0' to '/var/lib/laminar/.ansible/collections/ansible_collections/fkuep/deployers'
Created collection for fkuep.deployers at /var/lib/laminar/.ansible/collections/ansible_collections/fkuep/deployers
fkuep.deployers (1.0.0) was installed successfully
laminar@ci:/tmp$
laminar@ci:/tmp$ cat requiments.yml
collections:
- name: "git+https://git1.fkuep.de/fkuep/deployers.git"
version: 1.0.0
laminar@ci:/tmp$

This will use the git authentication from the user. (e.g cache credential-helper )

This has been working out great for us, as we can version control all of our collections and change them when we fully intend to.

We did recently find out that AWX 22.4 and higher switched the var name, so might be a good idea to update this post in case others find it down the road.

[DEPRECATION WARNING]: [defaults]collections_paths option, does not fit var
naming standard, use the singular form collections_path instead. This feature
will be removed from ansible-core in version 2.19.