Adding Configuration Split to a Drupal site using BLT and Acquia Cloud

Note: As of Config Split beta4, you no longer need to use drush csex/csim to export and import config accounting for splits. You instead install both Config Filter and Config Split, then use the normal Drush commands (drush cex/cim). There are also a few other tweaks to the guide below; I may update it when I get more time.

I've been looking at a ton of different solutions to using Drupal 8's Configuration Management in a way that meets the following criteria:

As easy (or almost as easy) as plain old drush cex -y to export and drush cim -y to import.
Allows a full config export/import (so you don't have to use update hooks to do things like enable modules, delete fields, etc.).
Allows environment-specific configuration and modules (so you don't have to have some sort of build system to tweak things post-config-import—Drupal should manage its own config).
Allows certain configurations to be ignored/not overwritten on production (so content admins could, for example, manage Webforms or Contact Forms on prod, but not have to have a developer pull the database back and re-export config just to account for a new form).

The Configuration Split module checks off the first three of those four requirements, so I've been using it on a couple Drupal 8 sites that I'm building using Acquia's BLT and hosting on Acquia Cloud. The initial setup poses a bit of a challenge due to the 'chicken-and-egg' problem of needing to configure Config Split before being able to use Config Split... therefore this blog post!

Installing Config Split

Configuration Split setup - Drupal 8

The first time you get things set up, you might already be using core CMI, or you might not yet. In my case, I'm not set up with config management at all, and BLT is currently configured out of the box to do a --partial config import, so I need to do a couple specific things to get started with Config Split:

Add the module to your project with composer require drupal/config_split:^1.0.
Deploy the codebase to production with the module in it (push a build to prod).
On production, install the module either through the UI or via Drush (assuming you're not already using core CMI to manage extensions).
On production, create one config split per Acquia Cloud environment, plus another one for local and ci (so I created local, ci, dev, test, and prod).
- For each split, make sure the machine name matches the Acquia Cloud environment name, and for the path, use ../config/[environment-machine-name]).
- For Local, use local, for CI (Travis, Pipelines, etc.), use ci (for the machine names).
Pull the production database back to all your other Acquia Cloud environments so Config Split will be enabled and configured identically in all of them. 6 On your local, run blt local:refresh to get prod's database, which has the module enabled.

Note that there may be more efficient (and definitely more 'correct') ways of getting Config Split installed and configured initially—but this way works, and is quick for new projects that don't necessarily have a custom install profile or module where you can toss in an update hook to do everything automated.

Configuring the Splits

Now that you have your local environment set up with the database version that has Config Split installed—and now that Config Split is installed in all the other environments using the same configuration, it's time to manage your first split—the local environment!

Enable a module on your local environment that you only use for local dev (e.g. Devel).
Configure the 'Local' config split (on http://local.example.com/admin/config/development/configuration/config-split/local/edit)
Select the module for the Local split (e.g. select Devel in the 'Modules' listing).
Select all the module's config items in the 'Blacklist' (use Command on Mac, or Ctrl on Windows to multi-select, e.g. select devel.settings, devel.toolbar.settings, and system.menu.devel).
Click 'Save' to save the config split.

Now comes the important part—instead of using Drush's config-export command (cex), you want to make it a little... spicier:

drush @project.local csex -y local

This command (configuration-split-export, csex for short) will dump all the configuration just like cex... but it splits out all the blacklisted config into the separate config/local directory in your repository!

Note: If you get Command csex needs the following extension(s) enabled to run: config_split., you might need to run drush @project.local cc drush. Weird drush bug.

Next up, you need to create a blank folder for each of the other splits—so create one folder each for ci, dev, test, and prod, then copy the .htaccess file that Config Split added to the config/local folder into each of the other folders.

We're not ready to start deploying config yet—we need to modify BLT to make sure it knows to run csim (short for config-split-import) instead of cim --partial when importing configuration on the other environments. It also needs to know which split to use for each environment.

Modifying BLT

For starters, see the following BLT issue for more information about trying to standardize the support for Configuration Split in BLT: Support Config Split for environment-specific Core CMI.

You need to override some BLT Phing tasks, so first things first, replace the import: null line in blt/project.yml with import: '${repo.root}/blt/build.xml'.
Add a file in the blt/ directory named build.xml, and paste in the contents of this gist: https://gist.github.com/geerlingguy/1499e9e260652447c8b5a936b95440fa
Since you'll be managing all the modules via Config Split, you don't want or need BLT messing with modules during deployment, so clear out all the settings in blt/project.yml as is shown in this gist: https://gist.github.com/geerlingguy/52789b6489d338cb3867e325e2e0a792

Once you've made those two changes to BLT's project.yml and added a blt/build.xml file with your custom Phing tasks, it's time to test if this stuff is all working correctly! Go ahead and run:

blt local:refresh

And see if the local environment is set up as it should be, with Devel enabled at the end of the process. If it is, congratulations! Time to commit this stuff and deploy it to the Cloud!

Once you deploy some code to a Cloud environment, in the build log, you should see something like:

The following directories will be used to merge configuration to import:
/mnt/www/html/project/docroot/../config/default
../config/dev
Import the configuration? (y/n): 
y
Configuration successfully imported from:                              [success]
/mnt/www/html/project/docroot/../config/default
../config/dev.

This means it's importing the default config, mixed in with all the dev config split directory. And that means it worked.

Start deploying with impunity!

The great thing about using Drupal 8's core CMI the way it is meant to be used (instead of using it with --partial) is that configuration management becomes a total afterthought!

Remember in Drupal 7 when you had to remember to export certain features? And when features-revert-all would sometimes bring with it a six hour debugging session as to what happened to your configuration?

Remember in Drupal 7 when you had to write hundreds of update hooks to do things like add field, delete a field, remove a content type, enable or disable a module?

With CMI, all of that is a distant memory. You do whatever you need to do—delete a field, add a view, enable a dozen modules, etc.—then you export configuration with drush csex local. Commit the code, push it up to prod, et voilà, it's magic! The same changes you made locally are on prod!

The one major drawback to this approach (either with Config Split or just using core CMI alone without --partial) is that, at least at this time, it's an all-or-nothing approach. You can't, for example, allow admins on prod to create new Contact forms, Webforms, blocks, or menus without also pulling the database back, exporting the configuration, then pushing the exported config back to prod. If you forget to do that, CMI will happily delete all the new configuration that was added on prod, since it doesn't exist in the exported configuration!

If I can find a way to get that working with Config Split (e.g. say "ignore configuration for the webform. config namespace" without using --partial), I think I'll have found configuration nirvana in Drupal 8!

drupal

drupal 8

configuration management

config

Add new comment

Comments

Jeff,

If I have a workflow where I want certain configs and modules in dev, but not prod, then I don't really need to define a config split for the prod environment, right?

During my BLT deploys I was able to keep things as they were, and the normal `drush config-import` worked as I wanted it to.

One thing I don't quite understand is how config-split is handling core-extensions.yml config, which tracks which modules are enabled/disabled.

@Brian - Yes, that's correct. In the example above, I don't technically need a split for every environment... it's just easier (IMO) to set things up that way initially so you don't have to go back and add in all the splits later (which again involves doing it first on prod then pulling things back per-environment).

It also allows you to switch the config-split-import --split value programmatically in the BLT Phing task that runs csim.

how config-split is handling core-extensions.yml config

I believe it merges in the current split's 'modules' array with the rest of core.extensions.yml, then imports that entire list.

Ah, and I actually do need to override the setup:config-import target anyway to make sure that blt local:refresh is aware that it needs to run the proper config split import.

I think the main downside to config-split at the moment is that it doesn't show list of created/deleted/updated in the confirmation prompt during the drush call. This is quite valuable information to see for sanity reasons, especially when looking at the post-code-deploy task details in Acquia Cloud where it lists that out.

I *think* this is soon to be resolved from what I've read in https://www.drupal.org/node/2830767

I think the main downside to config-split at the moment is that it doesn't show list of created/deleted/updated in the confirmation prompt during the drush call.

That is true. Nice for auditing purposes!

Thanks for all your great blogging recently. High quality stuff.

Drush team is working with config_split to harmonize so that split is just a regular module that plus into the config-import pipeline and the drush command you run always regular config-import.

Thank you for all your work on that recently! One of the issues in question: Use config.storage.sync to instantiate the FileStorage

It is actually very doable to create a confit split setup that allows content admins to create and change webforms without reverting things on a "deploy". You can even add other configs to this split

I've implemented the below solutions for a client and it is currently active and working superb on several sites:

You need to have the latest version of config_split installed: the one that allows wildcards selections. (beta2)
Create a content_split config split and blacklist webform.webform.*
Activate this split on all environments (local, dev, staging, live ...)

When doing a deploy to any environment use this snippet of bash code: (you will need to adjust the paths, because they are tailored to the way my client works)

cd /ROOT/http && drush state-set system.maintenance_mode 1
cd /ROOT/config && cp -Rp sync sync.backup
cd /ROOT/http && drush csex --split=content_split -y
cd /ROOT/config && rm -rf sync && mv sync.backup sync
cd /ROOT/http && drush csim --split=content_split -y && drush state-set system.maintenance_mode 0

I have installed 8.x-1.0-beta3 version and i get the message: 'Unknown option: --split.' when i try export with command "drush @drushalias.local csex --split=local".

Do you know this issue?

Can you try first running drush @drushalias.local cc drush? Sometimes Drush gets a little confused...

Yes, I did this, but it still shows the error. I tried this way "drush @drushalias.local csex site" and it worked perfectly. Thank you!

Ah you're correct—the latest version of Config Split uses split as an arg, so now it's drush @site csex [split] instead of drush @site csex --split=[split].

I also had to do slight changes in build.xml file: Replace "${environment}" with ${environment}

Unfortunately, HTML filter scrambled my previous comment. The correct one is: replace <option name="split">${environment}</option> with <param>${environment}</param>

I was checking the process and there seems to be one inconsistency I faced, so I'd like to share my experience (using latest stable BLT, 8.6.15):

At the beginning, I strictly followed the Jeff's howto: firstly deployed and installed config_split to prod, set only clean environments there (dev, test, prod, ci and local) without any blacklists set, then fetched DB to local environment using blt local:refresh, enabled devel module on local, added blacklist on devel on local and exported its configuration to local split. Then, after performing second blt local:refresh, I ended up with error "There were errors validating the config synchronization.' in /var/www/natgrid/docroot/core/lib/Drupal/Core/Config/ConfigImporter.php:728" .

After further investigation I found an issue: The primary problem was that after second blt local:refresh, config importer wanted to import configuration only without ability to enable devel module on local. Namely, that time, there was missing blacklist information to config_split module provided from prod. This information is namely stored as an entity in database, so it's being primarily read from database (not from configuration). So in short, there is still a "chicken-and-egg" issue in the howto.

I was able to resolve that by following (I'm pretty sure not very clean) solution:
After first deploy of the codebase to prod, I enabled devel module on prod. In point 4, I also added all the blacklist settings (module "devel" + 3 configuration variables) to the "local" config_split module already on prod envornment. Then I uninstalled devel module on prod again and continued according the guide.

The whole process finished successfully, because after second blt local:refresh you already get all the correct entity info about the local environment from prod.

I wanted to solve following use cases (intentionally should cover all the enabled/disabled modules and configurations we typically want to use):

On local environment, devel module is enabled, shield and acquia_connector is disabled. Error reporting (variable system.logging) is set up to display errors on screen (in shortened form)
On Acquia Cloud dev environment, devel, shield and acquia_connector modules are enabled. Error reporting is set up to display errors on screen (in shortened form)
On Acquia Cloud test environment, shield and acquia_connector modules are enabled, devel is disabled. Error reporting is set up to not display any errors
On Acquia Cloud prod environment, the settings are the same as on Acquia Cloud test environment
Everything else is common for all environments

After some tweaks and workarounds, I was able to use setup everything (almost) reliably with config_shield beta4 and BLT 8.6.15.

Firstly, on prod environment, I deployed code and enabled all the modules I intended to use anywhere: config_split, devel, shield and acquia_connector. I also set up error reporting to none, as this is the standard setting I wanted to keep (unless overridden).
On prod environment, in /admin/config/development/configuration/config-split, I entered these settings manually (or you can import them via CIM) for each environment: CI, Dev, Local, Prod, Test:
- CI: nothing set (https://gist.github.com/mirsoftacquia/6922299f27529ed5d0e9eb571bab46bd)
- Dev: https://gist.github.com/mirsoftacquia/25b0f4ac5eb2d1af1ddffbe7c718f9b3
- Local: https://gist.github.com/mirsoftacquia/aea3eeb2702a9c7818e39ff97528de29
- Prod: https://gist.github.com/mirsoftacquia/df162bbd6f134747982f0c5c9898e4d5
- Test: https://gist.github.com/mirsoftacquia/8ad219d9e7c50408e3fadda4e895549c
Following build.xml worked for me: https://gist.github.com/mirsoftacquia/555b9f5c508f5e64b17e5c94a4bd0262 . Using this, both blt local:setup and blt local:refresh worked fine. However, you can find some not-very-nice entries in there. Firstly, the config must be imported 3x (I simply wasn't able to remove anything from these without missing something) and afterwards I had to manually uninstall acquia_connector and shield modules, as config_split didn't reflect that and the modules ended in installed state on local.

Using this setup, I downloaded everthing from prod to local (e.g. using blt local:refresh) and on local environment continued to split configurations:

drush @local csex
drush @local csex local
drush @local csex ci
drush @local csex dev
drush @local csex test
drush @local csex prod

You should end with common config files in /config/default folder and environment-specific files in particular environment folder.

Then, I could use a following workflow: - Export config data from your local (e.g. to prepare for pull-request), without environment-specific config: drush @local csex - Import environment-specific data into Acquia Cloud (dev / test / prod) after deployment: drush @ cim -y vcs && drush @ csim -y (e.g. drush @acquiacloudsite.test cim -y vcs && drush @acquiacloudsite.test csim -y test) - Update the latest configuration changes from prod and import them to your local: blt local:refresh

This setup is stable on config import, which is the most important part. However, to proper export and split the configurations, there are still two issues that needed manual work:

The graylist still didn't work to me as expected. Either with or without setting of graylist_skip_equal variable, this still sets some configurations wrongly on drush csex (without parameter) in some cases.

However, on drush cim, everything worked fine if I manually inserted the config files to the right places: - https://gist.github.com/mirsoftacquia/fba6be551bb679ad6974e713bfa3040a (default, with disabled error reporting), copy to /config/default/ folder - https://gist.github.com/mirsoftacquia/2c638c8d7f29a935ea0a9e62e15e7c23 (overridden, with enabled error reporting), copy to /config/dev/ and /config/local/ folders

Exporting configurations using drush @local csex (without parameter) exported the /config/default properly, however removed configuration of modules currently uninstalled on local (acquia_connector and shield) from prod and test folders. This is definitely not desired. It can however be worked around by not commiting any changes made in config/ folder other than config/default, unless the environment-specific changes should be commited.

Maybe I overlooked something clear, or my process is completely wrong. But so far it's at least somehow usable, so it might be of help for anyone else. And, of course, any ideas of improvement are welcome :)

Steps taken (not on acquia):
1. Enable config_split on prod, set prod and local splits. Commit.
2. Pull db and code.

But I still get:
Configuration config_split.config_split.local depends on the Configuration split module that will not be installed after import.

Important note if using BLT and Acquia Servers. You'll likely need to give your staging server split a machine name of "stage" as opposed to "test" as Jeff does here. BLT is configured to split using the machine name "stage" so that should work out of the box. This confused me quit a bit since also, the "drush aliases" that Acquia provides calls staging "test" as well. But alas...a machine name of "stage" works with BLT by default.

Hi,

I think blt local:refresh does not anymore exist. Now it's: blt sync