Maintain local development configuration with Configuration Split

Fabian Bircher
23 Aug 2016
8 Comments
Fabian Bircher
23 Aug 2016
8 Comments
Fabian Bircher, 23 Aug 2016 - 8 Comments

Maintain local development configuration with Configuration Split

Answering the most commonly asked question about Configuration Management in Drupal 8; and a preview of our DrupalCon training.

This post is an excerpt from the topics covered by our DrupalCon Dublin training: Drupal 8 Development - Workflows and Tools.

For the last two years we have been giving trainings and presentations at various Drupal events about configuration management and its new workflows in Drupal 8. One of the recurring questions has been:

How do I keep some configuration from being deployed? For example the devel module and its configuration?

Until now we have answered to use drush with the --skip-modules flag and then to gitignore the configuration for devel. But then you can't share the development configuration and the process is error prone when there are more modules and other configuration items that depend on the configuration you gitignore.

For some simpler cases where configuration needs to be different between environments (for example the error verbosity) the configuration override system allows to override the configuration in settings.php. This is a solution for some cases, however, you can not override which modules are enabled or override configuration that doesn't exist.

Enter: Configuration split!

Our new module Configuration split splits the configuration when exporting it with a special Drupal Console command. It can be configured to split out enabled modules and given blacklisted configuration and it will then separate all the configuration that is dependent on the listed modules or configuration. The module's settings also allow to set a folder to which the separated configuration will be exported to.

That way the configuration set which you use to deploy configuration between different environments is a subset of your development configuration and the trusty configuration system of Drupal 8 can be used unharmed. Of course when importing with the special command the split configuration is merged back, allowing you to keep your development configuration in place.

"To split" is a synonym for "to break" and as such "Configuration split" has a dangerous ring to it. This is on purpose because the exported subset is on purpose not what you have on your development site and not what you have locally tested. So you need to compensate that with a workflow that imports and verifies the configuration you are going to deploy. This is better than to import and export individual configuration because Drupal needs the whole set of configuration to do its checks.

How do I use it?

Download and enable the Configuration split module like any other module. Then configure it under admin/config/development/configuration/config_split. Set the split folder to a different folder than your normal config sync folder. If you want a prettier interface, consider using Chosen. Then use the Drupal Console command config_split:export and config_split:import to export and import respectively.

Let's look at the devel example from above. We typically version the configuration outside of the webroot in ../config/sync as seen by Drupal. (In the project root when starting a project with drupal-composer/drupal-project.) So the folder we specify for config_split would be ../config/dev. The module to filter would be Devel and the rest can be left empty. The devel.settings will be detected automatically. Note though that the system.menu.devel does not depend on the devel module and can not be detected automatically, but it is easy to add it to the blacklisted config and it could theoretically be deployed without breaking anything.

Configuration Split settings form with chosen

The resulting config_split.settings.yml could look something like this when exporting it:

folder: ../config/dev
module:
  devel: 0
theme: {  }
blacklist:
  - system.menu.devel

Finally the following command will export the configuration to the default sync directory without the devel module enabled and export the devel configuration into the dev directory.
web$ ../vendor/bin/drupal config_split:export

Now if you would import the configuration through the UI or with drush cim the devel module would be un-installed, and you can do that to see the site without its development configuration. However, If you want the development configuration to stay or become active and the devel module installed use the following command:
web$ ../vendor/bin/drupal config_split:import

How does it work internally?

We implemented a StorageWrapper that allows filters to interact with the configuration after it has been read and before it is written to the wrapped FileStorage during the import and export operation. The SplitFilter has a secondary storage and decides where to read from or write to. This is a very similar concept to what drush does with its --skip-modules flag since we will want to easily integrate this with drush in the future.

What comes next?

The module is still in an early development stage and some more additions in the scope of splitting configuration could be added. The subset without the split configuration could be verified after exporting it and we could warn the user if it couldn't be imported. Or for example we currently use only config_split.settings but if the need arises we could support multiple split configurations. Or we could add a "gray list" to ignore some configuration that exists rather than removing it when splitting, essentially making it a configuration override outside of the scope of Drupal's runtime. This could be useful when maintaining several sites that are almost the same but all have their little "special snowflake" configuration which in turn could be synchronized with the normal workflow.

It is important to understand that the configuration system of Drupal has limitations that are there for a good reason. Most of them are to ensure data integrity and robustness and reliability of the synchronisation and so on. In other words measures to protect you from accidentally breaking your site. Using tools like config_split or drush --skip-modules you circumvent some of these security and integrity checks so use them with caution.

Since the module is not required for the regular functioning of a Drupal 8 site (it can even be used to blacklist itself) it can already be used in the development process of your current projects already. Your feedback is welcome, see you in the issue queue.

Comments

Comments

enzo - Eduardo Garcia
23 Aug 2016

Hi Fabian

Thanks for this excellent contribution.

Could you add a screenshot of the UI ?

I don't understand where the file config_split.settings.yml must be located.

Hi Fabian

Thanks for this excellent contribution.

Could you add a screenshot of the UI ?

I don't understand where the file config_split.settings.yml must be located.

enzo - Eduardo Garcia, 23 Aug 2016

Hi Fabian

Thanks for this excellent contribution.

Could you add a screenshot of the UI ?

I don't understand where the file config_split.settings.yml must be located.

enzo - Eduardo Garcia, 23 Aug 2016
Fabian Bircher
24 Aug 2016

Hello Eduardo Thank you for your comment. The UI depends on whether you use chosen or not and can be found here: https://example.com/admin/config/development/configuration/config_split

The yml file you get by exporting the configuration. Either by exporting all of it or with the "single export" of the "simple configuration" called config_split.settings in the UI or with drush cget config_split.settings. I hope I explained it better now.

Hello Eduardo Thank you for your comment. The UI depends on whether you use chosen or not and can be found here: https://example.com/admin/config/development/configuration/config_split

The yml file you get by exporting the configuration. Either by exporting all of it or with the "single export" of the "simple configuration" called config_split.settings in the UI or with drush cget config_split.settings. I hope I explained it better now.

Fabian Bircher, 24 Aug 2016

Hello Eduardo Thank you for your comment. The UI depends on whether you use chosen or not and can be found here: https://example.com/admin/config/development/configuration/config_split

The yml file you get by exporting the configuration. Either by exporting all of it or with the "single export" of the "simple configuration" called config_split.settings in the UI or with drush cget config_split.settings. I hope I explained it better now.

Fabian Bircher, 24 Aug 2016
Chris
28 Aug 2016

Great modul. Something like this should have been default in CM! I've downloaded it and tried it but since it is not working well with Drush right now, i'll have to wait. I don't want to install Drupal Console since Drush is the ultimate tool in my opinion and the only reason for DC would be this module.

I have a some thoughts about your module:

You might include support for the environment module and give users the ability to set certain settings according to the existing environments. This way it would be possible to have certain settings for Dev, Staging and Production and have them easily maintained. You then might include the environment hook and it would not need a new Drush command because people could use the existing "drush env-switch $environment" to switch to an environment with the correct settings imported thought your module and this hook.

Great modul. Something like this should have been default in CM! I've downloaded it and tried it but since it is not working well with Drush right now, i'll have to wait. I don't want to install Drupal Console since Drush is the ultimate tool in my opinion and the only reason for DC would be this module.

I have a some thoughts about your module:

You might include support for the environment module and give users the ability to set certain settings according to the existing environments. This way it would be possible to have certain settings for Dev, Staging and Production and have them easily maintained. You then might include the environment hook and it would not need a new Drush command because people could use the existing "drush env-switch $environment" to switch to an environment with the correct settings imported thought your module and this hook.

Chris, 28 Aug 2016

Great modul. Something like this should have been default in CM! I've downloaded it and tried it but since it is not working well with Drush right now, i'll have to wait. I don't want to install Drupal Console since Drush is the ultimate tool in my opinion and the only reason for DC would be this module.

I have a some thoughts about your module:

You might include support for the environment module and give users the ability to set certain settings according to the existing environments. This way it would be possible to have certain settings for Dev, Staging and Production and have them easily maintained. You then might include the environment hook and it would not need a new Drush command because people could use the existing "drush env-switch $environment" to switch to an environment with the correct settings imported thought your module and this hook.

Chris, 28 Aug 2016
Fabian Bircher
29 Aug 2016

Hello Chris Thank you for your feedback! Drush needs two pull requests to be merged (2313 and 2314) you can use the version I used to test it (with the two merged) https://github.com/bircher/drush/tree/merged. I could also add a simple new drush command that does a sub-set of the things the cex and cim commands do, just doing the same as the console command.

I have not used the environment module so far and I am not sure how this would integrate. From what I understand, the environment module changes the active configuration when invoked via the drush command and allows other modules to execute code via a hook when that happens. This approach is an imperative description of the state of the configuration and how to transition between them. The config_split approach is a declarative description of the subset of configuration you want to have merged or split off. Those are pretty different approaches and both have use-cases that overlap only partially.

As such, the only way I could see this work somehow is to have multiple configuration splits (as I mentioned in the post) and mapping environments and splits by name. Then performing a configuration export with the current environment split (maybe to a temporary directory) and importing the configuration with the new environment split. I am of course happy to discuss this further.

Hello Chris Thank you for your feedback! Drush needs two pull requests to be merged (2313 and 2314) you can use the version I used to test it (with the two merged) https://github.com/bircher/drush/tree/merged. I could also add a simple new drush command that does a sub-set of the things the cex and cim commands do, just doing the same as the console command.

I have not used the environment module so far and I am not sure how this would integrate. From what I understand, the environment module changes the active configuration when invoked via the drush command and allows other modules to execute code via a hook when that happens. This approach is an imperative description of the state of the configuration and how to transition between them. The config_split approach is a declarative description of the subset of configuration you want to have merged or split off. Those are pretty different approaches and both have use-cases that overlap only partially.

As such, the only way I could see this work somehow is to have multiple configuration splits (as I mentioned in the post) and mapping environments and splits by name. Then performing a configuration export with the current environment split (maybe to a temporary directory) and importing the configuration with the new environment split. I am of course happy to discuss this further.

Fabian Bircher, 29 Aug 2016

Hello Chris Thank you for your feedback! Drush needs two pull requests to be merged (2313 and 2314) you can use the version I used to test it (with the two merged) https://github.com/bircher/drush/tree/merged. I could also add a simple new drush command that does a sub-set of the things the cex and cim commands do, just doing the same as the console command.

I have not used the environment module so far and I am not sure how this would integrate. From what I understand, the environment module changes the active configuration when invoked via the drush command and allows other modules to execute code via a hook when that happens. This approach is an imperative description of the state of the configuration and how to transition between them. The config_split approach is a declarative description of the subset of configuration you want to have merged or split off. Those are pretty different approaches and both have use-cases that overlap only partially.

As such, the only way I could see this work somehow is to have multiple configuration splits (as I mentioned in the post) and mapping environments and splits by name. Then performing a configuration export with the current environment split (maybe to a temporary directory) and importing the configuration with the new environment split. I am of course happy to discuss this further.

Fabian Bircher, 29 Aug 2016
Chris
29 Aug 2016

Sure, not everyone needs Environment module. For me it was very helpful on larger D7 projects with more then 2 environments. I've always used it together with Features Module and features that contained settings for each environment.

In D8, the Environment module is rather useless if you work with CM and Git. You'll export the current environment to Git and you end up with the same environment on all environments. The only options you have is overriding it in settings.php or .gitignore. But that will not trigger any additional hooks and i need a hook to trigger because in larger projects, i would constantly import the database back from production down to each environment after each successful deployment. I do this because real content will only be submitted to production.

Having multiple configuration splits as a future development would be great. But then you would have to name each split anyway and that's where Environment module would be useful since you're creating multiple environment specific settings. Without Environment Module, you would have to manage the environments yourself and write custom Drush commands to switch between environments and import those specific settings. At one point users would come to you asking for a hook to do additional work and you would end up having the same functionality as Environment Module.

But apart from Environment Module, the best way to get multiple configuration splits going without much conflicts, would be to exclude selected settings on all environments. If you select the Google Analytics ID for example, you have to exclude this setting in the default at all and have to set it for each existing environment itself. This could be done by cloning but it would be less conflicting than with a existing default configuration in Git.

I'll give your Drush version a try.

Sure, not everyone needs Environment module. For me it was very helpful on larger D7 projects with more then 2 environments. I've always used it together with Features Module and features that contained settings for each environment.

In D8, the Environment module is rather useless if you work with CM and Git. You'll export the current environment to Git and you end up with the same environment on all environments. The only options you have is overriding it in settings.php or .gitignore. But that will not trigger any additional hooks and i need a hook to trigger because in larger projects, i would constantly import the database back from production down to each environment after each successful deployment. I do this because real content will only be submitted to production.

Having multiple configuration splits as a future development would be great. But then you would have to name each split anyway and that's where Environment module would be useful since you're creating multiple environment specific settings. Without Environment Module, you would have to manage the environments yourself and write custom Drush commands to switch between environments and import those specific settings. At one point users would come to you asking for a hook to do additional work and you would end up having the same functionality as Environment Module.

But apart from Environment Module, the best way to get multiple configuration splits going without much conflicts, would be to exclude selected settings on all environments. If you select the Google Analytics ID for example, you have to exclude this setting in the default at all and have to set it for each existing environment itself. This could be done by cloning but it would be less conflicting than with a existing default configuration in Git.

I'll give your Drush version a try.

Chris, 29 Aug 2016

Sure, not everyone needs Environment module. For me it was very helpful on larger D7 projects with more then 2 environments. I've always used it together with Features Module and features that contained settings for each environment.

In D8, the Environment module is rather useless if you work with CM and Git. You'll export the current environment to Git and you end up with the same environment on all environments. The only options you have is overriding it in settings.php or .gitignore. But that will not trigger any additional hooks and i need a hook to trigger because in larger projects, i would constantly import the database back from production down to each environment after each successful deployment. I do this because real content will only be submitted to production.

Having multiple configuration splits as a future development would be great. But then you would have to name each split anyway and that's where Environment module would be useful since you're creating multiple environment specific settings. Without Environment Module, you would have to manage the environments yourself and write custom Drush commands to switch between environments and import those specific settings. At one point users would come to you asking for a hook to do additional work and you would end up having the same functionality as Environment Module.

But apart from Environment Module, the best way to get multiple configuration splits going without much conflicts, would be to exclude selected settings on all environments. If you select the Google Analytics ID for example, you have to exclude this setting in the default at all and have to set it for each existing environment itself. This could be done by cloning but it would be less conflicting than with a existing default configuration in Git.

I'll give your Drush version a try.

Chris, 29 Aug 2016
Fabian Bircher
31 Aug 2016

Hello Chris

Yes, when working with CM and git it is best to adopt the workflow for which it was designed, namely exporting and importing. With Configuration split we interfere only at this point and allow for a more complex management of the files. I don't think we would add a hook to allow other modules to react since there are already events fired when configuration is imported.

Also I think we can limit the scope to entire configuration objects only: settings.php and the configuration override system is capable of dealing with most of the use cases like the GA-ID. The only time it falls short is when "settings" get compiled like the list of the enabled modules gets compiled into the container or the colour of bartik gets aggregated into css. In those cases we need to "override" the configuration at import time so that it becomes the canonical configuration. This could be achieved with a sort of "gray list" I mentioned in the blog post: We would not remove the gray-listed configuration and its dependencies from the production directory but just leave the original intact if it exists.

Feel free to open issues on drupal.org where we can discuss which of these improvements to tackle and how.

Thanks for your feedback and the insight into the environment module.

Hello Chris

Yes, when working with CM and git it is best to adopt the workflow for which it was designed, namely exporting and importing. With Configuration split we interfere only at this point and allow for a more complex management of the files. I don't think we would add a hook to allow other modules to react since there are already events fired when configuration is imported.

Also I think we can limit the scope to entire configuration objects only: settings.php and the configuration override system is capable of dealing with most of the use cases like the GA-ID. The only time it falls short is when "settings" get compiled like the list of the enabled modules gets compiled into the container or the colour of bartik gets aggregated into css. In those cases we need to "override" the configuration at import time so that it becomes the canonical configuration. This could be achieved with a sort of "gray list" I mentioned in the blog post: We would not remove the gray-listed configuration and its dependencies from the production directory but just leave the original intact if it exists.

Feel free to open issues on drupal.org where we can discuss which of these improvements to tackle and how.

Thanks for your feedback and the insight into the environment module.

Fabian Bircher, 31 Aug 2016

Hello Chris

Yes, when working with CM and git it is best to adopt the workflow for which it was designed, namely exporting and importing. With Configuration split we interfere only at this point and allow for a more complex management of the files. I don't think we would add a hook to allow other modules to react since there are already events fired when configuration is imported.

Also I think we can limit the scope to entire configuration objects only: settings.php and the configuration override system is capable of dealing with most of the use cases like the GA-ID. The only time it falls short is when "settings" get compiled like the list of the enabled modules gets compiled into the container or the colour of bartik gets aggregated into css. In those cases we need to "override" the configuration at import time so that it becomes the canonical configuration. This could be achieved with a sort of "gray list" I mentioned in the blog post: We would not remove the gray-listed configuration and its dependencies from the production directory but just leave the original intact if it exists.

Feel free to open issues on drupal.org where we can discuss which of these improvements to tackle and how.

Thanks for your feedback and the insight into the environment module.

Fabian Bircher, 31 Aug 2016
suresh kumara
11 Oct 2017

config split configuration path admin/config/development/configuration/config_split change to admin/config/development/configuration/config-split

config split configuration path admin/config/development/configuration/config_split change to admin/config/development/configuration/config-split

suresh kumara, 11 Oct 2017

config split configuration path admin/config/development/configuration/config_split change to admin/config/development/configuration/config-split

suresh kumara, 11 Oct 2017
Fabian Bircher
11 Oct 2017
new

Thank you for visiting our blog. This post was the introduction of the first public version of config split over a year ago. Since then we have made other blog posts where we communicated changes to the module. Many things have since changed and config split is now stable. You can find and contribute to the official documentation which is kept up to date here: https://www.drupal.org/docs/8/modules/configuration-split

new

Thank you for visiting our blog. This post was the introduction of the first public version of config split over a year ago. Since then we have made other blog posts where we communicated changes to the module. Many things have since changed and config split is now stable. You can find and contribute to the official documentation which is kept up to date here: https://www.drupal.org/docs/8/modules/configuration-split

Fabian Bircher, 11 Oct 2017
new

Thank you for visiting our blog. This post was the introduction of the first public version of config split over a year ago. Since then we have made other blog posts where we communicated changes to the module. Many things have since changed and config split is now stable. You can find and contribute to the official documentation which is kept up to date here: https://www.drupal.org/docs/8/modules/configuration-split

Fabian Bircher, 11 Oct 2017

Comments

Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

Get your project started today!

Contact us