Configuration Management and Features: a look at Drupal 8

Andrea Pescetti
3 Sep 2013
11 Comments
Andrea Pescetti
3 Sep 2013
11 Comments
Andrea Pescetti, 3 Sep 2013 - 11 Comments

Configuration Management and Features: a look at Drupal 8

What CMI will bring to D8 and "Poorman's Features" for packaging configuration in D8

An update to this post is now available at http://nuvole.org/blog/2014/jul/15/packaging-and-reusing-configuration-d....

We have always told attendees of our Drupal 7 "Code-driven development" trainings that embracing the idea that configuration must be stored in files and not in the database would make their Drupal development future-ready: the specific processes would necessarily change but the concept would stay and be reinforced in Drupal 8, thanks to the nice work being done in the Configuration Management Initiative.

And this was right. The progress done in CMI is very good, to the point that in our coming DrupalCon training in Prague, Code-Driven Development: an Effective Drupal Workflow, from D7 to D8 we will explore in detail how things work in Drupal 7, but for each topic we'll also describe what is going to improve/change in Drupal 8.

Configuration Management in a nutshell

From a site builder's point of view, Configuration Management in Drupal 8 is visible as two items in admin/config/development: "Configuration Export" and "Configuration Import".

The Export functionality has no further options, and it provides you with a config.tar.gz file containing hundreds (180+ on a clean installation, many more on a feature-rich test site) of files in YAML format, human- and machine-readable. All the site configuration is there: node types, their fields, permissions, Views, theme settings, variables...

The Import functionality, on the converse, will import a config.tar.gz file, show you any configuration changes between the currently active configuration and the one you are trying to import and allow you to replace the current configuration with the one in the provided files.

config-import

For many developers, this is a dream come true: with D8, Drupal Core will support a file-based configuration, allowing developers to track changes, put configuration under version control, update a production site cleanly... basically, all the advantages that we have described as selling points for the Features module in the last several years.

The Future of Features

So, if the major selling points of the Features module are now in core, does Features have a future in Drupal 8?

It still does, but it needs to be repurposed. So far, it had a dual functionality, serving as:

  • A machinery to export/import configuration from database to code. This will no longer be necessary in Drupal 8. Configuration lives in files and the database is used only for caching. For the same reason, an important problem in Drupal 7 development ("How do I make configuration exportable when there is no native support for Features?") is now solved for good.

  • A convenient way to package and reuse configuration across multiple sites. This is extremely useful to developers and not completely covered by Configuration Management in Drupal 8, even though the basic components are there.

Poorman's Features: emulating Features in Drupal 8

Let's see how it is possible to package configuration and emulate, manually, what Features and Drupal 7 allow today.

As an example use case, we will package a "Blog Feature".

1: Create and export configuration

Create the Blog content type, configure its fields (use proper naming for fields: field_blog_image and so on) and a classic view with a block and page display. Then go to admin/config/development and export your configuration into the config.tar.gz file.

2: Package relevant configuration into a module

Create a simple feature_blog module containing:

  • feature_blog.info.yml: a basic .info file in the Drupal 8 format.
name: Blog
type: module
description: 'A Blog content type and related configuration.'
package: Features
version: 8.x-0.1
core: 8.x
  • feature_blog.module: an empty module file, needed for correct Drupal operations.
<?php
  // Drupal needs this empty file.
  • a config subdirectory: copy to this folder all relevant files from config.tar.gz; filenames are quite helpful here, as it will be enough to copy all files that have "blog" in their name.
.
|-- config
|   |-- entity.display.comment.comment_node_blog.default.yml
|   |-- entity.display.node.blog.default.yml
|   |-- entity.display.node.blog.teaser.yml
|   |-- entity.form_display.comment.comment_node_blog.default.yml
|   |-- entity.form_display.node.blog.default.yml
|   |-- field.field.field_blog_image.yml
|   |-- field.instance.comment.comment_node_blog.comment_body.yml
|   |-- field.instance.node.blog.body.yml
|   |-- field.instance.node.blog.field_blog_image.yml
|   |-- node.type.blog.yml
|   `-- views.view.blog.yml
|-- feature_blog.info.yml
`-- feature_blog.module

3: Make the exported files portable (remove UUIDs)

The feature is ready, but it will only work if imported in the same Drupal 8 site. We need to make it portable to other sites.

The main problem here is that Drupal assigns UUIDs (Universally Unique Identifiers) to each component. When you add, say, an "image" field to a content type, you create an instance of the "image" field, and to describe this relationship Drupal will rely on the UUID of the "image" field. Since UUIDs are site-specific, trying to import this configuration on another site would result in an error message like:

Drupal\field\FieldException: Attempt to create an instance of
unknown field 84e904df-2f14-46e8-9700-e00c5ca3f7d3
in Drupal\field\Entity\FieldInstance->__construct()
(line 252 of .../core/modules/field/lib/Drupal/field/Entity/FieldInstance.php).

Fortunately, in this case configuration import is designed to accept either names or UUIDs for describing the field this is an instance of. So it is enough to edit all files containing field.instance in their name and manually replace UUIDs with machine names (or just add machine names), something like the following:

-field_uuid: 84e904df-2f14-46e8-9700-e00c5ca3f7d3
+field_name: field_blog_image

Note that this relies on the specific handling of the FieldInstance class and may not be a general solution.

Edit: also note (see comments) that it is important to include a hardcoded UUID if you want to avoid that your configuration items are seen by Drupal as different items between your development and production sites, causing configuration import/export between the two to fail.

4: Import the feature on another site

Our module/feature is now ready to be imported. Just copy it to the /modules folder on another site, or a subfolder of it (this is now the recommended location to add modules) and enable it.

Your content type and configuration will be immediately available.

Possible improvements

The Configuration Management in Drupal 8 will bring to Drupal Core a clean, unified, professional way to deal with configuration. There will still be the need for a module like Features for packaging and reusing configuration.

Some implementation details of Configuration Management do not behave well in this respect: for example, UUIDs are problematic and user permissions are stored in a packaging-unfriendly way (one file per each user role, with roles identified by UUID).

But, overall, the future looks bright and code-driven! And, as we have seen, it is already entirely possible to manually create basic "Features" (i.e., modules with configuration) that will work in Drupal 8.

Comments

Comments

Clemens Tolboom
3 Sep 2013

I was indeed curious about the feature-ness of CMI.

Do you now what happened to the content staging part of CMI?

I was indeed curious about the feature-ness of CMI.

Do you now what happened to the content staging part of CMI?

Clemens Tolboom, 3 Sep 2013

I was indeed curious about the feature-ness of CMI.

Do you now what happened to the content staging part of CMI?

Clemens Tolboom, 3 Sep 2013
Andrea Pescetti
4 Sep 2013

So far we only investigated the configuration management part that, as you have seen, is already quite stable. We haven't checked the content staging part yet.

So far we only investigated the configuration management part that, as you have seen, is already quite stable. We haven't checked the content staging part yet.

Andrea Pescetti, 4 Sep 2013

So far we only investigated the configuration management part that, as you have seen, is already quite stable. We haven't checked the content staging part yet.

Andrea Pescetti, 4 Sep 2013
Greg Dunlap (heyrocker)
4 Sep 2013

The content staging part of CMI never got done, we are already having trouble getting the configuration part finished. However there are many new features in Drupal 8 that will make content staging much easier than it was in the past.

- All entities can be serialized into a standard format.
- All entities will have UUIDs, which will make them more portable than relying on serial IDs.
- Built-in REST functionality will easily allow entities to be moved between environments.
- Improved APIs will mean that saving entities will no longer be reliant on form execution.
I expect a lot of interesting contrib solutions to appear in the next year.

As far as UUIDs vs machine names, one of the biggest problems we have right now is that in several places machine names can be changed, which means they are not reliable identifiers. You also run into several weird problems with machine names. One of the most notable is that with machine names it is impossible to determine the difference between an entity that has been modified and an entity that has been deleted and recreated. The actions that take place in those two scenarios are *vastly* different. Only UUIDs provide us a truly unique identifier we can use.

The content staging part of CMI never got done, we are already having trouble getting the configuration part finished. However there are many new features in Drupal 8 that will make content staging much easier than it was in the past.

- All entities can be serialized into a standard format.
- All entities will have UUIDs, which will make them more portable than relying on serial IDs.
- Built-in REST functionality will easily allow entities to be moved between environments.
- Improved APIs will mean that saving entities will no longer be reliant on form execution.
I expect a lot of interesting contrib solutions to appear in the next year.

As far as UUIDs vs machine names, one of the biggest problems we have right now is that in several places machine names can be changed, which means they are not reliable identifiers. You also run into several weird problems with machine names. One of the most notable is that with machine names it is impossible to determine the difference between an entity that has been modified and an entity that has been deleted and recreated. The actions that take place in those two scenarios are *vastly* different. Only UUIDs provide us a truly unique identifier we can use.

Greg Dunlap (heyrocker), 4 Sep 2013

The content staging part of CMI never got done, we are already having trouble getting the configuration part finished. However there are many new features in Drupal 8 that will make content staging much easier than it was in the past.

- All entities can be serialized into a standard format.
- All entities will have UUIDs, which will make them more portable than relying on serial IDs.
- Built-in REST functionality will easily allow entities to be moved between environments.
- Improved APIs will mean that saving entities will no longer be reliant on form execution.
I expect a lot of interesting contrib solutions to appear in the next year.

As far as UUIDs vs machine names, one of the biggest problems we have right now is that in several places machine names can be changed, which means they are not reliable identifiers. You also run into several weird problems with machine names. One of the most notable is that with machine names it is impossible to determine the difference between an entity that has been modified and an entity that has been deleted and recreated. The actions that take place in those two scenarios are *vastly* different. Only UUIDs provide us a truly unique identifier we can use.

Greg Dunlap (heyrocker), 4 Sep 2013
Adam Gerthel
5 Sep 2013

How will config man. handle things that shouldn't be version controlled?

For example, in Drupal 7 there is a variable called "language_default" which contains an array of settings, one of them being a randomized string which a language specific javascript file created in the files folder will be named after. If you clear all cache that string will be renewed, so keeping it in version control will be pointless. There are plenty of other scenarios like this with other modules.

Does D8 handle these?

How will config man. handle things that shouldn't be version controlled?

For example, in Drupal 7 there is a variable called "language_default" which contains an array of settings, one of them being a randomized string which a language specific javascript file created in the files folder will be named after. If you clear all cache that string will be renewed, so keeping it in version control will be pointless. There are plenty of other scenarios like this with other modules.

Does D8 handle these?

Adam Gerthel, 5 Sep 2013

How will config man. handle things that shouldn't be version controlled?

For example, in Drupal 7 there is a variable called "language_default" which contains an array of settings, one of them being a randomized string which a language specific javascript file created in the files folder will be named after. If you clear all cache that string will be renewed, so keeping it in version control will be pointless. There are plenty of other scenarios like this with other modules.

Does D8 handle these?

Adam Gerthel, 5 Sep 2013
yched
5 Sep 2013

Great article overall, thanks !

A couple important points though:

- Config import UI does not support partial imports. Any item absent in the imported set will be considered a deletion.
Separate tools can be built on top of that, that let you provide "just a couple new config files", will copy them on top of the existing config and import the thing as a whole, but the import mechanism built in core only reasons on "full config set".

- It is critical that "default config" files shipped in a module's /config directory, do include a hardcoded UUID. Otherwise, the resulting items, once imported into the active config directory, will be seen as different items on your dev and prod sites, and subsequent config imports between the two will fail badly.
D8 core itself is not consistent on that aspect yet, but this is being fixed - https://drupal.org/node/1969800.

Great article overall, thanks !

A couple important points though:

- Config import UI does not support partial imports. Any item absent in the imported set will be considered a deletion.
Separate tools can be built on top of that, that let you provide "just a couple new config files", will copy them on top of the existing config and import the thing as a whole, but the import mechanism built in core only reasons on "full config set".

- It is critical that "default config" files shipped in a module's /config directory, do include a hardcoded UUID. Otherwise, the resulting items, once imported into the active config directory, will be seen as different items on your dev and prod sites, and subsequent config imports between the two will fail badly.
D8 core itself is not consistent on that aspect yet, but this is being fixed - https://drupal.org/node/1969800.

yched, 5 Sep 2013

Great article overall, thanks !

A couple important points though:

- Config import UI does not support partial imports. Any item absent in the imported set will be considered a deletion.
Separate tools can be built on top of that, that let you provide "just a couple new config files", will copy them on top of the existing config and import the thing as a whole, but the import mechanism built in core only reasons on "full config set".

- It is critical that "default config" files shipped in a module's /config directory, do include a hardcoded UUID. Otherwise, the resulting items, once imported into the active config directory, will be seen as different items on your dev and prod sites, and subsequent config imports between the two will fail badly.
D8 core itself is not consistent on that aspect yet, but this is being fixed - https://drupal.org/node/1969800.

yched, 5 Sep 2013
Andrea Pescetti
5 Sep 2013

Thanks! Edited accordingly. Actually the partial import is out of scope here, so I just removed it. And about UUIDs handling, we'll follow the issue you mentioned.

Thanks! Edited accordingly. Actually the partial import is out of scope here, so I just removed it. And about UUIDs handling, we'll follow the issue you mentioned.

Andrea Pescetti, 5 Sep 2013

Thanks! Edited accordingly. Actually the partial import is out of scope here, so I just removed it. And about UUIDs handling, we'll follow the issue you mentioned.

Andrea Pescetti, 5 Sep 2013
Larry Garfield
10 Sep 2013

Adam: That sort of data was never really configuration to begin with; we just usually stuck it in the variables system for lack of anything better. In Drupal 8 there is now a State API, which provides an instance-specific key/value store for data should not be staged between sites. The last-cron-ran-time is the canonical example there, but there are plenty of other uses. It's also pluggable, so if you want a bit more performance you can swap out the default MySQL implementation for Redis, MongoDB, or whatever your favorite key/value store is.

As far as content staging, heyrocker is correct that the better way to handle that is via web services. With the REST module in core now and the many improvements to the Entity API it is possible to entirely round-trip any content entity to the HAL format and back, even on different sites. (Or at least it should be; if you find it's not the case please file a bug) That's not content staging in and of itself; rather, it means that building a robust content staging or content syndication system atop standard REST semantics should be WAY easier than in previous versions of Drupal. I expect the Deploy module, for instance, to make a come back with a far far cleaner implementation. Other models and workflows are left for contrib to experiment with.

Adam: That sort of data was never really configuration to begin with; we just usually stuck it in the variables system for lack of anything better. In Drupal 8 there is now a State API, which provides an instance-specific key/value store for data should not be staged between sites. The last-cron-ran-time is the canonical example there, but there are plenty of other uses. It's also pluggable, so if you want a bit more performance you can swap out the default MySQL implementation for Redis, MongoDB, or whatever your favorite key/value store is.

As far as content staging, heyrocker is correct that the better way to handle that is via web services. With the REST module in core now and the many improvements to the Entity API it is possible to entirely round-trip any content entity to the HAL format and back, even on different sites. (Or at least it should be; if you find it's not the case please file a bug) That's not content staging in and of itself; rather, it means that building a robust content staging or content syndication system atop standard REST semantics should be WAY easier than in previous versions of Drupal. I expect the Deploy module, for instance, to make a come back with a far far cleaner implementation. Other models and workflows are left for contrib to experiment with.

Larry Garfield, 10 Sep 2013

Adam: That sort of data was never really configuration to begin with; we just usually stuck it in the variables system for lack of anything better. In Drupal 8 there is now a State API, which provides an instance-specific key/value store for data should not be staged between sites. The last-cron-ran-time is the canonical example there, but there are plenty of other uses. It's also pluggable, so if you want a bit more performance you can swap out the default MySQL implementation for Redis, MongoDB, or whatever your favorite key/value store is.

As far as content staging, heyrocker is correct that the better way to handle that is via web services. With the REST module in core now and the many improvements to the Entity API it is possible to entirely round-trip any content entity to the HAL format and back, even on different sites. (Or at least it should be; if you find it's not the case please file a bug) That's not content staging in and of itself; rather, it means that building a robust content staging or content syndication system atop standard REST semantics should be WAY easier than in previous versions of Drupal. I expect the Deploy module, for instance, to make a come back with a far far cleaner implementation. Other models and workflows are left for contrib to experiment with.

Larry Garfield, 10 Sep 2013
André
22 Apr 2014

These instructions don't appear to work under alpha10. When enabling the module, the error "Attempt to create an instance of unknown field ___ in Drupal\field\Entity\FieldInstanceConfig->__construct() (line 240 of /Users/aangel/Sites/Gaz/core/modules/field/lib/Drupal/field/Entity/FieldInstanceConfig.php" occurs.

These instructions don't appear to work under alpha10. When enabling the module, the error "Attempt to create an instance of unknown field ___ in Drupal\field\Entity\FieldInstanceConfig->__construct() (line 240 of /Users/aangel/Sites/Gaz/core/modules/field/lib/Drupal/field/Entity/FieldInstanceConfig.php" occurs.

André, 22 Apr 2014

These instructions don't appear to work under alpha10. When enabling the module, the error "Attempt to create an instance of unknown field ___ in Drupal\field\Entity\FieldInstanceConfig->__construct() (line 240 of /Users/aangel/Sites/Gaz/core/modules/field/lib/Drupal/field/Entity/FieldInstanceConfig.php" occurs.

André, 22 Apr 2014
Andrea Pescetti
24 Apr 2014

There were many significant changes in CMI since this post was written. An updated post is coming in 1-2 weeks, stay tuned!

There were many significant changes in CMI since this post was written. An updated post is coming in 1-2 weeks, stay tuned!

Andrea Pescetti, 24 Apr 2014

There were many significant changes in CMI since this post was written. An updated post is coming in 1-2 weeks, stay tuned!

Andrea Pescetti, 24 Apr 2014
Ryan
3 Jun 2014

Looking forward to an update to this post, which was so helpful at the time.

Looking forward to an update to this post, which was so helpful at the time.

Ryan, 3 Jun 2014

Looking forward to an update to this post, which was so helpful at the time.

Ryan, 3 Jun 2014
Andrea Pescetti
15 Jul 2014
Andrea Pescetti, 15 Jul 2014
Andrea Pescetti, 15 Jul 2014

Get your project started today!

Contact us