Optimal deployment workflow for Composer-based Drupal 8 projects

Andrea Pescetti
19 Aug 2016
12 Comments
Andrea Pescetti
19 Aug 2016
12 Comments
Andrea Pescetti, 19 Aug 2016 - 12 Comments

Optimal deployment workflow for Composer-based Drupal 8 projects

Considerations following our Drupal Dev Day Milan and Drupalaton presentations; and a preview of our DrupalCon training.

This post is an excerpt from the topics covered by our DrupalCon Dublin training: Drupal 8 Development - Workflows and Tools.

During the recent Nuvole presentations at Drupal Dev Days Milan 2016 and Drupalaton Hungary 2016 we received a number of questions on how to properly setup a Drupal 8 project with Composer. An interesting case where we discovered that existing practices are completely different from each other is: "What is the best way to deploy a Composer-based Drupal 8 project?".

We'll quickly discuss some options and describe what works best for us.

What to commit

You should commit:

  • The composer.json file: this is obvious when using Composer.
  • The composer.lock file: this is important since it will allow you to rebuild the entire codebase at the same status it was at a given point in the past.

The fully built site is commonly left out of the repository. But this also means that you need to find a way for rebuilding and deploying the codebase safely.

Don't run Composer on the production server

You would clearly never run composer update on the production server, as you want to be sure that you will be deploying the same code you have been developing upon. For a while, we considered it to be enough to have Composer installed on the server and run composer install to get predictable results from the (committed) composer.lock file.

Then we discovered that this approach has a few shortcomings:

  • The process is not robust. A transient network error or timeout might result in a failed build, thus introducing uncertainty factors in the deploy scripts. Easy to handle, but still not desirable as part of a delicate step such as deployment.

  • The process will inevitably take long. If you run composer install in the webroot directly, your codebase will be unstable for a few minutes. This is orders of magnitude longer than a standard update process (i.e., running drush updb and drush cim) and it may affect your site availability. This can be circumvented by building in a separate directory and then symlinking or moving directories.

  • Even composer install can be unpredictable, especially on servers with restrictions or running different versions of Composer or PHP; in rare circumstances, a build may succeed but yield a different codebase. This can be mitigated by enforcing (e.g., through Docker or virtualization) a dev/staging environment that matches the production environment, but you are still losing control on a relatively lengthy process.

  • You have no way of properly testing the newly built codebase after building it and before making it live.

  • Composer simply does not belong in a production server. It is a tool with a different scope, unrelated to the main tasks of a production server.

Where to build the codebase? CI to the rescue

After ruling out the production server, where should the codebase be built then?

Building it locally (i.e., using a developer's environment) can't work: besides the differences between the development and the production (--no-dev) setup, there is the risk of missing possible small patches applied to the local codebase. And a totally clean build is always necessary anyway.

We ended up using Continuous Integration for this task. Besides the standard CI job, which operates after any push operation to the branches under active development, performs a clean installation and runs automated tests, another CI job builds the full codebase based on the master branch and the composer.lock file. This allows sharing it between developers, a fast deployment to production through a tarball or rsync, and opportunities for actually testing the upgrade (with a process like: automatically import the production database, run database updates, import the new configuration, run a subset of automated tests to ensure that basic site functionality has no regressions) for maximum safety.

Slides from our recent presentations, mostly focused on Configuration Management but covering part of this discussion too, are below.

Configuration Management in Drupal 8 - DDD2016.pdf (1.05 MB)Download

Comments

Comments

micbar
21 Aug 2016

So if you don't run composer on the production server and don't build your codebase locally, how do you get your clean build of your codebase for the live site?

So if you don't run composer on the production server and don't build your codebase locally, how do you get your clean build of your codebase for the live site?

micbar, 21 Aug 2016

So if you don't run composer on the production server and don't build your codebase locally, how do you get your clean build of your codebase for the live site?

micbar, 21 Aug 2016
Andrea Pescetti
22 Aug 2016

The Continuous Integration environment takes care of it. In our case, this is a separate environment from both the local and production server; and it is the best place for doing further operations (e.g., automated tests) on the build codebase.

The Continuous Integration environment takes care of it. In our case, this is a separate environment from both the local and production server; and it is the best place for doing further operations (e.g., automated tests) on the build codebase.

Andrea Pescetti, 22 Aug 2016

The Continuous Integration environment takes care of it. In our case, this is a separate environment from both the local and production server; and it is the best place for doing further operations (e.g., automated tests) on the build codebase.

Andrea Pescetti, 22 Aug 2016
Piotr Nowak
5 Sep 2016

I'm using buddy.works for it - for me is's the best and easiest solution to deploy php apps with composer :)

I'm using buddy.works for it - for me is's the best and easiest solution to deploy php apps with composer :)

Piotr Nowak, 5 Sep 2016

I'm using buddy.works for it - for me is's the best and easiest solution to deploy php apps with composer :)

Piotr Nowak, 5 Sep 2016
Johny
2 Dec 2016

As I'm new to the deployment topic I didn't understand the last part. What are CI Jobs and what are they doing exactly?

As I'm new to the deployment topic I didn't understand the last part. What are CI Jobs and what are they doing exactly?

Johny, 2 Dec 2016

As I'm new to the deployment topic I didn't understand the last part. What are CI Jobs and what are they doing exactly?

Johny, 2 Dec 2016
Andrea Pescetti
2 Dec 2016

Hi, CI is a common short form for Continuous Integration. You can find all details just searching the Web for "Continuous Integration": in the case of a Drupal project, we want to be able to build the codebase, install the project and see automated tests pass; all of this happens in a dedicated test environment and is completely automatic.

Hi, CI is a common short form for Continuous Integration. You can find all details just searching the Web for "Continuous Integration": in the case of a Drupal project, we want to be able to build the codebase, install the project and see automated tests pass; all of this happens in a dedicated test environment and is completely automatic.

Andrea Pescetti, 2 Dec 2016

Hi, CI is a common short form for Continuous Integration. You can find all details just searching the Web for "Continuous Integration": in the case of a Drupal project, we want to be able to build the codebase, install the project and see automated tests pass; all of this happens in a dedicated test environment and is completely automatic.

Andrea Pescetti, 2 Dec 2016
Johny
2 Dec 2016

That's the first thing I did ;), but I just find very general information. I suppose the automation process is done similar to this (https://chromatichq.com/blog/drupal-8-deployments-jenkins-github-slack)?
I'm just not aware of the single steps in the process. I suppose you are still installing the site with Composer on the staging server? How is the site transfered from staging to production? What is finally controlled by git and what not?

That's the first thing I did ;), but I just find very general information. I suppose the automation process is done similar to this (https://chromatichq.com/blog/drupal-8-deployments-jenkins-github-slack)?
I'm just not aware of the single steps in the process. I suppose you are still installing the site with Composer on the staging server? How is the site transfered from staging to production? What is finally controlled by git and what not?

Johny, 2 Dec 2016

That's the first thing I did ;), but I just find very general information. I suppose the automation process is done similar to this (https://chromatichq.com/blog/drupal-8-deployments-jenkins-github-slack)?
I'm just not aware of the single steps in the process. I suppose you are still installing the site with Composer on the staging server? How is the site transfered from staging to production? What is finally controlled by git and what not?

Johny, 2 Dec 2016
Andrea Pescetti
4 Dec 2016

The example you cite is quite different. This is a post about Composer and the post you mention is about CI but not in a Composer workflow.

To answer your 3 questions:

I suppose you are still installing the site with Composer on the staging server? No. The staging server does not run Composer. You don't need Composer to install a site. The "install" in Composer and Drupal are two entirely different things.

How is the site transfered from staging to production? The site is never transferred from staging to production. The same codebase (obtained on the CI server using "composer install") is deployed to staging and production.

What is finally controlled by git and what not? We version in Git the files Composer needs to recreate the codebase. So, Drupal and contrib modules are not in Git. We build the site codebase with "composer install" locally and on CI servers. From the CI servers the packaged codebase is deployed to the hosting servers (staging and production). This way we don't need to use composer on the hosting servers.

The example you cite is quite different. This is a post about Composer and the post you mention is about CI but not in a Composer workflow.

To answer your 3 questions:

I suppose you are still installing the site with Composer on the staging server? No. The staging server does not run Composer. You don't need Composer to install a site. The "install" in Composer and Drupal are two entirely different things.

How is the site transfered from staging to production? The site is never transferred from staging to production. The same codebase (obtained on the CI server using "composer install") is deployed to staging and production.

What is finally controlled by git and what not? We version in Git the files Composer needs to recreate the codebase. So, Drupal and contrib modules are not in Git. We build the site codebase with "composer install" locally and on CI servers. From the CI servers the packaged codebase is deployed to the hosting servers (staging and production). This way we don't need to use composer on the hosting servers.

Andrea Pescetti, 4 Dec 2016

The example you cite is quite different. This is a post about Composer and the post you mention is about CI but not in a Composer workflow.

To answer your 3 questions:

I suppose you are still installing the site with Composer on the staging server? No. The staging server does not run Composer. You don't need Composer to install a site. The "install" in Composer and Drupal are two entirely different things.

How is the site transfered from staging to production? The site is never transferred from staging to production. The same codebase (obtained on the CI server using "composer install") is deployed to staging and production.

What is finally controlled by git and what not? We version in Git the files Composer needs to recreate the codebase. So, Drupal and contrib modules are not in Git. We build the site codebase with "composer install" locally and on CI servers. From the CI servers the packaged codebase is deployed to the hosting servers (staging and production). This way we don't need to use composer on the hosting servers.

Andrea Pescetti, 4 Dec 2016
Johny
10 Feb 2017

Ok, I'm starting to understand how this is working... What CI server are you using? An external service or did you somehow set it up on your own?

Ok, I'm starting to understand how this is working... What CI server are you using? An external service or did you somehow set it up on your own?

Johny, 10 Feb 2017

Ok, I'm starting to understand how this is working... What CI server are you using? An external service or did you somehow set it up on your own?

Johny, 10 Feb 2017
Andrea Pescetti
10 Feb 2017

It is an own environment, but the concepts apply in general.

It is an own environment, but the concepts apply in general.

Andrea Pescetti, 10 Feb 2017

It is an own environment, but the concepts apply in general.

Andrea Pescetti, 10 Feb 2017
Doug Ouverson
23 Oct 2017

It would be nice to have more details on how you actually setup CI: tools, techniques, etc. Gracias!

It would be nice to have more details on how you actually setup CI: tools, techniques, etc. Gracias!

Doug Ouverson, 23 Oct 2017

It would be nice to have more details on how you actually setup CI: tools, techniques, etc. Gracias!

Doug Ouverson, 23 Oct 2017
Andrea Pescetti
24 Oct 2017
new

Covering CI details is outside the scope of this post, but you will be able to find a lot of documentation online. As for the tools, you can run your own server but you can even get CI for free on, e.g., Gitlab and if you are new to CI I would recommend starting with a hosted solution like that.

new

Covering CI details is outside the scope of this post, but you will be able to find a lot of documentation online. As for the tools, you can run your own server but you can even get CI for free on, e.g., Gitlab and if you are new to CI I would recommend starting with a hosted solution like that.

Andrea Pescetti, 24 Oct 2017
new

Covering CI details is outside the scope of this post, but you will be able to find a lot of documentation online. As for the tools, you can run your own server but you can even get CI for free on, e.g., Gitlab and if you are new to CI I would recommend starting with a hosted solution like that.

Andrea Pescetti, 24 Oct 2017
Andrea Pescetti
24 Oct 2017
new

Comments on this post are closed. Please check out more recent posts on this blog!

new

Comments on this post are closed. Please check out more recent posts on this blog!

Andrea Pescetti, 24 Oct 2017
new

Comments on this post are closed. Please check out more recent posts on this blog!

Andrea Pescetti, 24 Oct 2017

Get your project started today!

Contact us