Towards a 100% database free development workflow

Antonio De Marco
24 May 2010
8 Comments
Antonio De Marco
24 May 2010
8 Comments
Antonio De Marco, 24 May 2010 - 8 Comments

Towards a 100% database free development workflow

Introducing #codepower

Building a website using Drupal is essentially about tuning and extending its core functionalities to meet the project's specific requirements. During its development phase we create content types with CCK, display them using Views and extend the basic functionalities by setting up contributed modules. All these actions are recorded into the database in the form of a very heterogeneous collection of settings, each specific to the module it belongs to.

When the development phase is completed the database probably contains, besides all our valuable configuration, other Drupal "entities" such as content, users, etc... that we don't really want to carry along once the website goes live: a careful and time consuming cleanup is then necessary.

With time our website naturally grows in size; it contains much more content and several users are active on it: soon it might be necessary to extend its functionalities. How to do it when both settings and content are now valuable, when really nothing can be lost during development?

The immediate solution could be to work on a copy of the production database and restore it afterwards. This, obviously, means that activities on the production website have to be suspended until the new development is completed. Additionally, a pre-live clean-up will probably be necessary also in this case.

When a team of developers works on the same project issues multiply: if each developer keeps and works on his own database dump the risk of overriding each other's changes is high. Another option is to work all together the same shared database, but still hoping that nobody will mess things up.

It is clear that relying on the database makes the development flow really error prone, not easily scalable and definitely difficult to maintain.

Settings in code: the key to Drupal heaven

Keeping track of changes, working concurrently on the same project and resolving conflicts are all issues that are usually addressed by the adoption of a version control systems (VCS). Unfortunately they are not that useful when it comes to working with configuration stored in database: you can surely version a dump but change tracking and conflict solving are practically impossible.

Everything changes if configuration is stored "in code", meaning that, instead to have settings spread across different tables, they are translated to understandable PHP code that Drupal modules can easily consume. Exportable entities are not something new in the Drupal universe: modules like Views and CCK have already supported exporting setting for quite some time. What's missing is to streamline the export/import operations to, eventually, switch from a database to a code driven development. All this is today possible thanks to the Features project. This project introduces the concept of feature as a collection of settings and configuration that describe a very well defined functionality. In doing so it implements a very solid system of keeping track of changes that occurs to settings "owned" by a particular feature making easy to dump them to code. That's basically it, the rest is all about wisely organizing our project in features which keeps all our settings safely in code. At this point, by using our favorite VCS, changes get traceable and conflicts resolvable, exactly how it happens with standard PHP code.

Introducing #codepower

Projects like Features, Chaos tool suite or Strongarm are at the heart of the code driven revolution, even though a lot still depends on how actually we are going to use this technology: good code conventions, well defined features boundaries and other good practices can really make the difference, at the end of the day.

What we really need is to share each other's best practices to develop together patterns for a solid code driven development. For this reason Nuvole introduced, at Drupal VolCon in Antwerp, the #codepower Twitter tag. The idea is simple: we will tweet and tag with #codepower whatever we deem interesting to be shared with the community regarding code driven development. We ask you to do the same.

The Movement

Today we all have the chance to push Drupal even further by changing the way we work and develop applications with it. At Nuvole the adoption of a code driven development has completely revolutionized our working flow, making it more maintainable, solid and scalable. For this reason we want to contribute back to the community our experience about code driven development with a series of blog posts, starting in occasion of the first event on the matter: "Developer Session: Code Driven Development", the 27th of May 2010, in Antwerp (Belgium) hosted by the great guys at Krimson.

Comments

Comments

Christopher Pelham
24 May 2010

If one wanted to advocate for a completely settings-free database for Drupal 8, for example, where would one begin to do this? Where should this kind of discussion take place?

If one wanted to advocate for a completely settings-free database for Drupal 8, for example, where would one begin to do this? Where should this kind of discussion take place?

Christopher Pelham, 24 May 2010

If one wanted to advocate for a completely settings-free database for Drupal 8, for example, where would one begin to do this? Where should this kind of discussion take place?

Christopher Pelham, 24 May 2010
decibel.places
25 May 2010

I support keeping the settings and content in the database.

There are already file-based places for settings in the theme tpl and css files, as well as Features and CTools etc.

Keeping the nodes and relations and configuration in the db as discrete bits of info makes the filesystem more standardized with settings that can be exported and replicated more easily.

When I am migrating a Drupal site, I move the files (sometimes there are more than 10K files already) and export and import the database - adjust the db connection in settings.php and voila! the site is moved. I would not want to have to edit dozens or hundreds of files!

Just think about the difference between working on a Drupal site and on a Joomla (or Mambo!) site - and those are mainly db-driven as well!

Then consider the dead wood that accrues in a file-based CMS - I have just today worked on adding GA tracking code to a "custom" site. The content is not in the db, so I had to download 15K+ files and search them for the 2 anchor tags I needed to modify. I found 6 files with the anchor tags but only two of them were actually being used. I estimate in sites like this, up to 75% of the files are dead wood and are never used.

One might argue that professional developers would keep the filesystem lean and organized - well, the same argument has already been made about the database. It is more difficult for novice coders to introduce garbage in the database than it is for them to add revisions as new files without removing the older versions.

Drupal seems to be heading for a hybridization with file-based configs layered over the core, modules and database - which is fine with me!

I support keeping the settings and content in the database.

There are already file-based places for settings in the theme tpl and css files, as well as Features and CTools etc.

Keeping the nodes and relations and configuration in the db as discrete bits of info makes the filesystem more standardized with settings that can be exported and replicated more easily.

When I am migrating a Drupal site, I move the files (sometimes there are more than 10K files already) and export and import the database - adjust the db connection in settings.php and voila! the site is moved. I would not want to have to edit dozens or hundreds of files!

Just think about the difference between working on a Drupal site and on a Joomla (or Mambo!) site - and those are mainly db-driven as well!

Then consider the dead wood that accrues in a file-based CMS - I have just today worked on adding GA tracking code to a "custom" site. The content is not in the db, so I had to download 15K+ files and search them for the 2 anchor tags I needed to modify. I found 6 files with the anchor tags but only two of them were actually being used. I estimate in sites like this, up to 75% of the files are dead wood and are never used.

One might argue that professional developers would keep the filesystem lean and organized - well, the same argument has already been made about the database. It is more difficult for novice coders to introduce garbage in the database than it is for them to add revisions as new files without removing the older versions.

Drupal seems to be heading for a hybridization with file-based configs layered over the core, modules and database - which is fine with me!

decibel.places, 25 May 2010

I support keeping the settings and content in the database.

There are already file-based places for settings in the theme tpl and css files, as well as Features and CTools etc.

Keeping the nodes and relations and configuration in the db as discrete bits of info makes the filesystem more standardized with settings that can be exported and replicated more easily.

When I am migrating a Drupal site, I move the files (sometimes there are more than 10K files already) and export and import the database - adjust the db connection in settings.php and voila! the site is moved. I would not want to have to edit dozens or hundreds of files!

Just think about the difference between working on a Drupal site and on a Joomla (or Mambo!) site - and those are mainly db-driven as well!

Then consider the dead wood that accrues in a file-based CMS - I have just today worked on adding GA tracking code to a "custom" site. The content is not in the db, so I had to download 15K+ files and search them for the 2 anchor tags I needed to modify. I found 6 files with the anchor tags but only two of them were actually being used. I estimate in sites like this, up to 75% of the files are dead wood and are never used.

One might argue that professional developers would keep the filesystem lean and organized - well, the same argument has already been made about the database. It is more difficult for novice coders to introduce garbage in the database than it is for them to add revisions as new files without removing the older versions.

Drupal seems to be heading for a hybridization with file-based configs layered over the core, modules and database - which is fine with me!

decibel.places, 25 May 2010
Quevin
25 May 2010

@decibel.places I think the majority of problems that Developers experience isn't with the first deployment, but regularly scheduled deployments after that. There's also a problem with tracking settings changes in the database alongside content changes that may be happening on production servers. However, some sites don't even rely on Drupal for content management, but pull from an external DB (whitehouse.gov).

I usually need a plan for each project. Migrating the DB each time may work for some sites, but others don't want anyone to mess with the DB because content is changing by the second. That's where Features, or any kind of exportable helps tremendously. Otherwise, I'm doing work two or even three times before it gets to the Production site.

I'd like the option to keep things in the DB, or have everything I need as a packaged exportable that's not going to bring down the site when I activate it because of some unforeseen override.

@decibel.places I think the majority of problems that Developers experience isn't with the first deployment, but regularly scheduled deployments after that. There's also a problem with tracking settings changes in the database alongside content changes that may be happening on production servers. However, some sites don't even rely on Drupal for content management, but pull from an external DB (whitehouse.gov).

I usually need a plan for each project. Migrating the DB each time may work for some sites, but others don't want anyone to mess with the DB because content is changing by the second. That's where Features, or any kind of exportable helps tremendously. Otherwise, I'm doing work two or even three times before it gets to the Production site.

I'd like the option to keep things in the DB, or have everything I need as a packaged exportable that's not going to bring down the site when I activate it because of some unforeseen override.

Quevin, 25 May 2010

@decibel.places I think the majority of problems that Developers experience isn't with the first deployment, but regularly scheduled deployments after that. There's also a problem with tracking settings changes in the database alongside content changes that may be happening on production servers. However, some sites don't even rely on Drupal for content management, but pull from an external DB (whitehouse.gov).

I usually need a plan for each project. Migrating the DB each time may work for some sites, but others don't want anyone to mess with the DB because content is changing by the second. That's where Features, or any kind of exportable helps tremendously. Otherwise, I'm doing work two or even three times before it gets to the Production site.

I'd like the option to keep things in the DB, or have everything I need as a packaged exportable that's not going to bring down the site when I activate it because of some unforeseen override.

Quevin, 25 May 2010
Mike Bishop
25 May 2010

Every project was the same, we'd keep the DB clean, we'd plan, we even started abstracting the db writes ourself but at the end of the day, Drupal is still way to married to a database.

It's the philosophy of making everything configurable and dumb enough for a content administrator that kills drupal.

Be honest with your self. When was the last time you ever really wanted a form to change the title of some column in a view. Really. Did a client ever expect that level of control? So all of that horse power that could be spent making really bad ass features, is spent trying to figure out what table got updated when you or someone else made a change to the views configuration.

Deploying and re-deploying and trying to maintain content in multiple environments, eg, staging, testing, production... it's too much.

All of the tools that where just starting out when we said "f-you drupal" like "features" looked like they where heading down the right path. Even code based Views that could be stored in version control... but the fact of the matter is, even with those working 100% as designed, whats the point of drupal? Why go through all that heart ache?

Your opening sentance spells it all out: "tuning and extending its core functionalities to meet the project's specific requirements"

I am sorry, thats completely asinine and epitomizes everything that is wrong with Drupal. Thats like saying, Let's buy a bull dozer to plant our flower garden. It's ok, we'll just trim down and tune the bull dozer to be a smaller set of tools that we really need.

Every project was the same, we'd keep the DB clean, we'd plan, we even started abstracting the db writes ourself but at the end of the day, Drupal is still way to married to a database.

It's the philosophy of making everything configurable and dumb enough for a content administrator that kills drupal.

Be honest with your self. When was the last time you ever really wanted a form to change the title of some column in a view. Really. Did a client ever expect that level of control? So all of that horse power that could be spent making really bad ass features, is spent trying to figure out what table got updated when you or someone else made a change to the views configuration.

Deploying and re-deploying and trying to maintain content in multiple environments, eg, staging, testing, production... it's too much.

All of the tools that where just starting out when we said "f-you drupal" like "features" looked like they where heading down the right path. Even code based Views that could be stored in version control... but the fact of the matter is, even with those working 100% as designed, whats the point of drupal? Why go through all that heart ache?

Your opening sentance spells it all out: "tuning and extending its core functionalities to meet the project's specific requirements"

I am sorry, thats completely asinine and epitomizes everything that is wrong with Drupal. Thats like saying, Let's buy a bull dozer to plant our flower garden. It's ok, we'll just trim down and tune the bull dozer to be a smaller set of tools that we really need.

Mike Bishop, 25 May 2010

Every project was the same, we'd keep the DB clean, we'd plan, we even started abstracting the db writes ourself but at the end of the day, Drupal is still way to married to a database.

It's the philosophy of making everything configurable and dumb enough for a content administrator that kills drupal.

Be honest with your self. When was the last time you ever really wanted a form to change the title of some column in a view. Really. Did a client ever expect that level of control? So all of that horse power that could be spent making really bad ass features, is spent trying to figure out what table got updated when you or someone else made a change to the views configuration.

Deploying and re-deploying and trying to maintain content in multiple environments, eg, staging, testing, production... it's too much.

All of the tools that where just starting out when we said "f-you drupal" like "features" looked like they where heading down the right path. Even code based Views that could be stored in version control... but the fact of the matter is, even with those working 100% as designed, whats the point of drupal? Why go through all that heart ache?

Your opening sentance spells it all out: "tuning and extending its core functionalities to meet the project's specific requirements"

I am sorry, thats completely asinine and epitomizes everything that is wrong with Drupal. Thats like saying, Let's buy a bull dozer to plant our flower garden. It's ok, we'll just trim down and tune the bull dozer to be a smaller set of tools that we really need.

Mike Bishop, 25 May 2010
tizzo
25 May 2010

The exportables/features approach is a great start but what we really need is a way to be able to truly separate settings from content and have a homogenous system for all settings.

I think we need to beef up the exportables system heavily and make all settings exportable by default in one central system. I've started working on this in my system settings github project but there is a heck of a lot to do. When I get something more stable and usable I'll be releasing a Drupal module. I'm afraid things are still a bit broken in there.

The exportables/features approach is a great start but what we really need is a way to be able to truly separate settings from content and have a homogenous system for all settings.

I think we need to beef up the exportables system heavily and make all settings exportable by default in one central system. I've started working on this in my system settings github project but there is a heck of a lot to do. When I get something more stable and usable I'll be releasing a Drupal module. I'm afraid things are still a bit broken in there.

tizzo, 25 May 2010

The exportables/features approach is a great start but what we really need is a way to be able to truly separate settings from content and have a homogenous system for all settings.

I think we need to beef up the exportables system heavily and make all settings exportable by default in one central system. I've started working on this in my system settings github project but there is a heck of a lot to do. When I get something more stable and usable I'll be releasing a Drupal module. I'm afraid things are still a bit broken in there.

tizzo, 25 May 2010
decibel.places
25 May 2010

If the goal is to create export a clean and replicable base configuration, what about Installation Profiles? http://drupal.org/node/67921

@Mike Bishop - I would say that the granularity IS desirable - and even so, I am often adding bits of jQuery to adjust labels or visibility in the DOM when a module or view lacks the config (example: OG Group search block button value http://drupal.org/node/806182 ). But I find it preferable to add a bit of JS to my page.tpl.php file and I will remember where I made the customization, rather than search through 10K+ files for the right one(s) to modify. So this is a file-based config - but sometimes I add PHP or JavaScript to blocks that may otherwise be invisible for page-specific functions - and of course block code is stored in the db.

Regarding working on busy sites, that is what distributed servers and dbs are for. If you are storing user-generated content in files, then how would that be different? If the files are being written at the time of export, you could lose or corrupt the data. Many Drupal tools that deal directly with the db can lock it temporarily and/or take the site offline briefly (eg Backup & Migrate).

So what if WhiteHouse.gov uses some custom databases or tables? - I have done the same, and it's STILL a Drupal site.

I am not arguing that Drupal is the solution for every requirement - but it works really well at least 95% of the time...

If the goal is to create export a clean and replicable base configuration, what about Installation Profiles? http://drupal.org/node/67921

@Mike Bishop - I would say that the granularity IS desirable - and even so, I am often adding bits of jQuery to adjust labels or visibility in the DOM when a module or view lacks the config (example: OG Group search block button value http://drupal.org/node/806182 ). But I find it preferable to add a bit of JS to my page.tpl.php file and I will remember where I made the customization, rather than search through 10K+ files for the right one(s) to modify. So this is a file-based config - but sometimes I add PHP or JavaScript to blocks that may otherwise be invisible for page-specific functions - and of course block code is stored in the db.

Regarding working on busy sites, that is what distributed servers and dbs are for. If you are storing user-generated content in files, then how would that be different? If the files are being written at the time of export, you could lose or corrupt the data. Many Drupal tools that deal directly with the db can lock it temporarily and/or take the site offline briefly (eg Backup & Migrate).

So what if WhiteHouse.gov uses some custom databases or tables? - I have done the same, and it's STILL a Drupal site.

I am not arguing that Drupal is the solution for every requirement - but it works really well at least 95% of the time...

decibel.places, 25 May 2010

If the goal is to create export a clean and replicable base configuration, what about Installation Profiles? http://drupal.org/node/67921

@Mike Bishop - I would say that the granularity IS desirable - and even so, I am often adding bits of jQuery to adjust labels or visibility in the DOM when a module or view lacks the config (example: OG Group search block button value http://drupal.org/node/806182 ). But I find it preferable to add a bit of JS to my page.tpl.php file and I will remember where I made the customization, rather than search through 10K+ files for the right one(s) to modify. So this is a file-based config - but sometimes I add PHP or JavaScript to blocks that may otherwise be invisible for page-specific functions - and of course block code is stored in the db.

Regarding working on busy sites, that is what distributed servers and dbs are for. If you are storing user-generated content in files, then how would that be different? If the files are being written at the time of export, you could lose or corrupt the data. Many Drupal tools that deal directly with the db can lock it temporarily and/or take the site offline briefly (eg Backup & Migrate).

So what if WhiteHouse.gov uses some custom databases or tables? - I have done the same, and it's STILL a Drupal site.

I am not arguing that Drupal is the solution for every requirement - but it works really well at least 95% of the time...

decibel.places, 25 May 2010
Jeremy Epstein
28 May 2010

"Code driven development". The very name makes me laugh! (And cry).

I'm a developer, but I've never heard of this "code" concept before. Could you explain it? :P

CCK+Views was a really bad idea that's grown far too popular. Data belongs in databases. Models and queries belong in code. Break this simple rule, and you have deployment hell.

"Code driven development". The very name makes me laugh! (And cry).

I'm a developer, but I've never heard of this "code" concept before. Could you explain it? :P

CCK+Views was a really bad idea that's grown far too popular. Data belongs in databases. Models and queries belong in code. Break this simple rule, and you have deployment hell.

Jeremy Epstein, 28 May 2010

"Code driven development". The very name makes me laugh! (And cry).

I'm a developer, but I've never heard of this "code" concept before. Could you explain it? :P

CCK+Views was a really bad idea that's grown far too popular. Data belongs in databases. Models and queries belong in code. Break this simple rule, and you have deployment hell.

Jeremy Epstein, 28 May 2010
Anonymous
15 Oct 2010

I am really new to drupal.I wanted to develop a module so that it works with other databases other than mysql and pgsql so can any one help me what are the files i have to be written and function to be wrtitten in a module to connect drupal to different database.please give me an idea where to start exactly

I am really new to drupal.I wanted to develop a module so that it works with other databases other than mysql and pgsql so can any one help me what are the files i have to be written and function to be wrtitten in a module to connect drupal to different database.please give me an idea where to start exactly

Anonymous, 15 Oct 2010

I am really new to drupal.I wanted to develop a module so that it works with other databases other than mysql and pgsql so can any one help me what are the files i have to be written and function to be wrtitten in a module to connect drupal to different database.please give me an idea where to start exactly

Anonymous, 15 Oct 2010

Get your project started today!

Contact us