Comments on: How many environments?

By: Renjith

Renjith — Sun, 03 Sep 2006 09:52:46 +0000

I think Steeve gives the right idea

By: Douglas

Douglas — Wed, 03 May 2006 18:37:20 +0000

I would contend that two environments is enough for single or small teams of perhaps no more than four people or so. The critical aspect of that, however, is to ensure that the test data used in the development environment is the same, or at least very similar, to the production data. I suspect that all too often the test data consists of a handful of users and associated data and then no one can understand why their SQL queries run so slow or pages are slow to render. If your developers don’t want to develop using big DBs then the only alternative would be to go to three environments, development, QA/test and production. And your QA/test data would be a mirror of production data.

By: stand

stand — Tue, 02 May 2006 16:57:29 +0000

I’m sort of in line with Tim here. I think it’s best to concentrate on automating the process of building an environment up from nothing. In an idealized setup, you’d be able to run a single script on an empty box and at the end, you’d have a new environment. I suspect that Tony’s suggestion of virtual servers could greatly aid in achieving this. In fact, I’ve seen development shops that are sort of set up this way.

One problem with this approach I think is managing the time it takes to build the environment. There are certain operations (things like installing a web server, database server, etc.) that could be skipped by the build script if they were already in place, but there are intricacies here that I haven’t really thought out in full.

By: Steve Loughran

Steve Loughran — Tue, 02 May 2006 15:03:37 +0000

I think even small devs should have a split of dev/(staging/test)/production. I use a vmware images hosting luntbuild to do staging: within 30 minutes of checking something in, staging is running on that code with tests. Production is just a laptop, but it is a laptop hosted at home to let me test my long-haul WS stuff from work.

I dont agree with tim, that operations/deployment should be split from development, because the developers need to think about the needs of operations too -automated deploy/config, management instrumentation. What is important is that the operations’ tasks have to be treated as important use cases for an app as the user front end.

By: Tim Howland

Tim Howland — Tue, 02 May 2006 14:37:02 +0000

Actually, many of the things that make agile work make it pretty easy to support multiple deployment environments:

- network-oriented version control (subversion / cvs) make it easy to do checkout to lots of machines
- test driven development makes it simple to run automated tests to catch regressions
- continuous integration doesn’t work without automated deployment scripts

I think the key thing is that the developers shouldn’t be intimately involved with deployment once the team scales up to a certain size. If you have more than 5 developers, your burn rate is a few thousand bucks a day – and it makes sense to hire a full time or part time release engineer to properly manage your deployments. A smart release engineer will use these agile tools to simplify deployments to multiple environments.

There are some builds that you want to do that you simply can’t run in production: load testing, aggressive security testing, and so on- stuff that will absolutely break your live site. If you run these on staging, you risk compromising the network that production lives in.

I’ve seen a pretty common pattern of development -> integration -> staging -> production, with additional special purpose builds happening on the side to support load testing, security testing, multi-platform testing, and so forth- but these don’t get in the way of an agile process, they support it. If you separate the development team from the deployment team (to an extent; you always want the developers on hand if something blows up) then it’s easy to put in this infrastructure, and set it up so the developers don’t have to mess with it.

By: david

david — Tue, 02 May 2006 14:11:04 +0000

I agree, Tony, that big enterprise is higher risk, and thus, harder: the question is at what point adding more environments raises the risk rather than lowering it. Managing and coordinating so many environments (I’ve seen as many as six or seven) tends to soak up resources that could otherwise be used for fixing bugs, refactoring code, etc.; environments tend to fall out of sync (and stay that way), limiting their usefulness; and the presence of so many environments and the extra process associated with them slows down the people who are left coding, debugging, and testing, so that they have even less time to spend improving the application. They’ll frequently be called away for hours or days to debug a problem in with their code in a middle-level environment that turns out to be a configuration problem in that environment.

I wonder if, excluding the developers’ desktops, most enterprise apps should have at most three environments — development, combined testing/staging, and production. All unit testing and most regression testing can (and should) take place in the development environment automatically, at least once a day; everything else can happen on a combined staging/testing server before the code is rolled out into production use.

By: Anthony B. Coates

Anthony B. Coates — Tue, 02 May 2006 13:51:15 +0000

For big enterprises, it’s necessary. In the banking/finance world, if any stock market trade gets delayed significantly, someone is likely to be fined a 6 figure sum. For the petroleum industry, if your software stops you from fulfilling your oil orders, you have to fulfil them by purchasing off the open market, at a big premium to your own internal cost.

It’s not that the agile approach is bad, but I don’t think it scales up well to situations where the cost of a bug out in the field is too high. It works best where the bugs aren’t likely to shut down the business, and where the users don’t mind being treated as beta testers.

Personally, I’m moving in the direction of using virtual PC sessions (e.g. VMware) to allow me to test in test-only environments without having to increase the actual number of physical boxes I have to share my office with.

Cheers, Tony.