Assume that you are a lone developer, maintaining a small web site in a shared hosting account. How many software environments do you need from development to production?
One environment
On the simplest level, you could develop directly in your ISP account, loading and saving files remotely via SFTP, WebDav, etc. — in other words, your development and production environment would be the same. For anything non-trivial, that’s a pretty hairy way to work, since you have no way to test changes before they’re rolled out into the world.
Two environments
I normally use the two-environment approach that (I suspect) is the most common one for single-developer LAMP sites: I maintain a development environment on my notebook, and periodically upload changes to the production environment at the ISP. I try to run roughly the same version of Apache, PHP, MySQL, etc. as my ISP, but otherwise, I take no special steps to replicate the production environment. On my notebook, I set up the development directory as its own virtual host (e.g. http://localhost:8001/, etc.) so that I can test changes literally as I type.
Three environments?
Even though there are no other developers working with me right now, I sometimes wonder if it would make sense to start using a third environment between development and production (a separate directory and virtual host on my notebook). A third development would allow me to run major experiments and restructuring in the development code, while still making small bug fixes, typo corrections, etc. in the stable code before uploading them to my ISP production environment.
While this sounds like a good idea initially, there is a major coordination problem involved in backporting fixes from the middle environment to the development environment, and the middle environment will still become unstable while new changes are rolled out into it, tempting me to create another environment — complexity is, sadly, highly contagious. Have any other lone developers had success (or failure) with this approach?
More
Big organizations use an enormous number of environments to build and roll-out a system:
- Each developer’s desktop, where code generally lives for a few hours.
- The development server, typically a single server running database, application server, etc. as well as version control unit/regression tests.
- One or more test environments, covering integration testing, system testing, user-acceptance testing, etc. (these can range from single servers to small clusters to near-duplicates of the full production environment).
- The staging environment, which is typically very similar or identical to the production environment.
- The production environment, where the system runs.
I’m still undecided about whether enterprises help or hurt themselves by making things so complicated — coordinating a lot of people on a big system is hard, it’s even harder to imagine an agile process functioning under so many layers of pain.
Quoderat 
For big enterprises, it’s necessary. In the banking/finance world, if any stock market trade gets delayed significantly, someone is likely to be fined a 6 figure sum. For the petroleum industry, if your software stops you from fulfilling your oil orders, you have to fulfil them by purchasing off the open market, at a big premium to your own internal cost.
It’s not that the agile approach is bad, but I don’t think it scales up well to situations where the cost of a bug out in the field is too high. It works best where the bugs aren’t likely to shut down the business, and where the users don’t mind being treated as beta testers.
Personally, I’m moving in the direction of using virtual PC sessions (e.g. VMware) to allow me to test in test-only environments without having to increase the actual number of physical boxes I have to share my office with.
Cheers, Tony.
I agree, Tony, that big enterprise is higher risk, and thus, harder: the question is at what point adding more environments raises the risk rather than lowering it. Managing and coordinating so many environments (I’ve seen as many as six or seven) tends to soak up resources that could otherwise be used for fixing bugs, refactoring code, etc.; environments tend to fall out of sync (and stay that way), limiting their usefulness; and the presence of so many environments and the extra process associated with them slows down the people who are left coding, debugging, and testing, so that they have even less time to spend improving the application. They’ll frequently be called away for hours or days to debug a problem in with their code in a middle-level environment that turns out to be a configuration problem in that environment.
I wonder if, excluding the developers’ desktops, most enterprise apps should have at most three environments — development, combined testing/staging, and production. All unit testing and most regression testing can (and should) take place in the development environment automatically, at least once a day; everything else can happen on a combined staging/testing server before the code is rolled out into production use.
Actually, many of the things that make agile work make it pretty easy to support multiple deployment environments:
– network-oriented version control (subversion / cvs) make it easy to do checkout to lots of machines
– test driven development makes it simple to run automated tests to catch regressions
– continuous integration doesn’t work without automated deployment scripts
I think the key thing is that the developers shouldn’t be intimately involved with deployment once the team scales up to a certain size. If you have more than 5 developers, your burn rate is a few thousand bucks a day – and it makes sense to hire a full time or part time release engineer to properly manage your deployments. A smart release engineer will use these agile tools to simplify deployments to multiple environments.
There are some builds that you want to do that you simply can’t run in production: load testing, aggressive security testing, and so on- stuff that will absolutely break your live site. If you run these on staging, you risk compromising the network that production lives in.
I’ve seen a pretty common pattern of development -> integration -> staging -> production, with additional special purpose builds happening on the side to support load testing, security testing, multi-platform testing, and so forth- but these don’t get in the way of an agile process, they support it. If you separate the development team from the deployment team (to an extent; you always want the developers on hand if something blows up) then it’s easy to put in this infrastructure, and set it up so the developers don’t have to mess with it.
I think even small devs should have a split of dev/(staging/test)/production. I use a vmware images hosting luntbuild to do staging: within 30 minutes of checking something in, staging is running on that code with tests. Production is just a laptop, but it is a laptop hosted at home to let me test my long-haul WS stuff from work.
I dont agree with tim, that operations/deployment should be split from development, because the developers need to think about the needs of operations too -automated deploy/config, management instrumentation. What is important is that the operations’ tasks have to be treated as important use cases for an app as the user front end.
I’m sort of in line with Tim here. I think it’s best to concentrate on automating the process of building an environment up from nothing. In an idealized setup, you’d be able to run a single script on an empty box and at the end, you’d have a new environment. I suspect that Tony’s suggestion of virtual servers could greatly aid in achieving this. In fact, I’ve seen development shops that are sort of set up this way.
One problem with this approach I think is managing the time it takes to build the environment. There are certain operations (things like installing a web server, database server, etc.) that could be skipped by the build script if they were already in place, but there are intricacies here that I haven’t really thought out in full.
I would contend that two environments is enough for single or small teams of perhaps no more than four people or so. The critical aspect of that, however, is to ensure that the test data used in the development environment is the same, or at least very similar, to the production data. I suspect that all too often the test data consists of a handful of users and associated data and then no one can understand why their SQL queries run so slow or pages are slow to render. If your developers don’t want to develop using big DBs then the only alternative would be to go to three environments, development, QA/test and production. And your QA/test data would be a mirror of production data.
I think Steeve gives the right idea