I know that we all come from different walks and strategy when it comes to testing configuration changes prior to production. I wanted to open up what I've found that works well in my environment as well as hear out your success stories or worst nightmare scenarios.
My test environment has 2 tiers.
Tier 1 - 4 different virtual workstations running in VMware Workstation. I power these 4 on and run patches, software deployments, and scripts through them to ensure that nothing is majorly broken. These 4 machines span 3 Operating systems and run multiple different business applications. If something breaks here it's too easy to revert to the preinstall snapshot and try again.
Tier 2 - A test group of 10 physical production workstations that are at the same physical location as my department. Provided nothing broke in Tier 1 this is where I go next. I have 10 end users across multiple departments with different requirements. Once I've verified that they have received their change and they've gone through their checklist of items they do on a typical basis (prioritized by business need), our changes usually go to production level.
A success story using this method is that our Office 2010 deployment was totally debugged using this method. Across the 2 operating systems we deployed to everything worked as intended and remained silent.
A disaster was a software deployment that went production and triggered an automatic reboot that wasn't caught in the virtual environment. The push to the test group happened late one business day and rather than report the reboot the end users called it a day. The reboot also effected some production servers and critical systems that were down somewhere between 10-15 minutes. It is fortunate that when they came back online nothing else was wrong.
The use of real machines even lets test bios changes and WOL then deploy. We push to those and if all goes well the ITO staff gets the deployment next (tier 2) about 20 users. - SMal.tmcc 11 years ago
Currently we do a 4 tier approach for software deployments/upgrades to workstations.
Tier 1 - test to virtual and some physical workstations in a test lab
Tier 2 - test deploy to IT which can be less than useful since most everyone in IT runs Windows 7 instead of XP
Tier 3 - production deploy to Clinics (about 700 workstations)
Tier 4 - production deployment to the rest of the network which includes in-patient and ambulatory areas of the hospital
I wish we had a Tier 2 more like yours. It sounds much more useful than our approach. I think I'll look into trying something like that.
I absolutely have a horror story. We had to do an overnight upgrade of a software application that is managed in house. The backend and frontend had to be upgraded over the same evening. The morning after the scheduled deployment I came in and almost none of the ~3000 deployments didn't work. I learned since then to minimize issues always create a label ahead of time to hit machines that don't already have the new software. That seems obvious, but I was a total newb when I started. Experiance is the best teacher, right? O_O
Recently I have a success story where I did a deployment overnight for Ultra VNC. I targeted machines that had an old version or no VNC installed at all. I scheduled the install to run and then scheduled a "force check-in" to run and scheduled one more deployment well after the check-in directing the second deployment at the same label. This seemed to yield a MUCH higher successful deployment rate.
Thanks for your post, GeekSoldier! It's good to be thinking about ways to make testing better! - awingren 11 years ago
Needless to say ,i knocked 10 users off of their computers and it just restarted on them. I work in a very large law firm, so it was a great learning experience to just hit SAVE and not SAVE and RUN NOW.
Keep that in mind any newcomers who are reading this. - areiner 11 years ago