It was about 17 years ago in the old Microsoft Building C at an evening test talk delivered by an industry luminary that I personally became excited to be a tester. In this talk the presenter shared a story about how he and his test team were the gate keepers, the protectors, the champions of the customer. It was their job to root out all software bugs before the product shipped and, if need be, prevent the bad product from being released to the unsuspecting and hapless end users of the world.
This test manager bragged about how every month the development team would come up with a release candidate and every month he and his test team would find one or more ship-stopping bugs. That would cause the release window to reset and the process would start all over again. This was of course pre-Internet and from the presenters own past. We're talking about a desktop application that shipped on floppy disks and ran on DOS. Well the presenter went on to brag about how he and his small test team were able to keep this dance going for several months, causing the product to slip by about a year from its original release date. "Wow," I thought, "the power to prevent a product from shipping. Being a tester is brutally cool."
We were listening to this presenter because he was well respected, a published author and the product he was talking about had been the top software product in its category for many years. You can no longer purchase the product he mentioned, in fact the release he was talking about became the next to last version of the product ever shipped. What happened was that Windows came along and this company was late to move from text based application to a GUI based interface.
For years after that talk I believed my job as a tester was to be the champion of the customer, that I needed to be the gate keeper and block any bad code from seeing the light of day. Well that was then, and this is now, and the big difference is I now work on Bing and Bing is a massive cloud-based service. I have changed my opinion of the role of testing: I may still be a champion of quality for the customer, but I do not see myself as a the sole guardian of the gate to production.
Let me share a few Bing stats with you and then explain how the role of testing changes with the cloud.
Bing is a very large online service built on a highly automated platform we call AutoPilot (just Bing it here http://binged.it/kHlFFl). As of summer 2010 the Yahoo US web search traffic has been served by Bing with other countries and languages on-boarded since then. Comscore reports for April (http://bit.ly/jcq2t3) place the combined traffic of Yahoo and Bing at about 30% of all US web search traffic with millions of unique users and billions of queries per month. In writing this article I searched records on the internet and found blog posts that estimate Bing runs on more than 100,000 servers. I cannot confirm or deny this number but anyone familiar with how web search operates would easily assume a very large number like this.
What this nets out to is that Bing is big and highly automated. Bing is its own cloud and one of the largest cloud services in the world. So, what's it like to be a tester working on one of the largest cloud services in the world? It's very different than any other role I have ever had at Microsoft. Let me share with you some of the differences I find exciting about testing in the clouds.
At massive scale it is hard to break everything. Note that it isn't impossible to break a cloud, just harder. Bing had a public outage about a year and a half ago and recently Amazon.com had its own major outage. Even so, when you consider the number of changes we make to Bing every single day you realize we have a very robust and hardened cloud infrastructure.
At massive scale you can ship more frequently with less Big Up Front Testing (BUFT- http://bit.ly/lO5Cmt). Don't construe this comment to mean you test less, in fact with Bing following an agile development process with some teams on 1-2 week sprints, test automation as a frontline to detect regressions is critical. The approach we use in Bing is to invest heavily in upstream test automation that flows through the system from pre-check-in tests all the way to production as live site monitors. Yes, test automation can also be used as live site monitors.
Where we are lean is not necessarily in test automation but in testing as the gateway to production. The reality is that some bugs can't be found in test environments and so we emphasize getting new code into production quickly and safely so that we can find and fix those bugs found only in production as quickly and efficiently as possible.
The reality is that in a robust cloud environment at massive scale, whether it's the Bing cloud or a public cloud, proper architecture can reduce risk. By reducing the risk that a single change will have catastrophic negative customer impact, the entire organization, not just test, can focus on driving rapid innovation while driving quality up stream to design and code and downstream to the live service health through measurement and analysis.
As a tester in Bing we work to build infrastructure for test automation and quality analysis that deeply integrate into the daily workflow of developing and releasing Bing. We do write many automated functional test scripts but we rely on the deveolpers to write unit tests and many of the automated functional tests. Collectively we work across Dev, Test and Program Management to find a way to safely release new quality features to end users as absolutely quickly as possible.
We can do this with fewer test passes because we have massive scale and a cloud infrastructure that allows us to quickly roll out and roll back code as well as expose that code to slices of production usage before releasing it to all users. In this sense all the engineers working on a feature collaborate to develop, release, and delight our customers as a team.
For me, going from the gatekeeper mindset to the role of iterative partner with my Dev counterparts has been quite a journey. Add to that the rapid release cycles, the world wide impact Bing is having, and the scale of Bing and it was inevitable that I would end up working in the Bing team. I loved shipping Office 2010 but I must admit that I am having fun and learning so much by working in Bing that I'm confident it was the best choice for me. I love cloud services and I love evolving the practices and techniques of software testing. At Microsoft I find Bing to be just about the most vibrant and dynamic place I could work.
I hope to find time to write another post in the near future where I can go deeper on some particulars of Bing testing. If there is something you'd like details on please leave a comment or a questions so I can get back to you or just ping me on twitter @RKJohnston.
KJ
Ken Johnston is a Principal Test Manager for Bing's Commerce Team. He is also the author of How we Test Software at Microsoft. Since joining Microsoft in 1998 Johnston has filled many other roles, including test lead on Site Server and MCIS and test manager on Hosted Exchange, Knowledge Worker Services, Net Docs, and the Microsoft Billing and Subscription Platform service. For two and a half years (2004-2006) he served as the Microsoft Director of Test Excellence. In 2003 he earned his MBA from the University of Washington.
Source: http://www.bing.com/community/Site_Blogs/b/bingjobs/archive/2011/05/26/testing-in-the-bing-cloud.aspx
L1 IDENTITY SOLUTIONS KINGSTON TECHNOLOGY COMPANY KEY JDS UNIPHASE JDA SOFTWARE GROUP
Pew Internet, a think tank that regularly publishes research reports about technology, has released a new study today showing the steady growth in using VoIP and phone services online. According to the organization's report, a quarter of American adult internet users (24 percent) have placed phone calls online. That amounts to 19 percent of all American adults. On any given day 5 percent of internet users are going online to place phone calls, says Pew. And Pew says that usage has grown significantly from a few years ago. For example, Pew found in February 2007 that 8 percent of internet users (6 percent of all adults) had placed calls online and 2% of internet users were making calls on any given day. At various points during the 2000s, Pew held similar surveys and found that at most about a tenth of internet users had ever used the internet to place calls and the daily figure never rose above 1 percent of internet users.
Thanks to 
