Microsoft crowdsourced QA, and look what we got

Will the next update to Windows 10, which should roll out in the next few weeks, be bug-free? That’s never a reasonable expectation, for Microsoft or any other software maker.

Will it absolutely, positively not do something so horrendous to the world’s PCs that corporate productivity will be hampered? A much lower standard, for sure — but I wouldn’t bet on it.

Windows updates have had a checkered history in recent years. The worst was probably the October 2018 Update for Windows 10, which deleted people’s files permanently and without warning. But that is hardly the only memorably bad update.

What went wrong with Microsoft that so many people now look to updates with trepidation rather than excitement? Windows Insiders, that’s what. I’m not talking about the millions of people who have signed up to give Microsoft free reports on Windows’ pre-release problems.

I’m talking about the program itself, and the elimination of the real Microsoft insiders (as in, employees) who used to do what has now been crowdsourced.

Windows Insiders are people who sign up to test out pre-release versions of Windows and report bugs using what Microsoft calls the Feedback Hub.

What the Insiders get is the “privilege” of using pre-release versions of the operating system, which can be flaky, tend to be riddled with bugs, often lack polish and frequently include features that don’t work properly and may be killed before Windows is released to the public.

For many Insiders, it’s apparently enough that they get bragging rights about having seen what may be coming next in Windows.

It’s like Mark Twain said: “Work consists of whatever a body is obliged to do. Play consists of whatever a body is not obliged to do.” People aren’t obliged to become Insiders. But it feels like a privilege to be one — the name itself implies exclusivity, even though anyone can sign up.

Microsoft says Insiders have helped make sure that Windows runs better and more reliably than ever by finding bugs early — which, when you think about it, is an odd statement. It almost sounds as if Microsoft is saying that before Windows Insiders was established, it just shipped out updates without any testing at all: Good luck, world!

But of course, that’s not what it did. Before 2014, Microsoft had a team devoted to testing Windows in real-world situations and analysing the results.

That year, it started testing Windows on virtual machines, says Jerry Berg, a Microsoft senior software development engineer for testing who was on the team and worked at the company for 15 years. The VMs didn’t work as well, it found — but it didn’t bring back the team.

About that team: Berg says it put Windows builds on many different types of hardware and system configurations in labs, ran automated tests, and found countless bugs in a structured way. The team did more than that, though.

It also analysed the bugs and helped engineers understand what needed to be done in order to fix them. Virtual machines, Berg says, can’t uncover those kinds of bugs because they’re not running on many different kinds of hardware.

Enter the Windows Insider program. It was introduced late in 2014, and by a year later had 7 million people signed up. (The most recent figures, from March 2017, show 10 million people in the Windows Insider program.)

Microsoft relies on Insiders to report bugs, but Berg says that many Insiders don’t bother to do that, and that those who do it don’t report them in the timely, structured, useful way that the laid-off team had done. They don’t analyse the bugs, either.

It’s true that Microsoft also gathers telemetry data from Windows Insiders who are using pre-release versions of Windows and uses that both to squash bugs and to gain more insight into how people use Windows. But Berg says that, by itself, telemetry just isn’t good enough.

Here’s how he puts what’s wrong with Windows testing today: “Fundamentally, Microsoft has replaced flesh-and-blood people who create automatic test sequences … [with] ourselves, the consumers.”

Are Insiders by themselves responsible for the declining quality of Windows builds? Certainly not. They’re quite useful as an adjunct to professional testers. But Microsoft now uses them instead of professional testers. After all, they’re free. But as the saying goes, you get what you pay for.