Managing software updates for large fleets of PCs and servers is never easy, especially when you’ve written your own code or if you’re a developer with code running out there in the wild. Every Patch Tuesday, there’s a question that needs to be answered: will this set of updates break my code? 

You can get some answers by running test hardware on the various Insider rings, but they don’t offer all updates in advance. While that’s understandable when it comes to significant security updates, it means that you’re unable to be sure that your code will run. The alternative is to delay updates until applications are tested, but in many cases, you don’t have control over what users are doing — and any delay to updates increases the risk of malware or other attacks. 

Introducing Test Base 

Microsoft has been offering software vendors an alternate application testing platform, Test Base for Microsoft 365, for some time now, using secure virtual machines running in Azure that have been loaded with pre-release updates. You can be sure that the systems have been set up and validated; all you need to do is upload, install, and test your applications. Test Base provides deep analytics on your code, looking at both performance and reliability. 

Test Base offers the option of VMs that run the current release version of Windows (along with security updates) or for longer-term tests, from the Windows Insider Beta channel. Results can be compared between pre- and current releases, allowing you to see any differences using the Test Base results viewer in the Azure Portal. Tests include CPU regression analysis, checking for changes in CPU utilisation between Windows versions. 

The service has begun rolling out to IT professionals and service providers, in a private preview, bringing it to a larger audience and providing support for bespoke internal development and for validating fleet software loadouts. The underlying VMs are maintained by Microsoft, and Microsoft will help with investigating issues as well as providing debugging support. 

Running tests in Test Base 

You may have heard of Test Base under its original name, the Update Staging Lab. The initial release of the service offered only pre-defined tests driven by standard scripts. That was seen as relatively limited, and Microsoft rolled out an update in late 2020 that added support for custom tests, using your choice of test frameworks. 

This works by uploading a zip file with all the tools needed to run tests as a single zip file, for use as functional tests. There is a size limit of 2GB, which should be fine for most applications. You’ll need PowerShell scripts to handle running your code, installing, launching, and uninstalling, the app, along with up to eight test scripts. Windows updates can be applied at any point in a test, and you can control VM reboots. This way you can get a baseline result before you run an update, comparing before and after results. Microsoft will verify that packages install before letting you run tests, with results notified in the Azure Portal. 

SEE: The future of work: Tools and strategies for the digital workplace (free PDF) (TechRepublic)

While the initial release of Test Base focused on desktop applications and desktop Windows, as these are the biggest issue for IT departments, recent updates added support For Windows Server, both 2016 and 2019. Working with server OSs is very like working with desktop releases; all you need to do is choose the appropriate OS version and run the same tests as you would for a desktop. Server tests currently don’t support preview releases of the OS, so you’re restricted to just ensuring apps work on the current baseline Windows Server releases. That’s not as much of a problem for server applications, as most run on long-term support releases, and the next major Windows Server release is still some way off. If an application runs on both client and server, you can choose to test the same package on both. 

You’re not limited to testing against Windows. Test Base also allows you to test against Office releases, so you can ensure that code doesn’t interfere with users’ productivity tools or that integrations still work after Office updates. 

Analysing results and planning deployments 

Results are available in an insights section of the Test Base portal. You can drill down into specific tests and can even see all the processes that have run during a test. Not every failure is in your code — it can be a failure in a related application or in Windows. You can even compare a baseline known-good release with different Windows updates, helping you to track down performance issues as well as crashes. 

The plan is to make Test Base part of Microsoft Endpoint Manager, including it in Microsoft’s suite of security tools. For now, however, the process is limited to what Microsoft calls ‘critical third-party software vendors’. These include antivirus and other security tools, as well as applications with large audiences where issues could have serious impact on both the vendor and any sites that use their code. While this preview is closed, there is an interesting option for IT professionals, who have the ability to nominate their software vendors for the preview. This way you can help get your critical applications tested. 

There’s the option of integrating Test Base with Azure DevOps, so you can include tests as part of a build pipeline, automating operations and ensuring that all code releases go through a test. Results can be exported to external tools, like Power BI, so you can build them into a deployment dashboard, highlighting issues and tracking fixes. Microsoft is adding API-based access to the service to make integrating with CI/CD pipelines and DevOps tooling a lot easier, giving you more automation options. 

Once the service rolls out to a wider audience as part of Microsoft 365, you should be able to use it alongside other validation and test tools as part of an application deployment workflow. Having a way of automatically tracking issues before you roll out either application or Windows updates will keep your users happy and you less stressed. 

Microsoft Weekly Newsletter

Be your company’s Microsoft insider by reading these Windows and Office tips, tricks, and cheat sheets. Delivered Mondays and Wednesdays

Sign up today

Also See