Making Sense of Office 365 Pro Plus and Office 2019

Office 2019 recently released and accordingly, you may be looking into the upgrade process and any challenges it may bring. We recently found ourselves in this situation and the upgrade is, unfortunately, anything but simple. My usual disclaimer applies: some of the information below is a result solely of my own experiences in trying to upgrade our organization. If there are any errors, please comment and let me know.

MSI vs. Click-To-Run

Prior versions of the Office Suite used the well-known MSI format. MSI allows the use of Group Policy for network installs, and most large organizations have plenty of experience with the format. So of course Microsoft had to fix something that wasn’t broken.

Office Pro Plus 365 and Office 2019 (I’ll cover the differences in a moment) use Click-To-Run technology (CTR) which Microsoft touts as a “modern” deployment method.

Here’s a quick comparison of the two methods:

MSI

• Deployed via Group Policy
• Installs from local media
• Patches directly from Windows Update or through WSUS

CTR

• Deployed via the Office Deployment Tool
• Installs from local media or directly from the Internet
• Patches directly from the Internet or from a network share (not through WSUS)

There are plusses and minuses to both methods, but CTR is seemingly the future so we have no choice but to deal with it.

Product Channels

For all practical purposes, there are five variants of the current Office Suite, each of which is considered a “channel.”

• Office 365 Pro Plus Semi-Annual
• Office 365 Pro Plus Semi-Annual (Targeted)
• Office 365 Pro Plus Monthly
• Office 365 Pro Plus Monthly (Targeted)
• Office 2019

The 365 SKUs are retail, and their license is tied to an appropriate Office 365 subscription that includes the Office Suite. You can choose to get either monthly or semi-annual feature updates, while security patches are applied regularly regardless. The “targeted” channels are essentially “sneak-peeks” of upcoming changes.

Where it gets confusing is that Office 2019 is essentially nothing more than a point-in-time snapshot of Office 365 Pro Plus, that will not get any feature updates; it will receive security updates only. This version is also the only one that can be volume-licensed using KMS.

As of November 1st, the most recent O365 Pro Plus Semi-Annual channel is version 1803, Office 2019 is considered version 1808, while the O365 Pro Plus Monthly channel is 1809, and the monthly targeted channel is 1810. Each has different feature sets. See Microsoft’s update history for O365 Pro Plus for more info.

Licensing

As mentioned, any of the Pro Plus channels imply retail licensing, and they must be activated against an Office 365 account that is entitled to the appropriate applications. This is fairly straightforward in an office setting where people don’t switch computers too often, but it poses a challenge in a school or other shared computer setting. In this case, you want to use shared computer licensing, and ideally license token roaming as well.

Office 2019 implies volume licensing, either using MAK or KMS. In this sense it’s identical to prior volume-licensed versions of the Office Suite. Note your KMS server will need an update to activate Office 2019.

Installation

Whether you want to install Pro Plus or 2019, and regardless of the desired channel, you will need to use the Office Deployment Tool (ODT). Essentially you will need to create an XML file that defines various options, including the desired channel, the installation source, and the update method. Microsoft’s documentation is actually pretty thorough, so you shouldn’t run into too many problems. There are a few caveats though:

• The setup.exe that you run to install the suite is actually just a stub installer. In some cases (and I haven’t yet figured out the specifics), setup.exe will terminate successfully before the apps are installed. There is a further installer that will continue to run in the background.
• The XML file gives you an option to remove any MSI-installed versions found. In my testing, this setting is ignored and the installer will always remove any older versions of Office. Even better, it will remove applications like Visio or Project that aren’t part of the suite and are thus not going to get reinstalled.
  
• Normally, the suite is going to be configured to take updates directly from Microsoft. You can no longer use WSUS to patch Office. You can technically control patches by disabling updates from the Internet and then manually downloading them to a network share, but that’s likely impractical. Realistically administrators have no choice but to accept giving up control of Office patches if they want to stay reasonably current. As an example of how little control we now have over the patch process, I left Word open with this document being edited on Friday, running version 1809. I came in Monday morning to the document still open, with Word running version 1810.

Co-Existence With Visio or Project

While neither Visio nor Project are included in the O365 Pro Plus suite, or in Office 2019, they are considered part of Office and the ODT is used to install them as well. This has implications for installing either application alongside the Office Suite. Microsoft has a KB article about that here, but it’s missing one critical bit of info.

According to Microsoft, it is possible to install Office 365 Pro Plus with a retail (subscription-based) license alongside Project 2019 or Visio 2019 using a volume license. However in the “additional information” section of the linked article, Microsoft adds a major restriction: all installed Office applications must be using the same update channel.

The reality is that restriction precludes using O365 Pro Plus along with volume-licensed copies of Project or Visio, because the retail versions use one set of update channels, and the volume license versions use a different one. Perhaps I’m missing something here but I could not get retail O365 Pro Plus to work properly alongside volume licensed (either MAK or KMS) versions of Project or Visio.

If a volume-licensed edition of Project 2019 or Visio 2019 is already installed, the O365 Pro Plus install may succeed (it did for me), however none of the applications will activate. The volume-license apps generate an error that the license has changed, and the retail apps simply refuse to license with a generic error.

If O365 Pro Plus is installed first, the volume license versions will refuse to install due to the unavoidable channel mismatch.

If you install the retail versions of Project or Visio alongside O365 Pro Plus, you cannot activate them using KMS or MAK because only the volume license version supports volume activation.

So the long and short of it is that if you need either Project or Visio, you either have to use Office 2019 and the volume license edition of Project or Visio, or you need an O365 subscription that includes rights to all applicable products.

In Conclusion

Anytime a long-standing technology changes, there are growing pains. We’re still too new to using CTR to see any clear advantages, but the move itself all that painful. However, the changes to how applications are updated, licensing, and the restrictions on product coexistence all have significant implications to cost, both in dollars and labor.

Most corporations will probably be content to stay with the volume-license editions of Office, but in education, Microsoft continues to add – and quickly publicize – new features that are only available in the monthly retail channel. So while we have the option of staying volume-licensed, the reality is we would be doing our students a disservice by not offering the latest features. And that means dealing with all the accompanying quirks.

Advertisements

802.1x Certificate Based Authentication against NPS on Windows Server 2016

When we were preparing to upgrade our domain controllers from 2008 R2 to 2016, we of course began inventorying all functions that were going to be migrated. One of these was Network Policy Server (NPS). NPS is one of the easiest services to migrate to a new system, since it’s basically a simple backup and restore, and we weren’t expecting any difficulties. Of course had things gone smoothly, this post wouldn’t exist.

A little background:

We use NPS for multiple functions, including Cisco AAA and wireless device authentication. Our Meraki wireless infrastructure has multiple SSIDs, some of which are for domain-joined devices, and others for non-domain devices. The domain-joined devices use 802.1x certificate-based authentication and the non-domain joined devices use various other methods.

NPS was migrated from 2008 R2 to 2016 and everything other than the 802.1x certificate authentication worked. The 2016 servers were correctly setup as RAS servers and had certificates that were valid for client & server authentication, as needed. I spent quite a long time troubleshooting this with no success, and eventually had to back-burner it. The 2008 R2 servers were due to be retired, so as a temporary measure I migrated NPS to a couple 2012 R2 servers, all of which worked flawlessly after about two minutes worth of effort.

This was pretty early in 2016’s lifecycle, so since it worked in 2008 R2 and 2012 R2, we chalked it up to a bug in the OS and intended to pursue it with Microsoft at a later date.

Before I had a chance to call Microsoft, another project had us looking at our PKI configuration. We were having issues using LDAPS with this particular system (despite using it elsewhere), and after extensive troubleshooting the vendor told us they thought the hash algorithm used by our PKI was unsupported.

Our PKI was originally setup on a 2003 server back when that was the current server OS. It had been migrated to a newer server since then but the hash algorithm had never been changed; it was still using md5RSA. Now upgrading the hash algorithm isn’t something to be taken lightly, as it can break a lot of things, but in our case since our PKI is pretty simple I decided (after backing everything up) that the upgrade risk was minimal. I changed the hash algorithm to SHA256, reissued RAS certificates to the NPS servers and just like that things started working.

I can’t find any changelog indicating why NPS would not work with an older certificate hash on 2016 but empirical evidence shows the hash was the problem.

There are plenty of resources about upgrading your root CA’s hash algorithm, but this is a good start from Jason Bender at Microsoft.

Managing Mail-Enabled Security Groups in O365

Office 365 certainly has its plusses and minuses, and one of the areas that clearly falls in the latter category is how O365 handles mail-enabled security groups.

Consider the following scenario:

• You have an on-premise Active Directory sync’d to O365/Azure via ADSync.
• You have an on-premise security group that you mail-enable, creating an e-mail address @yourdomain.com.
• This mail-enabled group syncs to O365, and then exists in Exchange Online.
• You then mail-disable the group on-premise and wait for the change to sync to Exchange Online.

Logic would dictate that once the mail-enabled attribute was removed from the group, it would disappear from Exchange Online. Of course it’s not that simple.

What you will instead find is that the security group still shows in Exchange Online, however it now has a proxy address @yourdomain.onmicrosoft.com. It will also continue to be visible in both the online and offline address books. If you are trying to move groups to the cloud and/or cleanup your distribution lists, this can cause confusion.

The fix fortunately is easy, but it’s an extra step that shouldn’t be necessary IMO. Credit to Tim McMichael’s TechNet post for the details.

The process to properly remove a mail-enabled security group from Exchange Online is as follows:

• On-Premise, remove the mail-enabled attribute from the group using the Disable-DistributionGroup PowerShell command.
• On Exchange Online, find the group via Get-MsolGroup –SearchString “Group Name”.
• Once you’ve confirmed you have the right group, remove it via Get-MsolGroup –SearchString “Group Name” | Remove-MsolGroup

This will finally remove it from Exchange Online and your address books.

EDIT: There is a major caveat here to be aware of. If you are using mail-enabled security groups to grant permissions to shared calendars, you CANNOT remove the mail attributes or you will break access permissions. In that situation I would advise simply hiding the list from the address book.

Monitoring Microsoft NLB with PowerShell

Recently the need arose to monitor the status of a Microsoft NLB cluster. There are a number of ways of approaching this, but PowerShell seemed like the cleanest.

I found this script on TechNet but found that it did not work in our (2008 R2) environment. During troubleshooting, what I found was that instead of returning a single status result, the WMI query was returning an array of results for each server. It appears that the statuscode is only relevant for the specific hostpriority applicable to each server.

For instance, this is the server with hostpriority 1 as shown in the NLB Manager:

And this is the one with hostpriority 2:

As you can see, a proper statuscode is only returned for the current hostpriority for each server.

With that in mind I made a change to the script to enumerate through the array of results for each server and check the status accordingly. This has only been tested on ADFS 2.0 on 2008 R2 servers, and it would need to be revised if you have more than two servers in a NLB cluster.

#Define Nodes
$node1 = "SERVER01"
$node2 = "SERVER02"

#get NLB status on NLB Nodes
$Node1status = Get-WmiObject -Class MicrosoftNLB_Node -computername $node1 -namespace root\MicrosoftNLB | Select-Object __Server, statuscode, status, hostpriority
$Node2status = Get-WmiObject -Class MicrosoftNLB_Node -computername $node2 -namespace root\MicrosoftNLB | Select-Object __Server, statuscode, status, hostpriority

$node1converged = $false

Foreach ($status in $Node1status)
{
if(($status.hostpriority -eq "1" -and $status.statuscode -eq "1008") -or ($status.hostpriority -eq "2" -and $status.statuscode -eq "1007"))
{
$node1converged = $true
write-host "NLB Status of $node1 is: Converged"
}
}

Foreach ($status in $Node2status)
{
if(($status.hostpriority -eq "1" -and $status.statuscode -eq "1008") -or ($status.hostpriority -eq "2" -and $status.statuscode -eq "1007"))
{
$node2converged = $true
write-host "NLB Status of $node2 is: Converged"
}
}

Sorry about the indents; they show up in the WordPress editor but not on the actual page.

Let me know if you find this useful, or if you are able to use it on a platform other than 2008 R2.

IIS Application Pool Will Not Start – App Log Error 2307

After applying patches to a 2012 R2 server, I found one of our IIS websites was not working. While IIS was itself running, the DefaultAppPool was stopped. Attempts to start the app pool were unsuccessful; the pool would stop almost immediately, and error 2307 from “IIS-W3SVC-WP” was logged in the Application Log.

The error message itself contained the following information:

The worker process for application pool 'DefaultAppPool' encountered an error 'Cannot read configuration file' trying to read configuration data from file '\\?\<EMPTY>', line number '0'. The data field contains the error code.

Not the most helpful error message since it doesn’t specific the file that cannot be read. Some research online turned up references to specific IIS config files (web.config, etc.) having incorrect user permissions, but I looked into that and everything seemed correct. I used Sysinternals Process Monitor to see if I could determine the file in question, with no luck.

Eventually I stumbled on a post (which I unfortunately lost track of), which referenced a Microsoft KB article. The article pertains to Windows 10 and Server 2016, the 1709 Fall Creators Update, and a totally different event ID from a totally different service. That said, the resolution shown fixed the error I was having.

The actual fix is quick:

net stop WAS /y
rmdir /s /q C:\inetpub\temp\appPools
net start W3SVC

Evidently the patches I applied caused the same issue (broken symbolic links) as the Win 10 1709 upgrade.

Inaccessible Network Shares on Windows 10 1709

After upgrading a number of systems to the Fall Creators edition of Windows 10, version 1709, we began receiving reports of certain users being unable to access a specific set of shared folders. These were folders hosted on a 2012R2 file server, and the error received indicated the share was missing, rather than there being a permissions issue.

Searching for this type of error on the Fall Creators edition turns up two common results:

• SMBv1 has been disabled and/or removed from Win 10 1709
• Guest access in SMB2 disabled by default in Win 10 1709

We were pretty certain SMBv1 was not in use but ran Wireshark to sniff the SMB traffic and verify SMB2 was being used.

Likewise, we knew guest account access was already disabled, and the Microsoft KB article references log entries that would be present if this were causing a problem. There were no relevant log entries, so we crossed this off the list of possible causes.

Interestingly, not all shares were an issue; even other ones on the same server worked fine. That said, this share had been used for years without problems, with multiple versions of Windows, including three previous Win 10 builds.

I started reviewing the permissions on the share, just to see what might be different from other shares that were working, and quickly found a discrepancy.

The share in question was the parent folder for many sub-folders. The sub-folders have different permissions for different groups. We provide a shortcut directly to the parent folder and have access-based enumeration enabled so that users only see the folders they have access to. Nothing all that unusual here.

The parent folder had “Authenticated Users” being granted “Traverse Folder / Execute File” on the folder itself. (“This Folder Only”)

Certain user groups however had “Read” permissions on the parent folder. Some quick testing showed that the users with “Read” permissions could access the share, while those with “Traverse Folder” could not.

The entire purpose of “Traverse Folder” is to “[a]llow..moving through a restricted folder to reach files and folders beneath the restricted folder in the folder hierarchy.” Having this permission should be perfectly sufficient for what we were doing, and it has been for years, up until the Fall Creators version of Win 10.

Even stranger though is that “bypass traverse checking,” which is granted to everyone by default (and enabled in our domain) is supposed to entirely negate the need for the “Traverse Folder” right in the first place.

Something obviously changed in Fall Creators that has broken traverse checking; my guess is with all the file system work that Microsoft acknowledges they put into Fall Creators, they either inadvertently broke this or deliberately changed the behavior without communicating it to us. Although if this were a deliberate change it completely eliminates the value of traverse bypass in the first place.

The fix here was easy; I gave “Read” permissions to “Authenticated Users” for the parent folder only. This had no effect on any of the sub folders or users access to them, but it resolved the path not found error.

Oh well…back to beta testing the latest GA build of Win 10…

WSUS Error 0x80092026 – File Failed Post-Processing

As is typically the case with my sporadic blog posts, we recently ran into a situation where, as a colleague likes to put it, we ended up far down the rabbit hole. There are a few lessons to be learned here so I’m going to explain a bit about the troubleshooting process, not just how we eventually fixed the issue.

The initial complaint as reported was that one set of laptops was not working properly, but only during the first class period, and only in one classroom. Now, it’s not uncommon for mobile devices to take a little while to wake up, but in this case the machines were turning on an hour before class started, and again, we were told the problem was very limited in scope.

We were able to duplicate the problem, but again, only during the one period, and even stranger, only when the class was in the room. We use Meraki access points, and based on this particular AP not seeing its mesh neighbors all that well, we initially thought we were dealing with some localized RF interference. After extensive troubleshooting of the RF environment we could not find any explanation for the apparent interference. The AP was seemingly overloaded during this period of time, but there was no obvious cause.

One thing that did stand out was certain wireless clients were transferring an abnormal amount of data. We noted this with the intent to revisit it later, but because we had no reports of issues from other locations, we concluded this was unrelated to our current situation.

That was a mistake. We assumed that because there were no other problems reported, that there were no other problems. As it turns out, nobody else was using laptops at this time of day. The lesson here is to check your assumptions.

Once we realized that we could not draw any conclusions from the location of the problem, we expanded our investigation and revisited the previously mentioned excessively large amounts of data being downloaded. The Meraki APs do an excellent job of reporting certain info, and it quickly became obvious the data was coming from our WSUS patch server.

We had recently rebuilt WSUS, and with 2012 R2 there are (were) some important out-of-band patches that can result in failed downloads. We reviewed these and confirmed that on a fully-patched 2012 R2 install, these needed updates are now included in the normal security and critical updates that Microsoft pushes out. (Evidently you still must add the .esd mime type to IIS, but that’s unrelated to this post.)

We next turned to one of the laptops that was experiencing a problem. On Windows 10, getting a human-readable update log requires running the PowerShell command Get-WindowsUpdateLog; this creates the old-style WindowsUpdate.log you may be used to from earlier Windows versions.

Searching for errors, we came across the following:

Misc Validating signature for <file guid> with dwProvFlags 0x00000080:
Misc Error: 0x80092026 when verifying trust for <file guid>
DownloadManager File failed postprocessing, error = 80092026
DownloadManager Failed file: URL = <WSUSServer><File>, Local path = <local path>
DownloadManager Error 0x80092026 occurred while downloading update; notifying dependent calls. 

I should add that many of the affected laptops were not consistently reporting to the new WSUS server, we were not certain if this was related but it certainly seemed suspicious.

A number of online searches confirmed that this error had something to do with a certificate check failing for a particular update. Since our content filter does SSL decryption, we double-checked that we were not decrypting any traffic to Microsoft’s CRL or patch servers: we were not. Of course, if something were being blocked by the content filter we would expect every client to be impacted, but most of our PCs were working fine.

We found different files failing post-processing on different computers, so this did not seem to be related to a specific update. Was what more interesting is that despite only one downloaded update failing, WSUS (or BITS) seemingly discarded the entire batch of downloads, so the next time Windows Update started to run, it re-downloaded everything.

After further troubleshooting, I stumbled on this thread on Microsoft TechNet. User “lforbes” posted the following:

I found the offending key for those that are interested.
HKEY_USERS\S-1-5-18\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing
State SHOULD be 146432 or 0x00023c00
If it switches to 408576 or 0x00063c00 [t]hen it will cause these Crypto Errors 

Checking the registry on affected computers confirmed that the key in question was set to 0x00063c00. After changing it back to 0x0023c00, the trust provider errors stopped. I have no idea how he found that key, and what’s amazing is that this TechNet post seems to be the only reference to this error and this registry key out there.

I didn’t want to just make the change without understanding what it meant, so researching further lead to this post which clarifies that the “State” key is a bitwise flag for various settings of “Microsoft Trust verification services, which provide a common API for determining whether a specific subject can be trusted.” After a bit of hex to binary conversion and a comparison between the two sets of flags, I determined that the difference was the incorrect setting had enabled the flag to only “allow…items in personal trust database.” Seemingly this flag was preventing a system-level certificate from being used to validate a Microsoft signature. No clue why.

But of course things didn’t stop there.

Whether it was related to the original certificate error or not, many of the same PCs that had this error were also not reporting to WSUS. This at least I already had a partial fix for. Years ago I had put together a basic batch file that reset the SusClientId, deleted the SoftwareDistribution folder, and basically forced a full reset of the WSUS client. (I’m not taking credit for figuring this all out, but I have no idea where I first saw the info.)

The process involves running the following as a batch file on the affected PC:

net stop wuauserv

reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v AccountDomainSid /f

reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v PingID /f

reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v SusClientId /f

reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v SusClientIdValidation /f

del c:\windows\softwaredistribution /S /F /Q

net start wuauserv

wuauclt /resetauthorization /detectnow

This stops the Automatic Update service, clears out the registry keys that identity the system to WSUS, clears out the patch download cache, restarts the Automatic Update service, and then re-registers the system to WSUS. There’s one little problem. The “/detectnow” switch was apparently removed in the Win 10 version of wuauclt.exe, leaving no simple way to force a machine to scan WSUS for updates.

There are a couple of ways to get around this via VBScript or languages that directly call the Windows Update API. In our case however we already were licensed for WUInstall, which has this functionality built-in. We added an additional command at the end of the batch file to force WUInstall to scan for updates, and at this point, everything seems to be working properly.

Thanks for reading, and “lforbes,” if you’re out there, thank you for posting your fix.