Microsoft versus Linux

I think it's a great accomplishment of the Linux community that Microsoft has recently started to publish white papers about this operating system. They're even so worried that they `bench-marked' NT Server against Linux. They published their results. Read this as: they were pleased with them or able to make them pleased to them.

I think this paper is very interesting. Not for its results, but for the way Microsoft presents information. As many of you know from bitter experience, Microsoft systems and software are not noted for their reliability. Still Microsoft sells. I believe that an important reason for this is, that Microsoft is very good in communicating to management. So good that the You can't be fired for buying Microsoft thought is firmly entrenched there.

How does Microsoft convince management? I think Microsoft masters four arts that the Open Source community doesn't practice. Using the above mentioned NT against Linux benchmark paper I've identified these arts as:

1. The art of comparing: you should win.
2. The art of ballooning: making yourself big.
3. The art of statistics: stats will do anything.
4. The art of deception: or should we say lying?

In the following four sections we will closely study those arts. My purpose is not so you can imitate it or become a master of these arts yourself. As we have "free" in the sense of "free speech", Open Source also has to maintain its open in the sense of "I will be open with you, I will be honest." This paper also isn't written, because I believe there is something wrong with the benchmarks. That case is firmly settled now: Currently, NT performs better, in almost any case, as Linux.

This paper is about how Microsoft uses that results to present its case. My purpose is to give you something to say to your management when they will point to Microsoft's white paper. As this is a draft paper, comments, suggestions and additions are eagerly sought.

The art of comparing

In case some manager doesn't have much time, it helps to distill the most important findings. So a manager can quickly scan the paper and still get a good understanding of its contents. It goes without saying that this distillation should be honest. However, Microsoft defines distillation as: publish your best benchmarks when compared to the worst benchmark of your competitor. We will highlight two examples of Microsoft distillation:

1. NT server performs up to 680 percent better as a web server.
2. NT server provides up 96 percent better file server price/performance.

NT server performs up to 680% better as a web server?

1. The latest Microsoft NT Server edition, including all the latest service packs. I infer this because PC Magazine used Microsoft Transaction Server and Microsoft Message Queue, needing at least Service Pack 4 with Option Pack 4.
2. The latest Microsoft Internet Information Server (IIS) version.
3. The application on IIS was a DLL. That means it was precompiled and loaded just once. It used the IIS optimized API, not CGI.

On the Linux side we have:

1. PC Magazine used a Linux 2.0 kernel, in no way optimized for SMP architectures.
2. PC Magazine used an older version of Apache (before 1.2.4). This version of Apache didn't use threads, but processes. The overhead of processes is much larger. This version wasn't written nor optimized for multi-processor architectures.
3. The application on Apache seems to be a Perl script (loaded and recompiled every time) using CGI. There was no hint in PC Magazine's test that they used something like PHP or embedded Perl.

So take NT's best performing case and put it against Linux lowest performing case and you're there. When the comparison was a bit more fair, comparing web serving with CGI applications, PC Magazine wrote: "On our CGI tests, Stronghold (Apache) held its own against the competition."

Note, in this case only the CGI variable was changed. They still compared compared NT, according to Microsoft optimized for SMP architectures, against a Linux kernel, not optimized for SMP and a web server not written for SMP kernels. PC Week, in a first-of-its-kind open benchmark, compared Linux with NT on a single processor architecture. The difference was much smaller in this case.

NT server provides up to 96% better file server price/performance?

What did Microsoft compare in its statement that NT server provides up to 96 percent better file server price/performance? This statement is based on a recent test by Mindcraft. Interestingly, Mindcraft did not publish results using NT Workstation clients, but only for Windows 95 clients.

As previous tests have shown that, using NT Workstations, Linux does outperform NT. But if you have only Windows 95 clients, NT performs better. Mindcraft was payed by Microsoft to do this comparison. As Mindcraft's results were not verifiable in PC Week's test, we don't know how much better Windows 95 clients perform against NT Server. And if you have NT workstations in your network, this benchmark doesn't help you to decide.

In the same graph, Microsoft also used testsresults done by PC Week. When PC Week confirmed again that using NT Workstations Linux did outperform NT, Microsoft came up with some new tuning parameters so the NTFS filesystem would perform better. You might expect that the coming Linux 2.3 kernel will provide better performance too.

Another question is how Microsoft calculated the cost of $2807 for it's server. You get NT Server 4.0 with 25 client licenses for $1609. The remaining $1200 is probably more client licenses. That will by you some 40 licenses. So Microsoft compared $2807 with a limit of 65 clients to $0 with no client limit. What would you choose?

Comparing is an art, you've just seen that. And it's not just this article in which Microsoft deploys this art. It's a common theme. See for example there Meeting Enterprise Security Needs:Microsoft Windows NT and UNIX white paper. To which Unix do they compare??

The art of ballooning

1. A well-respected company that gathers Internet statistics is Netcraft. In their June 1999 Web Server Survey, they confirm that Apache has 61% of market share. And it's growing much faster as it competitors.
2. PC Magazine favored NT and IIS as their editorial choice. But why are they running Apache on Solaris? And so does PC Week.
3. PC Magazine favored NT. But it doesn't agree that you got more money for your bucks as Microsoft wants you to believe. See their just released "The latest version of Linux gives Windows NT a run for its money -- for free." article.
4. Sometime ago Microsoft acquired HotMail, a ten million user system. We all know NT is enterprise ready, can scale, has the hottest TPC-X benchmarks, whatever. Why does Microsoft use FreeBSD?
5. And yet another Microsoft owned company uses FreeBSD.
6. Kirch is with you. See his Microsoft Windows NT Server 4.0 versus Unix

The art of statistics

Microsoft's excellent understanding of statistics, can be clearly seen in their claim that Windows NT Server 4.0 Provides better Price/Performance than Linux. For web servers even up to - we've seen the `up to' before - 623%. In this section I take a look at how Microsoft calculated the file server price/performance statistics. Microsoft calculates the price/performance ration by dividing the costs of the server hardware and software by its peak Megabyte per second throughput. If the hardware costs $10000, the server $1000, and you have a throughput of 200MB per second, the price/performance ratio is $55: it costs you $55 per MB. Microsoft makes the following claims:

1. According to Mindcraft, Microsoft costs $93 per Mbs.
2. According to PC Week, Microsoft costs $114 per Mbs.
3. The costs for Linux are about $182.

When people learn that NT Server has a better price/performance ration as a free OS, people react really surprised. How can something expensive have a better price/performance than a free OS? Learn how you do this when we entangle these figures.

As discussed above, Microsoft has paid Mindcraft to do a NT versus Linux benchmark. This benchmark was very favorable for NT. Later on PC Week was not able to verify the enormous differences Mindcraft found.

Anyway, what should one do to compete against a free OS? To get a higher price/performance ratio, you have to make the costs of the software insignificant to the costs of hardware. Microsoft does this by:

1. Choosing really expensive servers. Do you spend $20,851 on a file server? Or the even more ridiculous $35868 server?
2. Only counting the most basic costs of its Operating System. Add email to a Linux server. Costs you nothing. Add Microsoft Exchange Server to an NT Server and you got a monster, oops, I meant something expensive.

PC week commented that they got Linux most favorable results on a single processor server with 256MB of RAM. If we assume a $5000 file server (probably some 27GB disk, 256MB, and a Pentium II 400MHz or better), we get a $47 price/performance ratio for NT and $46 for Linux. Assuming the new NT NTFS tuning parameters of course, so NT outperforms Linux on file serving. How realisticly these NTSF tuning parametesr are, is still under debate.

The art of deception

In its article Microsoft argues that performance is not all. There's more than performance. How about user satisfaction? Microsoft doesn't mention that, but makes numerous false and misleading claims in this section. I've picked sentences from this section and gave my comments. When a sentence appears under Linux column I start it with Linux, when it appears under the Windows NT Server 4.0 column, I start it with NT. As this is a draft paper, I expect this section to expand a lot.

"NT: Lack of extensive testing to guarantee compatibility across components and applications."

Does Microsoft tests all of the 4000 applications that run on Windows NT Server? Or the 8000 Windows NT compatible applications?

"NT: Lack of an enterprise clustering system for service and application availability."

With Beowulf clustering, using Extreme Linux, you can put together dozens of Linux machines. They have been used as rendering machines, supercomputers, and lots of other applications.

"NT: Support for high availability application clustering and TCP/IP-based load balancing."

If you don't know what TCP/IP-based load balancing has to do with load balancing, I don't know either. It's probably means about the same as that Windows-CE is a real-time OS, or using another Microword: a soft real-time OS.

"Linux: Lack of a journaling file system (JFS), which means file system may not recover after unplanned down time."

Microsoft makes available two files systems: FAT and NTFS. The FAT filesystem isn't a journaling filesystem, NTFS is. You can use either with NT.

Linux' standard file system isn't a JFS. But you can install a JFS. SGI has made available a journaling file system for Linux. It's Irix renowned XFS technology.

A few weeks ago I followed a discussion about `soft updates' versus journalling. Soft updates should do for a standard Unix file system what a journalling file system does in respect to file system health.