On Wednesday, Untangle, maker of security gateway products based on open source, will be holding its much publicized “Deep Throat Fight Club” of six Web content filtering solutions. Among those in the fight are WatchGuard , SonicWall, Fortinet, Barracuda, Websense and ScanSafe.

Needless to say, we’ll be watching closely for the results. If I had to wager, I’d say we’ll hear a lot more about it in the days to come and that most of it will favor Untangle at the expense of other vendors.

So much for the first rule of Fight Club being that you don't talk about Fight Club .

A few things to ask yourself about this:

1. Is this even a real fight? Untangle is not an independent, objective product reviewer. Untangle is a security vendor. A sentence from their own press release reads: “The Untangle Gateway platform, the world’s first commercial-grade open source solution for blocking spam, spyware, viruses, adware and unwanted content on the network, provides a free and better alternative to costly, inflexible proprietary appliances.” Untangle clearly has an agenda.

Last summer at Linux World, Untangle conducted a similar “fight club” with anti-virus products. To no one’s surprise ClamAV, an open source solution that Untangle uses in its gateway product, was among the winners.

The point is that any reputable security firm will more than happily agree to have their products and services tested by independent reviewers without an agenda—like test labs or publications. And often the best most accurate test of a solution comes from businesses that have deployed it in a real world environment.

2. Methodology. Little information has been made available about the test methodology. A page on Untangle’s website promises that the methods and results will be made publicly available “to encourage discussion.” To date nothing has been made public except their widely circulated press release that merely states that the fight club “…will establish a baseline metric for porn filtering.”

Statements by Untangle to the press indicate that the test will focus exclusively on pornography and the PCs protected by the various filters will be used to search for 5,000 popular porn URLs to uncover whether the PC blocks them or not. It also indicates that Untangle has done some preliminary testing.

I’m not sure what purpose is served by a “test” that it is so limited. I mean why just test porn? What about other potentially offensive content? Seemingly all you need to do to win the fight is block these sites. There’s no attention given to sites blocked erroneously (like a medical site blocked as porn) or to perhaps the biggest web threat---malware.  Anyone who thinks a web filtering solution can be accurately tested by visiting well known porn sites has a very small and dated view of the Web security market. Web threats are increasingly found on legitimate, trusted sites, not just dodgy sites.

3. The participants. While we’re flattered to be included in this “fight club” some of the other big names in web content filtering—including Blue Coat and Secure Computing—are notably absent.

We’re not sure why Untangle has omitted these vendors, but it does make one wonder about the validity, scope and purpose of this fight club.

While we here at ScanSafe are always up for a good fight (win or lose), I’m not sure that’s what we’ll see. I hope Untangle proves me wrong.