Re: Proxy 2.0 secure?

[tqbf () pobox com]

> Ok, I'll bite.  What would YOU describe as "real firewall testing", given
> that you had to test a number of independent, but NT-based firewall
> products, and provide advice on their performance?

First off, I would not claim that I was capable of presenting a real
comparitive review of the security of multiple firewall packages in a tiny
magazine article. Following that, I would not makes claims about the
security of firewall packages based on the assumption that I had done any
real testing of them. In other words, I'd be honest with my readers about
what my capabilities were, and what conclusions the model reader could
draw from my article.

Now, if you want to ask me how I think firewalls can be tested, you're
asking a big loaded question that has an even larger and more
controversial answer. 

Some people don't believe in firewall testing --- they think the results
are misleading, since it is both practically infeasable to completely test
many firewalls in the short amount of time available (ie, before
everything changes revisions again), and because the information obtained
from a real design review doesn't compare neatly with the information from
other reviews (unless your reviewing strategy is either revolutionary or
completely flawed). 

I agree with this viewpoint somewhat. Obviously, I have some experience
reviewing commercial security software (my work has focussed on IDS). My
opinion, drawn from my experience reviewing software, is that attempts to
present "quick" reviews of security software are usually going to fail. It
doesn't take a lot of research to see that almost every review of security
software to date HAS failed. 

In any case, real firewall testing is complicated and outside the scope of
this message. For-profit companies (like the ICSA) are making money by
selling the idea that they have competancy in this area. If there was an
obvious way to review firewalls, we wouldn't have this problem.

However, there are certainly techniques that are more valuable than
running ISS (or CCS) against a firewall and making sure the report doesn't
list any vulnerabilities. Techniques such as marked-packet analysis and
active probing are good examples of ways that automated analysis can draw
useful conclusions about firewall characteristics, while treating them as
a black box. However, these are just useful tools. The real issue boils
down to this:

You should not trust software that has neither open source nor published
peer review results. 

-----------------------------------------------------------------------------
Thomas H. Ptacek                           SNI Labs, Network Associates, Inc.
-----------------------------------------------------------------------------
http://www.pobox.com/~tqbf       "If you're so special, why aren't you dead?"