Making Sense of Anti-Malware Comparative Testing

[To return to ESET white papers page click here: http://www.eset.com/threat-center/blog.]

This is an Elsevier article preprint of an article on the main issues around comparative testing of antivirus/antimalware products, made available here by permission of Elsevier.

The fully formatted, proofed and reviewed version is available at http://dx.doi.org/10.1016/j.istr.2009.03.002.

Abstract:

If there’s a single problem illustrating the gulf between the anti-malware industry and the rest of the online world, it revolves around the difficulties and misunderstandings that plague product testing and evaluation. This article considers these issues and the initiatives taken by the anti-malware and testing sectors to resolve some of them.

Execution Context in Anti-Malware Testing

[Go back to ESET White Papers page.]
[Go back to ESET blog.]

This is one of my most recent papers, presented by Randy Abrams and myself on behalf of ESETat the EICAR 2009 Conference in Berlin.

Abstract

Anti-malware testing methodology remains a contentious area because many testers are insufficiently aware of the complexities of malware and anti-malware technology. This results in the frequent publication of comparative test results that are misleading and often totally invalid because they don’t accurately reflect the detection capability of the products under test. Because many tests are based purely on static testing, where products are tested by using them to scan presumed infected objects passively, those products that use more proactive techniques such as active heuristics, emulation and sandboxing are frequently disadvantaged in such tests, even assuming that sample sets are correctly validated.

Recent examples of misleading published statistical data include the ranking of anti-malware products according to reports returned by multi-scanner sample submission sites, even though the better examples of such sites are clear that this is not an appropriate use of their services, and the use of similar reports to generate other statistical data such as the assumed prevalence of specific malware. These problems, especially when combined with other testing problem areas such as accurate sample validation and classification, introduce major statistical anomalies.

In this paper, it is proposed to review the most common mainstream anti-malware detection techniques (search strings and simple signatures, generic signatures, passive heuristics, active heuristics and behaviour analysis) in the context of anti-malware testing for purposes of single product testing, comparative detection testing, and generation of prevalence and global detection data. Specifically, issues around static and dynamic testing will be examined. Issues with additional impact, such as sample classification and false positives, will be considered – not only false identification of innocent applications as malware, but also contentious classification issues such as (1) the trapping of samples, especially corrupted or truncated honeypot and honeynet samples intended maliciously but unable to pose a direct threat to target systems (2) use of such criteria as packing and obfuscation status as a primary heuristic for the identification of malware.

EICAR execution context paper

Phish Phodder: Is User Education Helping or Hindering?

[Go back to ESET White Papers page.]
[Go back to ESET blog.]

David Harley & Andrew Lee, “Phish Phodder: Is User Education Helping or Hindering?” (davidharleyandrewleevb2007), September 2007, Virus Bulletin. Copyright is held by Virus Bulletin Ltd, but the document is made available on this site for personal use free of charge by permission of Virus Bulletin.

ABSTRACT
Mostly, security professionals can spot a phish a mile off. If they do err, it’s usually on the side of caution, for instance when real organizations fail to observe best practice and generate phish-like marketing messages. Many sites are now addressing the problem with phishing quizzes, intended to teach the everyday user to distinguish phish from phowl (sorry). Academic papers on why people fall for phishing mails and sites are something of a growth industry. Yet phishing attacks continue to increase, and while accurate and up-to-date figures for financial loss are hard to come by, indications are that losses from phishing and other forms of identity theft continue to climb.

This paper:
1. Evaluates current research on how end users are susceptible to phishing attacks and ID theft.
2. Evaluates a range of web-based educational and informational resources in general and summarizes the pros and cons of the quiz approach in particular.
3. Reviews the shared responsibility of phished institutions and phishing mail targets for reducing the impact of phishing scams. What constitutes best practice for finance-related mail-outs and e-commerce transactions? How far can we rely on detection technology?

A Musical Interlude

After only 20 years (well, nearly) of being connected to the Internet, I’ve finally got around to making some music available that I recorded in the 80s (I did sell some copies at the time, so this isn’t completely unheard stuff). Although this is studio recorded stuff, I don’t have access to the master tapes, so these tracks are taken from cassettes. Still, they sound better than I expected through decent headphones.

Three tracks are from an unreleased album made with Bob Theil, Don MacLeod, Bob Cairns, and Pat Orchard. There are also a handful of tracks from “Sheer Bravado” (more to come) and, eventually, there’ll be more  from “Scriptwrecked”. There will also be some more recent stuff eventually: I’ve got some BOSS recording kit that I’m dying to do more work with.

The relevant page on the main Small Blue-Green World site is here.

Who Will Test The Testers?

[Go back to ESET White Papers page.]
[Go back to ESET blog.]

Who Will Test The Testers? is a paper by myself and Andrew Lee on making anti-malware testers more accountable to their audiences, presented at the Virus Bulletin Conference in 2008 and published in the conference proceedings.

David Harley BA CISSP FBCS CITP & Andrew Lee CISSP, “Who Will Test The Testers?”, October 2008, Virus Bulletin. Copyright is held by Virus Bulletin Ltd, but the paper is made available on this site for personal use free of charge by permission of Virus Bulletin.

ABSTRACT

The anti-malware industry has been plagued since its earliest days by one poorly designed comparative test after another. In 2007, some of the best anti-malware researchers, comparative testers and product certification specialists took the first steps towards raising product testing standards with the formation of a group specifically focused on establishing standards and methodologies, educating both consumers and testers in discrimination between good and bad practice, and providing objective analyses of current testing practices. This paper summarizes current initiatives by the Anti-Malware Testing
Standards Organization and other groups, but also considers next steps, going beyond objectifying methodology, educational issues and blowing away the fog of misinformation and fallacy, to the next level. Underlying these vital issues is a question: is it possible to make testers and certifying authorities more accountable for the quality of their testing methods and the accuracy of the conclusions they draw based on that testing?

This paper attempts to answer that question.

A Dose By Any Other Name

[Go back to ESET White Papers page.]
[Go back to ESET blog.] 

A Dose By Any Other Name is a paper by myself and Pierre-Marc Bureau (a fellow researcher at ESET), presented at the Virus Bulletin Conference in 2008, and published in the conference proceedings.

David Harley BA CISSP FBCS CITP & Pierre-Marc Bureau, “A Dose By Any Other Name”, October 2008, Virus Bulletin. Copyright is held by Virus Bulletin Ltd, but the paper is made available on this site for personal use free of charge by permission of Virus Bulletin.

ABSTRACT

Years ago, when alt.comp.virus was still useful, ‘Name that virus’ was a popular virtual party game, and virus names were, if not standardized, at least easy to cross-reference with tools like VGrep. In 2008, the numbers have escalated exponentially, analysis and detection have become increasingly generic, and naming, even for some WildList malware, has become nearly useless because of the difficulty of mapping samples to names.

The CME (Common Malware Enumeration) initiative], while attempting to achieve something many people wanted, seems to have foundered on the rocks of the reality. Yet we continue to provide ‘top ten’ threat lists that have virtually no commonality or consistency across different vendors and sites, so that our customers continue to ask whether we detect the media virus du jour, and the slashdotty community point to us and giggle at our incompetence in failing to provide information about what we detect.

Are all our solutions going generic? Are there ways to resolve this issue so that our customers can understand what’s happening and regain some faith in the industry without being hung up on the question ‘Do you detect virus X?’ We think so, and will discuss some possible approaches in this paper.

Teach Your Children Well

[Go back to ESET White Papers page.]
[Go back to ESET blog.]

teach-your-children-well is a paper by myself, Eddy Willems and Judith Harley presented by Eddy and myself at the Virus Bulletin Conference in 2005, and published in the conference proceedings.

David Harley, Eddy Willems & Judith Harley, “Teach Your Children Well – ICT Security And The Younger Generation”, October 2005, Virus Bulletin. Copyright is held by Virus Bulletin Ltd, but the document is made available on this site for personal use free of charge by permission of Virus Bulletin.

ABSTRACT

An article by Eddy Willems in the August 2004 edition of VB discussed his research into the security awareness of Belgian children. The authors have developed this theme by submitting a similar questionnaire to ICT pupils in the UK and using the results as a basis for an interactive presentation and discussion with several groups in the UK, and an assignment-based follow-up with different groups was undertaken early in March 2005.

The paper is not intended as a completed formal study, but considers this presentation and the issues that came up in this preliminary research as a basis for further study and teaching tools. It also considers a range of resources in the area of child safety, learning, attitudes and behaviour as they affect and are affected by the use of information and communications technology, and the influence of the media, government, and the Internet itself. While the preliminary research has largely focused on malware and email abuse, we will also consider how these areas are connected with other technologies and areas of concern among parents and educators.

Scared of SCADA

I was asked this week about the CIA’s announcement to a SANS Institute conference that money has been extorted outside the US for “cyberattacks” have been used to extort money by threatening to (or even demonstrating an ability to) disrupt the supply of power to “multiple cities”.

Potential attacks on SCADA (Supervisory Control And Data Acquisition) systems have been a major concern in national security circles for many years, and it’s not the first time that SANS have commented on the topic. Rob Rosenberger, on the other hand, was scathing (he usually is).

Rosenberger can sometimes be accused of not letting accuracy get in the way of satire, but on this occasion he has a point. Effectively, all that’s been disclosed here is that the CIA believe that at least one unnamed country has been subjected to attacks on power facilities that they believe to have been delivered over the internet (which
could mean almost anything: we don’t have enough even to determine whether “the internet” in this case means “the internet” or some other form of external connection.). We don’t know where or when. We don’t know what sort of attacks. We don’t know if they were aimed directly at SCADA systems, or at other supplementary or support systems within facilities (which are, perhaps, likelier to be directly connected to the wild and woolly internet).

Clearly, enquiring minds would like to know which countries are having these problems, and round here, one or two people have wondered aloud about the UK. Perhaps influenced by the UK’s highly publicized problems at present with lost sensitive data, unencrypted laptops and so on. Political alliance factors apart, SCADA security has been a preoccupation of both countries for quite a few years. There have been reports of UK law enforcement and security services involvement in developing defences against DDoS attacks and extortion demands from time to time, but they tend to be in more lucrative sectors like online gaming. In fact, even Bruce Schneier, who has a pretty good nose for “security theatre”, has made much the same point, though not in a UK context.

The trouble here is that Those In The Know are passing on a warning so generic that it reminds me of the way a virus hoax was described some years ago by (I think) Iolo Davidson on alt.comp.virus. Something like “Fire! Fire! Don’t know when, don’t know where! Fire!” This warning may be based on real events, but functionally, it doesn’t amount to any more than “Be careful out there!”

It’s not as though there isn’t some good proactive work out there. In the US, the FERC (Federal Energy Regulatory Commission) has approved some relevant security standards, after some years of assessment, which certainly should be useful. There again, the UK security services (CPNI) have been publishing good practice guides on Process Control, Firewall Deployment and so on for SCADA for a good while. It’s a large part of their job to try to protect the Critical National Infrastructure, by providing information and other services to the agencies and companies who own and maintain it. We could, of course, debate how _successfully_ they do that job, but that’s as much about politics as it is about technology.

A tip of the hat here to Davey Winder, who indirectly started me thinking about this topic. He has a piece on related topics coming out in his ‘Real World Computing – Security’ column in PC Pro issue 163, which is due out at the end of February or beginning of March. I don’t know what he’s going to say, but I expect it to be worth reading. :)

Mac Malware

You may be aware that I have a long-standing love/hate relationship with the Mac community (love Macs, hate the maulings I get whenever I comment on Mac security: somehow I keep forgetting that Macs are 100% secure and Mac users are 100% more intelligent that Windows users. Sigh…)

If you have an interest in Mac issues, you might find my Securiteam blog interesting. Or not.

AVIEN Guide published

Good day to you, my loyal readers.

How are you both?

It’s been a long time since I posted anything here, which doesn’t mean things have been happening (too many things have been happening!)

The “AVIEN Malware Defense Guide for the Enterprise” was published in the US by Syngress early in August, 2007. This is a major publishing project I put together with AVIEN (Anti-Virus Information Exchange Network) and AVIEWS (Anti-Virus Information and Early Warning System) members. It will be published in the UK in early September. Read more on the book’s own web site here. Some of the authors will be at Infosec New York (11th-12th September) signing copies at the ESET stand, and at the Virus Bulletin conference in Vienna later in September (where Andrew Lee and I are presenting a phishing paper, by the way).

I also got somewhat irritated by a poor, misleading comparative test of antivirus products presented by Untangled.com at the Linuxworld expo: so irritated that I put a white paper here and a blog entry at Technet on the subject of testing. This is probably not the last you’ll hear of this from me.

Talking of AVIEN, as of 15th August I became the interim Administrator there. Essentially, my job is to keep order while the membership decide whether they want to change the structure of the organization. More about that on the AVIEN site in due course.