Automatic Evaluation of Intrusion Detection Systems. F. Massicotte, F. Gagnon, Y. Labich, L. Briand, Computer Security Applications Conference, ACSAC ’06, pp 361-370, 2006. Presented by: Lei WEI. Summary.
F. Massicotte, F. Gagnon, Y. Labich, L. Briand,
Computer Security Applications Conference,
ACSAC ’06, pp 361-370, 2006.
Presented by: Lei WEI
This is an automatic IDS evaluation system. Because of automation, it is possible to efficiently and systematically create a large number of sample data .
Each of the collected traffic traces belongs to one of the type, TP, TN, FP and FN.
According to types of all traces collected from IDS evaluation tests, the authors suggested a 15-class taxonomy for IDSes, such as, alarmist, quiet, quietand complete detection, complete evasion etc.
( , ) , from which we know the percentage of attack being detected and the percentage about wrong alarms.
In this paper, the two diagrams, Figure 5 and Figure 1, and relevant description used to represent the working process of the whole system are not clear enough.
(a). A title should be “… an effective guide for scientists rapidly scanning lists of titles for information relevant to their interests.” (Scientific writing for graduate students: a manual on the teaching of scientific writing, edited by F. Peter Woodford. New York: Rockefeller University Press, 1968. )
However, neither the title nor the content provides clear explanation to the meaning of numbers in Figure5.
(b). Although the article describes the steps listed in Figure1, the provided diagram does confused us to understand the structure and working process of the system. The title is Virtual network infrastructure,but the figure actually covers more stuff than that. It does not only represent Virtual network infrastructure, but also shows the working process of the subsystem.
This system could be divided into two subsystems.
Provide the virtual attacking machine the proper attack configuration (e.g. Whether apply IDS Evasion Tech.)
Set up Virtual Network
Set up Attack Script
IDS Evaluator takes documented traffic traces from the Data Set
The collected IDS alarms are fetched by the IDS Results Analyser
IDS Evaluator provide traffic traces to each tested IDS
Compare the two groups of data sets and determine whether the IDS detection succeed
IDS Result Analyzer
Generate the evaluation report
This paper evaluated two open source IDSes by the new strategy. However, many IDSes have patent or copy right protection. Those creators would never reveal the weak points of their products.
Is it ethical or illegal to publish the evaluations of IDS programs so that others can know the truth?
Each traffic trace is documented by four characteristics: