Reviews and Comments on Paper 607

Paper information

Paper #607: Peter Jing Tan and David L. Dowe. Decision Forests with Oblique Decision Trees

Abstract: Ensemble learning schemes have shown impressive increases in prediction accuracy over single model schemes. We introduce a new decision forest learning scheme, whose base learners are Minimum Message Length (MML) oblique decision trees. Unlike other tree inference algorithms, MML oblique decision tree learning does not over-grow the inferred trees. The resultant trees thus tend to be shallow and do not require pruning. MML decision trees are known to be resistant to over-fitting and excellent at probabilistic predictions. A novel weighted averaging scheme is also proposed which takes advantage of high probabilistic prediction accuracy produced by MML oblique decision trees. The experimental results show that the new weighted averaging offers solid improvement over other averaging schemes, such as majority vote. Our MML decision forests scheme also returns favourable results compared to other ensemble learning algorithms on data sets with binary classes.
(file)

Summary of received reviews and comments

Reviews superseded by other reviews are shown in the grey color in the table.

	confidence	score
Review 1	3	1
Review 2	4	2
Review 3	2	3

Reviews and Comments

Review 1

PC member:
Reviewer:
Overall rating:	1 (weak accept: Vote accept, but wouldn't mind rejecting)
Confidence:	3
Relevance: Is this paper relevant for this conference?	2 (accept (I will argue for this paper))
Soundness: Is this paper technically sound and complete?	2 (accept (I will argue for this paper))
Are the claims sufficiently supported by experimental/theoretical results?	2 (accept (I will argue for this paper))
Significance: Are the results/ideas interesting for other AI researchers?	2 (accept (I will argue for this paper))
Originality: Are the results or ideas novel and previously unknown?	2 (accept (I will argue for this paper))
Readability: Is the paper well-organized and easy to understand?	3 (strong accept)
Language: Is the paper written in correct English and style?	2 (accept (I will argue for this paper))
Format: Is the paper correctly and consistently formatted?	2 (accept (I will argue for this paper))
Review:	CONTRIBUTION OF THE PAPER: The paper describes a new decision forest algorithm, that uses minimum message length oblique decision trees. The paper also introduces a new weighted average scheme based on prediction accuracy. The approach is tested on several data sets and compared against other ensemble techniques. POSITIVE ASPECTS: An ensemble algorithm with shallow trees and the weighting scheme. NEGATIVE ASPECTS: Seems very costly to produce and uses strong assumptions of the probability estimation. CHANGES TO IMPROVE THE PAPER: The MML oblique decision tree algorithm seems to be computationally very expensive. The authors should clarify what is the clear benefit of using a computationally more expensive algorithm. Section 4.2 talks about a figure with circles and rectangles that is not included in the paper. The approach seems adequate mainly for binary class problems. Why not convert the multiple class problems into binary? What improvements are needed to make the approach competitive with multiple classes. There is no clear improvement over other ensemble methods. What are then, the main benefits of the approach. It would be interesting to compare the results against an ensemble of naive Bayes classifiers. FURTHER COMMENTS: ITEMS BELOW ARE JUSTIFICATION OF THE SCORES IF NEGATIVE: (1) IS THIS PAPER RELEVANT FOR THIS CONFERENCE? (2) IS THIS PAPER TECHNICALLY SOUND AND COMPLETE? (3) ARE THE CLAIMS SUFFICIENTLY SUPPORTED BY EXPERIMENTAL OR THEORETICAL RESULTS? (4) ARE THE RESULTS/IDEAS INTERESTING FOR OTHER AI RESEARCHERS? (5) ARE THE RESULTS OR IDEAS NOVEL AND PREVIOUSLY UNKNOWN? (6) IS THE PAPER WELL-ORGANIZED AND EASY TO UNDERSTAND? (7) IS THE PAPER WRITTEN IN CORRECT ENGLISH AND STYLE? (8) IS THE PAPER CORRECTLY AND CONSISTENTLY FORMATTED?
PC only:
Time:	Jul 14, 06:39

Review 2

PC member:
Reviewer:
Overall rating:	2 (accept: I will argue for this paper)
Confidence:	4
Relevance: Is this paper relevant for this conference?	2 (accept (I will argue for this paper))
Soundness: Is this paper technically sound and complete?	2 (accept (I will argue for this paper))
Are the claims sufficiently supported by experimental/theoretical results?	2 (accept (I will argue for this paper))
Significance: Are the results/ideas interesting for other AI researchers?	1 (weak accept (vote accept but don't mind rejecting))
Originality: Are the results or ideas novel and previously unknown?	2 (accept (I will argue for this paper))
Readability: Is the paper well-organized and easy to understand?	2 (accept (I will argue for this paper))
Language: Is the paper written in correct English and style?	2 (accept (I will argue for this paper))
Format: Is the paper correctly and consistently formatted?	2 (accept (I will argue for this paper))
Review:	CONTRIBUTION OF THE PAPER: A new ensemble supervised classification method based on oblique decision trees and MML. POSITIVE ASPECTS: New ensemble method robist to overfitting and good as probabilistic predictor. NEGATIVE ASPECTS: The significance of the results could be improved by using the advices from Demstar (2006) (JMLR). CHANGES TO IMPROVE THE PAPER: FURTHER COMMENTS: ITEMS BELOW ARE JUSTIFICATION OF THE SCORES IF NEGATIVE: (1) IS THIS PAPER RELEVANT FOR THIS CONFERENCE? (2) IS THIS PAPER TECHNICALLY SOUND AND COMPLETE? (3) ARE THE CLAIMS SUFFICIENTLY SUPPORTED BY EXPERIMENTAL OR THEORETICAL RESULTS? (4) ARE THE RESULTS/IDEAS INTERESTING FOR OTHER AI RESEARCHERS? (5) ARE THE RESULTS OR IDEAS NOVEL AND PREVIOUSLY UNKNOWN? (6) IS THE PAPER WELL-ORGANIZED AND EASY TO UNDERSTAND? (7) IS THE PAPER WRITTEN IN CORRECT ENGLISH AND STYLE? (8) IS THE PAPER CORRECTLY AND CONSISTENTLY FORMATTED?
PC only:
Time:	Jul 14, 09:41

Review 3

PC member:
Reviewer:
Overall rating:	3 (strong accept)
Confidence:	2
Relevance: Is this paper relevant for this conference?	3 (strong accept)
Soundness: Is this paper technically sound and complete?	3 (strong accept)
Are the claims sufficiently supported by experimental/theoretical results?	2 (accept (I will argue for this paper))
Significance: Are the results/ideas interesting for other AI researchers?	2 (accept (I will argue for this paper))
Originality: Are the results or ideas novel and previously unknown?	2 (accept (I will argue for this paper))
Readability: Is the paper well-organized and easy to understand?	3 (strong accept)
Language: Is the paper written in correct English and style?	3 (strong accept)
Format: Is the paper correctly and consistently formatted?	2 (accept (I will argue for this paper))
Review:	CONTRIBUTION OF THE PAPER: An ensemble learning scheme with decision forests, based on oblique decision trees. POSITIVE ASPECTS: Well-written paper. Presents a new approach (building up from existing work), and shows that it works well. NEGATIVE ASPECTS: The classification accuracies in Table 2 (for the new algo.) are not all that convincing, although there is some explanation about these in sec.5.3. The results for the anomaly in classification percentage vs. P_cost are unclear, esply. for SAT_IMAGES data. CHANGES TO IMPROVE THE PAPER: FURTHER COMMENTS: ITEMS BELOW ARE JUSTIFICATION OF THE SCORES IF NEGATIVE: (1) IS THIS PAPER RELEVANT FOR THIS CONFERENCE? (2) IS THIS PAPER TECHNICALLY SOUND AND COMPLETE? (3) ARE THE CLAIMS SUFFICIENTLY SUPPORTED BY EXPERIMENTAL OR THEORETICAL RESULTS? (4) ARE THE RESULTS/IDEAS INTERESTING FOR OTHER AI RESEARCHERS? (5) ARE THE RESULTS OR IDEAS NOVEL AND PREVIOUSLY UNKNOWN? (6) IS THE PAPER WELL-ORGANIZED AND EASY TO UNDERSTAND? (7) IS THE PAPER WRITTEN IN CORRECT ENGLISH AND STYLE? (8) IS THE PAPER CORRECTLY AND CONSISTENTLY FORMATTED?
PC only:
Time:	Jul 17, 12:48