Statistical and Machine Learning Approaches for Network Analysis

11.11 Evaluation

We applied the patterns on the German Wikipedia corpus from November 2006, which contains about 500,000 articles. In total, we extracted 391,153 different hyponymy relations employing 22 deep and 19 shallow patterns. The deep patterns were matched to the SN representation, the shallow patterns to the tokens. Concept pairs that were also recognized by the compound analysis were excluded from the results, since such pairs can be recognized on the fly and need not be stored in the knowledge base. Thus, these concept pairs are disregarded for the evaluation. Otherwise, recall and precision would increase considerably. 149,900 of the extracted relations were only determined by the deep but not by the shallow patterns. If relations extracted by one rather unreliable pattern are disregarded, this number is reduced to 100,342. The other way around, 217,548 of the relations were determined by the shallow but not by the deep patterns. 23,705 of the relations were recognized by both deep and shallow patterns. Naturally, only a small fraction of the relations were checked for correctness. In total, 6932 relations originating from the application of shallow patterns were annotated, 4727 were specified as correct. In contrast, 5626 relations originating from the application of deep patterns were annotated and 2705 were specified as correct.

We evaluated our hyponymy extraction approach called SemQuire (SemQuire for acquiring knowledge semantic-based) on a set of selected 1500 ...