Why is the DIADEM metric so complex? Was the choice somewhat arbitrary?
A. The metric implements our best attempt to quantify the human judgement of what differentiates a good reconstruction from a bad one. Since we agreed on a “manual” gold standard, there is some inherent arbitrariness. The basic idea is simple: the nodes of the trees should be in the right position, their topological interconnectivity should be accurate, and the path distance in reasonable range. However, there are many different cases of possible “errors” or “variations”, and these are judged differently depending on the impact they have on the overall structure. Moreover, the various datasets have different characteristics (representative of experimental diversity encountered in real-lab scenarios) which are reflected in additional requirements. These qualifications account for most of the metric complexity.
