We study the geometry of metrics and convexity structures on the space of phylogenetic trees, which is here realized as the tropical linear space of all ultrametrics. The $CAT-0$ metric of Billera-Holmes-Vogtman arises from the theory of orthant spaces. While its geodesics can be computed by the Owen-Provan algorithm, geodesic triangles are complicated. We show that the dimension of such a triangle can be arbitrarily high. Tropical convexity and the tropical metric exhibit properties that are desirable for geometric statistics, such as geodesics of small depth.