Magnetic skyrmions are topological magnetic structures, which exhibit quasi-particle properties and can show enhanced stability against perturbation from thermal noise. Recently, thermal Brownian diffusion of these quasi-particles has been found in continuous films and applications in unconventional computing have received significant attention, which however require structured elements. Thus, as the next necessary step, we here study skyrmion diffusion in confined geometries and find it to be qualitatively different: The diffusion is governed by the interplay between the total number of skyrmions and the structure geometry. In particular, we ascertain the effect of circular and triangular geometrical confinement and find that for triangular geometries the behavior is drastically different for the cases when the number of skyrmions in the element is either commensurate or incommensurate with a symmetric filling of the element. This influence of commensurability is corroborated by simulations of a quasi-particle model.