Why is it harder for males to mate with females if the species is dimorphic?
You’ve got it backwards. In polygamous species where males face alot of competition for access to mates, larger body size is one (of several) traits that can offer a selective advantage… and thus can get passed on to future offspring of successful males. The dimorphism is a result of mating pressure, not the other way around. Think about the two examples of baboons and orangutans.