Whether it’s chickens and sheep, fantasy animals, or alien creatures from another planet, characters in a novel are likely to depend on some kind of species for food — if not their meat, then their milk or eggs, or other secondary products like hide, wool, horns, and so forth. Which means it might be useful to take a moment to consider what makes an animal suitable for domestication, and how we’ve tended to relate to them.
To begin with, I should note that domestication is not the same thing as taming. A tame animal has been socialized to accept human contact and control, but you have to go through the socialization process all over again with that animal’s offspring. Domestication is the process of permanently changing a population of animals over a long period of time, selectively breeding them to have more of the qualities humans like, and fewer of the ones we don’t.
The former set of characteristics depends on what use the animal is being put to. If you’re eating its meat and the wild ancestor is small, you breed it to be bigger. If you’re gathering its wool, you breed it to be fluffier, or to have a coat composed more of the types of fiber you want, with less of the types you don’t want (like coarser overhairs). We’ve bred for more abundant milk and egg production. We also breed for docility, for a willingness to follow the lead of a human or a herd dog, and so forth.
On the flip side of the coin, we definitely want to reduce their weaponry. Smaller teeth, smaller horns (or no horns at all) — it means the animals can’t defend themselves as well, but that’s a trade-off we historically accept in exchange for them not hurting us as often. Really big animals we’ve often bred to be smaller, i.e. more manageable by humans. We breed out aggression and territoriality, at least to some degree, unless that particular strain is being shaped for a purpose where those qualities are desired, as with certain breeds of dog.
But it isn’t a free field when it comes to transforming a wild species into one human beings can make use of. Globally, there’s a very limited set of large animals which have been successfully domesticated. In Guns, Germs, and Steel, Jared Diamond separates these into the Major Five (sheep, goats, cattle, pigs, and horses), which have played a significant role around the world, and the Minor Nine (Arabian camels, Bactrian camels, llamas and alpacas, donkeys, reindeer, water buffalo, yaks, Bali cattle, and gayals), whose use has been more limited in scope or geography.
Why those fourteen? Why not American bison, or zebras, or grizzly bears, or cheetahs?
Because all of those animals have one more more characteristics which make them really bad candidates for domestication. As Diamond points out, carnivores are less useful than herbivores or omnivores, because the conversion rate from food mass to animal mass is about ten to one. You also want an animal that matures fairly quickly; the historian Bret Devereaux has speculated that the slow maturation of elephants is part of the reason they never became a lasting feature of Mediterranean or Chinese armies.
Furthermore, you need them to breed in captivity. Even in the modern day, vicuña wool has to be harvested from wild animals, because it’s more or less impossible to make them reproduce under human control. The same is true of cheetahs: they’re amazing hunters and can absolutely be tamed, but it took until the 1960s for zoos to successfully manage the birth of a cheetah cub. If you can’t get a breeding population going, you don’t really have a domesticated species.
Then there’s the matter of disposition raised above. Aggression is only one of the problems a would-be domesticator might face — though it’s a significant one, and the reason why you don’t see zebras being ridden as commonly as their equine relatives. (Zebras are apparently vicious bastards, and nobody has figured out how to breed that out of them.) You also need your livestock to be relatively placid . . . because if they aren’t, then they’ll either bolt when something freaks them out, or bash themselves to death against their enclosures. Deer panic; reindeer don’t. Now see which one we’ve domesticated.
Finally, it helps if the wild species is gregarious and hierarchical. The former allows you to keep them in herds, and the latter helps you control them: either by controlling their leader, or by getting them to recognize you as their leader. Along with this, you don’t want them to be too rabidly territorial, to the extent that they’ll kill or run off any interlopers of their own species.
These requirements aren’t rigid. They depend in part on what the animal is being bred for, and also how large it is; smaller animals can be controlled more easily. Cats and ferrets are both employed as solitary hunters, so aggression is fine up to a point, and territoriality isn’t too much of an issue. (Then again, I’ve seen it persuasively argued that cats are not actually domesticated. They aren’t meaningfully different in either morphology or behavior from their wild relatives — which is why they go feral so easily.) And certainly once you get something like genetic engineering involved, you can potentially pull off changes in a single generation that would be impossible to achieve the slow way. Even without advanced technology, we have hybrids of coyotes and dogs, or bison and cattle, which bring some of the traits of the wild relative into the domesticated line. But if you decide to write about an army mounted on zebras or moose, you’ve stepped well away from the normal bounds of plausibility.
Which means that if you’re making up an animal to fill these kinds of roles in an alien or fantasy world . . . odds are good that they’ll wind up seeming a lot like horses or sheep with a different name. In speculative fiction critique, a common phrase for this is “calling a rabbit a smeerp” — but in this instance, smeerps may be somewhat unavoidable. If it’s a large animal bred for riding and draft or for meat and milk and fiber, then it will probably have many of the physical and behavioral characteristics of horses and sheep, simply because those are the characteristics that make such creatures useful to us. So long as the people herding these animals have requirements like ours, they too will gravitate towards species that look a great deal like our livestock.