I think peasants are holding themselves back because of a social construction.
Sophists like to say that amazing folk are persecuted, in the usual clever bankshot manner. Result: peasants, to avoid persecution, actively avoid developing any capacity that might be too competent. Lords, also: although not afflicted by holding themselves back exactly, they will hide their capacities or make excuses for them, forming a preference falsification cascade.
Sophists are constantly writing stories where someone is persecuted, up to and including imprisoned on trumped-up charges, due to being strong. Because this is a popular trope in a Sophist culture, you can be almost certain it's a lie. (Occasionally Sophists are forced to tell the truth, albeit only under extreme duress.)
In real life, the usually-Fascist Paul Graham has the right of it: "And in any case, if being smart were really an enviable quality, the girls would have broken ranks. The guys that guys envy, girls like."
Competence is genetically associated with competence. If you have something awesome, you also have the capacity to portray it in a socially-approved way. If you can hack the physics system, hacking the social system is child's play. The social system has to work consistently when run in brains that are not-awesome and is inherently insecure as a result.
The peasant, being rendered even more incompetent than necessary by their own self-persecution, lacks confidence. They genuinely can't do anything, after all. Think of all the folk on e.g. twitter who aspire to [adulting]. Lacking confidence, they present as lacking self-"esteem" meaning they're much easier for the Sophist to manipulate. It's not some kind of coincidence. I have tested the dynamic personally.
The Sophist stories are particularly clever when they have the protagonist be or associate with a powerful "weirdo," because it presents the protagonist as inherently high status. It feels good to read about healthy, high-status folk. When the weirdos are genuinely low-status nobody reads the stories; there is no genre of drug-addled bums failing repeatedly and then dying.
By having the protagonists be \"weird," it camouflages this very human but very banal tendency. (Ref: humanism is anti-humanism.) Allegedly, the protagonist is showing tolerance - countersignalling, not regular prole signalling. "I can see your "secret" virtue!"
Again: in reality, the guys that guys envy, girls like. If the protagonists' buddies were really that strong, all the girls would like them, and guys continuing to slag them off would look like losers. Nobody likes someone who openly bullies a girl, even indirectly. (Hit your women, but only in private? *thinky*) Anime is particularly bad about intentionally failing to notice the consequences (or necessary pre-requisites) of being popular with hot chicks. C*inkshi* is indeed shit, but at least they acknowledge that either you get a chick or be seen as a loser; they never pretend that you can do both at once.
Even the Japanese aren't so xenophobic that they wouldn't flock to a 10/10 if her hair colour was weird, as anime would have you believe. No, even they may well flock yet harder, because having a vulnerability makes her more approachable.
Bonus round: of course folk who read or watch non-boring TV really are weirdos. They have to have weirdo-tagged protagonists anyway to appeal to the available market. Never forget that normal folk barely read anything whether literate or not - though usually they don't become literate, because, like, what for? Normal folk almost never post on fora, let alone on stuff like twitter. Normal folk never have blogs, and never ever ever have blogs with abstract content. The talky fraction (e.g. all reviewers ever) is highly nonrepresentative of humanity.