What AI cannot do — and the myths that follow
Lesson 5 of 5 in What Is AI? From Rules to Reasoning Machines.
To govern something well you must know where it breaks. Modern AI's failures are as characteristic as its successes, and each one maps to a governance control you will meet later in the path.
It breaks when the world shifts. Models learn the statistics of their training data; when reality moves on — new slang, new fraud patterns, a pandemic — accuracy decays silently. No error message fires. This is distribution shift, its operational symptom is model drift, and it is the reason every framework demands post-deployment monitoring rather than one-time approval.
It fails on the rare and the underrepresented. Performance concentrates where data was plentiful. The landmark Gender Shades study found commercial face analysis erring on darker-skinned women at rates dozens of times higher than on lighter-skinned men — a data-representation failure, discovered only because someone measured it by subgroup. Hence disaggregated testing in every serious evaluation regime.
It computes without understanding. A model can write a flawless essay on fire safety and still tell you to store petrol next to a heater — there is no common-sense floor beneath the fluency. Capability in one slice of a task does not imply competence across it; governance must test what the system actually does, not what its fluency implies it knows.
Myth: “AI is objective — it removes human bias”
AI is consistent, which is different. A model trained on biased decisions reproduces the bias with perfect uniformity, at scale, with a veneer of mathematical neutrality that makes it harder to challenge than a biased human. Objectivity has to be engineered and evidenced — via representative data, disaggregated testing, and bias audits — never assumed from the fact that a machine did it.
Myth: “It can’t be explained, so it can’t be governed”
Full mechanistic transparency is impossible for deep models — and yet aviation governs jet engines whose fluid dynamics no one can solve exactly. Governance works through behavioural evidence: testing, monitoring, documentation, explainability approximations, and process rights for affected people. Opacity constrains how you govern; it is never an excuse not to.
Myth: “More data always fixes bias”
More of the same data entrenches the same skew. Fixing representation failures takes deliberately different data — and some bias lives in the labels or the objective itself, where no volume of examples helps. Sometimes the fix is changing what you ask the model to optimise.
Myth: “The AI decided — no one is responsible”
Every framework you will study exists to make the opposite true. Systems have providers and deployers; duties attach to each; and "the computer did it" has failed as a defence in courtrooms from the Dutch childcare-benefits scandal to Moffatt v. Air Canada (the airline was held to the refund policy its chatbot invented). Accountability is the one principle with no technical exception.
Myth: “It talks like a person, so it thinks like one”
Humans anthropomorphise fluent language reflexively — researchers call it the ELIZA effect, after a 1966 chatbot whose users confided in 200 lines of pattern-matching code. Fluency is a property of the training objective, not evidence of understanding, intention, or reliability. Calibrated trust — not affection, not fear — is the professional stance this entire curriculum builds toward.
Tool: Is It AI? — Ready to pressure-test the definition? Judge 18 real systems against Article 3(1), element by element.