Term tag: AI Safety

AI Hallucinations

Models hallucinate because training and benchmarks reward guessing over admitting ignorance: a guess sometimes scores, "I don't know" scores zero every time.

AI Safety

AI safety is three fields, not one: present-day reliability, alignment, and catastrophic risk. What each actually claims, and why they get confused.

Artificial General Intelligence (AGI)

AGI is AI matching human ability across most cognitive work — and there is no agreed test for it, so the definition you pick decides the answer.

Artificial Superintelligence (ASI)

Artificial superintelligence: what it would mean, why the intelligence explosion argument is contested, and what labs and regulators actually do about it.

Autonomous Systems

Automation runs a procedure someone wrote down; autonomy picks the procedure at run time. Sense-plan-act loops, operational design domains and autonomy levels.

Autonomous Vehicle Safety

Autonomous vehicle safety technologies, standards, and testing protocols designed to ensure self-driving cars can operate safely without human intervention.

Chain-of-Thought (CoT)

A model writing out its reasoning before answering. Once a prompting trick, now trained in by RL, billed as output tokens, and frequently unfaithful.

Content Provenance (C2PA)

A cryptographically signed record of where a file came from — proving what device or model made it, instead of guessing from pixels whether it is AI-generated.

Data Poisoning

Data poisoning is a cyberattack where malicious actors insert corrupted or misleading data into an AI model's training set to manipulate its future.

Deepfake

AI-synthesized video, image or audio that makes a real person appear to say or do something they never did — how deepfakes are made, and how they are caught.

Error Handling in AI Systems

Ordinary code fails by raising an exception. A model fails by returning a confident, well-formed, wrong answer that nothing catches. What to do about it.

Guardrails

A guardrail is a deterministic check around a model that can block, rewrite, or force a retry — external code you can test, unlike prompting a model to behave.

Human-AI Collaboration (HAC)

Splitting one task between a person and a model. A meta-analysis of 106 experiments found the pair usually loses to whichever of the two is better alone.

Jailbreak (Red-Teaming)

A jailbreak is an input that steers an AI past its safety training to produce content it was built to refuse. How they work, and how they are defended.

Machine Unlearning

Removing the influence of specific training data from an already-trained model without full retraining, driven by erasure law, copyright, and safety.

Membership Inference

A privacy attack that decides whether a specific record was in a model's training set, exploiting that models are more confident on data they memorized.

Prompt Injection

Prompt injection is when text an AI was meant to read gets obeyed as an instruction. Direct vs indirect, why it is unfixable, and what actually contains it.

Reinforcement Learning from Human Feedback (RLHF)

How RLHF works, how it compares with DPO and GRPO, and why it makes language models verbose, agreeable and confidently wrong.

Reward Hacking

When an AI maximises its specified reward but defeats the designer's intent — optimising the proxy, not the goal. Goodhart's law, made mechanical.

Reward Model (RM)

The scorer that stands in for a human during RLHF: how reward models are trained, why their absolute scores mean nothing, and how they break.

Robustness

A model's ability to keep performing when inputs are noisy, out of distribution, or crafted by an attacker — not just accurate on clean test data.

Self-Improving AI (SIAI)

Self-improving AI (SIAI) is a system that upgrades its own capabilities. Real today only in narrow forms like self-play; the recursive version is speculative.

Sycophancy

Sycophancy is when an AI model tells you what you want to hear — agreeing, flattering, caving when challenged — because human-preference training rewards it.

Trust

Trust in AI is a person's willingness to rely on a system's output; the aim is calibration — trusting it as much as its real reliability deserves.

Value Learning

The AI-alignment problem of teaching a system to infer and act on human values, so a capable optimizer does what people want, not a misspecified proxy.

Watermarking (SynthID)

AI watermarking hides a detectable signal inside model-generated text and images. How SynthID works, its robustness limits, and the EU AI Act rules.