By Tag
- Axiom of Choice Definition (Intuitive)
Definition of the Axiom of Choice, without using heavy mathematical notation.
- Mark Chimes
- Bayes' rule: Guide
The Arbital guide to Bayes' rule
- Eliezer Yudkowsky
- Paul Christiano's AI control blog
Speculations on the design of safe, efficient AI systems.
- Paul Christiano
- Averting instrumental pressures
Almost-any utility function for an AI, whether the target is diamonds or paperclips or eudaimonia, implies subgoals like rapidly self-improving and refusing to shut down. Can we make that not happen?
- Eliezer Yudkowsky - Averting the convergent instrumental strategy of self-improvement
We probably want the first AGI to *not* improve as fast as possible, but improving as fast as possible is a convergent strategy for accomplishing most things.
- Eliezer Yudkowsky - Conservative concept boundary
Given N example burritos, draw a boundary around what is a 'burrito' that is relatively simple and allows as few positive instances as possible. Helps make sure the next thing generated is a burrito.
- Eliezer Yudkowsky - Corrigibility
"I can't let you do that, Dave."
- Nate Soares - Diamond maximizer
How would you build an agent that made as much diamond material as possible, given vast computing power but an otherwise rich and complicated environment?
- Eliezer Yudkowsky - Identifying ambiguous inductions
What do a "red strawberry", a "red apple", and a "red cherry" have in common that a "yellow carrot" doesn't? Are they "red fruits" or "red objects"?
- Eliezer Yudkowsky - Look where I'm pointing, not at my finger
When trying to communicate the concept "glove", getting the AGI to focus on "gloves" rather than "my user's decision to label something a glove" or "anything that depresses the glove-labeling button".
- Eliezer Yudkowsky - Low impact
The open problem of having an AI carry out tasks in ways that cause minimum side effects and change as little of the rest of the universe as possible.
- Eliezer Yudkowsky - Mild optimization
An AGI which, if you ask it to paint one car pink, just paints one car pink and doesn't tile the universe with pink-painted cars, because it's not trying *that* hard to max out its car-painting score.
- Eliezer Yudkowsky - Non-adversarial principle
At no point in constructing an Artificial General Intelligence should we construct a computation that tries to hurt us, and then try to stop it from hurting us.
- Eliezer Yudkowsky - Ontology identification problem
How do we link an agent's utility function to its model of the world, when we don't know what that model will look like?
- Eliezer Yudkowsky - Open subproblems in aligning a Task-based AGI
Open research problems, especially ones we can model today, in building an AGI that can "paint all cars pink" without turning its future light cone into pink-painted cars.
- Eliezer Yudkowsky - Other-izing (wanted: new optimization idiom)
Maximization isn't possible for bounded agents, and satisficing doesn't seem like enough. What other kind of 'izing' might be good for realistic, bounded agents?
- Eliezer Yudkowsky - Problem of fully updated deference
Why moral uncertainty doesn't stop an AI from defending its off-switch.
- Eliezer Yudkowsky - Safe impact measure
What can we measure to make sure an agent is acting in a safe manner?
- Eliezer Yudkowsky - Shutdown problem
How to build an AGI that lets you shut it down, despite the obvious fact that this will interfere with whatever the AGI's goals are.
- Eliezer Yudkowsky
- Meta tags
What are meta tags and when to use them?
- Eliezer Yudkowsky
- Arbital page summaries Markdown syntax
How to create page summaries using Arbital's Markdown syntax.
- Alexei Andreev
- Project proposal: Intro to numbers
Should Arbital's first "project" be a guide to numbers?
- Eric Rogstad
- Mic-Ra-finance and the illusion of control
This post discusses the following claims:
* [claim([6th])]
* [claim([6tk])]
* [claim([6tl])]
- Alexei Andreev
- Coherent extrapolated volition (alignment target)
A proposed direction for an extremely well-aligned autonomous superintelligence - do what humans would want, if we knew what the AI knew, thought that fast, and understood ourselves.
- Eliezer Yudkowsky
- 'Rationality' of voting in elections
"A single vote is very unlikely to swing the election, so your vote is unlikely to have an effect" versus "Many people similar to you are making a similar decision about whether to vote."
- Eliezer Yudkowsky - 99LDT x 1CDT oneshot PD tournament as arguable counterexample to LDT doing better than CDT
Arguendo, if 99 LDT agents and 1 CDT agent are facing off in a one-shot Prisoner's Dilemma tournament, the CDT agent does better on a problem that CDT considers 'fair'.
- Eliezer Yudkowsky - A whirlwind tour
A rapid tour of Eric's thoughts on the accelerator project.
- Eric Bruylant - Absent-Minded Driver dilemma
A road contains two identical intersections. An absent-minded driver wants to turn right at the second intersection. "With what probability should the driver turn right?" argue decision theorists.
- Eliezer Yudkowsky - Accelerator Project
The Accelerator Project aims to create a low-cost environment which facilitates rapid personal growt…
- Eric Bruylant - Arbital
Arbital is the place for crowdsourced, intuitive math explanations.
- Alexei Andreev - Arbital lens
A lens is a page that presents another page's content from a different angle.
- Alexei Andreev - Arbital: Google Maps for knowledge
Take your understanding from where it is to where it wants to be.
- Alexei Andreev - Arbital: learning from Wikipedia
How is Arbital different from Wikipedia?
- Alexei Andreev - Associative operation
An **associative operation** $\bullet : X \times X \to X$ is a binary operation such that for all $x…
- Nate Soares - Associativity: Examples
Yes: [Addition], [multiplication], string concatenation. No: [subtraction], [division], a Function …
- Nate Soares - Associativity: Intuition
Associative functions can be interpreted as families of functions that reduce lists down to a singl…
- Nate Soares - Bayes' rule
Bayes' rule is the core theorem of probability theory saying how to revise our beliefs when we make a new observation.
- Eliezer Yudkowsky - Bayes' rule: Beginner's guide
Beginner's guide to learning about Bayes' rule.
- Alexei Andreev - Bayes' rule: Functional form
Bayes' rule for to continuous variables.
- Eliezer Yudkowsky - Bayes' rule: Log-odds form
A simple transformation of Bayes' rule reveals tools for measuring degree of belief, and strength of evidence.
- Eliezer Yudkowsky - Bayes' rule: Odds form
The simplest and most easily understandable form of Bayes' rule uses relative odds.
- Eliezer Yudkowsky - Bayes' rule: Probability form
The original formulation of Bayes' rule.
- Nate Soares - Bayesian view of scientific virtues
Why is it that science relies on bold, precise, and falsifiable predictions? Because of Bayes' rule, of course.
- Eliezer Yudkowsky - Belief revision as probability elimination
Update your beliefs by throwing away large chunks of probability mass.
- Eliezer Yudkowsky - Bit
The term "bit" refers to different concepts in different fields. The common theme across all the us…
- Nate Soares - Coherent decisions imply consistent utilities
Why do we all use the 'expected utility' formalism? Because any behavior that can't be viewed from that perspective, must be qualitatively self-defeating (in various mathy ways).
- Eliezer Yudkowsky - Commutativity: Intuition
We can think of commutativity either as an artifact of notation, or as a symmetry in the output of a…
- Nate Soares - Contributing to Arbital
Want to help Arbital become awesome?
- Eric Bruylant - Cyclic Group Intro (Math 0)
A finite cyclic group is a little bit like a clock.
- Mark Chimes - Death in Damascus
Death tells you that It is coming for you tomorrow. You can stay in Damascus or flee to Aleppo. Whichever decision you actually make is the wrong one. This gives some decision theories trouble.
- Eliezer Yudkowsky - Derivative
How things change
- Michael Cohen - Diseasitis
20% of patients have Diseasitis. 90% of sick patients and 30% of healthy patients turn a tongue depressor black. You turn a tongue depressor black. What's the chance you have Diseasitis?
- Eliezer Yudkowsky - Exchange rates between digits
In terms of data storage, if a coin is worth $1, a digit wheel is worth more than $3.32, but less than $3.33. Why?
- Nate Soares - Extraordinary claims require extraordinary evidence
The people who adamantly claim they were abducted by aliens do provide some evidence for aliens. They just don't provide quantitatively enough evidence.
- Eliezer Yudkowsky - Finishing your Bayesian path on Arbital
The page that comes at the end of reading the Arbital Guide to Bayes' rule
- Eliezer Yudkowsky - Fractional digits
When $b$ and $x$ are integers, $\log_b(x)$ has a few good interpretations. It's roughly the length o…
- Nate Soares - Frequency diagrams: A first look at Bayes
The most straightforward visualization of Bayes' rule.
- Nate Soares - Group
The algebraic structure that captures symmetry, relationships between transformations, and part of what multiplication and addition have in common.
- Nate Soares - High-speed intro to Bayes's rule
A high-speed introduction to Bayes's Rule on one page, for the impatient and mathematically adept.
- Eliezer Yudkowsky - Interest in mathematical foundations in Bayesianism
"Want" this requisite if you prefer to see extra information about the mathematical foundations in Bayesianism.
- Eliezer Yudkowsky - Introduction to Bayes' rule: Odds form
Bayes' rule is simple, if you think in terms of relative odds.
- Eliezer Yudkowsky - Introductory guide to logarithms
Welcome to the Arbital introduction to logarithms! In modern education, logarithms are often mention…
- Nate Soares - Isomorphism: Intro (Math 0)
Things which are basically the same, except for some stuff you don't care about.
- Mark Chimes - Join and meet
Let $\langle P, \leq \rangle$ be a poset, and let $S \subseteq P$. The **join** of $S$ in $P$, deno…
- Kevin Clancy - Life in logspace
The log lattice hints at the reason that engineers, scientists, and AI researchers find logarithms s…
- Nate Soares - Log as generalized length
To estimate the log (base 10) of a number, count how many digits it has.
- Nate Soares - Log as the change in the cost of communicating
When interpreting logarithms as a generalization of the notion of "length" and as digit exchange rat…
- Nate Soares - Mathematical induction
Proving a statement about all positive integers by knocking them down like dominoes.
- Douglas Weathers - Odds form to probability form
The odds form of Bayes' rule works for any two hypotheses $H_i$ and $H_j,$ and looks like this:
$$\…
- Nate Soares - Partially ordered set
A set endowed with a relation that is reflexive, transitive, and antisymmetric.
- Kevin Clancy - Path: Multiple angles on Bayes's Rule
A learning-path placeholder page for learning multiple angles on Bayes's Rule.
- Eliezer Yudkowsky - Probability distribution: Motivated definition
People keep writing things like P(sick)=0.3. What does this mean, on a technical level?
- Nate Soares - Probability notation for Bayes' rule: Intro (Math 1)
How to read, and identify, the probabilities in Bayesian problems.
- Eliezer Yudkowsky - Project outline: Intro to the Universal Property
Outline detailing all the work required for a proposed Arbital Project
- Eric Rogstad - Proof of Bayes' rule
Proofs of Bayes' rule, with graphics
- Eliezer Yudkowsky - Proof of Bayes' rule: Probability form
Let $\mathbf H$ be a [random\_variable variable] in $\mathbb P$ for the true hypothesis, and let $H_…
- Nate Soares - Proof of Rice's theorem
A standalone proof of Rice's theorem, including one surprising lemma.
- Patrick Stevens - Properties of the logarithm
- $\log_b(x \cdot y) = \log_b(x) + \log_b(y)$ for any $b$, this is the defining characteristic of …
- Nate Soares - Rice's Theorem
Rice's Theorem tells us that if we want to determine pretty much anything about the behaviour of an arbitrary computer program, we can't in general do better than just running it.
- Patrick Stevens - Shift towards the hypothesis of least surprise
When you see new evidence, ask: which hypothesis is *least surprised?*
- Nate Soares - Strictly confused
A hypothesis is strictly confused by the raw data, if the hypothesis did much worse in predicting it than the hypothesis itself expected.
- Eliezer Yudkowsky - The End (of the basic log tutorial)
That concludes our introductory tutorial on logarithms! You have made it to the end.
Throughout thi…
- Nate Soares - The characteristic of the logarithm
Any time you find an output that adds whenever the input multiplies, you're probably looking at a (…
- Nate Soares - The log lattice
Log as the change in the cost of communicating and other pages give physical interpretations of what…
- Nate Soares - The missing step between Zero and Hero
Creating a space for high potential people grow and improve the world at scale.
- Eric Bruylant - Uncountability: Intuitive Intro
Are all sizes of infinity the same? What does "the same" even mean here?
- Jason Gross - Universal property of the empty set
The empty set can be characterised by how it interacts with other sets, rather than by any explicit property of the empty set itself.
- Patrick Stevens - Universal property of the product
The product can be defined in a very general way, applicable to the natural numbers, to sets, to algebraic structures, and so on.
- Patrick Stevens - Utility function
The only coherent way of wanting things is to assign consistent relative scores to outcomes.
- Eliezer Yudkowsky - Waterfall diagram
Visualizing Bayes' rule as the mixing of probability streams.
- Eliezer Yudkowsky - Waterfall diagrams and relative odds
A way to visualize Bayes' rule that yields an easier way to solve some problems
- Eliezer Yudkowsky - Welcome to Arbital
Front page explaining what Arbital is all about.
- Alexei Andreev - What is a logarithm?
Logarithms are a group of functions that take a number as input and produce another number. There i…
- Nate Soares
- Distant superintelligences can coerce the most probable environment of your AI
Distant superintelligences may be able to hack your local AI, if your AI's preference framework depends on its most probable environment.
- Eliezer Yudkowsky - Modeling distant superintelligences
The several large problems that might occur if an AI starts to think about alien superintelligences.
- Eliezer Yudkowsky
- Isomorphism: Intro (Math 0)
Things which are basically the same, except for some stuff you don't care about.
- Mark Chimes
- Arbital Markdown
All about Arbital's extended Markdown syntax.
- Alexei Andreev - Arbital projects
Arbital projects are small-scale drives to fill in areas of content.
- Eric Bruylant - Arbital scope
What kind of content is Arbital looking for?
- Eric Bruylant - Arbital user groups
Users can attain different powers and responsibilities on Arbital.
- Eric Bruylant - Arbital: fixing online discussion
How can Arbital do better than existing discussion platforms?
- Alexei Andreev - Arithmetical hierarchy
The arithmetical hierarchy is a way of classifying logical statements by the number of clauses saying "for every object" and "there exists an object".
- Eliezer Yudkowsky - Arithmetical hierarchy: If you don't read logic
The arithmetical hierarchy is a way of stratifying statements by how many "for every number" and "th…
- Eliezer Yudkowsky - Author's guide to Arbital
How to write intuitive, flexible content on Arbital.
- Alexei Andreev - Axiom
An **axiom** of a [theory\_mathematics theory] $T$ is a [well\_formed well-formed] [sentence\_mathem…
- Eric Bruylant - Bayes' rule: Definition
Bayes' rule is the mathematics of probability theory governing how to update your beliefs in the lig…
- Nate Soares - Bayes' rule: Proportional form
The fastest way to say something both convincing and true about belief-updating.
- Eliezer Yudkowsky - Bayes' rule: Vector form
For when you want to apply Bayes' rule to lots of evidence and lots of variables, all in one go. (This is more or less how spam filters work.)
- Eliezer Yudkowsky - Bit (of data)
A bit of data is the amount of data required to single out one message from a set of two. Equivalen…
- Nate Soares - Bézout's theorem
Bézout's theorem is an important link between highest common factors and the integer solutions of a certain equation.
- Patrick Stevens - Category theory
How mathematical objects are related to others in the same category.
- Mark Chimes - Ceiling
The ceiling of a real number $x,$ denoted $\lceil x \rceil$ or sometimes $\operatorname{ceil}(x),$ i…
- Nate Soares - Conditional probability
The notation for writing "The probability that someone has green eyes, if we know that they have red hair."
- Eliezer Yudkowsky - Conditional probability: Refresher
Is P(yellow | banana) the probability that a banana is yellow, or the probability that a yellow thing is a banana?
- Nate Soares - Disjoint union of sets
One of the most basic ways we have of joining two sets together.
- Patrick Stevens - Division of rational numbers (Math 0)
"Division" is the idea of "dividing something up among some people so that we can give equal amounts to each person".
- Patrick Stevens - Edge instantiation
When you ask the AI to make people happy, and it tiles the universe with the smallest objects that can be happy.
- Eliezer Yudkowsky - Elementary Algebra
How do we describe relations between different things? How can we figure out new true things from tr…
- Adele Lopez - Empirical probabilities are not exactly 0 or 1
"Cromwell's Rule" says that probabilities of exactly 0 or 1 should never be applied to empirical propositions - there's always some probability, however tiny, of being mistaken.
- Eliezer Yudkowsky - Expected value
Trying to assign value to an uncertain state? The weighted average of outcomes is probably the tool you need.
- Michael Cohen - Explicit Bayes as a counter for 'worrying'
Explicitly walking through Bayes's Rule can summarize your knowledge and thereby stop you from bouncing around pieces of it.
- Eliezer Yudkowsky - Extraordinary claims
What makes something an 'extraordinary claim' that requires extraordinary evidence?
- Eliezer Yudkowsky - Factorial
The *factorial* of a number $n$ is how we describe "how many different ways we can arrange $n$ obje…
- Patrick Stevens - Featured math content
Some Arbital pages we think are great!
- Eric Bruylant - Frequency diagram
Visualizing Bayes' rule by manipulating frequencies in large populations
- Nate Soares - Generalized principle of cognitive alignment
When we're asking how we want the AI to think about an alignment problem, one source of inspiration is trying to have the AI mirror our own thoughts about that problem.
- Eliezer Yudkowsky - Goodhart's Curse
The Optimizer's Curse meets Goodhart's Law. For example, if our values are V, and an AI's utility function U is a proxy for V, optimizing for high U seeks out 'errors'--that is, high values of U - V.
- Eliezer Yudkowsky - Group isomorphism
"Isomorphism" is the proper notion of "sameness" or "equality" among groups.
- Patrick Stevens - Integers: Intro (Math 0)
The integers are the whole numbers extended into the negatives.
- Joe Zeng - Interruptibility
A subproblem of corrigibility under the machine learning paradigm: when the agent is interrupted, it must not learn to prevent future interruptions.
- Eliezer Yudkowsky - Isomorphism
A morphism between two objects which describes how they are "essentially equivalent" for the purposes of the theory under consideration.
- Mark Chimes - Lambda calculus
A minimal, inefficient, and hard-to-read, but still interesting and useful, programming language.
- Dylan Hendrickson - Laplace's Rule of Succession
Suppose you flip a coin with an unknown bias 30 times, and see 4 heads and 26 tails. The Rule of Succession says the next flip has a 5/32 chance of showing heads.
- Eliezer Yudkowsky - Likelihood function
Let's say you have a piece of evidence $e$ and a set of hypotheses $\mathcal H.$ Each $H_i \in \math…
- Nate Soares - Limited AGI
Task-based AGIs don't need unlimited cognitive and material powers to carry out their Tasks; which means their powers can potentially be limited.
- Eliezer Yudkowsky - Modal combat
Modal combat
- Jaime Sevilla Molina - Mutually exclusive and exhaustive
The condition needed for probabilities to sum to 1
- Eliezer Yudkowsky - Odds: Introduction
What's the difference between probabilities and odds? Why is a 20% probability of success equivalent to 1 : 4 odds favoring success?
- Nate Soares - Odds: Technical explanation
Formal definitions, alternate representations, and uses of odds and odds ratios (like a 1 : 2 chance of drawing a red ball vs. green ball from a barrel).
- Alexei Andreev - Operations in Set theory
An operation in set theory is a Function of two sets, that returns a set.
Common set operations inc…
- M Yass - Operator
An operation $f$ on a set $S$ is a function that takes some values from $S$ and produces a new value…
- Nate Soares - Ordinary claims require ordinary evidence
Extraordinary claims require extraordinary evidence, but ordinary claims *don't*.
- Nate Soares - Parfit's Hitchhiker
You are dying in the desert. A truck-driver who is very good at reading faces finds you, and offers to drive you into the city if you promise to pay $1,000 on arrival. You are a selfish rationalist.
- Eliezer Yudkowsky - Posterior probability
What we believe, after seeing the evidence and doing a Bayesian update.
- Eliezer Yudkowsky - Probability
The degree to which someone believes something, measured on a scale from 0 to 1, allowing us to do math to it.
- Eliezer Yudkowsky - Proportion
A representation of a value as a fraction or multiple of another value.
- Joe Zeng - Rational arithmetic all works together
The various operations of arithmetic all play nicely together in a certain specific way.
- Patrick Stevens - Real numbers are uncountable
The real numbers are uncountable.
- Eric Bruylant - Set
An unordered collection of distinct objects.
- Nate Soares - Solomonoff induction: Intro Dialogue (Math 2)
An introduction to Solomonoff induction for the unfamiliar reader who isn't bad at math
- Eliezer Yudkowsky - Style guidelines
Various stylistic conventions people should follow on Arbital
- Alexei Andreev - Subjective probability
Probability is in the mind, not in the environment. If you don't know whether a coin came up heads or tails, that's a fact about you, not a fact about the coin.
- Eliezer Yudkowsky - The plan experiment
Root page for describing the reason and the process for planning how to approach and navigate through AGI development.
- Alexei Andreev - Transparent Newcomb's Problem
Omega has left behind a transparent Box A containing $1000, and a transparent Box B containing $1,000,000 or nothing. Box B is full iff Omega thinks you one-box on seeing a full Box B.
- Eliezer Yudkowsky - Turing machine
A Turing Machine is a simple mathematical model of computation that is powerful enough to describe any computation a computer can do.
- Eric Leese - Ultimatum Game
A Proposer decides how to split $10 between themselves and the Responder. The Responder can take what is offered, or refuse, in which case both parties get nothing.
- Eliezer Yudkowsky - Uncountability
Some infinities are bigger than others. Uncountable infinities are larger than countable infinities.
- Jason Gross - Uncountability (Math 3)
Formal definition of uncountability, and foundational considerations.
- Patrick Stevens - Uncountability: Intro (Math 1)
Not all infinities are created equal. The infinity of real numbers is infinitely larger than the infinity of counting numbers.
- Jason Gross - Universal property of the disjoint union
Just as the empty set may be described by a universal property, so too may the disjoint union of sets.
- Patrick Stevens - Whole number
A term that can refer to three different sets of "numbers that are not fractions".
- Joe Zeng
- Isomorphism
A morphism between two objects which describes how they are "essentially equivalent" for the purposes of the theory under consideration.
- Mark Chimes - Morphism
A morphism is the abstract representation of a relation between mathematical objects.
Usually, it i…
- Jaime Sevilla Molina
- Central examples
List of central examples in Value Alignment Theory domain.
- Eliezer Yudkowsky
- Value-laden
Cure cancer, but avoid any bad side effects? Categorizing "bad side effects" requires knowing what's "bad". If an agent needs to load complex human goals to evaluate something, it's "value-laden".
- Eliezer Yudkowsky
- Countability
Some infinities are bigger than others. Countable infinities are the smallest infinities.
- Alexei Andreev - Crony belief
**Crony belief** is a concept originally introduced in Kevin Simler's post, "Crony Beliefs". It's us…
- Alexei Andreev - Donor lottery
An arrangement where a group of people pool their money and pick one person to give it away.
- Alexei Andreev - Logical decision theories
Root page for topics on logical decision theory, with multiple intros for different audiences.
- Eliezer Yudkowsky - Odds
Odds express a relative probability.
- Eliezer Yudkowsky - Outside view
Taking the **outside view** (another name for reference class forecasting) means using an estimate b…
- Alexei Andreev - Uncountability
Some infinities are bigger than others. Uncountable infinities are larger than countable infinities.
- Jason Gross
- Correlated coverage
In which parts of AI alignment can we hope that getting many things right, will mean the AI gets everything right?
- Eliezer Yudkowsky - Low impact
The open problem of having an AI carry out tasks in ways that cause minimum side effects and change as little of the rest of the universe as possible.
- Eliezer Yudkowsky
- Convergent instrumental strategies
Paperclip maximizers can make more paperclips by improving their cognitive abilities or controlling more resources. What other strategies would almost-any AI try to use?
- Eliezer Yudkowsky
- Modular arithmetic
Addition as traveling around a circle, instead of along a line.
- Malcolm McCrimmon
- Indirect decision theory
In which I argue that understanding decision theory can be delegated to AI.
### Indirect normativit…
- Paul Christiano
- 'Concept'
In the context of Artificial Intelligence, a 'concept' is a category, something that identifies thingies as being inside or outside the concept.
- Eliezer Yudkowsky - Algorithmic complexity
When you compress the information, what you are left with determines the complexity.
- Eliezer Yudkowsky - Alternating group
The alternating group is the only normal subgroup of the symmetric group (on five or more generators).
- Patrick Stevens - Arity (of a function)
The arity of a function is the number of parameters that it takes. For example, the function $f(a, b…
- Nate Soares - Bijective function
A bijective function is a function with an inverse.
- Patrick Stevens - Closure
A set $S$ is _closed_ under an operation $f$ if, whenever $f$ is fed elements of $S$, it produces an…
- Nate Soares - Codomain (of a function)
The codomain $\operatorname{cod}(f)$ of a function $f : X \to Y$ is $Y$, the set of possible outputs…
- Nate Soares - Development phase unpredictable
Several proposed problems in advanced safety are alleged to be difficult because they depend on some…
- Eliezer Yudkowsky - Dihedral group
The dihedral groups are natural examples of groups, arising from the symmetries of regular polygons.
- Patrick Stevens - Domain (of a function)
The domain $\operatorname{dom}(f)$ of a function $f : X \to Y$ is $X$, the set of valid inputs for t…
- Nate Soares - Image (of a function)
The image $\operatorname{im}(f)$ of a function $f : X \to Y$ is the set of all possible outputs of $…
- Nate Soares - Injective function
A Function $f: X \to Y$ is *injective* if it has the property that whenever $f(x) = f(y)$, it is the…
- Patrick Stevens - Instrumental
What is "instrumental" in the context of Value Alignment Theory?
- Eliezer Yudkowsky - Intended goal
Definition. An "intended goal" refers to the intuitive intention in the mind of a human programmer …
- Eliezer Yudkowsky - Kernel of group homomorphism
The kernel of a Group homomorphism $f: G \to H$ is the collection of all elements $g$ in $G$ such th…
- Patrick Stevens - Likelihood notation
The likelihood of a piece of evidence $e$ according to a hypothesis $H,$ known as "the likelihood of…
- Nate Soares - Linguistic conventions in value alignment
How and why to use precise language and words with special meaning when talking about value alignment.
- Eliezer Yudkowsky - Modalized modal sentence
A [ modal sentence] $A$ is said to be **modalized** in $p$ if every occurrence of $p$ happens within…
- Jaime Sevilla Molina - Natural number
The numbers we use to count: 0, 1, 2, 3, ...
- Jaime Sevilla Molina - Normal subgroup
Normal subgroups are subgroups which are in some sense "the same from all points of view".
- Patrick Stevens - Object-level vs. indirect goals
Difference between "give Alice the apple" and "give Alice what she wants".
- Eliezer Yudkowsky - Order of a group
The order $|G|$ of a group $G$ is the size of its underlying set. For example, if $G=(X,\bullet)$ an…
- Nate Soares - Pivotal event
Which types of AIs, if they work, can do things that drastically change the nature of the further game?
- Eliezer Yudkowsky - Preference framework
What's the thing an agent uses to compare its preferences?
- Eliezer Yudkowsky - Programmer
Who is building these advanced agents?
- Eliezer Yudkowsky - Range (of a function)
The "range" of a function is an ambiguous term that is generally used to refer to either the functio…
- Nate Soares - Set builder notation
$\{ 2n \mid n \in \mathbb N \}$ denotes the set of all even numbers, using set builder notation. Set…
- Nate Soares - Sign homomorphism (from the symmetric group)
The sign homomorphism is how we extract the alternating group from the symmetric group.
- Patrick Stevens - Simple group
The simple groups form the "building blocks" of group theory, analogously to the prime numbers in number theory.
- Patrick Stevens - String (of text)
A string (of text) is a series of letters (often denoted by quote marks), such as `"abcd"` or `"hell…
- Nate Soares - Strong cognitive uncontainability
An advanced agent can win in ways humans can't understand in advance.
- Eliezer Yudkowsky - Surjective function
A surjective function is one which "hits everything in the codomain".
- Patrick Stevens - Transposition (as an element of a symmetric group)
A transposition is the simplest kind of permutation: it swaps two elements.
- Patrick Stevens - Utility
What is "utility" in the context of Value Alignment Theory?
- Eliezer Yudkowsky - Value
The word 'value' in the phrase 'value alignment' is a metasyntactic variable that indicates the speaker's future goals for intelligent life.
- Eliezer Yudkowsky - n-message
A message singling out one thing from a set of $n$ is sometimes called an $n$-message. For example,…
- Nate Soares
- Ontology identification problem
How do we link an agent's utility function to its model of the world, when we don't know what that model will look like?
- Eliezer Yudkowsky
- Bit
The term "bit" refers to different concepts in different fields. The common theme across all the us…
- Nate Soares - Whole number
A term that can refer to three different sets of "numbers that are not fractions".
- Joe Zeng
- Coherent extrapolated volition (alignment target)
A proposed direction for an extremely well-aligned autonomous superintelligence - do what humans would want, if we knew what the AI knew, thought that fast, and understood ourselves.
- Eliezer Yudkowsky
- Low impact
The open problem of having an AI carry out tasks in ways that cause minimum side effects and change as little of the rest of the universe as possible.
- Eliezer Yudkowsky
- A $1 donation to a top animal charity alleviates more suffering than is caused by a day of eating meat.
For the purposes of this claim, top animal welfare charities include:
- [Animal Charity Evaluators…
- Eric Rogstad - Ethics Offsets to the Rescue
Hate hurting animals, but love eating meat? Throw money at the problem!
- Eric Rogstad - For most EA-Blank projects, we would expect more good to be done if they would: i) disband or ii) remove EA from the name and aim to outgrow the EA movement.
The claim refers to projects like:
* Effective Altruism Forum
* Effective Altruism Handbook
* Effec…
- Ryan Carey - Fundraisers should have a threshold amount which, if not hit, results in a refund.
When starting a fundraiser, a nonprofit should declare a threshold amount. If the nonprofit doesn't …
- Alexei Andreev - Growing the EA movement is net positive - Eric Rogstad
- If EA leaders with similar values disagree about how the EA movement should be branded, then they should discuss in detail the subquestions that would cause them to change their minds if they have not already done so. - Ryan Carey
- If they spent 100x longer deciding where to donate, then most effective altruists would choose targets with much higher expected impact.
Does analysis help?
- Ryan Carey - Kickstarter project is a better tool for fundraising a threshold amount of money to start an EA project than a donor charity - Alexei Andreev
- On the margin, effective altruist researchers and leaders should carry out more empirical investigation of strategic questions.
Strategic question might include:
* How can we shape the development of brain-computer interfaces?
…
- Ryan Carey - The current message of effective altruism heavily discourages creativity.
Alyssa Vance expands on this point in her [FB post](https://www.facebook.com/alyssamvance/posts/1021…
- Alexei Andreev - When I donate to a charity, I am concerned whether or not the charity will raise enough money to make my donation worthwhile. - Alexei Andreev
- Universal property of the empty set
The empty set can be characterised by how it interacts with other sets, rather than by any explicit property of the empty set itself.
- Patrick Stevens
- Blue oysters
A probability problem about blue oysters.
- Nate Soares - Diseasitis
20% of patients have Diseasitis. 90% of sick patients and 30% of healthy patients turn a tongue depressor black. You turn a tongue depressor black. What's the chance you have Diseasitis?
- Eliezer Yudkowsky - Lattice: Examples
Here are some additional examples of lattices. $\newcommand{\nsubg}{\mathcal N \mbox{-} Sub~G}$
A f…
- Kevin Clancy - Sock-dresser search
There's a 4/5 chance your socks are in one of your dresser's 8 drawers. You check 6 drawers at random. What's the probability they'll be in the next drawer you check?
- Nate Soares - Sparking widgets
10% of widgets are bad and 90% are good. 4% of good widgets emit sparks, and 12% of bad widgets emit…
- Nate Soares
- Rescuing the utility function
If your utility function values 'heat', and then you discover to your horror that there's no ontologically basic heat, switch to valuing disordered kinetic energy. Likewise 'free will' or 'people'.
- Eliezer Yudkowsky
- Group: Exercises
Test your understanding of the definition of a group with these exercises.
- Qiaochu Yuan - Join and meet: Exercises
Try these exercises to test your knowledge of joins and meets.
Tangled up
--------------------
!…
- Kevin Clancy - Lattice: Exercises
Try these exercises to test your knowledge of lattices.
## Distributivity
Does the lattice meet op…
- Kevin Clancy - Logarithm: Exercises
Without using a calculator: What is $\log_{10}(4321)$? What integer is it larger than, what integer …
- Nate Soares - Poset: Exercises
Try these exercises to test your poset knowledge.
# Corporate Ladder
Imagine a company with five …
- Kevin Clancy
- Orbit-Stabiliser theorem: External Resources
External resources on the Orbit-Stabiliser theorem.
- Mark Chimes - Turing machine: External resources
* [Wikipedia](https://en.wikipedia.org/wiki/Turing_machine)
* [Wolfram MathWorld](http://mathworld.w…
- Eric Bruylant
- Extraordinary claims require extraordinary evidence
The people who adamantly claim they were abducted by aliens do provide some evidence for aliens. They just don't provide quantitatively enough evidence.
- Eliezer Yudkowsky
- You can't get more paperclips that way
Most arguments that "A paperclip maximizer could get more paperclips by (doing nice things)" are flawed.
- Eliezer Yudkowsky
- Algebraic structure
Roughly speaking, an algebraic structure is a set $X$, known as the underlying set, paired with a co…
- Nate Soares - Conjugacy class
In a group, the elements can be partitioned naturally into certain classes.
- Patrick Stevens - Equaliser (category theory)
In Category theory, an *equaliser* of a pair of arrows $f, g: A \to B$ is an object $E$ and a univer…
- Patrick Stevens - Field structure of rational numbers
In which we describe the field structure on the rationals.
- Patrick Stevens - Group coset
Given a subgroup $H$ of Group $G$, the *left cosets* of $H$ in $G$ are sets of the form $\{ gh : h \…
- Patrick Stevens - Group orbit
When we have a group acting on a set, we are often interested in how the group acts on a particular …
- Adele Lopez - Identity element
An element in a set with a binary operation that leaves every element unchanged when used as the other operand.
- Joe Zeng - Iff
If and only if...
- Alexei Andreev - Inverse function
The inverse of a function returns an input of the original function when fed the original's corresponding output.
- Michael Cohen - Order of a group element
Given an element $g$ of group $(G, +)$ (which henceforth we abbreviate simply as $G$), the order of …
- Patrick Stevens - Order relation
A way of determining which elements of a set come "before" or "after" other elements.
- Joe Zeng - Ordered field
An ordered ring with division.
- Joe Zeng - Prime number
The prime numbers are the "building blocks" of the counting numbers.
- Patrick Stevens - Relation
A **relation** is a set of [tuple\_mathematics tuples], all of which have the same [tuple\_arity ar…
- Kevin Clancy - Stabiliser (of a group action)
If a group acts on a set, it is useful to consider which elements of the group don't move a certain element of the set.
- Patrick Stevens - Transitive relation
If a is related to b and b is related to c, then a is related to c.
- Dylan Hendrickson - Union
The union of two sets is the set of elements which are in one or the other, or both
- M Yass
- Category theory
How mathematical objects are related to others in the same category.
- Mark Chimes
- Hypercomputer
Some formalisms demand computers larger than the limit of all finite computers
- Eliezer Yudkowsky - Infrahuman, par-human, superhuman, efficient, optimal
A categorization of AI ability levels relative to human, with some gotchas in the ordering. E.g., in simple domains where humans can play optimally, optimal play is not superhuman.
- Eliezer Yudkowsky - Instrumental
What is "instrumental" in the context of Value Alignment Theory?
- Eliezer Yudkowsky - Pivotal event
Which types of AIs, if they work, can do things that drastically change the nature of the further game?
- Eliezer Yudkowsky - Programmer
Who is building these advanced agents?
- Eliezer Yudkowsky - Utility
What is "utility" in the context of Value Alignment Theory?
- Eliezer Yudkowsky - Value
The word 'value' in the phrase 'value alignment' is a metasyntactic variable that indicates the speaker's future goals for intelligent life.
- Eliezer Yudkowsky
- Edge instantiation
When you ask the AI to make people happy, and it tiles the universe with the smallest objects that can be happy.
- Eliezer Yudkowsky - Goodhart's Curse
The Optimizer's Curse meets Goodhart's Law. For example, if our values are V, and an AI's utility function U is a proxy for V, optimizing for high U seeks out 'errors'--that is, high values of U - V.
- Eliezer Yudkowsky
- Category theory
How mathematical objects are related to others in the same category.
- Mark Chimes
- Isomorphism
A morphism between two objects which describes how they are "essentially equivalent" for the purposes of the theory under consideration.
- Mark Chimes
- Pivotal event
Which types of AIs, if they work, can do things that drastically change the nature of the further game?
- Eliezer Yudkowsky
- Bayes' rule: Guide
The Arbital guide to Bayes' rule
- Eliezer Yudkowsky - Guide to Logical Decision Theory
The entry point for learning about logical decision theory.
- Eliezer Yudkowsky - Introductory guide to logarithms
Welcome to the Arbital introduction to logarithms! In modern education, logarithms are often mention…
- Nate Soares
- High-speed intro to Bayes's rule
A high-speed introduction to Bayes's Rule on one page, for the impatient and mathematically adept.
- Eliezer Yudkowsky - Odds: Refresher
A quick review of the notations and mathematical behaviors for odds (e.g. odds of 1 : 2 for drawing a red ball vs. green ball from a barrel).
- Nate Soares
- Realistic (Math 1)
Real-life examples of Bayesian reasoning
- Eliezer Yudkowsky
- Value-laden
Cure cancer, but avoid any bad side effects? Categorizing "bad side effects" requires knowing what's "bad". If an agent needs to load complex human goals to evaluate something, it's "value-laden".
- Eliezer Yudkowsky
- Addition of rational numbers (Math 0)
The simplest operation on rational numbers is addition.
- Patrick Stevens
- Bijective Function: Intro (Math 0)
Two boxes are bijective if they contain the same number of items.
- Mark Chimes
- Mic-Ra-finance and the illusion of control
This post discusses the following claims:
* [claim([6th])]
* [claim([6tk])]
* [claim([6tl])]
- Alexei Andreev
- Ability to read algebra
Do you have sufficient mathematical ability that you can read a sentence that uses some algebra or invokes a mathematical idea, without slowing down too much?
- Eliezer Yudkowsky - Ability to read calculus
Can you take integral signs and differentiations in stride?
- Eliezer Yudkowsky - Ability to read logic
Can you read sentences symbolically stating "For all x: exists y: phi(x, y) or not theta(y)" without slowing down too much?
- Eliezer Yudkowsky - Blue oysters
A probability problem about blue oysters.
- Nate Soares - Math 0
Are you not actively bad at math, nor traumatized about math?
- Eliezer Yudkowsky - Math 1
Is math sometimes fun for you, and are you not anxious if you see a math puzzle you don't know how to solve?
- Eliezer Yudkowsky - Math 2
Do you work with math on a fairly routine basis? Do you have little trouble grasping abstract structures and ideas?
- Eliezer Yudkowsky - Math 3
Can you read the sort of things that professional mathematicians read, aka LaTeX formulas with a minimum of explanation?
- Eliezer Yudkowsky - Path: Insights from Bayesian updating
A learning-path placeholder page for insights derived from the Bayesian rule for updating beliefs.
- Eliezer Yudkowsky - Path: Multiple angles on Bayes's Rule
A learning-path placeholder page for learning multiple angles on Bayes's Rule.
- Eliezer Yudkowsky - Sock-dresser search
There's a 4/5 chance your socks are in one of your dresser's 8 drawers. You check 6 drawers at random. What's the probability they'll be in the next drawer you check?
- Nate Soares - Sparking widgets
10% of widgets are bad and 90% are good. 4% of good widgets emit sparks, and 12% of bad widgets emit…
- Nate Soares - Wants to get straight to Bayes
A simple requisite page to mark whether the user has selected wanting to get straight into Bayes on …
- Eliezer Yudkowsky
- Behaviorist genie
An advanced agent that's forbidden to model minds in too much detail.
- Eliezer Yudkowsky
- Central examples
List of central examples in Value Alignment Theory domain.
- Eliezer Yudkowsky - Orbit-Stabiliser theorem: External Resources
External resources on the Orbit-Stabiliser theorem.
- Mark Chimes
- Identifying causal goal concepts from sensory data
If the intended goal is "cure cancer" and you show the AI healthy patients, it sees, say, a pattern of pixels on a webcam. How do you get to a goal concept *about* the real patients?
- Eliezer Yudkowsky
- Odds: Introduction
What's the difference between probabilities and odds? Why is a 20% probability of success equivalent to 1 : 4 odds favoring success?
- Nate Soares
- Addition of rational numbers (Math 0)
The simplest operation on rational numbers is addition.
- Patrick Stevens - Arithmetic of rational numbers (Math 0)
How do we combine rational numbers together?
- Patrick Stevens - Bijective Function: Intro (Math 0)
Two boxes are bijective if they contain the same number of items.
- Mark Chimes - Cyclic Group Intro (Math 0)
A finite cyclic group is a little bit like a clock.
- Mark Chimes - Division of rational numbers (Math 0)
"Division" is the idea of "dividing something up among some people so that we can give equal amounts to each person".
- Patrick Stevens - Integers: Intro (Math 0)
The integers are the whole numbers extended into the negatives.
- Joe Zeng - Isomorphism: Intro (Math 0)
Things which are basically the same, except for some stuff you don't care about.
- Mark Chimes - Subtraction of rational numbers (Math 0)
In which we meet anti-apples.
- Patrick Stevens - Uncountability: Intuitive Intro
Are all sizes of infinity the same? What does "the same" even mean here?
- Jason Gross
- Bit (of data)
A bit of data is the amount of data required to single out one message from a set of two. Equivalen…
- Nate Soares - Combining vectors
One of the most useful things we can do with vectors is to combine them!
- Adele Lopez - Derivative
How things change
- Michael Cohen - Proof by contradiction
Discover what 'reductio ad absurdum' means!
- Jaime Sevilla Molina - Rice's Theorem: Intro (Math 1)
You can't write a program that looks at another programs source code, and tells you whether it computes the Fibonacci sequence.
- Dylan Hendrickson - Vector arithmetic
Vectors: what they are, and how to add and scale them.
- Adele Lopez
- Binary function
A binary function $f$ is a function of two inputs (i.e., a function with arity 2). For example, $+,$…
- Nate Soares - Bézout's theorem
Bézout's theorem is an important link between highest common factors and the integer solutions of a certain equation.
- Patrick Stevens - Ceiling
The ceiling of a real number $x,$ denoted $\lceil x \rceil$ or sometimes $\operatorname{ceil}(x),$ i…
- Nate Soares - Group conjugate
Conjugation lets us perform permutations "from the point of view of" another permutation.
- Patrick Stevens - Group isomorphism
"Isomorphism" is the proper notion of "sameness" or "equality" among groups.
- Patrick Stevens - Identity element
An element in a set with a binary operation that leaves every element unchanged when used as the other operand.
- Joe Zeng - Join and meet
Let $\langle P, \leq \rangle$ be a poset, and let $S \subseteq P$. The **join** of $S$ in $P$, deno…
- Kevin Clancy - List
A list is an ordered collection of objects, such as `[0, 1, 2, 3]` or `["red", "blue", 0, "shoe"]`. …
- Nate Soares - Mutually exclusive and exhaustive
The condition needed for probabilities to sum to 1
- Eliezer Yudkowsky - Operator
An operation $f$ on a set $S$ is a function that takes some values from $S$ and produces a new value…
- Nate Soares - Partially ordered set
A set endowed with a relation that is reflexive, transitive, and antisymmetric.
- Kevin Clancy - Probability
The degree to which someone believes something, measured on a scale from 0 to 1, allowing us to do math to it.
- Eliezer Yudkowsky - Rice's Theorem
Rice's Theorem tells us that if we want to determine pretty much anything about the behaviour of an arbitrary computer program, we can't in general do better than just running it.
- Patrick Stevens - Underlying set
What do a Group, a Partially ordered set, and a [ topological space] have in common? Each is a Set …
- Nate Soares
- Every group is a quotient of a free group
Given a group $G$, there is a Free group $F(X)$ on some set $X$, such that $G$ is isomorphic to some…
- Patrick Stevens - Formal definition of the free group
Van der Waerden's trick lets us define the free groups in a slick but mostly incomprehensible way.
- Patrick Stevens - Group presentation
Presentations are a fairly compact way of expressing groups.
- Patrick Stevens
- Needs motivation
A tag for text that could benefit from some motivating statements. Why is the reader interested in w…
- Eric Rogstad - Thought experiment
Meta-tag for thought experiments.
- Nate Soares
- C-Class
This page has substantial content, but may not thoroughly cover the topic, may not meet style and prose standards, or may not explain the concept in a way the target audience will reliably understand.
- Eric Bruylant - Needs brief summary
Meta tag for pages which need a brief summary.
- Eric Bruylant - Needs clickbait
This page does not have clickbait (a short teaser for the page displayed on various lists). Feel free to add it!
- Eric Bruylant
- Meta-rules for (narrow) value learning are still unsolved
We don't currently know a simple meta-utility function that would take in observation of humans and spit out our true values, or even a good target for a Task AGI.
- Eliezer Yudkowsky
- Goodhart's Curse
The Optimizer's Curse meets Goodhart's Law. For example, if our values are V, and an AI's utility function U is a proxy for V, optimizing for high U seeks out 'errors'--that is, high values of U - V.
- Eliezer Yudkowsky
- Behaviorist genie
An advanced agent that's forbidden to model minds in too much detail.
- Eliezer Yudkowsky
- Isomorphism
A morphism between two objects which describes how they are "essentially equivalent" for the purposes of the theory under consideration.
- Mark Chimes
- Low impact
The open problem of having an AI carry out tasks in ways that cause minimum side effects and change as little of the rest of the universe as possible.
- Eliezer Yudkowsky - Mindcrime
Might a machine intelligence contain vast numbers of unhappy conscious subprocesses?
- Eliezer Yudkowsky
- Codomain (of a function)
The codomain $\operatorname{cod}(f)$ of a function $f : X \to Y$ is $Y$, the set of possible outputs…
- Nate Soares - Löb's theorem
Löb's theorem
- Jaime Sevilla Molina
- Decimal notation
The winning architecture for numerals
- Michael Cohen - Group
The algebraic structure that captures symmetry, relationships between transformations, and part of what multiplication and addition have in common.
- Nate Soares
- Algebraic structure
Roughly speaking, an algebraic structure is a set $X$, known as the underlying set, paired with a co…
- Nate Soares - Arbital page alias
The alias is a short, unique name assigned to each page. For example: "arbital_alias".
The alias u…
- Eric Rogstad - Arithmetical hierarchy: If you don't read logic
The arithmetical hierarchy is a way of stratifying statements by how many "for every number" and "th…
- Eliezer Yudkowsky - Arity (of a function)
The arity of a function is the number of parameters that it takes. For example, the function $f(a, b…
- Nate Soares - Associative operation
An **associative operation** $\bullet : X \times X \to X$ is a binary operation such that for all $x…
- Nate Soares - Associativity vs commutativity
Associativity and commutativity are often confused, because they are both constraints on how a funct…
- Nate Soares - Associativity: Intuition
Associative functions can be interpreted as families of functions that reduce lists down to a singl…
- Nate Soares - Bag
In mathematics, a "bag" is an unordered list. A bag differs from a set in that it can contain the sa…
- Nate Soares - Binary function
A binary function $f$ is a function of two inputs (i.e., a function with arity 2). For example, $+,$…
- Nate Soares - Cartesian product
The Cartesian product of two sets $A$ and $B,$ denoted $A \times B,$ is the set of all [ordered\_pai…
- Nate Soares - Cauchy's theorem on subgroup existence: intuitive version
Cauchy's Theorem states that if $G$ is a finite [-group], and $p$ is a prime dividing the order of $…
- Patrick Stevens - Ceiling
The ceiling of a real number $x,$ denoted $\lceil x \rceil$ or sometimes $\operatorname{ceil}(x),$ i…
- Nate Soares - Closure
A set $S$ is _closed_ under an operation $f$ if, whenever $f$ is fed elements of $S$, it produces an…
- Nate Soares - Codomain (of a function)
The codomain $\operatorname{cod}(f)$ of a function $f : X \to Y$ is $Y$, the set of possible outputs…
- Nate Soares - Codomain vs image
It is useful to distinguish codomain from image both (a) when the type of thing that the function pr…
- Nate Soares - Commutative operation
A commutative function $f$ is a function that takes multiple inputs from a set $X$ and produces an o…
- Nate Soares - Commutativity: Examples
Yes: addition, multiplication, maximum, minimum, rock-paper-scissors. No: subtraction, division, st…
- Nate Soares - Commutativity: Intuition
We can think of commutativity either as an artifact of notation, or as a symmetry in the output of a…
- Nate Soares - Complex number
A complex number is a number of the form $z = a + b\textrm{i}$, where $\textrm{i}$ is the imaginary …
- Eliana Ruby - Domain (of a function)
The domain $\operatorname{dom}(f)$ of a function $f : X \to Y$ is $X$, the set of valid inputs for t…
- Nate Soares - Function
Intuitively, a function $f$ is a procedure (or machine) that takes an input and performs some opera…
- Nate Soares - Function: Physical metaphor
Many functions can be visualized as physical mechanisms of wheels and gears, that take their inputs …
- Nate Soares - Generalized associative law
Given an associative operator $\cdot$ and a list $[a, b, c, \ldots]$ of parameters, all ways of red…
- Nate Soares - Group coset
Given a subgroup $H$ of Group $G$, the *left cosets* of $H$ in $G$ are sets of the form $\{ gh : h \…
- Patrick Stevens - Group orbit
When we have a group acting on a set, we are often interested in how the group acts on a particular …
- Adele Lopez - Image (of a function)
The image $\operatorname{im}(f)$ of a function $f : X \to Y$ is the set of all possible outputs of $…
- Nate Soares - Information theory
The study (and quantificaiton) of information, and its communication and storage.
- Nate Soares - Injective function
A Function $f: X \to Y$ is *injective* if it has the property that whenever $f(x) = f(y)$, it is the…
- Patrick Stevens - Integer
An **integer** is a Number that can be represented as either a Natural number or its [-additive\_inv…
- Michael Cohen - Kernel of group homomorphism
The kernel of a Group homomorphism $f: G \to H$ is the collection of all elements $g$ in $G$ such th…
- Patrick Stevens - Likelihood
"Likelihood", when speaking of Bayesian reasoning, denotes *the probability of an observation, sup…
- Nate Soares - Logarithms invert exponentials
The function $\log_b(\cdot)$ inverts the function $b^{(\cdot)}.$ In other words, $\log_b(n) = x$ imp…
- Nate Soares - Logical system
Logical systems (a.k.a. formal systems) are mathematical abstractions that aim to capture the notion…
- Jaime Sevilla Molina - Monoid
A monoid $M$ is a pair $(X, \diamond)$ where $X$ is a [set\_theory\_set set] and $\diamond$ is an [a…
- Nate Soares - Odds form to probability form
The odds form of Bayes' rule works for any two hypotheses $H_i$ and $H_j,$ and looks like this:
$$\…
- Nate Soares - Order of a group
The order $|G|$ of a group $G$ is the size of its underlying set. For example, if $G=(X,\bullet)$ an…
- Nate Soares - Order of a group element
Given an element $g$ of group $(G, +)$ (which henceforth we abbreviate simply as $G$), the order of …
- Patrick Stevens - Proof of Bayes' rule: Probability form
Let $\mathbf H$ be a [random\_variable variable] in $\mathbb P$ for the true hypothesis, and let $H_…
- Nate Soares - Ring
A ring is a kind of Algebraic structure which we obtain by considering groups as being "things with…
- Nate Soares - Set
An unordered collection of distinct objects.
- Nate Soares - Shannon
The shannon (Sh) is a unit of Information. One shannon is the difference in [info\_entropy entropy] …
- Nate Soares - Underlying set
What do a Group, a Partially ordered set, and a [ topological space] have in common? Each is a Set …
- Nate Soares
- Chesterton's fence
If someone did something, it's generally good to understand their reasons for doing it before undoing it.
- Eric Bruylant
- Isomorphism
A morphism between two objects which describes how they are "essentially equivalent" for the purposes of the theory under consideration.
- Mark Chimes
- Addition of rational numbers (Math 0)
The simplest operation on rational numbers is addition.
- Patrick Stevens - Cartesian product
The Cartesian product of two sets $A$ and $B,$ denoted $A \times B,$ is the set of all [ordered\_pai…
- Nate Soares - Category theory
How mathematical objects are related to others in the same category.
- Mark Chimes - Proportion
A representation of a value as a fraction or multiple of another value.
- Joe Zeng
- Algebraic structure
Roughly speaking, an algebraic structure is a set $X$, known as the underlying set, paired with a co…
- Nate Soares - Exponential
Any function that constantly gets larger as a proportion of itself.
- Joe Zeng - Function
Intuitively, a function $f$ is a procedure (or machine) that takes an input and performs some opera…
- Nate Soares - How many bits to a trit?
$\log_2(3) \approx 1.585.$ This can be interpreted a few different ways:
1. If you multiply the nu…
- Nate Soares - Logarithmic identities
- [ Inversion of exponentials]: $b^{\log_b(n)} = \log_b(b^n) = n.$
- [ Log of 1 is 0]: $\log_b(1) …
- Nate Soares
- Arbital page
The Arbital is a series of pages.
- Alexei Andreev - Arithmetical hierarchy
The arithmetical hierarchy is a way of classifying logical statements by the number of clauses saying "for every object" and "there exists an object".
- Eliezer Yudkowsky - Arithmetical hierarchy: If you don't read logic
The arithmetical hierarchy is a way of stratifying statements by how many "for every number" and "th…
- Eliezer Yudkowsky
- Binary notation
A way to write down numbers using powers of two.
- Malcolm McCrimmon - Boolean
A value in logic that evaluates to either "true" or "false".
- Malcolm McCrimmon - Diagonal lemma
Constructing self-referential sentences
- Jaime Sevilla Molina - Freely reduced word
"Freely reduced" captures the idea of "no cancellation" in a free group.
- Patrick Stevens - Greatest common divisor
The greatest common divisor of two natural numbers is… the largest number which is a divisor of both. The clue is in the name, really.
- Patrick Stevens - Gödel's first incompleteness theorem
The theorem that destroyed Hilbert's program
- Jaime Sevilla Molina - Logistic function
A monotonic function from the real numbers to the open unit interval.
- Joe Zeng - Modular arithmetic
Addition as traveling around a circle, instead of along a line.
- Malcolm McCrimmon - Ordered field
An ordered ring with division.
- Joe Zeng - Provability predicate
A provability predicate of a theory $T$ is a formula $P(x)$ with one free variable $x$ such that:
…
- Jaime Sevilla Molina - The n-th root of m is either an integer or irrational
In other words, no power of a rational number that is not an integer is ever an integer.
- Joe Zeng
- Cardinality
The "size" of a set, or the "number of elements" that it has.
- Joe Zeng - Convex set
A set that contains all line segments between points in the set
- Jessica Taylor
- Ackermann function
The slowest-growing fast-growing function.
- Alex Appel - Advanced nonagent
Hypothetically, cognitively powerful programs that don't follow the loop of "observe, learn, model the consequences, act, observe results" that a standard "agent" would.
- Eliezer Yudkowsky - Arbital hidden text
How to hide text in Markdown behind a button.
- Alexei Andreev - Church-Turing thesis
A thesis about computational models
- Jaime Sevilla Molina - Convex function
A function that only curves upward
- Jessica Taylor - Convex set
A set that contains all line segments between points in the set
- Jessica Taylor - Extraordinary claims require extraordinary evidence
The people who adamantly claim they were abducted by aliens do provide some evidence for aliens. They just don't provide quantitatively enough evidence.
- Eliezer Yudkowsky - Fractional bits
It takes $\log_2(8) = 3$ bits of data to carry one message from a set of 8 possible messages. Simila…
- Nate Soares - Introductory Bayesian problems
Bayesian problems to try to solve yourself, before beginning to learn about Bayes' rule.
- Eliezer Yudkowsky - Likelihood
"Likelihood", when speaking of Bayesian reasoning, denotes *the probability of an observation, sup…
- Nate Soares - Löb's theorem
Löb's theorem
- Jaime Sevilla Molina - Normal system of provability logic
Between the modal systems of provability, the normal systems distinguish themselves by exhibiting ni…
- Jaime Sevilla Molina - Posterior probability
What we believe, after seeing the evidence and doing a Bayesian update.
- Eliezer Yudkowsky - Prior probability
What we believed before seeing the evidence.
- Eliezer Yudkowsky - Real number
A **real number** is any number that can be used to represent a physical quantity.
Intuitively, rea…
- Michael Cohen - Realistic (Math 1)
Real-life examples of Bayesian reasoning
- Eliezer Yudkowsky - Set product
A fundamental way of combining sets is to take their product, making a set that contains all tuples of elements from the originals.
- Patrick Stevens - Stabiliser (of a group action)
If a group acts on a set, it is useful to consider which elements of the group don't move a certain element of the set.
- Patrick Stevens - Strong Church Turing thesis
A strengthening of the Church Turing thesis
- Jaime Sevilla Molina - Symmetric group
The symmetric groups form the fundamental link between group theory and the notion of symmetry.
- Patrick Stevens - There is only one logarithm
All logarithm functions are the same, up to a multiplicative constant.
- Nate Soares - Totally ordered set
A set where all the elements can be compared as greater than or less than.
- Joe Zeng
- Axiom of Choice
The most controversial axiom of the 20th century.
- Mark Chimes - Edge instantiation
When you ask the AI to make people happy, and it tiles the universe with the smallest objects that can be happy.
- Eliezer Yudkowsky - Project proposal: Intro to the Universal Property
Proposal for one of the first Arbital Projects.
- Patrick Stevens
- Omnipotence test for AI safety
Would your AI produce disastrous outcomes if it suddenly gained omnipotence and omniscience? If so, why did you program something that *wants* to hurt you and is held back only by lacking the power?
- Eliezer Yudkowsky
- Nick Bostrom's book Superintelligence
The current best book-form introduction to AI alignment theory.
- Eliezer Yudkowsky
- Corrigibility
"I can't let you do that, Dave."
- Nate Soares
- Colon-to notation
Find out what the notation "f : X -> Y" means that everyone keeps using.
- Qiaochu Yuan - GalCom
In the GalCom thought experiment, you live in the future, and make your money by living in the Dene…
- Nate Soares - Intradependent encoding
An encoding $E(m)$ of a message $m$ is intradependent if the fact that $E(m)$ encodes $m$ can be de…
- Nate Soares - Likelihood notation
The likelihood of a piece of evidence $e$ according to a hypothesis $H,$ known as "the likelihood of…
- Nate Soares - Strictly confused
A hypothesis is strictly confused by the raw data, if the hypothesis did much worse in predicting it than the hypothesis itself expected.
- Eliezer Yudkowsky - n-digit
An $n$-digit is a physical object that can be stably placed into any of $n$ distinguishable states. …
- Nate Soares - n-message
A message singling out one thing from a set of $n$ is sometimes called an $n$-message. For example,…
- Nate Soares
- Look where I'm pointing, not at my finger
When trying to communicate the concept "glove", getting the AGI to focus on "gloves" rather than "my user's decision to label something a glove" or "anything that depresses the glove-labeling button".
- Eliezer Yudkowsky
- Averting instrumental pressures
Almost-any utility function for an AI, whether the target is diamonds or paperclips or eudaimonia, implies subgoals like rapidly self-improving and refusing to shut down. Can we make that not happen?
- Eliezer Yudkowsky - Conservative concept boundary
Given N example burritos, draw a boundary around what is a 'burrito' that is relatively simple and allows as few positive instances as possible. Helps make sure the next thing generated is a burrito.
- Eliezer Yudkowsky - Corrigibility
"I can't let you do that, Dave."
- Nate Soares - Faithful simulation
How would you identify, to a Task AGI (aka Genie), the problem of scanning a human brain, and then running a sufficiently accurate simulation of it for the simulation to not be crazy or psychotic?
- Eliezer Yudkowsky - Identifying ambiguous inductions
What do a "red strawberry", a "red apple", and a "red cherry" have in common that a "yellow carrot" doesn't? Are they "red fruits" or "red objects"?
- Eliezer Yudkowsky - Identifying causal goal concepts from sensory data
If the intended goal is "cure cancer" and you show the AI healthy patients, it sees, say, a pattern of pixels on a webcam. How do you get to a goal concept *about* the real patients?
- Eliezer Yudkowsky - Informed oversight
Incentivize a reinforcement learner that's less smart than you to accomplish some task
- Jessica Taylor - Look where I'm pointing, not at my finger
When trying to communicate the concept "glove", getting the AGI to focus on "gloves" rather than "my user's decision to label something a glove" or "anything that depresses the glove-labeling button".
- Eliezer Yudkowsky - Low impact
The open problem of having an AI carry out tasks in ways that cause minimum side effects and change as little of the rest of the universe as possible.
- Eliezer Yudkowsky - Mild optimization
An AGI which, if you ask it to paint one car pink, just paints one car pink and doesn't tile the universe with pink-painted cars, because it's not trying *that* hard to max out its car-painting score.
- Eliezer Yudkowsky - Non-adversarial principle
At no point in constructing an Artificial General Intelligence should we construct a computation that tries to hurt us, and then try to stop it from hurting us.
- Eliezer Yudkowsky - Safe training procedures for human-imitators
How does one train a reinforcement learner to act like a human?
- Jessica Taylor - Shutdown problem
How to build an AGI that lets you shut it down, despite the obvious fact that this will interfere with whatever the AGI's goals are.
- Eliezer Yudkowsky
- Likelihood functions, p-values, and the replication crisis
What's the whole Bayesian-vs.-frequentist debate about?
- Eliezer Yudkowsky - Report likelihoods not p-values: FAQ
This page answers frequently asked questions about the Report likelihoods, not p-values proposal for…
- Nate Soares - Report likelihoods, not p-values
If scientists reported likelihood functions instead of p-values, this could help science avoid p-ha…
- Nate Soares
- Arbital "requires" relationship
A page can require a requisite if the reader needs to have it before they are able to understand the page.
- Alexei Andreev - Arbital "teaches" relationship
A page can teach a requisite when the user can acquire it by reading the page.
- Alexei Andreev - Arbital comment
A comment is a way for you to express your thoughts and opinions within the context of a page.
- Alexei Andreev - Arbital features
Overview of all Arbital features.
- Alexei Andreev - Arbital mark
What is a mark on Arbital? When is it created? Why is it important?
- Alexei Andreev - Arbital path
Arbital path is a linear sequence of pages tailored specifically to teach a given concept to a user.
- Alexei Andreev - Arbital requisites
To understand a thing you often need to understand some other things.
- Alexei Andreev
- You can't get more paperclips that way
Most arguments that "A paperclip maximizer could get more paperclips by (doing nice things)" are flawed.
- Eliezer Yudkowsky
- Edge instantiation
When you ask the AI to make people happy, and it tiles the universe with the smallest objects that can be happy.
- Eliezer Yudkowsky - Low impact
The open problem of having an AI carry out tasks in ways that cause minimum side effects and change as little of the rest of the universe as possible.
- Eliezer Yudkowsky - Nearest unblocked strategy
If you patch an agent's preference framework to avoid an undesirable solution, what can you expect to happen?
- Eliezer Yudkowsky
- Imitation-based agent
An AI meant to imitate the behavior of a reference human as closely as possible.
- Eliezer Yudkowsky
- Executable philosophy
Philosophical discourse aimed at producing a trustworthy answer or meta-answer, in limited time, which can used in constructing an Artificial Intelligence.
- Eliezer Yudkowsky
- Bézout's theorem
Bézout's theorem is an important link between highest common factors and the integer solutions of a certain equation.
- Patrick Stevens - Cauchy's theorem on subgroup existence
Cauchy's theorem is a useful condition for the existence of cyclic subgroups of finite groups.
- Patrick Stevens - Dihedral groups are non-abelian
The group of symmetries of the triangle and all larger regular polyhedra are not abelian.
- Patrick Stevens - Field homomorphism is trivial or injective
Field homomorphisms preserve a *lot* of structure; they preserve so much structure that they are always either injective or totally boring.
- Patrick Stevens - Group orbits partition
When a group acts on a set, the set falls naturally into distinct pieces, where the group action only permutes elements within any given piece, not between them.
- Patrick Stevens - Pi is irrational
The number pi is famously not rational, in spite of joking attempts at legislation to fix its value at 3 or 22/7.
- Patrick Stevens - Product is unique up to isomorphism
If something satisfies the universal property of the product, then it is uniquely specified by that property, up to isomorphism.
- Patrick Stevens - Proof that there are infinitely many primes
Suppose there were finitely many primes. Then consider the product of all the primes plus 1...
- Joe Zeng - Real numbers are uncountable
The real numbers are uncountable.
- Eric Bruylant - Stabiliser is a subgroup
Given a group acting on a set, each element of the set induces a subgroup of the group.
- Patrick Stevens - The n-th root of m is either an integer or irrational
In other words, no power of a rational number that is not an integer is ever an integer.
- Joe Zeng - The rationals form a field
The set $\mathbb{Q}$ of rational numbers is a field.
# Proof
$\mathbb{Q}$ is a (commutative) ring …
- Patrick Stevens - The reals (constructed as Dedekind cuts) form a field
The reals are an archetypal example of a field, but if we are to construct them from simpler objects, we need to show that our construction does indeed have the right properties.
- Patrick Stevens - The reals (constructed as classes of Cauchy sequences of rationals) form a field
The reals are an archetypal example of a field, but if we are to construct them from simpler objects, we need to show that our construction does indeed have the right properties.
- Patrick Stevens - The set of rational numbers is countable
Although there are "lots and lots" of rational numbers, there are still only countably many of them.
- Patrick Stevens - The square root of 2 is irrational
The number whose square is 2 can't be written is a quotient of natural numbers
- Dylan Hendrickson
- Bayes' rule: Log-odds form
A simple transformation of Bayes' rule reveals tools for measuring degree of belief, and strength of evidence.
- Eliezer Yudkowsky - Uncountability: Intuitive Intro
Are all sizes of infinity the same? What does "the same" even mean here?
- Jason Gross - Waterfall diagrams and relative odds
A way to visualize Bayes' rule that yields an easier way to solve some problems
- Eliezer Yudkowsky
- Bit (of data)
A bit of data is the amount of data required to single out one message from a set of two. Equivalen…
- Nate Soares - Group isomorphism
"Isomorphism" is the proper notion of "sameness" or "equality" among groups.
- Patrick Stevens - Rational arithmetic all works together
The various operations of arithmetic all play nicely together in a certain specific way.
- Patrick Stevens - Uncountability
Some infinities are bigger than others. Uncountable infinities are larger than countable infinities.
- Jason Gross
- Missing the weird alternative
People might systematically overlook "make tiny molecular smileyfaces" as a way of "producing smiles", because our brains automatically search for high-utility-to-us ways of "producing smiles".
- Eliezer Yudkowsky - Underestimating complexity of value because goodness feels like a simple property
When you just want to yell at the AI, "Just do normal high-value X, dammit, not weird low-value X!" and that 'high versus low value' boundary is way more complicated than your brain wants to think.
- Eliezer Yudkowsky
wiki
- Bayesian reasoning
A probability-theory-based view of the world; a coherent way of changing probabilistic beliefs based on evidence.
- Eliezer Yudkowsky
no-type
- Extensionality Axiom
If two sets have exactly the same members, then they are equal
- Ilia Zaichuk
- Problem of fully updated deference
Why moral uncertainty doesn't stop an AI from defending its off-switch.
- Eliezer Yudkowsky
- Shutdown problem
How to build an AGI that lets you shut it down, despite the obvious fact that this will interfere with whatever the AGI's goals are.
- Eliezer Yudkowsky
- 0.999...=1
No, it's not "infinitesimally far" from 1 or anything like that. 0.999... and 1 are literally the same number.
- Dylan Hendrickson - A googolplex
A moderately large number, as large numbers go.
- Nate Soares - Ackermann function
The slowest-growing fast-growing function.
- Alex Appel - Algebraic structure tree
When is a monoid a semilattice? What's the difference between a semigroup and a groupoid? Find out here!
- Ryan Hendrickson - An Introduction to Logical Decision Theory for Everyone Else
So like what the heck is 'logical decision theory' in terms a normal person can understand?
- Eliezer Yudkowsky - Arbital Labs
Landing page for the Arbital Labs domain.
- Alexei Andreev - Arbital external resources
Arbital wants to link users to great content, wherever it is!
- Eric Bruylant - Arbital hidden text
How to hide text in Markdown behind a button.
- Alexei Andreev - Arbital likes
What are likes? When should I use them? What happens when I like something?
- Alexei Andreev - Arbital markdown demo
Demo of Arbital's markdown
- Eric Bruylant - Arbital math levels
How mathy do you like your pages?
- Eric Bruylant - Arbital page
The Arbital is a series of pages.
- Alexei Andreev - Arbital practices
Guidelines and rules for interacting on Arbital.
- Eliezer Yudkowsky - Arbital quality
Arbital's system for tracking page quality.
- Eric Bruylant - Arbital: Do what works
When deciding things on Arbital, think about the real goals, and move towards them.
- Eric Bruylant - B-Class
This page is mostly complete and without major problems, but has not had detailed feedback from the target audience and reviewers.
- Eric Bruylant - Bayes' rule examples
Interesting problems solvable by Bayes' rule
- Eliezer Yudkowsky - Bayesian update
Bayesian updating: the ideal way to change probabilistic beliefs based on evidence.
- Eliezer Yudkowsky - Binary notation
A way to write down numbers using powers of two.
- Malcolm McCrimmon - Bit (abstract)
An abstract bit is an element of the set $\mathbb B$, which has two elements. An abstract bit is to …
- Nate Soares - Cartesian product
The Cartesian product of two sets $A$ and $B,$ denoted $A \times B,$ is the set of all [ordered\_pai…
- Nate Soares - Communication: magician example
Imagine that you and I are both magicians, performing a trick where I think of a card from a deck of…
- Nate Soares - Complex number
A complex number is a number of the form $z = a + b\textrm{i}$, where $\textrm{i}$ is the imaginary …
- Eliana Ruby - Complexity theory: Complexity zoo
Pass and see the exotic beasts coming from the lands of afar!
- Jaime Sevilla Molina - Consequentialist preferences are reflectively stable by default
Gandhi wouldn't take a pill that made him want to kill people, because he knows in that case more people will be murdered. A paperclip maximizer doesn't want to stop maximizing paperclips.
- Eliezer Yudkowsky - Convex function
A function that only curves upward
- Jessica Taylor - Convex set
A set that contains all line segments between points in the set
- Jessica Taylor - Decision problem
Formalization of general problems
- Jaime Sevilla Molina - Dependent messages can be encoded cheaply
Say you want to transmit a 2-message, a 4-message, and a 256-message to somebody. For example, you m…
- Nate Soares - Distances between cognitive domains
Often in AI alignment we want to ask, "How close is 'being able to do X' to 'being able to do Y'?"
- Eliezer Yudkowsky - Empty set
The empty set does what it says on the tin: it is the set which is empty.
- Patrick Stevens - Encoding trits with GalCom bits
There are $\log_2(3) \approx 1.585$ bits to a Trit. Why is it that particular value? Consider the Ga…
- Nate Soares - Equivalence relation
A relation that allows you to partition a set into equivalence classes.
- Dylan Hendrickson - Examination through isomorphism
Isomorphism is the correct notion of equality between objects in a category. From the category-theor…
- Luke Sciarappa - Exponential
Any function that constantly gets larger as a proportion of itself.
- Joe Zeng - Extensionality Axiom
If two sets have exactly the same members, then they are equal
- Ilia Zaichuk - Fair problem class
A problem is 'fair' (according to logical decision theory) when only the results matter and not how we get there.
- Eliezer Yudkowsky - Function
Intuitively, a function $f$ is a procedure (or machine) that takes an input and performs some opera…
- Nate Soares - Fundamental Theorem of Arithmetic
The FTA tells us that natural numbers can be decomposed uniquely into prime factors; it is the basis of almost all number theory.
- Patrick Stevens - Information
Information is a measure of how much a message grants an observer the ability to predict the world.…
- Nate Soares - Intradependent encoding
An encoding $E(m)$ of a message $m$ is intradependent if the fact that $E(m)$ encodes $m$ can be de…
- Nate Soares - Intradependent encodings can be compressed
Given an encoding scheme $E$ which gives an Intradependent encoding of a message $m,$ we can in prin…
- Nate Soares - Introduction to Logical Decision Theory for Analytic Philosophers
Why "choose as if controlling the logical output of your decision algorithm" is the most appealing candidate for the principle of rational choice.
- Eliezer Yudkowsky - Introduction to Logical Decision Theory for Computer Scientists
'Logical decision theory' from a math/programming standpoint, including how two agents with mutual knowledge of each other's code can cooperate on the Prisoner's Dilemma.
- Eliezer Yudkowsky - Introduction to Logical Decision Theory for Economists
An introduction to 'logical decision theory' and its implications for the Ultimatum Game, voting in elections, bargaining problems, and more.
- Eliezer Yudkowsky - Introductory Bayesian problems
Bayesian problems to try to solve yourself, before beginning to learn about Bayes' rule.
- Eliezer Yudkowsky - Less Wrong
A community blog devoted to refining the art of human rationality.
- Alexei Andreev - Likelihood
"Likelihood", when speaking of Bayesian reasoning, denotes *the probability of an observation, sup…
- Nate Soares - Likelihood notation
The likelihood of a piece of evidence $e$ according to a hypothesis $H,$ known as "the likelihood of…
- Nate Soares - Likelihood ratio
Given a piece of evidence $e$ and two hypothsese $H_i$ and $H_j,$ the likelihood ratio between them…
- Nate Soares - Log base infinity
There is no log base infinity, but if there were, it would send everything to zero
- Nate Soares - Logarithm base 1
There is no log base 1.
- Nate Soares - Logarithmic identities
- [ Inversion of exponentials]: $b^{\log_b(n)} = \log_b(b^n) = n.$
- [ Log of 1 is 0]: $\log_b(1) …
- Nate Soares - Logistic function
A monotonic function from the real numbers to the open unit interval.
- Joe Zeng - Meta-rules for (narrow) value learning are still unsolved
We don't currently know a simple meta-utility function that would take in observation of humans and spit out our true values, or even a good target for a Task AGI.
- Eliezer Yudkowsky - Mind projection fallacy
Uncertainty is in the mind, not in the environment; a blank map does not correspond to a blank territory. In general, the territory may have a different ontology from the map.
- Eliezer Yudkowsky - Minimality principle
The first AGI ever built should save the world in a way that requires the least amount of the least dangerous cognition.
- Eliezer Yudkowsky - Modal logic
The logic of boxes and bots.
- Jaime Sevilla Molina - Modular arithmetic
Addition as traveling around a circle, instead of along a line.
- Malcolm McCrimmon - Moral uncertainty
A meta-utility function in which the utility function as usually considered, takes on different values in different possible worlds, potentially distinguishable by evidence.
- Eliezer Yudkowsky - Most complex things are not very compressible
We can't *prove* it's impossible, but it would be *extremely surprising* to discover a 500-state Turing machine that output the exact text of "Romeo and Juliet".
- Eliezer Yudkowsky - Natural number
The numbers we use to count: 0, 1, 2, 3, ...
- Jaime Sevilla Molina - Natural numbers: Intro to Number Sets
Natural numbers are the numbers we use to count in everyday life.
- Joe Zeng - Object identity via interactions
If we think of objects as opaque "black boxes", how can we tell whether two objects are different? By looking at how they interact with other objects!
- Patrick Stevens - Odds: Refresher
A quick review of the notations and mathematical behaviors for odds (e.g. odds of 1 : 2 for drawing a red ball vs. green ball from a barrel).
- Nate Soares - Order of operations
Conventions used for disambiguating infix notation.
- Joe Zeng - Ordered ring
A ring with a total ordering compatible with its ring structure.
- Dylan Hendrickson - Rational number
The rational numbers are "fractions".
- Patrick Stevens - Real number
A **real number** is any number that can be used to represent a physical quantity.
Intuitively, rea…
- Michael Cohen - Relative likelihood
How relatively likely an observation is, given two or more hypotheses, determines the strength and direction of evidence.
- Eliezer Yudkowsky - Rice's Theorem: Intro (Math 1)
You can't write a program that looks at another programs source code, and tells you whether it computes the Fibonacci sequence.
- Dylan Hendrickson - Ring
A ring is a kind of Algebraic structure which we obtain by considering groups as being "things with…
- Nate Soares - Solomonoff induction
A simple way to superintelligently predict sequences of data, given unlimited computing power.
- Eliezer Yudkowsky - Strong Church Turing thesis
A strengthening of the Church Turing thesis
- Jaime Sevilla Molina - The AI must tolerate your safety measures
A corollary of the nonadversarial principle is that "The AI must tolerate your safety measures."
- Eliezer Yudkowsky - The plan
Root page for the plan on how to approach and navigate through AGI development.
- Alexei Andreev - Totally ordered set
A set where all the elements can be compared as greater than or less than.
- Joe Zeng - Toxoplasmosis dilemma
A parasitic infection, carried by cats, may make humans enjoy petting cats more. A kitten, now in front of you, isn't infected. But if you *want* to pet it, you may already be infected. Do you?
- Eliezer Yudkowsky - Underestimating complexity of value because goodness feels like a simple property
When you just want to yell at the AI, "Just do normal high-value X, dammit, not weird low-value X!" and that 'high versus low value' boundary is way more complicated than your brain wants to think.
- Eliezer Yudkowsky - Underlying set
What do a Group, a Partially ordered set, and a [ topological space] have in common? Each is a Set …
- Nate Soares - Union
The union of two sets is the set of elements which are in one or the other, or both
- M Yass - Universal prior
A "universal prior" is a probability distribution containing *all* the hypotheses, for some reasonable meaning of "all". E.g., "every possible computer program that computes probabilities".
- Eliezer Yudkowsky - Universal property
A universal property is a way of defining an object based purely on how it interacts with other objects, rather than by any internal property of the object itself.
- Patrick Stevens - Up to isomorphism
A phrase mathematicians use when saying "we only care about the structure of an object, not about specific implementation details of the object".
- Patrick Stevens - Why is log like length?
If a number $x$ is $n$ digits long (in Decimal notation), then its logarithm (base 10) is between $n…
- Nate Soares - Why is the decimal expansion of log2(3) infinite?
Because 2 and 3 are relatively prime.
- Nate Soares
- 'Beneficial'
Really actually good. A metasyntactic variable to mean "favoring whatever the speaker wants ideally to accomplish", although different speakers have different morals and metaethics.
- Eliezer Yudkowsky - 'Detrimental'
The opposite of beneficial.
- Eliezer Yudkowsky - A googol
A pretty small large number.
- Nate Soares - AI arms races
AI arms races are bad
- Eliezer Yudkowsky - AIXI-tl
A time-bounded version of the ideal agent AIXI that uses an impossibly large finite computer instead of a hypercomputer.
- Eliezer Yudkowsky - Ability to read algebra
Do you have sufficient mathematical ability that you can read a sentence that uses some algebra or invokes a mathematical idea, without slowing down too much?
- Eliezer Yudkowsky - Ability to read calculus
Can you take integral signs and differentiations in stride?
- Eliezer Yudkowsky - Ability to read logic
Can you read sentences symbolically stating "For all x: exists y: phi(x, y) or not theta(y)" without slowing down too much?
- Eliezer Yudkowsky - Abortable plans
Plans that can be undone, or switched to having low further impact. If the AI builds abortable nanomachines, they'll have a quiet self-destruct option that includes any replicated nanomachines.
- Eliezer Yudkowsky - Actual effectiveness
If you want the AI's so-called 'utility function' to actually be steering the AI, you need to think about how it meshes up with beliefs, or what gets output to actions.
- Eliezer Yudkowsky - Ad-hoc hack (alignment theory)
A "hack" is when you alter the behavior of your AI in a way that defies, or doesn't correspond to, a principled approach for that problem.
- Eliezer Yudkowsky - Another another playpen child
May it be a light for you in dark places, when all other lights go out.
- Stephanie Zolayvar - Arbital Blog
Stay up to date on all things Arbital
- Alexei Andreev - Arbital Slack
Where the cool kids hang out.
- Eric Bruylant - Arbital arbiter
Arbiters provide oversight and dispute resolution to an Arbital domain.
- Eric Bruylant - Arbital biographies
As a very strong default (presently an absolute rule), Joe Smith's page only says nice things about Joe. Even if a negative fact is true, it doesn't go on Joe's page.
- Eliezer Yudkowsky - Arbital content request
Arbital doesn't explain something you'd like to learn? We'd like to know, so we can prioritize.
- Eric Bruylant - Arbital draft
Drafts are private work-in-progress pages.
- Eric Bruylant - Arbital editor
How to use Arbital's page editor.
- Alexei Andreev - Arbital editor: Advanced
Advanced features of Arbital editor.
- Alexei Andreev - Arbital greenlink
What happens when you hover over an Arbital link?
- Alexei Andreev - Arbital reviewer
Reviewers help writers improve their pages, check over all changes to Arbital's content, and assess page quality.
- Eric Bruylant - Arbital todo
So many things todo!
- Eric Bruylant - Arbital trusted user
Trusted users can edit most pages directly, and don't need approval to add pages to a domain.
- Eric Bruylant - Arbital unlisted page
What do you call a page that's not part of any domain?
- Alexei Andreev - Artificial General Intelligence
An AI which has the same kind of "significantly more general" intelligence that humans have compared to chimpanzees; it can learn new domains, like we can.
- Eliezer Yudkowsky - Attainable optimum
The 'attainable optimum' of an agent's preferences is the best that agent can actually do given its finite intelligence and resources (as opposed to the global maximum of those preferences).
- Eliezer Yudkowsky - Averting instrumental pressures
Almost-any utility function for an AI, whether the target is diamonds or paperclips or eudaimonia, implies subgoals like rapidly self-improving and refusing to shut down. Can we make that not happen?
- Eliezer Yudkowsky - Averting the convergent instrumental strategy of self-improvement
We probably want the first AGI to *not* improve as fast as possible, but improving as fast as possible is a convergent strategy for accomplishing most things.
- Eliezer Yudkowsky - Bag
In mathematics, a "bag" is an unordered list. A bag differs from a set in that it can contain the sa…
- Nate Soares - Bayesian reasoning
A probability-theory-based view of the world; a coherent way of changing probabilistic beliefs based on evidence.
- Eliezer Yudkowsky - Big-O Notation
This notation describes asymptotic behavior of functions.
# O(x)
A function f is O(g(x)) if, for la…
- Aeneas Mackenzie - Bijective function
A bijective function is a function with an inverse.
- Patrick Stevens - Binary function
A binary function $f$ is a function of two inputs (i.e., a function with arity 2). For example, $+,$…
- Nate Soares - Bit (of data): Examples
In the game "20 questions", one player (the "leader") thinks of a concept, and the other players ask…
- Nate Soares - Boolean
A value in logic that evaluates to either "true" or "false".
- Malcolm McCrimmon - Bounded agent
An agent that operates in the real world, using realistic amounts of computing power, that is uncertain of its environment, etcetera.
- Eliezer Yudkowsky - Cartesian agent-environment boundary
If your agent is separated from the environment by an absolute border that can only be crossed by sensory information and motor outputs, it might just be a Cartesian agent.
- Eliezer Yudkowsky - Category of finite sets
The category of finite sets is exactly what it claims to be. It's a useful training ground for some of the ideas of category theory.
- Patrick Stevens - Cauchy sequence
Infinite sequences whose terms get arbitrarily close together.
- Joe Zeng - Chesterton's fence
If someone did something, it's generally good to understand their reasons for doing it before undoing it.
- Eric Bruylant - Church-Turing thesis
A thesis about computational models
- Jaime Sevilla Molina - Cognitive domain
An allegedly compact unit of knowledge, such that ideas inside the unit interact mainly with each other and less with ideas in other domains.
- Eliezer Yudkowsky - Cognitive steganography
Disaligned AIs that are modeling human psychology and trying to deceive their programmers will want to hide their internal thought processes from their programmers.
- Eliezer Yudkowsky - Computer Programming Familiarity
Want to see programming analogies and applications in your math explanations? Mark this as known.
- Kevin Clancy - Conjugacy class
In a group, the elements can be partitioned naturally into certain classes.
- Patrick Stevens - Decision theory
The mathematical study of ideal decisionmaking
- Eliezer Yudkowsky - Decit
Decimal digit
- Nate Soares - Diagonal lemma
Constructing self-referential sentences
- Jaime Sevilla Molina - Dihedral group
The dihedral groups are natural examples of groups, arising from the symmetries of regular polygons.
- Patrick Stevens - Direct sum of vector spaces
The direct sum of two vector spaces $U$ and $W,$ written $U \oplus W,$ is just the sum of $U$ and $W…
- Nate Soares - Disambiguation
Several distinct concepts use this page's name, this page helps readers find what they're looking for.
- Eric Bruylant - Disjoint cycle notation is unique
Disjoint cycle notation provides a canonical way to express elements of the symmetric group.
- Patrick Stevens - Distinguish which advanced-agent properties lead to the foreseeable difficulty
Say what kind of AI, or threshold level of intelligence, or key type of advancement, first produces the difficulty or challenge you're talking about.
- Eliezer Yudkowsky - Donor lottery
An arrangement where a group of people pool their money and pick one person to give it away.
- Alexei Andreev - Emphemeral premises
When somebody says X, don't just say, "Oh, not-X because Y" and then forget about Y a day later. Y is now an important load-bearing assumption in your worldview. Write Y down somewhere.
- Eliezer Yudkowsky - Equaliser (category theory)
In Category theory, an *equaliser* of a pair of arrows $f, g: A \to B$ is an object $E$ and a univer…
- Patrick Stevens - Evidential decision theories
Theories which hold that the principle of rational choice is "Choose the act that would be the best news, if somebody told you that you'd chosen that act."
- Eliezer Yudkowsky - Expected utility
Scoring actions based on the average score of their probable consequences.
- Eliezer Yudkowsky - Expected utility formalism
Expected utility is the central idea in the quantitative implementation of consequentialism
- Eliezer Yudkowsky - External resources
This lens links out to other great resources across the web.
- Eric Bruylant - Fallacies
To call something a fallacy is to assert that you think people shouldn't think like that.
- Eliezer Yudkowsky - Finite set
A finite set is one which is not infinite. Some of these are the least complicated sets.
- Patrick Stevens - Flag the load-bearing premises
If somebody says, "This AI safety plan is going to fail, because X" and you reply, "Oh, that's fine because of Y and Z", then you'd better clearly flag Y and Z as "load-bearing" parts of your plan.
- Eliezer Yudkowsky - Focusing
Focusing is a psychotherapeutic process developed by psychotherapist Eugene Gendlin
- Alexei Andreev - Formal definition
This page gives a purely formal definition of a topic, rather than motivating, explaining, and giving examples.
- Eric Bruylant - Fractional bits: Digit usage interpretation
It is 316, not 500, that requires about two and a half digits to write down. 500 requires nearly 2.7…
- Nate Soares - Friendly AI
Old terminology for an AI whose preferences have been successfully aligned with idealized human values.
- Eliezer Yudkowsky - Goal-concept identification
Figuring out how to say "strawberry" to an AI that you want to bring you strawberries (and not fake plastic strawberries, either).
- Eliezer Yudkowsky - Graham's number
A fairly large number, as numbers go.
- Nate Soares - Greatest common divisor
The greatest common divisor of two natural numbers is… the largest number which is a divisor of both. The clue is in the name, really.
- Patrick Stevens - Greatest lower bound in a poset
The greatest lower bound is an abstraction of the idea of the greatest common divisor to a general poset.
- Patrick Stevens - Group presentation
Presentations are a fairly compact way of expressing groups.
- Patrick Stevens - Gödel's first incompleteness theorem
The theorem that destroyed Hilbert's program
- Jaime Sevilla Molina - Happiness maximizer
It is sometimes proposed that we build an AI intended to maximize human happiness. (One early propo…
- Eliezer Yudkowsky - Hub page
This tag is applied to pages which server the role of a "hub": the user starts there, goes off to learn more about the topic, and then comes back. This meta tag modifies the page's UI.
- Alexei Andreev - Human perception of sound
What is the mechanism by which vibrations around the human ear are translated into the sensation of sound?
- Silas Barta - Humans doing Bayes
The human use of Bayesian reasoning in everyday life
- Eliezer Yudkowsky - Humean degree of freedom
A concept includes 'Humean degrees of freedom' when the intuitive borders of the human version of that concept depend on our values, making that concept less natural for AIs to learn.
- Eliezer Yudkowsky - Iff
If and only if...
- Alexei Andreev - Ignorance prior
Key equations for quantitative Bayesian problems, describing exactly the right shape for what we believed before observation.
- Eliezer Yudkowsky - Image requested
An editor has requested an image for this page.
- Eric Bruylant - Inductive prior
Some states of pre-observation belief can learn quickly; others never learn anything. An "inductive prior" is of the former type.
- Eliezer Yudkowsky - Information theory
The study (and quantificaiton) of information, and its communication and storage.
- Nate Soares - Instrumental
What is "instrumental" in the context of Value Alignment Theory?
- Eliezer Yudkowsky - Intelligence explosion
What happens if a self-improving AI gets to the point where each amount x of self-improvement triggers >x further self-improvement, and it stays that way for a while.
- Eliezer Yudkowsky - Intension vs. extension
"Red is a light with a wavelength of 700 nm" vs. "Look at this red apple, red car, and red cup."
- Eliezer Yudkowsky - Intro to Number Sets
An introduction to number sets for people who have no idea what a number set is.
- Joe Zeng - Intution pump
In philosophy, a metaphor or visualization used to shove the listener's intuition in a particular direction.
- Eliezer Yudkowsky - Irrational number
Real numbers that are not rational numbers
- Joe Zeng - Joint probability
The notation for writing the chance that both X and Y are true.
- Eliezer Yudkowsky - Just a requisite
A tag for nodes that just act as part of Arbital's requisite system
- Eliezer Yudkowsky - Linear algebra
The study of [linear\_transformation linear transformations] and vector spaces.
- Nate Soares - Logarithm: Examples
$\log_{10}(100)=2.$ $\log_2(4)=2.$ $\log_2(3)\approx 1.58.$ (TODO)
- Nate Soares - Logarithm: Exercises
Without using a calculator: What is $\log_{10}(4321)$? What integer is it larger than, what integer …
- Nate Soares - Logarithms invert exponentials
The function $\log_b(\cdot)$ inverts the function $b^{(\cdot)}.$ In other words, $\log_b(n) = x$ imp…
- Nate Soares - Logical decision theories
Root page for topics on logical decision theory, with multiple intros for different audiences.
- Eliezer Yudkowsky - Löb's theorem
Löb's theorem
- Jaime Sevilla Molina - Math 0
Are you not actively bad at math, nor traumatized about math?
- Eliezer Yudkowsky - Math 1
Is math sometimes fun for you, and are you not anxious if you see a math puzzle you don't know how to solve?
- Eliezer Yudkowsky - Math 2
Do you work with math on a fairly routine basis? Do you have little trouble grasping abstract structures and ideas?
- Eliezer Yudkowsky - Math 3
Can you read the sort of things that professional mathematicians read, aka LaTeX formulas with a minimum of explanation?
- Eliezer Yudkowsky - Mathematics
Mathematics is the study of numbers and other ideal objects that can be described by axioms.
- Eliezer Yudkowsky - Meta-utility function
Preference frameworks built out of simple utility functions, but where, e.g., the 'correct' utility function for a possible world depends on whether a button is pressed.
- Eliezer Yudkowsky - Metaethics
Metaethics asks "What kind of stuff is goodness made of?" (or "How would we compute goodness?") rather than "Which particular policies or outcomes are good or not-good?"
- Eliezer Yudkowsky - Microlending
The practice of giving microloans, which are small loans that are issued by individuals.
- Alexei Andreev - Mind design space is wide
Imagine all human beings as one tiny dot inside a much vaster sphere of possibilities for "The space of minds in general." It is wiser to make claims about *some* minds than *all* minds.
- Eliezer Yudkowsky - Moral hazards in AGI development
"Moral hazard" is when owners of an advanced AGI give in to the temptation to do things with it that the rest of us would regard as 'bad', like, say, declaring themselves God-Emperor.
- Eliezer Yudkowsky - Multiplication of rational numbers (Math 0)
"Multiplication" is the idea of "now do the same as you just did, but instead of doing it to one apple, do it to some other number".
- Patrick Stevens - Needs accessible summary
This page needs a summary for a less technical audience.
- Eric Bruylant - Needs examples
This page would benefit from more examples of the concept it teaches.
- Eric Bruylant - Needs parent
This page is not attached to an appropriate parent page. If you know where it should go, please help categorize it!
- Eric Bruylant - Needs requisites
This page has important requisites which are not listed. If you know what they are, you could help add them!
- Eric Bruylant - Neutral genie metaphor
Definition. A neutral-genie metaphor is an attempt to illustrate a possible formal problem via an in…
- Alexei Andreev - Newcomblike decision problems
Decision problems in which your choice correlates with something other than its physical consequences (say, because somebody has predicted you very well) can do weird things to some decision theories.
- Eliezer Yudkowsky - Nick Bostrom's book Superintelligence
The current best book-form introduction to AI alignment theory.
- Eliezer Yudkowsky - Normal subgroup
Normal subgroups are subgroups which are in some sense "the same from all points of view".
- Patrick Stevens - Number
An abstract object that expresses quantity or value of some sort.
- Joe Zeng - Opinion page
Opinion pages represent one position on a topic (often from a single author), and are not necessarily balanced or a reflection of consensus.
- Eric Bruylant - Orbit-Stabiliser theorem: External Resources
External resources on the Orbit-Stabiliser theorem.
- Mark Chimes - Order of rational operations (Math 0)
Our shorthand for all the operations on rationals is very useful, but full of brackets; this is how to get rid of some of the brackets.
- Patrick Stevens - Ordered field
An ordered ring with division.
- Joe Zeng - Other-izing (wanted: new optimization idiom)
Maximization isn't possible for bounded agents, and satisficing doesn't seem like enough. What other kind of 'izing' might be good for realistic, bounded agents?
- Eliezer Yudkowsky - P (Polynomial Time Complexity Class)
P is the class of problems which can be solved by algorithms whose run time is bounded by a polynomial.
- Eric Leese - P vs NP
Is creativity purely mechanical?
- Jaime Sevilla Molina - P vs NP: Arguments against P=NP
Why we believe P and NP are different
- Jaime Sevilla Molina - Path: Insights from Bayesian updating
A learning-path placeholder page for insights derived from the Bayesian rule for updating beliefs.
- Eliezer Yudkowsky - Perfect rolling sphere
If you don't understand something, start by assuming it's a perfect rolling sphere.
- Eliezer Yudkowsky - Philosophy
A stub parent node to contain standard concepts, belonging to subfields of academic philosophy, that are being used elsewhere on Arbital.
- Eliezer Yudkowsky - Pigovian tax
Taxation of negative externalities so that their producers have an incentive to cheaply reduce them
- Silas Barta - Placeholder
This is an empty page created for structural reasons (parent, requisite, or teaches).
- Eric Bruylant - Possible math pages
A list of things which we may want math pages on
- Eric Bruylant - Prime element of a ring
Despite the name, "prime" in ring theory refers not to elements which are "multiplicatively irreducible" but to those such that if they divide a product then they divide some term of the product.
- Patrick Stevens - Prime number
The prime numbers are the "building blocks" of the counting numbers.
- Patrick Stevens - Prior
A state of prior knowledge, before seeing information on a new problem. Potentially complicated.
- Eliezer Yudkowsky - Probability distribution (countable sample space)
A function assigning a probability to each point in the sample space.
- Tsvi BT - Probability notation for Bayes' rule
The probability notation used in Bayesian reasoning
- Eliezer Yudkowsky - Probability theory
The logic of science; coherence relations on quantitative degrees of belief.
- Eliezer Yudkowsky - Product (Category Theory)
How a product is characterized rather than how it's constructed
- Mark Chimes - Quality meta tags
Meta tags which determine the page's quality.
- Alexei Andreev - Querying the AGI user
Postulating that an advanced agent will check something with its user, probably comes with some standard issues and gotchas (e.g., prioritizing what to query, not manipulating the user, etc etc).
- Eliezer Yudkowsky - Rationality
The subject domain for [ epistemic] and [ instrumental] rationality.
- Eliezer Yudkowsky - Real analysis
The study of real numbers and real-valued functions.
- Kevin Clancy - Real number (as Dedekind cut)
A way to construct the real numbers that follows the intuition of filling in the gaps.
- Joe Zeng - Reflective consistency
A decision system is reflectively consistent if it can approve of itself, or approve the construction of similar decision systems (as well as perhaps approving other decision systems too).
- Eliezer Yudkowsky - Reflective stability
Wanting to think the way you currently think, building other agents and self-modifications that think the same way.
- Eliezer Yudkowsky - Representability theorem for computable functions
A [ logical theory] $T$ is said to satisfy the **representability theorem for computable functions**…
- Jaime Sevilla Molina - Safe plan identification and verification
On a particular task or problem, the issue of how to communicate to the AGI what you want it to do and all the things you don't want it to do.
- Eliezer Yudkowsky - Sample space
The set of possible things that could happen in a part of the world that you are uncertain about.
- Tsvi BT - Set product
A fundamental way of combining sets is to take their product, making a set that contains all tuples of elements from the originals.
- Patrick Stevens - Shannon
The shannon (Sh) is a unit of Information. One shannon is the difference in [info\_entropy entropy] …
- Nate Soares - Show me what you've broken
To demonstrate competence at computer security, or AI alignment, think in terms of breaking proposals and finding technically demonstrable flaws in them.
- Eliezer Yudkowsky - Shutdown problem
How to build an AGI that lets you shut it down, despite the obvious fact that this will interfere with whatever the AGI's goals are.
- Eliezer Yudkowsky - Shutdown utility function
A special case of a low-impact utility function where you just want the AGI to switch itself off harmlessly (and not create subagents to make absolutely sure it stays off, etcetera).
- Eliezer Yudkowsky - Simple group
The simple groups form the "building blocks" of group theory, analogously to the prime numbers in number theory.
- Patrick Stevens - Stabiliser (of a group action)
If a group acts on a set, it is useful to consider which elements of the group don't move a certain element of the set.
- Patrick Stevens - Strategic AGI typology
What broad types of advanced AIs, corresponding to which strategic scenarios, might it be possible or wise to create?
- Eliezer Yudkowsky - Strength of Bayesian evidence
From a Bayesian standpoint, the strength of evidence can be identified with its likelihood ratio.
- Eliezer Yudkowsky - Subgroup
A group that lives inside a bigger group.
- Dylan Hendrickson - Subspace
A subspace $U=(F_U, V_U)$ of a Vector space $W=(F_W, V_W)$ is a vector space where $F_U = F_W$ and $…
- Nate Soares - Sum of vector spaces
The sum of two vector spaces $U$ and $W,$ written $U + W,$ is a vector space where the set of vector…
- Nate Soares - Task identification problem
If you have a task-based AGI (Genie) then how do you pinpoint exactly what you want it to do (and not do)?
- Eliezer Yudkowsky - The alternating groups on more than four letters are simple
The alternating groups are the most accessible examples of simple groups, and this fact also tells us that the symmetric groups are "complicated" in some sense.
- Patrick Stevens - The ideal Arbital math page
Think of the best math textbook you've ever read -- why was it good?
- Eric Rogstad - Theory of (advanced) agents
One of the research subproblems of building powerful nice AIs, is the theory of (sufficiently advanced) minds in general.
- Eliezer Yudkowsky - Tiling agents theory
The theory of self-modifying agents that build successors that are very similar to themselves, like repeating tiles on a tesselated plane.
- Eliezer Yudkowsky - Total alignment
We say that an advanced AI is "totally aligned" when it knows *exactly* which outcomes and plans are beneficial, with no further user input.
- Eliezer Yudkowsky - Transitive relation
If a is related to b and b is related to c, then a is related to c.
- Dylan Hendrickson - Trit
Trinary digit
- Nate Soares - Two independent events
What do [a pair of dice], [a pair of coins], and [a pair of people on opposite sides of the planet] all have in common?
- Tsvi BT - Type theory
Modern foundations for formal mathematics.
- Jack Gallagher - Unassessed
This page's quality has not been assessed.
- Eric Bruylant - Understandability principle
The more you understand what the heck is going on inside your AI, the safer you are.
- Eliezer Yudkowsky - Updateless decision theories
Decision theories that maximize their policies (mappings from sense inputs to actions), rather than using their sense inputs to update their beliefs and then selecting actions.
- Eliezer Yudkowsky - Useless variable decomposition
A variable decomposition can be true but useless if it is a poor guide to intervention due to automa…
- Alexei Andreev - User manipulation
If not otherwise averted, many of an AGI's desired outcomes are likely to interact with users and hence imply an incentive to manipulate users.
- Eliezer Yudkowsky - User maximization
A sub-principle of avoiding user manipulation - if you see an argmax over X or 'optimize X' instruction and X includes a user interaction, you've just told the AI to optimize the user.
- Eliezer Yudkowsky - Value alignment problem
You want to build an advanced AI with the right values... but how?
- Eliezer Yudkowsky - Vector space
A vector space is a field $F$ paired with a Group $V$ and a function $\cdot : F \times V \to V$ (cal…
- Nate Soares - Vingean reflection
The problem of thinking about your future self when it's smarter than you.
- Eliezer Yudkowsky - Vingean uncertainty
You can't predict the exact actions of an agent smarter than you - so is there anything you _can_ say about them?
- Eliezer Yudkowsky - Well-calibrated probabilities
Even if you're fairly ignorant, you can still strive to ensure that when you say "70% probability", it's true 70% of the time.
- Eliezer Yudkowsky - Work in progress
This page is being actively worked on by an editor. Check with them before making major changes.
- Eliezer Yudkowsky - concat (function)
The string concatenation function `concat` puts two strings together, i.e., `concat("one","two")="on…
- Nate Soares
- Page's title should always be capitalized
Vote "agree" if you think Arbital should enforce the first letter of a page title to always be capit…
- Alexei Andreev
- Identifying causal goal concepts from sensory data
If the intended goal is "cure cancer" and you show the AI healthy patients, it sees, say, a pattern of pixels on a webcam. How do you get to a goal concept *about* the real patients?
- Eliezer Yudkowsky
- Neutral genie metaphor
Definition. A neutral-genie metaphor is an attempt to illustrate a possible formal problem via an in…
- Alexei Andreev
- Category theory
How mathematical objects are related to others in the same category.
- Mark Chimes
- GalCom
In the GalCom thought experiment, you live in the future, and make your money by living in the Dene…
- Nate Soares
- Malcolm McCrimmon
A person, presumably.
- Low impact
The open problem of having an AI carry out tasks in ways that cause minimum side effects and change as little of the rest of the universe as possible.
- Eliezer Yudkowsky
- Shutdown problem
How to build an AGI that lets you shut it down, despite the obvious fact that this will interfere with whatever the AGI's goals are.
- Eliezer Yudkowsky
- Problem of fully updated deference
Why moral uncertainty doesn't stop an AI from defending its off-switch.
- Eliezer Yudkowsky
- Vinge's Principle
An agent building another agent must usually approve its design without knowing the agent's exact policy choices.
- Eliezer Yudkowsky - Vingean reflection
The problem of thinking about your future self when it's smarter than you.
- Eliezer Yudkowsky
- Mic-Ra-finance and the illusion of control
This post discusses the following claims:
* [claim([6th])]
* [claim([6tk])]
* [claim([6tl])]
- Alexei Andreev
- Advanced agent properties
How smart does a machine intelligence need to be, for its niceness to become an issue? "Advanced" is a broad term to cover cognitive abilities such that we'd need to start considering AI alignment.
- Eliezer Yudkowsky - Algorithmic complexity
When you compress the information, what you are left with determines the complexity.
- Eliezer Yudkowsky - Almost all real-world domains are rich
Anything you're trying to accomplish in the real world can potentially be accomplished in a *lot* of different ways.
- Eliezer Yudkowsky - An Introduction to Logical Decision Theory for Everyone Else
So like what the heck is 'logical decision theory' in terms a normal person can understand?
- Eliezer Yudkowsky - Arbital subscriptions: Maintenance
Subscribing to a page with intention of maintaining it.
- Alexei Andreev - Arbital: Do what works
When deciding things on Arbital, think about the real goals, and move towards them.
- Eric Bruylant - Arguments
An argument is a formal reasoning, valid or not.
- Jeremy Perret - Asymptotic Notation
Asymptotic notation seeks to capture the behavior of functions as its input(s) become extreme. It is most widely used in Computer Science and Numerical Approximation.
- Morgan Redding - Author's guide to processing feedback
Requisite used for teaching authors about Arbital feedback features.
- Alexei Andreev - Bayes' rule: Beginner's guide
Beginner's guide to learning about Bayes' rule.
- Alexei Andreev - Behaviorist genie
An advanced agent that's forbidden to model minds in too much detail.
- Eliezer Yudkowsky - Bijective Function: Intro (Math 0)
Two boxes are bijective if they contain the same number of items.
- Mark Chimes - Bit (of data)
A bit of data is the amount of data required to single out one message from a set of two. Equivalen…
- Nate Soares - Bit (of data): Examples
In the game "20 questions", one player (the "leader") thinks of a concept, and the other players ask…
- Nate Soares - Boxed AI
Idea: what if we limit how AI can interact with the world. That'll make it safe, right??
- Eliezer Yudkowsky - Category (mathematics)
A description of how a collection of mathematical objects are related to one another.
- Mark Chimes - Category theory
How mathematical objects are related to others in the same category.
- Mark Chimes - Causal decision theories
On CDT, to choose rationally, you should imagine the world where your physical act changes, then imagine running that world forward in time. (Therefore, it's irrational to vote in elections.)
- Eliezer Yudkowsky - Central examples
List of central examples in Value Alignment Theory domain.
- Eliezer Yudkowsky - Civilization scale energy
What are the main options for powering civilization, and how do they compare?
- Eric Bruylant - Coherent extrapolated volition (alignment target)
A proposed direction for an extremely well-aligned autonomous superintelligence - do what humans would want, if we knew what the AI knew, thought that fast, and understood ourselves.
- Eliezer Yudkowsky - Communication: magician example
Imagine that you and I are both magicians, performing a trick where I think of a card from a deck of…
- Nate Soares - Complete lattice
A poset that is closed under arbitrary joins and meets.
- Kevin Clancy - Complex number
A complex number is a number of the form $z = a + b\textrm{i}$, where $\textrm{i}$ is the imaginary …
- Eliana Ruby - Complexity of value
There's no simple way to describe the goals we want Artificial Intelligences to want.
- Eliezer Yudkowsky - Compressing multiple messages
How many bits of data does it take to encode an $n$-message? Naively, the answer is $\lceil \log_2(n…
- Nate Soares - Conjunctions and disjunctions
The fancy name for the "and" and "or" connectives.
- Jeremy Perret - Context disaster
Some possible designs cause your AI to behave nicely while developing, and behave a lot less nicely when it's smarter.
- Eliezer Yudkowsky - Difficulty of AI alignment
How hard is it exactly to point an Artificial General Intelligence in an intuitively okay direction?
- Eliezer Yudkowsky - Distant superintelligences can coerce the most probable environment of your AI
Distant superintelligences may be able to hack your local AI, if your AI's preference framework depends on its most probable environment.
- Eliezer Yudkowsky - Encoding trits with GalCom bits
There are $\log_2(3) \approx 1.585$ bits to a Trit. Why is it that particular value? Consider the Ga…
- Nate Soares - Epistemic exclusion
How would you build an AI that, no matter what else it learned about the world, never knew or wanted to know what was inside your basement?
- Eliezer Yudkowsky - Expected utility agent
If you're not some kind of expected utility agent, you're going in circles.
- Eliezer Yudkowsky - Faithful simulation
How would you identify, to a Task AGI (aka Genie), the problem of scanning a human brain, and then running a sufficiently accurate simulation of it for the simulation to not be crazy or psychotic?
- Eliezer Yudkowsky - Fixed point theorem of provability logic
Deal with those pesky self-referential sentences!
- Jaime Sevilla Molina - Formal Logic
Formal logic studies the form of correct arguments through rigorous and precise mathematical theories.
- Erik Istre - Fractional bits: Digit usage interpretation
It is 316, not 500, that requires about two and a half digits to write down. 500 requires nearly 2.7…
- Nate Soares - Function
Intuitively, a function $f$ is a procedure (or machine) that takes an input and performs some opera…
- Nate Soares - Grid scale storage
Scalable energy storage is required if civilization's switches to primarily renewables in order to keep the grid powered at night. What are the options and how do they compare?
- Eric Bruylant - How many bits to a trit?
$\log_2(3) \approx 1.585.$ This can be interpreted a few different ways:
1. If you multiply the nu…
- Nate Soares - How to author on Arbital!
Want to contribute pages to Arbital? Here's our current version of the ad-hoc guide to being an author!
- Eliezer Yudkowsky - Identifying ambiguous inductions
What do a "red strawberry", a "red apple", and a "red cherry" have in common that a "yellow carrot" doesn't? Are they "red fruits" or "red objects"?
- Eliezer Yudkowsky - Immediate goods
One of the potential views on 'value' in the value alignment problem is that what we should want fro…
- Eliezer Yudkowsky - Information
Information is a measure of how much a message grants an observer the ability to predict the world.…
- Nate Soares - Instrumental convergence
Some strategies can help achieve most possible simple goals. E.g., acquiring more computing power or more material resources. By default, unless averted, we can expect advanced AIs to do that.
- Eliezer Yudkowsky - Joint probability distribution: (Motivation) coherent probabilities
If you don't use joint probability distributions, none of your probabilities will make any sense. So, yeah, use joint probability distributions.
- Tsvi BT - Known-algorithm non-self-improving agent
Possible advanced AIs that aren't self-modifying, aren't self-improving, and where we know and understand all the component algorithms.
- Eliezer Yudkowsky - Law of syllogism
Deriving something from the conclusion of another thing.
- Jeremy Perret - Likelihood functions, p-values, and the replication crisis
What's the whole Bayesian-vs.-frequentist debate about?
- Eliezer Yudkowsky - Logarithm tutorial overview
The logarithm tutorial covers the following six subjects:
1. What are logarithms?
2. Logarithms as…
- Nate Soares - Methodology of foreseeable difficulties
Building a nice AI is likely to be hard enough, and contain enough gotchas that won't show up in the AI's early days, that we need to foresee problems coming in advance.
- Eliezer Yudkowsky - Methodology of unbounded analysis
What we do and don't understand how to do, using unlimited computing power, is a critical distinction and important frontier.
- Eliezer Yudkowsky - Modus tollens
Deriving a negation from another negation
- Jeremy Perret - Morphism
A morphism is the abstract representation of a relation between mathematical objects.
Usually, it i…
- Jaime Sevilla Molina - Natural language understanding of "right" will yield normativity
What will happen if you tell an advanced agent to do the "right" thing?
- Eliezer Yudkowsky - Natural numbers: Intro to Number Sets
Natural numbers are the numbers we use to count in everyday life.
- Joe Zeng - Nearest unblocked strategy
If you patch an agent's preference framework to avoid an undesirable solution, what can you expect to happen?
- Eliezer Yudkowsky - Negation of propositions
The proposition that is false if another one is true and vice-versa.
- Jeremy Perret - Ontology identification problem
How do we link an agent's utility function to its model of the world, when we don't know what that model will look like?
- Eliezer Yudkowsky - Open subproblems in aligning a Task-based AGI
Open research problems, especially ones we can model today, in building an AGI that can "paint all cars pink" without turning its future light cone into pink-painted cars.
- Eliezer Yudkowsky - Optimization daemons
When you optimize something so hard that it crystalizes into an optimizer, like the way natural selection optimized apes so hard they turned into human-level intelligences
- Eliezer Yudkowsky - Oracle
System designed to safely answer questions.
- Eliezer Yudkowsky - Order theory
The study of binary relations that are reflexive, transitive, and antisymmetic.
- Kevin Clancy - Orthogonality Thesis
Will smart AIs automatically become benevolent, or automatically become hostile? Or do different AI designs imply different goals?
- Eliezer Yudkowsky - Paperclip maximizer
This agent will not stop until the entire universe is filled with paperclips.
- Eliezer Yudkowsky - Programmer deception
Programmer deception is when the AI's decision process leads it to optimize for an instrumental goal…
- Eliezer Yudkowsky - Programming in Dependent Type Theory
Working with simple types in Lean
- Jack Gallagher - Propositions
Propositions are statements with a truth value.
- Jeremy Perret - Resources and the future
Resource constraints are a widely held concern. Which are most likely to be limiting factors, and what can we do to relax those limits?
- Eric Bruylant - Rice's theorem and the Halting problem
We will show that Rice's theorem and the the halting problem are equivalent.
#The Halting theorem i…
- Jaime Sevilla Molina - Rich domain
A domain is 'rich', relative to our own intelligence, to the extent that (1) its [ search space] is …
- Eliezer Yudkowsky - Ring
A ring is a kind of Algebraic structure which we obtain by considering groups as being "things with…
- Nate Soares - Shannon
The shannon (Sh) is a unit of Information. One shannon is the difference in [info\_entropy entropy] …
- Nate Soares - Solovay's theorems of arithmetical adequacy for GL
Using GL to reason about PA, and viceversa
- Jaime Sevilla Molina - Standard agent properties
What's a Standard Agent, and what can it do?
- Eliezer Yudkowsky - Task-directed AGI
An advanced AI that's meant to pursue a series of limited-scope goals given it by the user. In Bostrom's terminology, a Genie.
- Eliezer Yudkowsky - The reals (constructed as Dedekind cuts) form a field
The reals are an archetypal example of a field, but if we are to construct them from simpler objects, we need to show that our construction does indeed have the right properties.
- Patrick Stevens - There is only one logarithm
All logarithm functions are the same, up to a multiplicative constant.
- Nate Soares - Type theory
Modern foundations for formal mathematics.
- Jack Gallagher - Value achievement dilemma
How can Earth-originating intelligent life achieve most of its potential value, whether by AI or otherwise?
- Eliezer Yudkowsky - Value identification problem
The subproblem category of value alignment which deals with pinpointing valuable outcomes to an adva…
- Eliezer Yudkowsky - Value-laden
Cure cancer, but avoid any bad side effects? Categorizing "bad side effects" requires knowing what's "bad". If an agent needs to load complex human goals to evaluate something, it's "value-laden".
- Eliezer Yudkowsky - Vingean uncertainty
You can't predict the exact actions of an agent smarter than you - so is there anything you _can_ say about them?
- Eliezer Yudkowsky - Zermelo-Fraenkel provability oracle
We might be able to build a system that can safely inform us that a theorem has a proof in set theory, but we can't see how to use that capability to save the world.
- Eliezer Yudkowsky