Title: Theory / Important / CS / Algo / Artificial intelligence / Embedded Agency

Name: A dualistic agent
Definition: only set interactions between agent-stuff and environment-stuff
Page: https://www.alignmentforum.org/s/Rm6oQRJJmhGCcLvxh/p/i3BTagvt3HbPMx6PN
Definition2: is larger than the environment
Example: Playing a video game/board game
Definition3: don't tend to be self-referential
Definition4: is made of different stuff than what it reasons about
DefType: An agent

Name: An embedded agent
Definition: do not have well-defined i/o channels;
Page: https://www.alignmentforum.org/s/Rm6oQRJJmhGCcLvxh/p/i3BTagvt3HbPMx6PN
Definition2: are smaller than their environment;
Example: A robot in the physical world
Conjdef: all
Definition3: are able to reason about themselves and self-improve;
Intuition: An agent belonging to the worlds it models
Definition4: are made of parts similar to the environment.
DefType: An agent

Definition: An element \(a\in A\) such that \(f(a)\) is maximal
Page: https://www.alignmentforum.org/s/Rm6oQRJJmhGCcLvxh/p/i3BTagvt3HbPMx6PN
Variables: \(A,B\) sets
\(f:A\to B\)
Notation: \(\mathtt{argmax}_{a\in A}f(a)\)

Question: Topics of embedded agency
Answer: Decision theory
Page: 5
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1
Answer2: Embedded world-models
Answer3: Robust delegation
Answer4: subsystem alignment

Question: Major open problems in decision theory
Answer: Logical counterfactuals
Page: 5
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1.2
Answer2: How to deal with copy of the agent/trustworthy prediction of the agent
Answer3: logical updatelessness

Name: Logical counterfactual problem
Definition: how do you reason about what would happen if you take action B, given that you can prove that you will instead take action A?
Page: 5
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1.2
Example: Five and ten problem

Name: logical updatelessness problem
Definition: how to combine the very nice but very Bayesian world of Wei Dai’s (2009) updateless decision theory, with the much less Bayesian world of logical uncertainty.
Page: 5
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1.2

Name: Embedded world-models problem
Definition: How to create a good model of the world in an agent much smaller than the world/contained in it
Page: 5
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1.2
Example: Logical uncertainty
Example2: Multi-level modeling
Example3: Ontological crises

Name: logical uncertainty
Definition: how to combine the world of logic with the world of probability
Page: 66
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1.2
Context: Embedded agency

Name: multi-level modeling
Definition: how to have multiple models of the same world at different levels of description, and transition nicely between them
Page: 66
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1.2

Name: ontological crises
Definition: what to do when you realize that your model, or even your goal, was specified using a different ontology than the real world
Page: 66
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1.2

Name: Vingean reflection
Definition: how to reason about and trust agents that are much smarter than you, in spite of the Lobian obstacle to trust
Page: 6
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1.2

Name: Value learning
Definition: the successor agent’s task of learning the goals of the initial agent in spite of the initial agent’s inconsistencies and lower intelligence.
Page: 6
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1.2

Name: corrigibility
Definition: about how an initial agent can get a successor agent to allow (or even help with) modifications, in spite of an instrumental incentive not to
Page: 6
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1.2

Name: Robust delegation
Definition: How to create something and ensure it will share our goal
Page: 6
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1.2
Example: Smarter agent
Example2: the initial agent when it learned more
Extra: second agent problem ?

Name: Subsystem alignment
Definition: Assuming a system has truthworthy subsystem, how can the system use them ?
Page: 6, 28
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 1.2
Extra: principal agent problem ?
Counterexample: Having a subsystem to generate money, which fight the initial system's goal

Question: What is the trouble with having a copy of an agent in the same universe
Answer: When you act, do you believe you control only yourself, or that you also control the copy, since it'll take the same decisions as you
Page: 7
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 2

Name: Five and ten problem
Definition: Suppose we have the option of taking a five dollar bill or a ten dollar bill, and all we care about in the situation is how much money we get.
If you take 5$, the sentence (If I take 10$ I've 0$) holds, hence you should not take 10$, and 5$ is logically a good choice
Page: 7
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 2.1

Name: \(P\) is provable
Page: 9
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 2.1
Variables: \(P\) a proposition
Context: Embedded agency
Notation: \(\Box P\)

Smaller0: \(\Box(\Box P\implies P)\)
Greater0: \(\Box P\)
Page: 9
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 2.1
Variables: \(P\) a proposition
Name: Löb's theorem

Name: Spurious
Page: 9
Chapter: https://arxiv.org/pdf/1902.09469.pdf
Section: 2.1
French: Faux

Question: What's the main limit of updateless decision theory
Answer: It requires to be able to foresee all possible situations
Page: 11
Chapter: Embedded agency
Section: 2

Name: Realizability assumption
Definition: The true underlying environment is assumed to have at least some probability in the prior
Page: 12
Chapter: Embedded agency
Section: 3.1
Context: AI

Name: Liar paradoxe
Definition: «This sentence is not true»
Page: 13
Chapter: Embedded agency
Section: 3.2
Context: AI

Question: Exemple of real liar paradox
Answer: rock-paper-scissors with two people able to predict the other one with >50% success
Page: 13
Chapter: Embedded agency
Section: 3.2
Context: AI
Answer2: You must create a map of a road, knowing the person doing the road will look at your map and do other thing

Name: Grain of truth problem
Definition: the problem of formulating a reasonably bound prior probability distribution which would allow agents playing games to place some positive probability on each other’s true (probabilistic) behavior, without knowing it precisely from the start
Page: 14
Chapter: 3
Section: 2

Name: Logical uncertainty
Definition: Uncertainty about the consequences of your beliefs
Page: 14
Chapter: 3
Example: You can now game of life rules and starting position without knowing its entire futur

Name: Logical Omniscience
Definition: Knowing every logical consequences of your beliefs
Page: 14
Chapter: 3
Intuition: Generally Bayesian hypothesis

Name: Naturalized induction
Definition: the problem of learning world-models which include yourself in the environment

Name: Vingean reflection
Definition: being able to trust that our future self is applying its intelligence to the pursuit of our goals without being able to predict precisely what our future self will do.
Page: 18
Chapter: 4
Section: 1
Example: you might plan your driving route before visiting a new city, but you do not plan your steps. You plan to some level of detail, and trust that your future self can figure out the rest.

Name: The Löbian Obstacle
Definition: Agents who trust their future self because they trust the output of their own reasoning are inconsistent.
Page: 18
Chapter: 4
Section: 1

Name: The Procrastination Paradox
Definition: Agents who trust their future selves without reason tend to be consistent but unsound and untrustworthy, and will put off tasks forever because they can do them later.
Page: 18
Chapter: 4
Section: 1

Question: Main problems in the context of value loading
Answer: We don’t know what we want
Page: 19
Chapter: 4
Section: 2
Answer2: Goodhart's law

Name: Goodhart's law
Definition: Optimization amplifies slight differences between what we say we want and what we really want
Page: 19
Chapter: 4
Section: 2

Name: Optimizer's curse
Definition: When a goal is to optimise some value \(X\), and we use proxy \(Y\) which is highly correlated with \(X\), it is still possible that a great optimization of \(Y\) is a poor optimization of \(X\).
Page: 19
Chapter: 4
Section: 2
Name2: Regressional Goodhart

Name: Causal Goodhart
Definition: when you observes a correlation between proxy and goal, but upon intervening to increase the proxy, you fail to increase the goal because the observed correlation was not causal in the right way.

Name: Extremal Goodhart
Definition: optimization pushes you outside the range where the correlation exists, into portions of the distribution which behave very differently

Question: What's difference between Extremal Goodhart and regressional Goodhart
Answer: Extremal can't be corrected by Bayesian correction because we don't even have a model which cover extremal case
Page: 21
Chapter: 4
Section: 2

Name: A proxy for \(X\)
Definition: A variable \(Y\) such that there is a high correlation between \(X\) and \(Y\) and thus you optimize \(Y\) instead
Page: 21
Chapter: 4
Section: 2
Variables: \(X\) a variable you want to optimize
Context: Bayes

Question: How to protect oneself from Extremal goodhart
Answer: Don't actually try to optimize ase much as possible, but allow less good than optimal, in hope to be in part of the world you know.
Page: 21
Chapter: 4
Section: 2
Context: Bayes

Name: Adversarial Goodhart
Definition: An agent actively make our proxy worse by intelligently manipulating it.
Page: 22
Chapter: 4
Section: 2

Question: List of Goodhart effects
Answer: Regressional
Page: 22
Chapter: 4
Section: 2
Answer2: Causal
Answer3: Extremal
Answer4: Adversarial

Name: A robust to scale up approach
Definition: An approch which still works or fail gracefully as you scale up the capability.
Page: 22
Chapter: 4
Section: 3
Intuition: If the function it should maximize is indeed maximized, everything is allright

Name: A robust to scale down approach
Definition: An approch which still works or fail gracefully when the maximized function is not actually maximal
Page: 22
Chapter: 4
Section: 3
Example: A system which is safe when humans are totally understood should not be unsafe if human is misunderstood

Name: A robust to relative scale approach
Definition: An approch which does not rely on subagent to be as powerful as itself.
Page: 22
Chapter: 4
Section: 3