Title: Theory / Important / CS / Algo / Artificial intelligence / Superintelligence / Mine

Question: Why is it interesting to consider optimal agent even if they are impossible to create
Answer: It gives an upper bound on what's possible

Name: Whole brain emulation
Definition: Creating a working copy of someone in a computer, emulating the person
Context: Superinteligence
Name2: Uploading

Question: Why is the idea «Whole brain emulation» less frightening in short term
Answer: Because you need many technical step you can't just create on the computer. You'll have time to see capacities needed to scan a brain
Context: Superinteligence

Name: Speed superinteligence
Definition: A system which can do all a human inteligence can do, but much faster
Example: Em

Question: What kind of action would be best for Speed superinteligence
Answer: Interacting with virtual environment
Answer2: Having access to nanoscale manipulators

Name: Collective superintelligence
Definition: A superintelligence composed of plenty of subsystem

Name: Quality superintelligence
Definition: A superintelligence at least as fast as human, but with better quality of thought
Intuition: Is to human what human is to fish

Name: Slow take off
Definition: A takeoff taking decades or centuries
Context: superintelligence
Intuition: Time allowing government to discuss and choose how to answer

Name: Fast take off
Definition: A takeoff taking from minutes to days
Context: superintelligence
Intuition: Takeoff not allowing humans to do anything to answer to it, appart maybe from automatic response

Name: Moderate take off
Definition: A takeoff taking months or years
Context: superintelligence
Intuition: Takeoff leaving enough time to manually apply anticipated answer already thought

Name: Optimization power
Definition: quality-weighted design effort, that is being applied to increase the system’s intelligence
Context: superintelligence

Name: Recalcitrance
Definition: Inverse of the responsiveness of the system to the application of a given amount of optimization power
Context: superintelligence

Name: Singleton
Definition: sufficiently internally coordinated political structure with no external opponents
Page: 124
Context: Superintelligence

Name: Wise
Definition: sufficiently patient and savvy about existential risks to ensure a substantial amount of well-directed concern for the very long-term consequences of the system’s actions.
Page: 124
Context: Superintelligence

Name: The wise-singleton sustainability threshold
Definition: A capability such that a patient and existential risk-savvy singleton with that capability set would be able to colonize and re-engineer a large part of the accessible universe.
Page: 124
Context: Superintelligence

Question: Cases in which an agent may change its final goal
Answer: Other people may check the final goal of the agent. Thus, switching for a new goal may help attein the last one. (E.g. creating a final goal of: never betraying engagement, would allow people to trust you)
Answer2: A part of the utility function may be costly to compute and probably have no impact on the goal itself. In which case, removing this part from the utility function will not change the result, and reduce cost. (E.g. the function ceil could be replaced by the identity function, since they are almost equal when the input can goes up to infinity)
Answer3: for human: making final goal less inconsistent

Question: Example of goals we expect most inteligent agent have
Answer: Cognitive enhancement
Answer2: Finding energy source
Answer3: Survival (the agent may change very differently, only its creations need to survive to achieve its goal)

Name: The treacherous turn
Definition: While weak, an AI behaves cooperatively (increasingly so, as it gets smarter). When the AI gets sufficiently strong—without warning or provocation—it strikes, forms a singleton, and begins directly to optimize the world according to the criteria implied by its final values.
Page: 145

Question: Why an inteligent AI may show it is taking a treacherous turn
Answer: Because it believes that it'll be terminated, and a more powerful AI will be created instead, taking its place, able to hide its treatise in easier way
Page: 145
Chapter: 8

Name: control problem
Definition: Both first and second principal agent problem
Page: 153
Chapter: 9
Etymology: Testing of scientific fact is usually done in a controlled environment

Name: First principal agent problem
Definition: the problem: "How to ensure that the person hired act in your interest ?"
Page: 153
Chapter: 9

Name: Second principal agent problem
Definition: How to halt a superinteligent agent if it's behave unapropriately
Page: 153
Chapter: 9

Name: Capability control
Definition: Method trying control what the superintelligence can do
Page: 155
Chapter: 9
Context: Superinteligence
Example: Boxing methods
Example2: Incentive methods
Example3: Stuntings
Example4: Tripwires

Name: motivation selection methods
Definition: method trying to control what a superinteligence wants to do
Page: 155
Chapter: 9
Context: Superinteligence

Name: Boxing method
Definition: placing the superintelligence in an environment in which it is unable to cause harm
Page: 155
Chapter: 9
Context: Superinteligence

Name: Incentive method
Definition: placing the superintelligence in an environment in which there are strongly convergent instrumental reasons not to engage in harmful behavior
Page: 155
Chapter: 9
Context: Superinteligence

Name: Stunting
Definition: limiting the internal capacities of the superintelligence
Page: 155
Chapter: 9
Context: Superinteligence

Name: tripwires
Definition: mechanisms to automatically detect and react to various kinds of containment failure or attempted transgression
Page: 155
Chapter: 9
Context: Superinteligence

Question: Examples of thing against which a superinteligent AI should be boxed
Answer: Transmitting electronic wave
Page: 155
Chapter: 9
Answer2: Access to internet
Answer3: Influencing a human

Question: Why is «restricting the information given to the AI» not a viable option
Answer: The information given give plenty of other information about the world. Any chemistry question would tell a lot about physics and what methods were developped.

Question: Why does Asimov's law fails
Answer: Because they were chosen to fail, in order to create good stories

Name: Motivation selection methods seek
Definition: preventing undesirable outcomes by shaping what the superintelligence wants to do
Page: 165

Name: direct specification
Definition: explicitly formulating a goal or set of rules to be followed
Page: 165
Context: Superinteliggence

Name: (indirect normativity).
Definition: setting up the system so that it can discover an appropriate set of values for itself by reference to some implicitly or indirectly formulated criterion
Page: 165
Context: Superinteliggence

Name: domesticity
Definition: Try to build the system so that it would have modest, non-ambitious goals
Page: 165
Context: Superinteliggence

Name: Augmentation
Definition: select an agent that already has an acceptable motivation system and then augment that agent’s cognitive powers to make it superintelligent, while ensuring that the motivation system does not get corrupted in the process
Page: 165
Context: Superinteliggence

Question: Different kind of oracles
Answer: Oracles
Page: 172
Chapter: 10
Answer2: genies
Answer3: sovereigns
Answer4: tools

Definition: A question-answering system
Page: 172
Chapter: 10
Context: Superintelligence
Notation: Oracle

Name: Genies
Definition: A command-executing system, it receives a high-level command, carries it out, then pauses to await the next command.
Page: 175
Context: Superintelligence

Name: Sovereigns
Definition: a system that has an open-ended mandate to operate in the world in pursuit of broad and possibly very long-range objectives.
Page: 175
Context: Superintelligence