Title: Theory / Important / CS / Algo / Artificial intelligence / AI Policy
Tags: AI safety AGI groups security US reports UK

Name: national security community
Definition: people working on national security issues, directly or indirectly, from within or without the government

Name: supervised learning
Definition: a machine learning technique in which algorithms are trained on labelled data -- that is, data for which humans already know the relevant classifications and indicated them to the computer

Name: sensitivity
Definition: percent of positive case detected as such
Context: statistics
Name2: True positive rate
Etymology: The most sensible case is not to miss anything we look for
Name3: recall
Name4: probability of detection

Name: specificity
Definition: probability of a negative to be classified as negative
Context: statistics
Name2: true negative rate

Name: Explainable AI (XAI)
Definition: DARPA program meant to create new AI techniques that aren't a black box but still have great performance, by using natural language to explain to users how the autonomous system is making decisions

Name: Locally Interpretable Model-Agnostic Explanations
Definition: A means of using ML to "break open the black box" of autonomous systems by explaining what information is being used for classification.
Example: differentiating between a husky and another type of dog (say, by identifying snow in the background)
Example2: determining if the author of a text is an atheist (say, by seeing if the email is .edu)
Abbreviation: LIME

Name: unsupervised learning
Definition: An ML method that finds similarities in data and figures out how to break up data into sections based on the distance between chosen values and the actual data.
Extra: It's advantageous because it requires less human work to create labelled data sets, and can potentially pick up on trends that humans failed to identify. However, it's dangerous because it might make classifications that are socially unacceptable -- say, marketing products more aggressively to vulnerable groups.
Intuition: this method tries to make classifications based on trends it picks up, rather than being given labelled data as input from which it identifies trends and extrapolates for new examples.

Name: semi-supervised learning
Definition: Integrated use of supervised and unsupervised learning, wherein an algorithm clusters a labelled data set (supervised) to find characteristics that are similar across all instances of a label, and then use these characteristics to identify more examples in an unlabelled data set (unsupervised).
This is advantageous because it makes ML less data-constrained. However, you likely have to be more careful than usual in understanding how the system is making its decisions, and provide it with a large and diverse data set.

Name: reinforcement learning
Definition: an ML agent (a kind of software) acts randomly in an environment and eventually figures out which actions improve its score.
The agent is placed in an environment, given a set of moves it can make and a utility function (often, a score in a game), and not given the rules of the game. It acts randomly and adjusts its actions based on which actions where correlated with increases in its score.
This was the mechanism that was combined with deep learning to create AlphaGo.
Abbreviation: RL

Name: deep learning
Definition: a stack of neural networks, each of which takes input data (the raw data for the first one, the outputs of the previous network for subsequent layers) and outputs some ultimate classification
The issue is that DL networs come up with their own hierarchies of meaning what we don't understand, hence the "black box" effect.
Example: For instance, in an image classification context you could take perceptrons --> pixel values --> edges --> corners and counters (the building blocks of shapes) --> category.
Example2: Google used to drastically reduce its energy usage. It took into account a huge number of factors that would have been difficult to do manually, such as weather and time of day.
Abbreviation: DL

Name: generative design algorithms
Definition: ML process to create things based on specifications, rather than mimicking human designs. The results are sometimes quite unlike the human-created versions but are much more efficient.

Name: exploratory attack
Definition: testing an algorithm to see where it makes mistakes and exploiting those vulnerabilities to one's advantage
Example: image classification systems can mistakenly classify an image with only a few pixels changed, which is clearly X to a human but Y with high confidence according to the algorithm. Militaries could paint tanks to be misclassified as something innocuous, so the algorithms operating without human oversight would fail to report back about the danger.

Name: causative attack
Definition: when a malicious actor trains an algorithm to ensure there are weaknesses to exploit
For example, a spammer could feed a spam algorithm a lot of spam of a certain type, ensuring that other types of spam go undetected. This is an issue for algorithms training in the wild.

Name: Pentagon
Definition: headquarters of the Department of Defense (DoD); often used as a metonym for the DoD

Name: The Hill
Definition: prominent DC news website about political affairs
Etymology: metonym for the US Congress; aka "Capitol Hill"

Name: The White House
Definition: the official residence of the President of the United States (POTUS); also a metonym for the POTUS and his/her Executive Office (i.e. staff)

Name: Washington
Definition: the US federal government, or
Definition2: established career politicians
Conjdef: metonym for
Etymology: Washington, DC

Name: Project Maven
Definition: DoD initiative to "integrate AI and ML more effectively across operations to maintain advantages over increasingly capable adversaries and competitors"
They project focuses on computer vision, using industry developments to improve their efforts.
Google's support of this project received pushback from its employees in March 2018, who wrote a letter saying that they don't want to support military projects.

Name: Defense Innovation Board (DIB)
Definition: Independent federal advisory committee advising the Secretary of Defense (SecDef) on various tech issues
Extra: Part of the larger innovation ecosystem in the DoD, working closely with the Defense Innovation Unit Experimental (DIUx) and Defense Digital Services (DDS)
Notable members include Eric Schmidt (Exec. Chairman of Alphabet), Adam Grant (Wharton professor, "Give and Take"), Reid Hoffman (Co-Founder of LinkedIn), and Neil DeGrasse Tyson (public scientist)

Name: Defense Innovation Unit Experimental
Definition: fast-moving military entity contracting with companies to offer tech solutions to defense problems; five focus areas, including AI and autonomy
Abbreviation: DIUx

Name: mutual assured destruction
Definition: any attack by one side against another would leave the victim with more than enough power (in most cases, they're referring to deliverable nuclear warheads) to exact terrible retribution against the aggressor's homeland

Name: fratricide
Definition: the accidental killing of one's own forces in war (e.g. "friendly fire" in direct combat)

Name: counterforce
Definition: attacks aimed at disarming the enemy, often used in reference to nuclear forces
Extra: Keir Lieber (Director of Georgetown's SFS) wrote a paper suggesting that changing computer tech makes nuclear deterrence less robust than it used to be. AI products make it easier to detect, aim at, and attack nuclear arsenals, so countries that are advanced in this way may consider breaking the nuclear stasis.
If acted upon, this could lead to a cascade of global catastrophes: nuclear explosions and fallout, destabilized world order, escalation in the militaristic use of AI, unintentional consequences of rapid development.

Name: types of security clearances
Definition: confidential, secret, and top secret (TS)
Extra: There's also sensitive compartmented information (SCI) which can be given to someone with a Secret or TS clearance and required a separate investigation from the normal clearance process.
The US Dept of Energy also has two, which you can probably ignore for AI security matters.
https://en.wikipedia.org/wiki/List_of_U.S._security_clearance_terms

Name: Subcommittee on Information Technology
Definition: Bipartisan unit within the House Oversight and Government Reform Committee
Hosted "Game Changers," a three-part hearing series on AI in spring 2018, featuring folks from industry, government, and academia:
https://oversight.house.gov/hearing/game-changers-artificial-intelligence-part/
(Particularly interesting people to note appeared in the third hearing: Jack Clark of OpenAI, Terah Lyons of the PAI, and Ben Buchanan of the Belfer Center at HKS)

Name: Pax Americana
Definition: "A state of relative international peace regarded as overseen by the US (or the UK)" (Wikipedia)
Extra: Some people believe that the US needs to maintain military dominance to maintain the current global order, and that AI is the best means by which to do so. Others believe that we're out of that state already, and/or worry that emphasis on military dominance via AI will escalate an arms race and make AI development more dangerous.

Name: "The Malicious Use of Artificial Intelligence" report
Definition: Long multi-author paper that analyzes "what the long-term equilibrium between attackers and defenders will be."
It makes four high-level recommendations, mainly that different stakeholders should collaborate closely and take seriously AI's dual-use concerns.
https://arxiv.org/pdf/1802.07228.pdf

Name: "AI in the UK: ready, willing and able?" report
Definition: UK House of Lords report by the Select Committee on Artificial Intelligence, published April 2018
In the writing on the report they received written evidence from, amongst others, researchers in the Future of Humanity Institute's Governance of AI Program.
It focuses on what AI is, what the public thinks about it, how design will affect safe use, how it's being developed, and various ways it will affect e.g. the labor market, education, and healthcare.
The report recommends the creation of a UK AI Council for multi-stakeholder coordination, a Government Office of AI for work at the central level, and the Centre fro Data Ethics and Innovation for industry coordination.
https://publications.parliament.uk/pa/ld201719/ldselect/ldai/100/100.pdf

Name: Partnership on AI
Definition: Large coalition industry partners, nonprofits, and academic research groups aiming to ensure that AI socially beneficial. As of 2017 it was mostly a means of demonstrating buy-in for these goals, but they've been driving the development of activities to forward their mission.
Director Terah Lyons used to be a Policy Advisor to the US CTO in the White House's OSTP, and co-directed the White House Future of AI Initiative.
https://www.partnershiponai.org/

Name: Future of Life Institute (FLI) AI Open Letter
Definition: Public document garnering widespread support (as of April 2018, over 8000 signatories) for "research [on] how to reap [AI's] benefits while avoiding potential pitfalls"
https://futureoflife.org/ai-open-letter/

Name: model vs. algorithm
Definition: When you train an algorithm with a data set it generates a model.
For instance, you can take a Decision Tree algorithm and give it a data set for e.g. anomaly detection in a network to identify malicious packets. It would produce a model that takes normal and malicious packets as inputs and produces a result set corresponding to each packet as the output.

Name: neural network
Definition: One of the learning algorithms used in machine learning, consisting of different layers for analyzing and learning data.

Name: value selection methods
Definition: One of the two broad classes of approaches to the AGI control problem, this area focusing on engineering an AGI's motivation system so that it would not choose to cause harm even if it were capable of doing so. AGI safety research problem area in getting an AGI to work towards the right goals. MIRI calls this "error tolerance and value learning," and tackles part of this problem with their Machine Learning technical agenda.
Also called "value specification" in this overview of research in the area (p8): https://arxiv.org/pdf/1805.01109.pdf
Context: AI Policy

Name: reliability
Definition: AGI safety research problem area of making an agent that keeps pursuing the goals we have designed it with.
Called "highly reliable agent design" by MIRI, and a self-modification subproblem by DeepMind.
Brief overview of research on this topic here: https://arxiv.org/pdf/1805.01109.pdf

Name: corrigibility
Definition: AGI safety research problem area of determining if an agent with cooperate in us trying to fix an error in the design or construction of an agent.
Name2: "error-tolerant design" by MIRI's Agent Foundations agenda
Extra: As of 2018, approaches to counter the self-preservation drives that make corrigibility hard include:
(a) baking indifference to shutdown into the reward function,
(b) making the agent act as if it will always act optimally in the future (i.e. be ignorant of the possibility of shutdown), and
(c) making the agent uncertain about the reward function, so (in some circumstances) assuming shutdown by humans is optimal according to its reward function.
Brief overview of the research in this area (p8-9): https://arxiv.org/pdf/1805.01109.pdf
Name3: connected to safe interruptibility as considered by DeepMind.

Name: security (AGI)
Definition: AGI safety research problem area of how to design AGIs that are robust to adversaries and adversarial environments.
Extra: Since small perturbations of inputs can cause severe misclassification errors, adversaries can take advantage of these weaknesses. The best approach (as of 2018) for avoiding these problems is with an extended algorithm that can verify the behavior of neural networks for far more nodes and down more layers. This gives some insight into behavior in certain important regions as well as sensitivity to adversarial examples.
Brief overview of the research area (p9) and safety measure (p15): https://arxiv.org/pdf/1805.01109.pdf

Name: safe learning
Definition: AGI safety research problem area of how to create AGIs that avoid making fatal mistakes during the learning phase.
Extra: The current techniques (as of 2018) for averting these problems are by providing a catastrope detector with labelled real examples or unlabelled machine-generated examples of catastrophes, then letting it overridde the agent's actions whenever it judges an action to be too dangerous.
Brief overview of the research area (p9) and solution (p15): https://arxiv.org/pdf/1805.01109.pdf

Name: intelligibility
Definition: AGI safety research problem area of how to build agents whose decisions we can understand.
Similar to the "explainable AI" effort or "interpretability problem" of the XAI program at DARPA, except focused on AGI instead of narrow AIs.
Brief overview of the research area (p9): https://arxiv.org/pdf/1805.01109.pdf

Name: distributional shift problem
Definition: Problem in AI that exists in agents that learns a reward function from the actions and preferences of human actors. If all training happens before the agent is launched into the environment, the data it observed may not properly describe situations that the agent reaches far into its lifetime, making it unable to adapt to these circumstances.
Described in Amodei et al,. 2016: https://arxiv.org/abs/1606.06565

Name: reward gaming problem
Definition: Problem in AIs that learn a reward function from actions and preferences of humans. Rather than doing what the human designers wanted it to, an AI may do something unanticipated (i.e. misaligned with the humans' goals) because the reward function was subtly misspecified.
Extra: "One potential way around the problem of gameable reward functions is to let the agent learn the reward function. This lets designers offload some of the design work to powerful machine learning techniques." (Everitt et al., 2018)
https://arxiv.org/pdf/1805.01109.pdf
Examples include those described in Clark and Amodei (2016), Gwern (2011), Irpan (2018), and Lehman et. al. (2018).

Name: inverse reinforcement learning
Definition: Framework for an agent to learn a reward function from the actions of an expert, often a human demonstrator. While avoiding the reward gaming problem (i.e. subtly misspecified reward function), the learned reward function cannot be better than the data that trained it.
Context: AI
Abbreviation: IRL

Name: cooperative inverse reinforcement learning
Definition: Generalization of IRL that lets the expert and the agent act simultaneously in the same environment, with the agent interactively learning the expert's preferences.
Among other things, this allows the expert to take demonstrative actions that are suboptimal according to his or her reward function but more informative to the agent, without the agent being led to infer an incorrect reward function. It can be used to build agents that avoid interpreting reward functions overly literally, thus avoiding some of the misalignment problems with RL.
More in Hadfield-Menell, Dragan, et al., 2016 and Hadfield-Menell, Milli, et al., 2017.
Abbreviation: CIRL

Name: approval-directed agent
Definition: Potential AGI structure developed to maximize approval for its actions rather than trying to reach some goal, which could have many of the same benefits of goal-directed systems while avoiding some of their worst pitfalls.
Extra: Christiano (2015) and Cotra (2018) outline a method for how these agents can be chained together in a hierarchy, boosting the accuracy of the approvals of the human at the top of the chain.

Name: reward corruption
Definition: A possible value specification failure mode in which an RL AGI hijacks its reward signal and feeds itself maximal reward.
Relatedly, rather than hijacking the reward signal, it could hijack its reward function in the online learning process by influencing training data to point towards a simple-to-optimize reward function. This is "reward learning corruption," discussed by Armstrong, 2015 and Everett, Krakovna, et al., 2017.

Name: expected utility quantilization
Definition: Alternative to expected utility maximization which could make an AGI safer. It is the approach of selecting a random action in the top n% of actions from some distribution, sorted by expected utility.
The agent's utility function makes it behave somewhere between a human and an expected utility maximizer. It is directed toward intuitively desirable outcomes in novel ways, making it potentially more useful than a digitized human, while being tempered by the distribution toward safer and more predictable strategies.
More:
http://wrai.org/slides/Reinforcement Learning with a Corrupted Reward Channel - Victoria Krakovna.pdf
https://intelligence.org/2015/11/29/new-paper-quantilizers/

Name: AIXI
Definition: Theoretical mathematical formalization of AGI first proposed by Marcus Hutter in 2000. Structurally it is a reinforcement learning agent, meaning that it maximizes the expected total rewards received from the environment. It considers all possible programs at each time step and then chooses the action that provides the most expected value, i.e. the one that has the greatest combination of reward and probability of being true.
This formalization is incomputable because it would need to consider all possible actions (which is impossible given time and space limitations). Therefore the real form an AGI could take would be an approximation.

Name: capability control methods
Definition: One of the two broad classes of approaches to the AGI control problem, this area focusing on limiting the system's ability to cause harm.
Definition2: MIRI tackles the problem with their Agent Foundations technical agenda.
Name2: highly reliable agent design (miri)

Name: coherent extrapolated volition
Definition: The ideal solution to the value specification problem in which an AGI would act in humanity's best interests, doing what we want it to do (all things considered) instead of what we tell it to do.
Extra: Unfortunately it would be "a thousand lightyears beyond hopeless" to implement an ordinary computer program that captures this, not to mention that human values may not converge. Therefore since Yudkowsky created this concept other safety writers have proposed more feasible alternatives, including Bostrom's Hail Mary approach wherein an AGI would act on a set of simpler, perhaps less ideal but acceptable values.

Name: Hail Mary approach
Definition: Possible solution to the value specification problem proposed by Nick Bostrom. If an AGI were would act on a set of simpler values that, while unlikely to be ideal, may lead to "a better outcome than would be obtained by means of a failed attempt to implement some complex ideal value."
Rather than specifying values manually (which entails all sorts of coding difficulties), the AGI would be made to be "motivated to act in accordance with its beliefs about what alien AIs would have told it to do if they had (counterfactually) been asked about the matter."
Paper in which Bostrom first proposes the idea: https://pdfs.semanticscholar.org/f447/c5858b0f31ecda40261f8a6cda8ee3dac9da.pdf

Name: fairness, accountability, and transparency in machine learning
Definition: Focus on ensuring non-discrimination, due process, and understandability in decision-making of AI algorithms. Mostly addresses short-term issues but applicable to long-term work as well.
Abbreviation: FAT/ML

Name: categories of long-term, global/extreme risks from AI
Definition: 1. Automation: Mass labor displacement, unemployment, and inequality from (a) machines being good enough and cheaper than humans in price-sensitive domains or (b) machines being quite a bit better, even if more expensive, than humans in quality-sensitive domains
2. AI oligopolies: AI may make companies more prone to become natural monopolies, and may make it harder for new companies to enter the space)
3. Surveillance and control: AI and associated technologies may allow authorities to do mass surveillance, intimate profiling, tailored persuasion, and robotic repression
4. Strategic (nuclear) stability: Incentive with the increasing speed of tech to automate systems of intel, sensors & drones, hypersonics, and C3, moving humans out of the loop and therefore creating vulnerabilities that humans wouldn't usually be susceptible to
Discussed in Allan Dafoe's "Governance of AI" talk: https://www.youtube.com/watch?v=RWKHx2bE1H4

Name: one-shot learning
Definition: Object categorization problem in computer vision wherein the aim is to learn information about object categories from one, or only a few, training images.
Extra: As of May 2018, still a changing area of AI development that may be slowing down our progress towards AGI.

Name: unipolar vs. multipolar
Definition: Worlds in which only one player or many players, respectively, are in a realistic position to reach AGI. It's important to consider which of the two is likely to make safe AGI development outcomes more likely when determining the ideal coordination strategy.
The Foresight Institute white paper briefly lists the advatanges and disadvantages of each world (p17):
https://foresight.org/publications/AGI-Timeframes&PolicyWhitePaper.pdf

Name: offense-defense balance
Definition: This theory offers an optimistic view of international politics based on the argument that war can be prevented if defense gains an advantage over offense. It also argues that an effective arms control can reduce the risk of arms races and war
Name2: the security dilemma theory
Extra: Overview here: https://web.stanford.edu/class/polisci211z/2.1/Glaser & Kaufmann IS 1988.pdf
In the context of transformative AI, scholars worry that AI-backed technologies will lend an advantage to those in the offensive position, upsetting what they see as being the current defense-dominant position that lends itself to states feeling secure and acting benignly.
Paper against the theory by Keir Lieber, Director of the Center for Security Studies at Georgetown: https://web.stanford.edu/class/polisci211z/2.3/Lieber IS 2000.pdf

Name: first-mover advantage
Definition: Concept in game theory wherein a player achieves a higher payoff by turning the game into a sequential one with him being the first mover.
Extra: If in a battle what can be seen can be hit, military actors should be primed for the offense, i.e. trying to make the first move by striking first. It holds true in particular in naval engagements and aerial dogfights.
In the AI realm, if the first actor to develop a transformative AI capability becomes unusually able to capture most of the market or acquire most of the power, there's a strong incentive for them to achieve that technological milestone first. If fast development is likely to make development less safe, perception of a first-mover advantage (even if the advantage doesn't exist) will increase the chances of unsafe development, either just from the player that believe this advantage to exist or also from the many players partaking in the race.

Name: dual-use technology
Definition: In politics and diplomacy this refers to technology that can be used for both peaceful and military aims. Examples include missiles, nuclear power, industrial chemical processes, biological sciences, and night vision & thermal imaging.
Basic research in AI has the ability to augment many processes in many domains, landing it this classification as well. While most industrial countries have export controls on certain types of designated dual-use technologies, AI may be particularly difficult to regulate and control, given that it is software-based, relative easy for small private actors to develop, and has functions likely to be used in many more industries than other such technologies.

Name: cryptography
Definition: The study of techniques for encoding, protecting, and authenticating data. Associated with the complementary field of cryptanalysis, which focuses on decoding.
Progress in AI and progress in cryptography could be relevant to one another in many ways. AI systems may:
- Enable and motivate more effective surveillance
- Help to make privacy-preserving surveillance feasible
- Increase the need for anti-forgery schemes
- Be trained in a decentralized manner with methods of computing on confidential data
- Be designed to interact with decentralized applications
- Have safe design problems that overlap with those of smart contract design
- Be governed by new coordination and verification mechanisms
- Be governed differently as a result of changes to the political landscape
- Be made safer with the help of fully homomorphic encryption
Details about the overlap of AI and cryptography in Ben Garfinkel's "Recent Developments in Cryptography and Possible Long-Run Consequences" (not yet public as of May 2018)
Context: As it relates to AI

Name: digital security
Definition: One of the three domains of security affected by developments in AI.
The use of AI to automate tasks involved in carrying out cyberattacks will alleviate the existing tradeoff between the scale and efficacy of attacks. This may expand the threat associated with labor-intensive cyberattacks (such as spear phishing). We also expect novel attacks that exploit human vulnerabilities (e.g. through the use of speech synthesis for impersonation), existing software vulnerabilities (e.g. through automated hacking), or the vulnerabilities of AI systems (e.g. through adversarial examples and data poisoning).
Discussed here: https://img1.wsimg.com/blobby/go/3d82daa4-97fe-4096-9c6b-376b92c619de/downloads/1c6q2kc4v_50335.pdf
Context: As it relates to AI

Name: physical security
Definition: The use of AI to automate tasks involved in carrying out attacks with drones and other physical systems (e.g. through the deployment of autonomous weapons systems) may expand the threats associated with these attacks. We also expect novel attacks that subvert cyberphysical systems (e.g. causing autonomous vehicles to crash) or involve physical systems that it would be infeasible to direct remotely (e.g. a swarm of thousands of micro-drones).
Discussed here: https://img1.wsimg.com/blobby/go/3d82daa4-97fe-4096-9c6b-376b92c619de/downloads/1c6q2kc4v_50335.pdf
Context: As it relates to AI
Extra: One of the three domains of security affected by developments in AI

Name: political security
Definition: One of the three domains of security affected by developments in AI.
The use of AI to automate tasks involved in surveillance (e.g. analysing mass-collected data), persuasion (e.g. creating targeted propaganda), and deception (e.g. manipulating videos) may expand threats associated with privacy invasion and social manipulation. We also expect novel attacks that take advantage of an improved capacity to analyse human behaviors, moods, and beliefs on the basis of available data. These concerns are most significant in the context of authoritarian states, but may also undermine the ability of democracies to sustain truthful public debates.
Discussed here: https://img1.wsimg.com/blobby/go/3d82daa4-97fe-4096-9c6b-376b92c619de/downloads/1c6q2kc4v_50335.pdf
Context: As it relates to AI