Paul Christiano

PaulChristiano.json

https://arbital.com/p/PaulChristiano

by Paul Christiano Sep 4 2015

Index
Paul Christiano

Automatically generated page for "Paul Christiano" user. Click here to edit.

Related

Imitation-based agent An AI meant to imitate the behavior of a reference human as closely as possible. - Eliezer Yudkowsky

Created

wiki

A possible stance for AI control research I think that AI control research should focus on finding [scalable](https://arbital.com/pages/492374…
AI control on the cheap Ideally, we will build aligned AI systems without sacrificing any efficiency. I think this is a rea…
Abstract approval-direction Consider the following design for an agent, which I first described [here](https://arbital.com/p/1t7…
Act based agents I’ve recently discussed three kinds of learning systems: - [Approval-directed agents](https://arbit…
Active learning for opaque, powerful predictors (An open theoretical question relevant to AI control.) Suppose that I have a very powerful predicti…
Adversarial collaboration Suppose that I have hired a group of employees who are much smarter than I am. For some tasks it’s…
Advisor games Machine learning algorithms often learn models or policies that are inscrutable to humans. We belie…
Against mimicry One simple and apparently safe AI system is a “copycat:” an agent that predicts what its user woul…
Ambitious vs. narrow value learning Suppose I’m trying to build an AI system that “learns what I want” and helps me get it. I think tha…
Apprenticeship learning and mimicry This post compares my [recent proposal](https://arbital.com/p/1vp/mimicry_meeting_halfway) with [Ab…
Approval directed agents Research in AI is steadily progressing towards more flexible, powerful, and autonomous goal-directe…
Approval-based agents An alternative to goal-directed behavior
Approval-directed bootstrapping Approval-directed behavior works best when the overseer is very smart. Where can we find a smart o…
Automated assistants In my [last post](https://arbital.com/p/1th?title=implementing-our-considered-judgment), I describ…
Challenges for safe AI from RL In this post, I’ll describe and discuss a few big problems for the proposal from [my last post](htt…
Concrete approval-directed agents This post lays out my current concrete “[approval-directed agents](https://arbital.com/p/1t7)” propo…
Counterfactual oversight vs. training data I have written a lot recently about [counterfactual human oversight](https://arbital.com/p/1tj?titl…
Delegating to a mixed crowd ### Suppose I have ten programs, each a human-level agent. I suspect that at least one or two of…
Efficient feedback In some machine learning domains, such as image classification, we can produce a bunch of labelled t…
Elaborations on apprenticeship learning Apprenticeship learning (AL) is an intuitively appealing approach to AI control. In AL, a human expe…
Handling adversarial errors Even a very powerful learning system can’t do everything perfectly at first — it requires time to l…
Handling errors with arguments My [recent proposal](https://arbital.com/p/1v7?title=steps-towards-safe-ai-from-online-learning) f…
How common is imitation? How often do we train machine learning systems to imitate human behavior? Some researchers explicit…
Human arguments and AI control ### Explanation and AI control Consider the definition: > An action is good to the extent that I w…
Human in counterfactual loop Consider an autonomous system which is buying or selling assets, operating heavy machinery, or mak…
Humans consulting HCH Consider a human who has access to a question-answering machine. Suppose the machine answers questio…
IRL and VOI Consider the following straightforward algorithm based on inverse reinforcement learning: - Given…
Imitation and justification Suppose that I am training an AI system to play Go. One approach is to have the AI observe human m…
Implementing our considered judgment Suppose I had a very powerful prediction algorithm. How might I use this algorithm to build a smar…
Implicit consequentialism Consider a machine that does exactly what its user [would tell it to do](https://arbital.com/p/1tj?t…
In defense of maximization I’ve been thinking about [AI systems that take actions their users would most approve of](https://…
Indirect decision theory In which I argue that understanding decision theory can be delegated to AI. ### Indirect normativit…
Learn policies or goals? I’ve [recently proposed](https://arbital.com/p/1t7/approval_directed_agents) training agents to ma…
Learning and logic In most machine learning tasks, the learner maximizes a concrete, empirical performance measure: i…
Learning representations Many AI systems form internal representations of their current environment or of particular data. Pr…
Mimicry and meeting halfway I’ve talked recently about two different model-free decision procedures: - At each step, pick the …
Modeling AI control with humans I’ve been trying to build an aligned AI out of reward-maximizing modules. A successful scheme could …
Of arguments and wagers (In which I explore an unusual way of combining the two.) Suppose that Alice and Bob disagree, and …
Of simulations and inductive definitions _(Warning: weird.)_ Consider a simple AI system, named A, that carries out a task by predicting wha…
On heterogeneous objectives Eliezer Yudkowsky [has said](https://www.facebook.com/yudkowsky/posts/10153748345169228): > If you …
Online guarantees and AI control I’m interested in claims of the form: “If we had an AI that could do X well, then we could build a…
Optimization and goals If we want to write a program that _doesn’t_ pursue a goal, we can have two kinds of trouble: 1. We…
Optimizing with comparisons I could [elicit a user’s approval](https://arbital.com/p/1w5) of an action _a_ by having them supply…
Paul Christiano's AI control blog Speculations on the design of safe, efficient AI systems.
Problem: safe AI from episodic RL In [a previous post](https://arbital.com/p/1tv?title=the-steering-problem), I posed the steering pr…
Reinforcement learning and linguistic convention Existing machine learning techniques are most effective when we can provide concrete feedback — such…
Research directions in AI control What research would best advance our understanding of AI control? I’ve been thinking about this qu…
Reward engineering This post gestures at a handful of research questions with a loose thematic connection. ### The id…
Safe AI from question-answering _(Warning: minimal new content. Just a clearer framing.)_ Suppose that I have a question-answering…
Scalable AI Control By AI control, I mean the problem of getting AI systems to do what we want them to do, to the best o…
Scalable AI control By AI control, I mean the problem of getting AI systems to do what we want them to do, to the best…
Stable self-improvement as an AI safety problem “Stable self-improvement” seems to be a primary focus of MIRI’s work. As I understand it, the proble…
Steps towards safe AI from online learning ### Steps towards safe AI from online learning Suppose that we have a good algorithm for episodic r…
Synthesizing training data [Counterfactual oversight](https://arbital.com/p/1tj?title=human-in-counterfactual-loop) requires th…
Technical and social approaches to AI safety I often divide solutions to the AI control problem into two parts: technical and social. I think a…
The absentee billionaire Once each day, Hugh wakes for 10 minutes. During these 10 minutes, he spends 10 million dollars. The…
The easy goal inference problem is still hard Goal inference and inverse reinforcement learning ------------------------------------------------…
The state of the steering problem The [steering problem](https://arbital.com/p/1tv?title=the-steering-problem) asks: given some powe…
The steering problem Most AI research focuses on reproducing human abilities: to learn, infer, and reason; to perceive,…
Unsupervised learning and AI control Reinforcement learning systems optimize for an objective defined by external feedback — anything fro…

group

Paul Christiano

comment

(This is hard without threaded conversations. Responding to the "agree/disagree" from Eliezer) >The…
(Understandable to focus on explanation for now. Threaded replies to replies would also be great eve…
> If you want to demonstrate competence... you should first think in terms of exposing technically …
> >This is like one step of ten in the act-based approach, and so to the extent that we disagree it …
> Even so, while the outputs are still abstract and not-yet-computed, Alice doesn't have much of a p…
> on my view it seems extremely probable that, whatever we have in the way of AI algorithms short of…
A 5% change in mortality / 15% change in other endpoints would be surprisingly large to me. Does thi…
Act-based is a more general designation, that includes e.g. imitation learning (and value learning w…
As I see it, there are two cases that are meaningfully distinct: (1) what we want is so simp…
Consider an AI system composed of many interacting subsystems, or a world containing many AI systems…
Eliezer [objects](https://arbital.com/p/2fr/?l=2fr#subpage-2h4) to this post's optimism about robust…
Eliezer seems to have, and this page seems to reflect, strong intuitions about "self-modification" b…
Eliezer, I find your position confusing. Consider the first AI system that can reasonably predict y…
I agree that reflective degrees of freedom won't "fix themselves" automatically, and that this is a …
I agree. You can use the results of easier/earlier inferences to guide harder/later inferences, but …
I am pretty surprised by how confident the voters are! Is "arbitrarily powerful" intended to includ…
I can imagine this concept becoming relevant one day. But it seems sufficiently improbable that it d…
I don't see indirect specifications as encountering these difficulties; all of the contenders so far…
I don't see why getting the satisfying assignment really matters. If your AI sometimes declines to a…
I don't think the existence of such a colony would directly mitigate AI risk, but it could help in t…
I don't think you've correctly diagnosed the disagreement yet (your strawman position is obviously c…
I expect you know my answer on this one. I agree that if there is a *really* fast transition (e.g. …
I obviously disagree with "under intelligence explosion scenarios a Singleton seems like a quite pro…
I responded [here](https://arbital.com/p/1w4/?l=1w4#subpage-2hr). > Some of this probably reflects …
I share the concern that people working on value alignment won't understand what has been done befor…
I think it's going to be hard to talk or think clearly about these problems (even at the level of se…
I think that using the uniform prior over observers constitutes a critical learning failure. Calling…
I think the [key question](https://medium.com/ai-control/the-informed-oversight-problem-1b51b4f66b35…
I was comparing act-based agents to what you are calling a genie. Both get objectives from humans an…
I was talking to Chelsea Finn about IRL a few weeks ago, and she said that they had encountered the …
I wouldn't call this "Christiano's hack." I appreciate the implicit praise that I can think up esote…
If the distinguishing characteristic of a genie is "primarily relying on the human ability to discer…
If there aren't side effects, it seems like the answer is probably yes, since vitamin D deficiency s…
In practice, Eliezer often invokes this concept in settings where there *isn't* yet an intelligent a…
In the long run automation will increase the share of income going to capital. I think theory is agn…
In the sudoku and first OWF example, the agent can justify their answer, and its easy to incentivize…
It seems critical to distinguish the cases where 1. We are hoping the AI generalizes the concept of…
It seems unlikely we'll ever build systems that "maximize X, but rule out some bad solutions with th…
It's easy to equivocate between "can be viewed as" and "is." Indeed, any rational agent "can be view…
It's worth pointing out that in our discussions of AI safety, the author (I assume Eliezer, hereafte…
Methodologically, I am trying to understand what approaches may or may not work and what the key dif…
My knee-jerk response to this problem (just as with mind crime and corrigibility) is to try to build…
My views about Eliezer's preferences may depend on the reason that I am running X, rather than merel…
Normally I think that you set the bar too high for yourself. In this case, I think that you would be…
Of course, the game is typically about costs and benefits. Saying "it is good to adopt the security …
Often complaints are with the particular problems which purportedly will require novel solutions or …
On the act-based model, the user would say something like "paint all the cars pink," and the AI woul…
On this definition, what is the difference between "communicating a goal concept" and "communicating…
One natural standard: it should be hard to distinguish an adequate model from the system-to-be-model…
Presumably the advantage of this approach---rather than simply learning to imitate the human burrito…
Re: "poking holes in things," what is an example of a proposal you would ask people to poke a hole i…
Re: simulating a hostile superintelligence: I find this concern really unconcerning. Some points:…
Regarding corporations: I have seen very few arguments about superintelligence that rest on epistem…
Six months and several discussions later this still seems like a serious concern (Nick Bostrom seeme…
Sorry, I tried to be concrete about what we were discussing, but I will try harder: Consider some p…
Superficially, there are two quite different concerns: 1. You optimize a system for X. You are unha…
The ZFC provability box is equivalent to a good SAT solver, up to a constant factor (and I don't see…
The obvious patch is for a sufficiently sophisticated system to have preferences over its own behavi…
There seems to be some equivocation here between two motivations for studying corrigibility. As far…
These arguments seem weak to me. - I think the basic issue is that you are not properly handling u…
This (and many of your concerns) seem basically sensible to me. But I tend to read them more broadly…
This is a more general pattern in theoretical research. When you first start to attack a hard proble…
This isn't the case in modern cryptography, except perhaps for the design of ciphers. It seems at be…
This seems like a good example to have at hand. I'm skeptical that it's much easier than what we rea…
This seems like a straw alternative. More realistically, we could imagine an agent which avoids pert…
This topic consistently frustrates me; the proposed typology is obviously incomplete, and I don't th…
To me, the most natural way to approach this is to take a probability distribution over "what it mea…
To the extent that humans can imagine these kinds of scenarios, it seems pretty futile to try to pre…
We can imagine two regimes of this problem: in the weak regime the AI may make a small number of err…
Yeah, thanks.
Your characterization of utility indifference doesn't seem quite right. More accurate would be: the …

Edited

Neutral genie metaphor Definition. A neutral-genie metaphor is an attempt to illustrate a possible formal problem via an in… - Alexei Andreev
Researchers in value alignment theory Who's working full-time in value alignment theory? - Eliezer Yudkowsky