{
localUrl: '../page/reward_engineering.html',
arbitalUrl: 'https://arbital.com/p/reward_engineering',
rawJsonUrl: '../raw/1vd.json',
likeableId: 'IvanShekerev',
likeableType: 'page',
myLikeValue: '0',
likeCount: '0',
dislikeCount: '0',
likeScore: '0',
individualLikes: [],
pageId: 'reward_engineering',
edit: '5',
editSummary: '',
prevEdit: '4',
currentEdit: '5',
wasPublished: 'true',
type: 'wiki',
title: 'Reward engineering',
clickbait: '',
textLength: '3675',
alias: 'reward_engineering',
externalUrl: '',
sortChildrenBy: 'likes',
hasVote: 'false',
voteType: '',
votesAnonymous: 'false',
editCreatorId: 'PaulChristiano',
editCreatedAt: '2016-03-04 02:04:42',
pageCreatorId: 'PaulChristiano',
pageCreatedAt: '2016-02-03 08:39:15',
seeDomainId: '0',
editDomainId: '705',
submitToDomainId: '0',
isAutosave: 'false',
isSnapshot: 'false',
isLiveEdit: 'true',
isMinorEdit: 'false',
indirectTeacher: 'false',
todoCount: '0',
isEditorComment: 'false',
isApprovedComment: 'true',
isResolved: 'false',
snapshotText: '',
anchorContext: '',
anchorText: '',
anchorOffset: '0',
mergedInto: '',
isDeleted: 'false',
viewCount: '16',
text: '\nThis post gestures at a handful of research questions with a loose thematic connection.\n\n### The idea\n\nConsider the following frameworks:\n\n- [Temporal difference learning](https://en.wikipedia.org/wiki/Temporal_difference_learning): learn to predict the future by predicting tomorrow’s prediction.\n- [Generative adversarial models](http://arxiv.org/abs/1406.2661): learn to sample from a distribution by fooling a distinguisher.\n- [Predictability minimization](http://cognet.mit.edu/system/cogfiles/journalpdfs/neco.1992.4.6.863.pdf): learn to represent data efficiently by making each part of the representation unpredictable given the others.\n\nThese algorithms replace a hard-to-optimize objective with a nicer proxy. These proxies are themselves defined by machine learning systems rather than being specified explicitly. I think this is a really nice paradigm, and my guess is that it will become more important if large-scale supervised and reinforcement learning continues to be a dominant methodology.\n\nFollowing Daniel Dewey, I’ll call this flavor of research “[reward engineering](http://www.danieldewey.net/reward-engineering-principle.pdf).” In terms of tools and techniques I don’t know if this is a really a distinct category of research; but I do think that it might be a useful heuristic about where to look for problems relevant to AI control.\n\n### Relevance to AI control\n\nThough reward engineering seems very broadly useful in AI, I expect it to be especially important for AI control:\n\n- A key goal of AI control is using AI systems to optimize objectives which are defined implicitly or based on expensive human feedback. We will probably need to use complex proxies for this feedback if we want to apply reinforcement learning.\n- Reward engineering seems relatively robust to changes in AI techniques. Uncertainty about future techniques if often a major obstacle to doing meaningful work on AI control in advance (even if only a little bit in advance).\n\n### Applications\n\nI see a few especially interesting opportunities for reward engineering for AI control:\n\n- [Making efficient use of human feedback](https://arbital.com/p/1w1). Here we have direct access to the objective we really care about, and it is just too expensive to frequently evaluate. (_Simple proposal_: train a learner to predict human judgments, then use those predicted judgments in place of real feedback.)\n- [Combining the benefits of imitation and approval-direction](https://arbital.com/p/1vp/mimicry_meeting_halfway). I suspect it is possible to avoid perverse instantiation concerns while also providing a flexible training signal. (_Simple proposal_: use the adversarial generative models framework, and have the operator accomplish the desired task in a way optimized to fool the distinguisher.)\n- [Increasing robustness](https://arbital.com/p/1w9). If our ML systems are sufficiently sophisticated to foresee possible problems, then we might be able to leverage those predictions to avoid the problems altogether. (_Simple proposal_: train a generative model to produce data from the test distribution, with an extra reward for samples that “trip up” the current model.)\n\nIn each case I’ve made a preliminary simple proposal, but I think it is quite possible that a clever trick could make the problem look radically more tractable. A search for clever tricks is likely to come up empty, but hits could be very valuable (and would be good candidates for things to experiment with).\n\nBeyond these semi-specific applications, I have a more general intuition that thinking about this aspect of the AI control problem may turn up interesting further directions.',
metaText: '',
isTextLoaded: 'true',
isSubscribedToDiscussion: 'false',
isSubscribedToUser: 'false',
isSubscribedAsMaintainer: 'false',
discussionSubscriberCount: '1',
maintainerCount: '1',
userSubscriberCount: '0',
lastVisit: '',
hasDraft: 'false',
votes: [],
voteSummary: 'null',
muVoteSummary: '0',
voteScaling: '0',
currentUserVote: '-2',
voteCount: '0',
lockedVoteType: '',
maxEditEver: '0',
redLinkCount: '0',
lockedBy: '',
lockedUntil: '',
nextPageId: '',
prevPageId: '',
usedAsMastery: 'false',
proposalEditNum: '0',
permissions: {
edit: {
has: 'false',
reason: 'You don't have domain permission to edit this page'
},
proposeEdit: {
has: 'true',
reason: ''
},
delete: {
has: 'false',
reason: 'You don't have domain permission to delete this page'
},
comment: {
has: 'false',
reason: 'You can't comment in this domain because you are not a member'
},
proposeComment: {
has: 'true',
reason: ''
}
},
summaries: {},
creatorIds: [
'PaulChristiano'
],
childIds: [
'technical_socail_approach_ai_safety'
],
parentIds: [
'paul_ai_control'
],
commentIds: [],
questionIds: [],
tagIds: [],
relatedIds: [],
markIds: [],
explanations: [],
learnMore: [],
requirements: [],
subjects: [],
lenses: [],
lensParentId: '',
pathPages: [],
learnMoreTaughtMap: {},
learnMoreCoveredMap: {},
learnMoreRequiredMap: {},
editHistory: {},
domainSubmissions: {},
answers: [],
answerCount: '0',
commentCount: '0',
newCommentCount: '0',
linkedMarkCount: '0',
changeLogs: [
{
likeableId: '0',
likeableType: 'changeLog',
myLikeValue: '0',
likeCount: '0',
dislikeCount: '0',
likeScore: '0',
individualLikes: [],
id: '8279',
pageId: 'reward_engineering',
userId: 'JessicaChuan',
edit: '5',
type: 'newEdit',
createdAt: '2016-03-04 02:04:42',
auxPageId: '',
oldSettingsValue: '',
newSettingsValue: ''
},
{
likeableId: '0',
likeableType: 'changeLog',
myLikeValue: '0',
likeCount: '0',
dislikeCount: '0',
likeScore: '0',
individualLikes: [],
id: '7785',
pageId: 'reward_engineering',
userId: 'JessicaChuan',
edit: '4',
type: 'newEdit',
createdAt: '2016-02-25 02:01:40',
auxPageId: '',
oldSettingsValue: '',
newSettingsValue: ''
},
{
likeableId: '0',
likeableType: 'changeLog',
myLikeValue: '0',
likeCount: '0',
dislikeCount: '0',
likeScore: '0',
individualLikes: [],
id: '7754',
pageId: 'reward_engineering',
userId: 'JessicaChuan',
edit: '3',
type: 'newEdit',
createdAt: '2016-02-24 22:58:46',
auxPageId: '',
oldSettingsValue: '',
newSettingsValue: ''
},
{
likeableId: '0',
likeableType: 'changeLog',
myLikeValue: '0',
likeCount: '0',
dislikeCount: '0',
likeScore: '0',
individualLikes: [],
id: '6886',
pageId: 'reward_engineering',
userId: 'JessicaChuan',
edit: '2',
type: 'newEdit',
createdAt: '2016-02-11 22:19:47',
auxPageId: '',
oldSettingsValue: '',
newSettingsValue: ''
},
{
likeableId: '0',
likeableType: 'changeLog',
myLikeValue: '0',
likeCount: '0',
dislikeCount: '0',
likeScore: '0',
individualLikes: [],
id: '6258',
pageId: 'reward_engineering',
userId: 'JessicaChuan',
edit: '1',
type: 'newChild',
createdAt: '2016-02-03 08:40:07',
auxPageId: 'technical_socail_approach_ai_safety',
oldSettingsValue: '',
newSettingsValue: ''
},
{
likeableId: '0',
likeableType: 'changeLog',
myLikeValue: '0',
likeCount: '0',
dislikeCount: '0',
likeScore: '0',
individualLikes: [],
id: '6257',
pageId: 'reward_engineering',
userId: 'JessicaChuan',
edit: '1',
type: 'newEdit',
createdAt: '2016-02-03 08:39:15',
auxPageId: '',
oldSettingsValue: '',
newSettingsValue: ''
},
{
likeableId: '0',
likeableType: 'changeLog',
myLikeValue: '0',
likeCount: '0',
dislikeCount: '0',
likeScore: '0',
individualLikes: [],
id: '6256',
pageId: 'reward_engineering',
userId: 'JessicaChuan',
edit: '0',
type: 'deleteParent',
createdAt: '2016-02-03 08:38:43',
auxPageId: '1vc',
oldSettingsValue: '',
newSettingsValue: ''
},
{
likeableId: '0',
likeableType: 'changeLog',
myLikeValue: '0',
likeCount: '0',
dislikeCount: '0',
likeScore: '0',
individualLikes: [],
id: '6254',
pageId: 'reward_engineering',
userId: 'JessicaChuan',
edit: '0',
type: 'newParent',
createdAt: '2016-02-03 08:38:41',
auxPageId: 'paul_ai_control',
oldSettingsValue: '',
newSettingsValue: ''
},
{
likeableId: '0',
likeableType: 'changeLog',
myLikeValue: '0',
likeCount: '0',
dislikeCount: '0',
likeScore: '0',
individualLikes: [],
id: '6252',
pageId: 'reward_engineering',
userId: 'JessicaChuan',
edit: '0',
type: 'newParent',
createdAt: '2016-02-03 08:38:10',
auxPageId: '1vc',
oldSettingsValue: '',
newSettingsValue: ''
}
],
feedSubmissions: [],
searchStrings: {},
hasChildren: 'true',
hasParents: 'true',
redAliases: {},
improvementTagIds: [],
nonMetaTagIds: [],
todos: [],
slowDownMap: 'null',
speedUpMap: 'null',
arcPageIds: 'null',
contentRequests: {}
}