Agent loitering around goal rather than reaching it

[gdaddi]

Hello! Can a problem be set up to terminate once a goal is reached?

I have a reward function that increases as the agent's position gets closer to a goal and a terminal condition set to trigger within a certain radius of the goal. The behavior I get with this reward/terminal structure is that the agent moves around the terminal condition accumulating reward but never entering the terminal condition. Is there a way to have the agent move into the terminal-goal condition? Thanks

function POMDPs.reward(p, s, a)
   if distance(agent, goal) < threshold
      return 100.
   end
   else
       return f(s)  ### f(s) < 100 for all s
   end
end

function POMDPs.isterminal(p, s)
   if distance(agent, goal) < threshold
       return true
   else
      return false
   end
end

[lassepe]

It looks like your agent just realized that it will no longer receive a reward once the problem terminated. So the optimal thing to do is to stay close to the goal and never terminate. You could set a very high goal reward and a discount factor < 1 or a step cost.

[gdaddi]

Thanks! I am using a discount factor of 0.9 and tried changing the goal's reward to something 8 orders of magnitude larger than the upper bound of f(s). Unfortunately, the agent still avoids terminal conditions.

[zsunberg]

I think the issue is that in POMDPs.jl you can't receive any reward from a terminal state. As explained here, the value of a terminal state is always zero. In a simulation, if a terminal state is reached, the simulation never reaches another line where a reward is assigned.

So, you need to modify your problem to either assign reward upon entering a terminal state or you need to make the goal state non-terminal and add an extra state to transition to after the goal.

The first option would look something like this: (note that I added sp and the threshold calculation is done with sp.agent)

function POMDPs.reward(p, s, a, sp)
   if distance(sp.agent, p.goal) < threshold
      return 100.
   else
       return f(s)  ### f(s) < 100 for all s
   end
end

The second option is implemented here: https://github.com/JuliaPOMDP/POMDPModels.jl/blob/master/src/gridworld.jl . There is a special [-1,-1] state that is terminal, so the agent first enters a goal state (e.g. [9,3]) and then on the next step receives the reward and always transitions to [-1, -1].

[gdaddi]

@zsunberg Thank you so much for the clear explanation!!! Tried out the second solution you suggested, and it worked out perfectly.