Belief associated with initial state and restricting actions for certain states

[Manavvora]

Hi,

I am using the BasicPOMCP algorithm for solving my POMDP. The state for my POMDP is a tuple (s,c) and I declare my POMDP using the QuickPOMDPs interface. I assign the initial state as Deterministic((100,0)). I have two questions:

1. What is the belief associated with my initial state and how can I get it? More specifically I want the belief associated with my initial state to also be Deterministic((100,0)). Is there a way to do this?
2. When I'm at some state (s,c) such that c is equal to some constant b, I want the solver to consider only certain actions. Is there a way to do this?

[zsunberg]

What is the belief associated with my initial state and how can I get it? More specifically I want the belief associated with my initial state to also be Deterministic((100,0)). Is there a way to do this?

By default, if you run simulate(HistoryRecorder(), pomdp, policy, belief_updater), it will start the belief out at the belief as you wanted. If you want to get an action from that belief with BasicPOMCP, you can use

planner = solve(POMCPSolver(#=params=#), pomdp)
a, info = action_info(planner, Deterministic((100, 0)))

When I'm at some state (s,c) such that c is equal to some constant b, I want the solver to consider only certain actions. Is there a way to do this?

Yes, you can define the POMDP with an actions function with a second argument that is the belief, for instance

function actions(m, b)
    if all(state[2] == b for state in support(b))
        return # limited action space
    else
        return # full action space
    end
end

[zsunberg]

Hi I had a doubt regarding the implementation. How will I write an if condition involving the state variable outside the QuickPOMDP? The state will not be defined outside right? And also the condition is not on the belief but just some constant number b_0. So that would just be the same right?

I'm not 100% sure what you mean here. It sounds like you may want to create a new POMDP each time b_0 changes. In QuickPOMDPs, you could do something like

if b_0 == 1.0
    as = # limited action space
else
    as = # full action space
end

QuickPOMDP(
    actions = as
    # ...
)

But at this point it might be better to upgrade to the full object oriented (non-quick) interface so that the POMDP can have a parameter b_0:

struct MyPOMDP <: POMDP{Tuple{S, C}, A, O}
    b_0::Float64
end

function POMDPs.actions(m::MyPOMDP)
    if m.b_0 == 1.0
        return # limited action space
    else
        return # full action space
    end
end

Or is it the case that POMCP calls the QuickPOMDP multiple times for different states?

When the POMCP tree is created at the root, it calls actions with a b argument that is the belief it is currently planning at: https://github.com/JuliaPOMDP/BasicPOMCP.jl/blob/120d9cbcb2f487e7951c266b7ff42c1afaae4512/src/BasicPOMCP.jl#L134 and then whenever a new leaf argument is created, it also calls actions with a belief argument: https://github.com/JuliaPOMDP/BasicPOMCP.jl/blob/120d9cbcb2f487e7951c266b7ff42c1afaae4512/src/BasicPOMCP.jl#L180

[Manavvora]

Thank you for your response. The constant b_0 is actually fixed. What I want to do is that:
If state[2] >= b_0:
Limited action space
Else
Full action space

So my doubt is, if I’m using a QuickPOMDP, can I define such a function that references the state before I create the POMDP? Also, inside the QuickPOMDP I declare actions = some action space. So if I call the above defined function here, will the action space change for different states or will it just become fixed once I call it? What I mean exactly is if my initial state is such that I can consider the entire action space, my a = {a1,a2,a3}. However further down the line, I encounter a state where I don’t want to consider a3. So if my code is something like:

`function selective_actions(s,b_0)
#logic for code here
end

POMDP = QuickPOMDP {
actions = selective_actions(s,b_0)
}`

will such a code ensure that when I run simulations and s changes, the action space will also change accordingly?

I hope this makes my exact question clear.

[zsunberg]

Ah, I see, I accidentally used b in two places which probably confused you. How about this:

const b_0 = 3.0

function selective_actions(belief)
    if all(state[2] == b_0 for state in support(belief))
        return # limited action space
    else
        return # full action space
    end
end

selective_actions() = # full action space if no argument given

QuickPOMDP(
    actions = selective_actions,
    # ...
end

The actions function will be called to get the action space each time a new belief is encountered.

when I run simulations and s changes, the action space will also change accordingly?

Yes, you can verify this yourself with something like:

m = QuickPOMDP(...)

initial_belief = Deterministic((100, 0))
actions(m, initial_belief)

future_belief = Deterministic((87, 3))
actions(m, future_belief)

[zsunberg]

In the code above, we are "capturing" variable b_0 in a "closure". That way it doesn't need to be fed in as an argument. See more here:

https://docs.julialang.org/en/v1/devdocs/functions/#Closures
https://docs.julialang.org/en/v1/manual/performance-tips/#man-performance-captured

[Manavvora]

This worked for me. Thanks a lot for your help!