How can I model this problem of delivering assets by choosing a route with reinforcement learning?

Question

I would like to build a model based on reinforcement learning (RL) for the following scenario

Recommend the best route (of cities listed for a given country) that satisfies the required criteria (museum, beaches, food, etc) for a total budget of $2000.

Based on the recommendation, the user will provide its feedback (as a reward), so the recommendations can be fine-tuned (by reinforcement learning) the next time. I modeled the system this way:

• States = (c,cr), where $c$ is the city and $cr$ is the criteria (history, beach, food, etc)

• Actions = (p) is the price of visiting the city

• Reward: acceptance of the cities selected by end user as a route (1 or 0)

The objective is to decide which list of cities together satisfy the given budget.

Is this MDP model right and how can I implement this? May be the only option is using Monte Carlo methods and linear/dynamic programming.. Is there any other way?

Answer 1

I do not see how you came to choose prices as actions. Normally, actions are something like go left, go right, jump, stay etc. Analogously, I would say that in your case the actions are visiting a certain location, whereas locations are what you referred to as states. I'd go for something like that:

locations = {location1=(c1,cr1), location2=(c1,cr2), ...}
Actions = {Visit location1, Visit location2, ...}
States = {--set of all the possible paths the model can generate until the budget is possibly exhausted--}

The reward function could then be a combination of both the acceptance (vs. rejection) of a route/path by the user and the inverse of the cost associated with the suggested path (because you want the model to favor cheap paths in order to keep your own business costs low). How you balance these two terms is up to exploration.

For rapid prototyping, check out stable baselines, which offers a bunch of highly optimized RL algorithms.

Answer 2

I don't know what is your dataset exactly look like. But based on assumption, I would like to refer something --

You can think your MDP environment this way

action = {stay, go}

reward = {something like based on visitor's satisfaction maybe rating}

state = {current money in hand, city, other some variable those key feature to make next iteration action}

I am working on a project-based stock market trading (sorry, I can not share detail). In stock market trading problem, we need to decide action sequentially ( like per hour or maybe day). If your problem (and data as I assume) is a sequential action selection problem.

More detail: the first day you visited NY and enjoyed your moment and cost $800. Now you wanted to continue the tour in NY or want to go Washington D.C/ Miami in next day. You need to take action based on several subjects (transportation, time in transport vehicle etc). So, action is either stay or go. what is state possibly? state = {NY, $1200, tired or not maybe etc...}

your reward function design may be trickier to design.

I will suggest studying some RL-based real-life problem which will improve understanding more. You can study this one and try to relate with your problem.