master-degree-notes/Autonomous Networking/notes/7 RL.md

Case study: battery-free smart home

• each device produces a new data sample with a rate that depends on the environment and the user (continuously, event based / on demand...)
• a device should only transmit when it has new data
  • but in backscattering-based networks they need to be queried by the receiver

In which order should the reader query tags?

• assume prefixed time slots
• TDMA with random access performs poorly
• TDMA with fixed assignment also does (wasted queries)
• we want to query devices that have new data samples and avoid
  • data loss
  • redundant queries

Goal: design a mac protocol that adapts to all of this. One possibility is to use Reinforcement Learning

Reinforcement learning

How can an intelligent agent learns to make a good sequence of decisions

• an agent can figure out how the world works by trying things and see what happens
• is what people and animals do
• we explore a computational approach to learning from interaction
  • goal-directed learning from interaction

RL is learning what to do, it presents two main characteristics:

• trial and error search

• delayed reward

• sensation, action and goal are the 3 main aspects of a reinforcement learning method

• a learning agents must be able to

  • sense the state of the environment
  • take actions that affects the state

Difference from other ML

• no supervisor
• feedback may be delayed
• time matters
• agent action affects future decisions
• a sequence of successful decisions will result in the process being reinforced
• RL learns online

Learning online

• learning while interacting with an ever changing world
• we expect agents to get things wrong, to refine their understanding as they go
• the world is not static, agents continuously encounter new situations

Rewards

• a reward is a scalar feedback signal (a number)
• reward Rt indicates how well the agent is doing at step t
• the agent should maximize cumulative reward

RL based on the reward hypotesis all goals can be described by the maximization of expected cumulative rewards

communication in battery free environments

• positive rewards if the queried device has new data
• else negative

Challenges:

• tradeoff between exploration and exploitation
• to obtain a lot of reward a RL agent must prefer action that it tried in the past
• but better actions may exist... So the agent has to exploit!

exploration vs exploitation dilemma:

• comes from incomplete information: we need to gather enough information to make best overall decisions while keeping the risk under control
• exploitation: we take advanced of the best option we know
• exploration: test new decisions

A general RL framework

at each timestamp the agent:

• executes action At
• receives observation Ot
• receives scalar reward Rt

the environment:

• receives action At
• emits observation Ot
• emits scalar reward Rt

agent state: the view of the agent on the environment state, is a function of history

• the history is involved in taking the next decision:
  • agent selects actions
  • environment selects observations/rewards
• the state information is used to determine what happens next
  • state is a function of history: S_t = f(H_t)

Inside the agent

one or more of these components

• Policy: agent's behavior function
  • defines what to do (behavior at a given time)
  • maps state to action
  • core of the RL agent
  • the policy is altered based on the reward
  • may be
    • deterministic: single function of the state
    • stochastic: specifying probabilities for each actions
      • reward changes probabilities
• Value function:
  • specifies what's good in the long run
  • is a prediction of future reward
  • used to evaluate the goodness/badness of states
  • values are prediction of rewards
  • V_\pi(s) = Ep[yRt+1 + y^2Rt+2 ... | St = s]
    • better explained later
• Model:
  • predicts what the environment will do next
  • may predict the resultant next state and/or the next reward
  • many problems are model free

back to the original problem:

• n devices
• each devices produces new data with rate_i
• in which order should the reader query tags?
• formulate as an RL problem
  • agent is the reder
  • one action per device (query)
  • rewards:
    • positive when querying a device with new data
    • negative if it has no data
    • what to do if the device has lost data?
  • state?