master-degree-notes/Autonomous Networking/notes/7 RL.md

Case study: battery-free smart home
- each device produces a new data sample with a rate that depends on the environment and the user (continuously, event based / on demand...)
- a device should only transmit when it has new data
	- but in backscattering-based networks they need to be queried by the receiver

In which order should the reader query tags?
- assume prefixed time slots
- TDMA with random access performs poorly
- TDMA with fixed assignment also does (wasted queries)
- we want to query devices that have new data samples and avoid
	- data loss
	- redundant queries

Goal: design a mac protocol that adapts to all of this.
One possibility is to use Reinforcement Learning

#### Reinforcement learning
How can an intelligent agent learns to make a good sequence of decisions

- an agent can figure out how the world works by trying things and see what happens
- is what people and animals do
- we explore a computational approach to learning from interaction
	- goal-directed learning from interaction

RL is learning what to do, it presents two main characteristics:
- trial and error search
- delayed reward

- sensation, action and goal are the 3 main aspects of a reinforcement learning method
- a learning agents must be able to
	- sense the state of the environment
	- take actions that affects the state

Difference from other ML
- no supervisor
- feedback may be delayed
- time matters
- agent action affects future decisions
- a sequence of successful decisions will result in the process being reinforced
- RL learns online

Learning online
- learning while interacting with an ever changing world
- we expect agents to get things wrong, to refine their understanding as they go
- the world is not static, agents continuously encounter new situations

RL applications:
- self driving cars
- engineering
- healthcare
- news recommendation
- ...

Rewards
- a reward is a scalar feedback signal (a number)
- reward Rt indicates how well the agent is doing at step t
- the agent should maximize cumulative reward

RL based on the reward hypotesis
all goals can be described by the maximization of expected cumulative rewards

communication in battery free environments
- positive rewards if the queried device has new data
- else negative

Challenge:
- tradeoff between exploration and exploitation
- to obtain a lot of reward a RL agent must prefer action that it tried in the past
- but better actions may exist... So the agent has to exploit!

exploration vs exploitation dilemma:
- comes from incomplete information: we need to gather enough information to make best overall decisions while keeping the risk under control
- exploitation: we take advanced of the best option we know
- exploration: test new decisions

### A general RL framework
**at each timestamp the agent:**
- executes action At
- receives observation Ot
- receives scalar reward Rt

**the environment:**
- receives action At
- emits observation Ot
- emits scalar reward Rt



**agent state:** the view of the agent on the environment state, is a function of history
- the function of the history is involved in taking the next decision
- the state representation defines what happens next
- ...

#### Inside the agent
one or more of these components
- **Policy:** agent's behavior function
	- defines what to do (behavior at a given time)
	- maps state to action
	- core of the RL agent
	- the policy is altered based on the reward
	- may be
		- deterministic: single function of the state
		- stochastic: specifying probabilities for each actions
			- reward changes probabilities
- **Value function:**
	- specifies what's good in the long run
	- is a prediction of future reward
	- used to evaluate the goodness/badness of states
	- values are prediction of rewards
	- Vp(s) = Ep[yRt+1 + y^2Rt+2 ... | St = s]
- **Model:**
	- predicts what the environment will do next
	- many problems are model free

back to the original problem:
- n devices
- each devices produces new data with rate_i
- in which order should the reader query tags?
- formulate as an RL problem
	- agent is the reder
	- one action per device (query)
	- rewards:
		- positive when querying a device with new data
		- negative if it has no data
		- what to do if the device has lost data?
	- state?


### Exploration vs exploitation trade-off
- Rewards evaluate actions taken
- evaluative feedback depends on the action taken
- no active exploration

Let's consider a simplified version of an RL problem: K-armed bandit problem.
- K different options
- every time need to chose one
- maximize expected total reward over some time period
- analogy with slot machines
	- the levers are the actions
	- which level gives the highest reward?
- Formalization
	- set of actions A (or "arms")
	- reward function R that follows an unknown probability distributions
	- only one state
	- ...

Example: doctor treatment
- doctor has 3 treatments (actions), each of them has a reward.
- for the doctor to decide which action to take is best, we must define the value of taking each action
- we call these values the action values (or action value function)
- action value: ... 

Each action has a reward defined by a probability distribution.
- the red treatment has a bernoulli probability
- the yellow treatment binomial
- the blue uniform
- the agent does not know the distributions!
- the estimated action for action a is the sum of rewards observed divided by the total time the action has been taken (add formula ...)
	- 1predicate denotes the random variable (1 if true else 0)

- greedy action:
	- doctors assign the treatment they currently think is the best
	- ...
	- the greedy action is computed as the argmax of Q values
	- greedy always exploits current knowledge
- epsilon-greedy:
	- with a probability epsilon sometimes we explore
		- 1-eps probability: we chose best greedy action
		- eps probability: we chose random action

exercises ...

exercise 2: k-armed bandit problem.
K = 4 actions, denoted 1,2,3 and 4
eps-greedy selection
initial Q estimantes = 0 for all a.

Initial sequenze of actions and rewards is:
A1 = 1   R1 = 1
A2 = 2  R2 = 2
A3 = 2  R3 = 2
A4 = 2  R4 = 2
A5 = 3  R5 = 0

---
step A1: action 1 selected. Q of action 1 is 1
step A2: action 2 selected. Q(1) = 1, Q(2) = 1
step A3: action 2 selected. Q(1) = 2, Q(2) = 1.5
step A4: action 2. Q(1) = 1, Q(2) = 1.6
step A5: action 3. Q(1) = 1, Q(2) = 1.6, Q(3) = 0

For sure A2 and A5 are epsilon cases, system didn't chose the one with highest Q value.
A3 and A4 can be both greedy and epsilon case.

#### Incremental formula to estimate action-value
- to simplify notation we concentrate on a single action
- Ri denotes the reward received after the i(th) selection of this action. Qn denotes the estimate of its action value after it has been selected n-1 times (add Qn formula ...)
- given Qn and the reward Rn, the new average of rewards can be computed by (add formula with simplifications...) $Q_(n+1) = Q_{n} + \frac{1}{n}[Rn - Qn]$
	- NewEstimate <- OldEstimate + StepSize (Target - OldEstimate)
	- Target - OldEstimate is the error

Pseudocode for bandit algorithm:
```
Initialize for a = 1 to k:
	Q(a) = 0
	N(a) = 0
Loop forever:
	with probability 1-eps:
		A = argmax_a(Q(a))
	else:
		A = random action
	R = bandit(A) # returns the reward of the action A
	N(A) = N(A) + 1
	Q(A) = Q(A) + 1\N(A) * (R - Q(A))
```

Nonstationary problem: rewards probabilities change over time.
- in the doctor example, a treatment may not be good in all conditions
- the agent (doctor) is unaware of the changes, he would like to adapt to it

An option is to use a fixed step size. We remove the 1/n factor and add an $\alpha$ constant factor between 0 and 1.
And we get $Q_{n+1} = (1-\alpha)^{n}Q_1 + \sum_{i=1}^{n}{\alpha(1 - \alpha)^{(n-1)} R_i}$


... ADD MISSING PART ...