Confused About Math Notations in RL

Hi all,

While learning reinforcement learning, I've been struggling with the math, especially expectation notation. For example, the value function is often written as:

$$

V^\pi(s)=\mathbb{E}_\pi\left\{R_t \mid s_t=s\right\}=\mathbb{E}_\pi\left\{\sum_{k=0}^{\infty} \gamma^k r_{t+k+1} \mid s_t=s\right\}

$$

What exactly does $\mathbb{E}_\pi$ mean? I thought the subscript should be a probability distribution or random variable, but $\pi$ is a policy (a function).

This confusion also comes up with trajectory definitions like:

$$

P(\tau \mid \pi)=\rho_0\left(s_0\right) \prod_{t=0}^{T-1} P\left(s_{t+1} \mid s_t, a_t\right) \pi\left(a_t \mid s_t\right)

$$

The problem is that, while I can *understand* these notations, it seems that writing things like E_\pi and \underset{\tau \sim \pi}{\mathrm{E}}\left[R(\tau) \mid s_0=s, a_0=a\right] is not rigorous mathematically.

Any advice about that?