# DeepLearning.ai作业:(5-1)-- 循环神经网络（Recurrent Neural Networks）（1）

• 手动建一个RNN模型
• 搭建一个字符级的语言模型来生成恐龙的名字
• 用LSTM生成爵士乐

# Part1:Building a recurrent neural network - step by step

## 1 - Forward propagation for the basic Recurrent Neural Network

RNN cell

1. Compute the hidden state with tanh activation: $a^{\langle t \rangle} = \tanh(W_{aa} a^{\langle t-1 \rangle} + W_{ax} x^{\langle t \rangle} + b_a)$
2. Using your new hidden state $a^{\langle t \rangle}$, compute the prediction $\hat{y}^{\langle t \rangle} = softmax(W_{ya} a^{\langle t \rangle} + b_y)$. We provided you a function: softmax.
3. Store $(a^{\langle t \rangle}, a^{\langle t-1 \rangle}, x^{\langle t \rangle}, parameters)$ in cache
4. Return $a^{\langle t \rangle}$ , $y^{\langle t \rangle}$ and cache

We will vectorize over $m$ examples. Thus, $x^{\langle t \rangle}$ will have dimension $(n_x,m)$, and $a^{\langle t \rangle}$ will have dimension $(n_a,m)$.

RNN forward pass

• 先把 a ,y_pred置为0
• 然后初始化a_next = a0
• 然后经过Tx个循环，求得每一步的a和y以及cache

## 2 - Long Short-Term Memory (LSTM) network

$$\Gamma_f^{\langle t \rangle} = \sigma(W_f[a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_f)$$

$$\Gamma_u^{\langle t \rangle} = \sigma(W_u[a^{\langle t-1 \rangle}, x^] + b_u)$$

$$\tilde{c}^{\langle t \rangle} = \tanh(W_c[a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_c)$$

$$c^{<t>} = \Gamma_f^{<t>} c^{<t-1>} + \Gamma_u ^{<t>} \tilde {c}^{<t>}$$

$$\Gamma_o^{\langle t \rangle}= \sigma(W_o[a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_o)$$
$$a^{\langle t \rangle} = \Gamma_o^{\langle t \rangle}* \tanh(c^{\langle t \rangle})$$

LSTM 单元

• 先将$a^{\langle t-1 \rangle}$ and $x^{\langle t \rangle}$连接在一起变成$concat = \begin{bmatrix} a^{\langle t-1 \rangle} \ x^{\langle t \rangle} \end{bmatrix}$
• 计算以上的6个公式
• 然后预测输出y

Forward pass for LSTM

## 3 - Backpropagation in recurrent neural networks

RNN backward pass

LSTM backward pass