# Proximal Policy Optimization!

**URL:** https://discourse.processing.org/t/proximal-policy-optimization/42565
**Category:** Gallery
**Created:** [August 6, 2023, 7:17pm UTC](https://discourse.processing.org/t/proximal-policy-optimization/42565 "2023-08-06T19:17:46Z")
**Posts on this page:** 1
**Page:** 1

<div class="post-metadata">

### Author: ![SNICKRS](https://avatars.discourse-cdn.com/v4/letter/s/43a26b/32.png) [@SNICKRS](https://discourse.processing.org/u/SNICKRS)
#### Post date: [August 6, 2023, 7:17pm UTC](https://discourse.processing.org/t/proximal-policy-optimization/42565/1 "2023-08-06T19:17:46Z")

</div>

Hey guys! So I decided that I wasn’t gonna take a break after all (lol), because I wanted to challenge myself and code an AI for Pong with PPO in only one day, learning as I go. Since I’d never exposed myself to such a big-boy algorithm before, I was expecting it to be the ultimate challenge, but it is really similar to A2C and A3C, just that we have a fixed trajectory length (number of steps in the environment before training), and we have a couple extra variables, like lambda (smoothing parameter) for the generalized advantage function (new!!), and a new loss function. We just don’t want huge policy updates, so we say the ratio of the probability of one action occurring in now versus before is restricted to a certain trust region. Then the loss function is similar to that of A2C.

Bruh the math notation was wack as heck, and I spent a good hour or two just trying to decipher what the heck was up with the letters with hats and all. After I kind of understood what was going on, I tried implementing it, but of course it didn’t work. I pretty much failed my challenge, but I didn’t not expect that. I went at it again in the morning, and I was able to get the algorithm to work!

However, its performance was dogwater bruh when I tried putting it on snake and pong. It did WORSE than A3C on snake, getting a max score of 12 after 125 episodes before going haywire. It worked kind of well on pong, being able to beat a simple bot I coded 4 times out of 10 after 200 episodes (this is better than me, who could only do it twice out of 10).

However, I was kind of foolish hoping that my pong ai would beat a bot that basically plays perfectly (can’t win all the time though), especially with only 6 inputs.

I am thinking about looking into convolutions in the future, and maybe throwing some convolutional layers onto PPO to make this algorithm work better (since it is good with many inputs - powerful algorithm), but I don’t know the first thing about CNNs, so that will be the next thing I research.

Here’s a picture of pong:

 ![pongai](https://canada1.discourse-cdn.com/flex036/uploads/processingfoundation1/original/3X/f/5/f5337d0e33f543ad2deedbf4be1276dac8ed5e07.png)

If anyone sees this, have a great day!