Scrolling sometimes hits the bottom of the scroll bar without reaching the end of the buffer

Repro:

- Write a lot of text to terminal
- Scroll up and down quickly until the prompt isn't visible

<img width="881" alt="Screen Shot 2021-04-23 at 9 22 00 AM" src="https://user-images.githubusercontent.com/2193314/115900904-5dce4600-a415-11eb-8d27-f3579842c76b.png">
