Once you've given the first banana to the monkey, the monkey will stay in place, and the next item of your inventory (the second banana) will automatically get surrounded by the yellow box. This means you can give all the other bananas to the monkey without moving the cursor, using just the keyboard.