Enter Quick Mask mode and Gaussian Blur the rectangle. I've used 8 pixels. The larger the number, the rounder the rectangle will be.