Billing address can't be entered with pay later & payment processor enabled

To reproduce (did so on the official demo):
On a contribution setting page, select a payment processor (tested with Paypal and Stripe) AND the pay later option AND "Billing address required".

Result:
On the contribution page, if you select "pay later" the block where users are supposed to provide the billing address disappears. As a result you can not make the payment/continue to the next page but get the error "Billing First Name is a required field; Billing Last Name is a required field....."
If you select the Payment processor instead of Pay later, it's working fine.

I only found one Forum Post describing this problem.