Deleting membership doesn't delete contribution - but it says it does

If I create a pending membership record and then delete it I get this message:

WARNING: Deleting this membership will also delete related membership log and payment records. This action cannot be undone. Consider modifying the membership status instead if you want to maintain a record of this membership. Do you want to continue?

However, that doesn't happen - the contribution record is not deleted.