Custom fields can't be edited by a user without "access all custom data" permission

I've got some regular fields and custom data fields represented in a profile which has used for set to View/Edit User Account. A user without "access all custom data" will not see the custom data fields on the view or edit pages of the profile. If you turn on "access all custom data", it works, but the unintentional side effect is that the "View Contact Record" link on the profile view is turned on, which allows the user to see the CiviCRM profile page (not what I want).

I'm not sure if this is a design decision (only one permission that controls read/write to custom fields), but it seems like it's a bug. One should be able to edit custom fields that are represented in a profile without having to turn on that rather global permission.