Custom individuals & groups not saved in custom relationships

Using CiviCRM 4.5.8 under Joomla!, on my system and on the demo CiviCRM install, when I create a Custom Relationship and try to set Subtypes, they don't save, and the search filter doesn't work as expected when creating a new Relationship of that type.

For example, on the demo site, I created a Fills role for / Role filled by Relationship. If I try to set Contact Type A to Student, and Contact Type B to Team, it saves A as "All Contacts" and B as "Organization." On my system, I can use phpmyadmin to edit the civicrm_relationship_type -> contact_sub_type_a and b to make it work as desired, but if I edit the Relationship through CiviCRM interface, it loses the sub_types again.