Allow CiviCRM contact reference custom fields to be restricted to particular groups

It would be nice to limit contact reference fields to a particular group or smart group, so you could do things like have a field that is a list of a particular type of contact. Maybe exposing those fields to hook_civicrm_customFieldOptions would do the trick?