Creating profile to search locations for contacts uses primary instead of specified fields

When creating a profile to search for contacts based on location it seems the primary location for the user is taken into account when it shouldn't be. I have contacts with their home address set as their primary. They all have a work address which I wanted to be searchable through a profile I created that contained all work address fields. If a contact has a home address in say Maryland and a work address in Virginia and the Home is set to primary. When you search for that contain in the state of Virginia (work) it will not find any results. If you then change the state to Maryland (even though that is a home address) it will find the contact but then continue to show the work state in the fields listing.

The only way I could work around this is by changing all contacts to have their work address as their primary location. This is going to be a problem with what I am currently working on now because I am going to be giving contacts multiple work addresses but only one will be set as the primary. This means the search will only show one work location when it should list all of them.