When using Yes/ No Custom field and Search Builder Field != 1, does not return null records

See the demo site. I created a custom field called "Contact is Removed" and then I went to search builder and added the criteria "Contact is Removed" != 1. I get 0 results. I understand this is because it is joining on civicrm_custom_field_value, but what does someone do, if they add a field (even with a default value like I did) after the contact has been created, and want to do a negative test?