It looks like a Smart Group can be created from a "custom search" (depending on what methods are defined in the custom search class). However, we need to test / verify this and document the exact requirements in the class methods for making this work properly. Also we need to test / verify that smart groups created in this manner can be used as "Mailing" groups in a CiviMail mailing (i.e. that required info is available to for the mailing to happen and the correct recipients are targeted etc.).