After the standalone install process, an foreign key constraint error is triggered before the "First User Setup" is shown, if there is an active PHP session.
After corrupting a test install, I decided to reinstall. I cleared the database and removed civicrm.settings.php, and went through the installation process again. The installation completed, but I was unable to create the initial user, because an insert to civicrm_acl_cache failed with a foreign key constraint violation. Deleting the session cookie resolved the issue.
The correct behavior would be to discard sessions that belong to a non-existing user.