I have 2 systems running side by side, but they're using 2 different log in systems. One's a custom built system (PHP), running at for example, 'customscript.example.com', and then SilverStripe running at example.com.
Depends how your custom system is built and what you want to use as your 'source' database and your slave. I would recommend SilverStripe be your master. In that case you need to push any SS logins to your custom app. SilverStripe does give you a hook to do this in Member 'memberLoggedIn' (https://github.com/silverstripe/silverstripe-framework/blob/3.1/security/Member.php#L401) which you could push the login state too. To get users for custom.example.com to login to the main app you could try link them to example.com/Security/login?BackURL=http://custom.example.com
if you want to use it as a slave then SilverStripe includes an ability to log users in programmatically which you could write a script for.