This has been bugging me so I've just done a little digging into how profiler actually works. If anyone's still interested.
Profiler works by starting and stopping timers around code execution, quite simple really, however, the clever part is that you can have nested timers. When you start a timer within another, the parent timer is suspended.
This is the case with all_execution, which is started in main.php, so essentially the time it reports is really unprofiled time ... i.e. code that does not use profile timers and is not contained within another timer.
To get more detail you really need to dig into the code starting at Director::direct($url); and add timers - although I'm not sure how many levels you can "nest" timers I don't think there's a limit.
Obviously this involves editing core files, however, if you find some code without a profile timer that has a significant chunk of execution time then I'm sure it would be appreciated by the wider community. I'll let you know if I turn up anything.
EDIT: you can also use profile_trace to see the trace and find potential gaps between all_execution resuming and suspending e.g. ?debug_profile=1&profile_trace=1 at the end of your URL