I now use this URL pattern for my custom pagination:
http://example.com/sub/dir/xyz/?page=5

I would rather use this:
http://example.com/sub/dir/xyz/page5/

However that URL gives a 404 Not Found error.

How can I get a page ending in /page5/ to be controlled by the appropriate controller?

First, you need to tell the controller class for your page that it is allowed to take parameters.

e.g. add this to the contrioller section:

static $allowed_actions = array(
'page'
);

...then you can access the parameters like this...

$action = Director::URLParam('action');
$page = Director::URLParam('ID');
$recsperpage = Director::URLParam('OtherID');

e.g. if you browse to...

http://example.com/sub/dir/xyz/page/5/50

...then using the above assignment code

$action is 'page'
$page is 5
$recsperpage is 50

See http://doc.silverstripe.org/security#limiting_url-access_to_controller_methods

Alternatively, use url handlers: http://doc.silverstripe.org/controller#url_handling

All this is ripe for a tutorial. I wonder if one exists.

Thanks!