I'm still somewhat hazy about this, but I think in general extensions are being invoked in 3.0 by calling Object::add_extension() rather than trying to directly extend a core SS php class. So instead of overloading getCMSActions() in a child class, somehow you have to define updateCMSActions(FieldList $actions) in your extension class, which ends up being called by the base class' getCMSActions() from core SilverStripe to incorporate the extra actions you have defined.
But don't take my word on anything yet: I'm still trying to figure this out.
If you do get it figured out, please post an example. There are definitely mysteries to low level stuff in the CMS for me.
In my case I specifically want to have the actions happen in ModelAdmin and it appears that ModelAdmin doesn't call getCMSActions when it is generated its ItemEditForm so I don't know how external code could inject actions.
My solution, derived from that link I posted, is working for me, but for sure that doesn't mean it is the best one.