You might want to get in touch with Andrew who developed the Wordpress Connector and see if he has had any experience with that many pages. It's a relatively new module so may not have been tried on a site that big.
I did contact Andrew and he was kind enough to look into the problem. Well according to him the the default Wordpress XML-RPC API does not allow to fetch any sub-set of posts hence we can't get limited number of post at a time. He adviced increasing the memory for PHP to few GB but again no luck.
However as a work around I deleted my post and brought it down to 600 post. Now atleast these post show up in External Content >> WordpressPostSource. But again the actual import fails. I tried queuing it and got an error, I have attached the file.
I can say definitively that an Import of Wordpress Blog content using a WordpressPostContentSource connector with all the latest from github:
External Content Module 0.2.7
Blog Module 0.4.1
Word Press Connector Module 0.1-beta1
Queued Jobs Module 1.2.3
Does not work. You get a message saying the import is starting and then nothing happens after that and nothing is imported, even if you kick the task off using http://path.to.silverstripe/dev/tasks/ProcessJobQueueTask.
Any guidance on this would be greatly appreciated. I have too many posts to do manually.