installing new toilet

And what seems to be the problem ? You should have plumbers mate to re seal

 

A new adaptor is always a good choice

 

If that is cement mortar used to 'seal' the end, then take care in its removal so's you don't crack the cast pipe. That shouldn't happen, but just don't go crazy with a club hammer and chisel.

Once the old plastic adaptor is out and you've test-fitted the new pan, it should become clear which kind of adaptor you'll need to bridge the two.

You should be able to get a perfect seal into that cast pipe without the use of any mastic around the outside; just choose a good-quality adaptor with a snug-fitting rubber gasket to go in there. Check that the inside of the cast pipe is in good order with no lumps of mortar or anything stuck to it that could cause leaks, etc etc.

Once you've cleared away what's there and placed the new pan in position, you could take another photo and post it on here - I'm sure peeps will advise on what adaptor to go for.

Sometimes - annoyingly - the outlet of the new pan will be lower than the old one, so your adaptor pipe will be sloping slightly upwards. If this turns out to be the case, take measurements and come back for advice.