Whether you take out all or only part of your ISAs is a decision only you can make. Once taken out of the ISA wrapper, that money can't be put back, only next year's new allowance.

As far as bank of mattress is concerned, forget it. If you're after safety as a priority, then I don't think there's anything safer than Treasury backed Premium Bonds. You can't lose (except for inflation eating your principle if you don't win anything), but any prizes you do win are tax free. It's certainly safer than under the mattress, and you stand as good a chance as anybody in winning something. With a block as big as £20,000 you should receive a few irregular £50 prizes (as is the experience of someone I know), and this would possibly be a better return than the rotten present interest rates in banks. You can always live in hope of the million!