If I remember from previous threads don't the lenders say something like the terms and conditions are available on the website and it is up to you to go read them and it states the fees and charging process within these T&Cs and that's how they got around it and apparently it is legal for them to do so.
I have found a letter from a previous thread which was recommended to be used for such cases. You may even want to read the thread. You will see that you are not the first and un likely to be the last in this position and what you are going through is common practice with personal loan lenders, specifically those that deal with client with a touch of adverse credit.
(Courtesy of Resurrection)
[Your Name & Address]
On the [Date] I made an application to your company for a loan of [Amount]. I have a copy of the agreement which was signed and sent back to you. The [Fee £ Amount] administration fee was paid.
I would like to ask that you refund my administration fee minus £5 which by law you are entitled to keep.
I would at this time remind you that Section 155 of the Consumer Credit Act 1974 provides that a credit-broker may charge no more than £5 for his services if the consumer does not enter into a relevant agreement within six months of an introduction to a prospective source of credit or hire. If the broker has already charged more than £5 for his services and section 155 applies, the excess must be refunded to the consumer upon request.
As I have never entered into any relevant agreement the above applies and you are lawfully bound to refund the original administration fee, minus £5.
Should you fail to do so, I will have no option but to report your company to the Office of Fair Trading to whom you are licensed by.
You have 21 days from today in which to refund the requested monies. Please make any cheques payable to [Your Name]
I would suggest you give this a go as this will show them that you are aware of the law under which they operate and your rights to refund and absolutely follow through with it if they do not refund in a timely manner.
Should you receive no action from this then you should take it to the Office of Fair Trading and include all copies of correspondence both too and from the company.