If you try but do not end up with a policy, you may at some stage decide to contact your bank and ask them to reverse the payment but this should only be done as a last resort.
Just be aware that if you do this the insurance company may instantly end any cover that you have in place and you will need to source a new policy immediately, if you take this route it may be worth making sure you have doen some reearch first as to a new policy provider.
So far as the first company, they are in breach of the contract and legislation for not having provided you with any policy documentation, especially as they have now taken money for the policy so I would find it surprising if they tried to charge you any cancellation fees.
It is also definitely worth remembering that you get what you pay for. If something is far cheaper than other policies it almost certainly means they are not offering something that a more expensive policy will offer, most insurance companies know that people shop on price and don't look to see what it contains and therefore they offer basic bones policies for low prices. Make sure that you do your research based on the type and amount of cover and features that you want and not just on who has the best price.