Hi, I wouldn't advise cancelling your direct debit in an effort to dodge the fee. Most car insurance polices are unfortunately annual, and thus you do incur an early cancellation charge if you decide to terminate any time during the course of the year/before your renewal date. In addition to this, depending on the final balance, most companies will have no qualms about issuing you with a CCJ or a default if you continue to ignore their letters chasing for an outstanding balance.
As mentioned above, Elephant are required to make you aware of this at point of sale as well as in your original documentation, so scan your policy and see if it is mentioned. If it isn't then by all means dispute it, but be aware that the adviser who sold you the policy usually includes it in the scripts at point of sale. Likewise the internet website where you originally set it up will most likely include it in the small print/terms & conditions, and thus from a legal standpoint the company has done everything they are required to do.