Cheapest to fit will be electric storage heaters (by some considerable margin) -
Oil is only really used as a last resort - when no mains gas is available - it is expensive and dirty - but if there is no choice - electric is the only way to go.....
Incidentally - you don't heat the water and circulate it like a gas powered system - that would be too expensive - simple (reliable) storage heaters are used