Police say Sheikh Salim Bakari was confronted by the gunmen and shot at close range.
He was pronounced dead as he was being rushed to a private hospital.
Mombasa County Commander Robert Kitur says detectives have launched investigations and a manhunt for the attackers.
Bakari’s family claims he had received threats to his life.
“They just shot him and fled…he used to receive threats. He had even said that he feared he may be killed any day,” a family member said.
Police sources confirmed that the sheikh who was an imam at Bilal mosque might have been killed by radical Muslim youths behind recent terror attacks in the coastal city.
Detectives have linked a local terror gang to gun attacks targeting security agencies and suspected State informers whom they accuse of being government sympathisers.
Police are investigating radical youths suspected of involvement in Sunday’s attempted invasion of the Nyali Barracks in Mombasa, where a soldier and seven attackers were killed and one arrested.
The incident occurred at about 5.30am when 10 machete-wielding youths attempted to gain access to the heavily guarded military base.
“Soldiers at the sentry responded killing five of the men. The sixth was pursued and shot dead in a nearby forest,” the military said in a statement.
The seventh attacker was arrested and is being interrogated on their mission, the military statement said.
County Commissioner Nelson Marwa has since urged the public to volunteer information on the attackers who are believed to have escaped with injuries.