Although the Duke won some popular support on the Island for his regard of the Manx, he was less popular for filling well paid official posts on the Island with his own Scottish dependents.

The appointment of the Duke of Atholl as the Governor-in-Chief of the Isle of Man, in 1793, brought him into direct conflict with the House of Keys over the extent of his rights and those of the Manx people.

The most unpopular of Atholls dependents was his nephew George Murray, the Bishop of Sodor and Man, who, in 1825, despite a very poor harvest, declared a tithe payment (Church tax) of 5 times the rate per acre in the North of England. The resulting famine and poverty led to a series of riots around the Island. A mob attacked the Bishops residence, Bishopscourt near Kirk Michael, and the Bishop had to flee for his life. The tithes were finally repealed and Bishop Murray was moved to another Diocese.

The unhappy saga of the Duke of Atholl and the House of Keys finally came to an end in 1828 when the Duke accepted a final payment of £ 417,000 for the remainder of his rights on the Isle of Man, and never again returned to the Island.