Typically, who pays income tax where is based on residence. The only country that I know of that does not follow this principle is the United States. As a US citizen (or green card holder), you have to pay income tax to the IRS wherever you live, and on your worldwide income.
Fortunately, many expatriates are able to benefit from some tax treaties between the US and their country of residence: they can deduct their foreign taxes, in principle. But this is by far not the case for everyone. But if you do not live where your taxes are used, what do you get in return for your taxes?
Depending where you live, the US offers protection: an embassy where one can find refuge, or repatriation to the US in case of serious trouble. This sounds like some insurance policy. If it were offered on the free market, such a policy would, however, be dirt cheap for most expatriates, for example those in Canada. This seems like a pure tax grab to me.
Note 1: many countries bill their citizens for repatriation service. Not Canada, which lead to some debates after the last war in South Lebanon. Some people were quite unhappy about subsidizing others living in dangerous places.
Note 2: an obvious solution for expatriates to avoid paying US taxes is to renounce citizenship. It is, however, illegal doing so for tax reason. And in a sly move, the Heroes Act of 2008 which increased benefits to veterans and their survivors also included a provision that anyone giving one's citizenship up voluntarily would be taxed on all assets as if their were sold.