Though it has been decades since most union jacks came down across Britain’s once-vast empire, 14 disparate territories remain under London’s control. Though largely unknown to most of the British public, the decisions they take can still have a considerable impact on the 265,000 inhabitants of these surviving imperial relics. Brexit is the major event threatening change.

Of course, not every territory will be affected. With no permanent inhabitants, nothing will change in the British Antarctic Territory and the South Georgia and Sandwich Islands. Nor is Brexit the greatest problem some of the territories face. Britain’s last Pacific colony, the tiny Pitcairn Islands,  with just 42 inhabitants, faces an uphill battle just to retain its existing population in the face of continued emigration and the inability to attract newcomers. This difficult task has not been assisted by an abuse scandal earlier this century that saw nearly a third of the adult male population jailed for child sex offences.

One Brexit concern is the potential loss of diplomatic allies to assist Britain in retaining several territories whose sovereignty is disputed. As an EU member, Britain has enjoyed Brussels’ diplomatic support in its longstanding feud with Argentina over the Falklands. Theoretically once Britain is out of the bloc, there is no reason for that diplomatic support to continue. Although, no matter what happens, Argentina is in no state to capitalise on the opportunity.

The British Indian Ocean Territory, carved out of Mauritius shortly before the latter gained independence in 1968, is the source of another controversy. Mauritius regards the separation of the BIOT from its territory as illegal. Some commentators suggested that Brexit, by reducing the UK’s influence abroad, will make it more difficult to hold the territory. This is in light of growing calls for the BIOT to be returned to Mauritius and its former inhabitants to be allowed back, after the British shamefully expelled them to make way for a US military base. Britain recently lost a non-binding International Court of Justice case that called for BIOT’s return to Mauritius and was humiliated at the UN General Assembly on the subject. This defeat had little to do with Brexit, however. Ultimately, though, as with the Falklands, it is difficult to see what major difference any international pressure would make, especially with the large US base on Diego Garcia going nowhere fast.

A more serious dispute concerns Gibraltar, the only British overseas territory formally part of the EU, and the only one to vote in the 2016 EU referendum. Spain has been trying to regain the tiny but strategic peninsula ever since it ceded Gibraltar in perpetuity to Britain in 1713 under the Treaty of Utrecht. Gibraltarians, however, are very proud of their British identity, and want nothing to do with Spain. Sentiments were not helped when Spain closed the border entirely between 1969. It did not fully reopen until 1985. Diplomatic rows periodically have erupted ever since.

The concern is that after Britain leaves the EU that Spain can apply pressure to the territory, especially since Spanish consent will be needed for any final EU-UK settlement. So far Spain has declined to use Brexit as a tool to settle the dispute, though it caused anger when draft EU regulations concerning post-Brexit visa-free arrangements for UK citizens referred to Gibraltar as a ‘colony of the British Crown’. The draft withdrawal agreement would have provided for some continuity for Gibraltarians, 96% of whom voted Remain in 2016. While border checks already exist between Gibraltar and mainland Spain since Gibraltar falls outside the EU’s customs area, these are quick enough to allow workers to commute daily from one side to the other. The local economy relies heavily on cross-border trade. A no-deal scenario would be thus far more problematic than a managed exit, with the UK Government’s Operation Yellowhammer document predicting up to four-hour delays to border crossings and disruptions to essential supplies.

A related concern is the possible economic impact of Brexit, especially for those territories bordering the EU. Gibraltar is the most obvious example, but not the only one. The island of Anguilla in the Caribbean—which somewhat uniquely launched a successful rebellion in 1969 in order to stay British—is dependent on the neighbouring Franco-Dutch island of Saint Martin for basic supplies. Because it lacks a long enough runway of its own, Anguilla also relies on Saint Martin’s airport to bring in tourists and much-needed tourism revenue with them. Family ties also run deeply across the three neighbouring territories. Should it be cut off from the Single Market, Anguillans worry that their access to goods and essential services will be cut. While the British Government regards such fears as unrealistic—many goods are in imported from the United States rather than from Saint Martin—these reassurances have not been accepted by the islanders.

Trade is also a worry for several territories far away from any other European territory. Though much closer to South America than to Europe, more than 90% of the Falkland Islands’ fishing exports go to the European Union, as do more than three quarters of its wool exports. Should tariffs be imposed on those industries, the impact on the territory’s 3,000 inhabitants could be considerable. Britain has not guaranteed that it would compensate the Falklands for any loss of export revenue should easy market access to the EU be lost.

Other territories are likely to be less severely affected, even in the unlikely event of a hard Brexit. Several of these—like the Cayman Islands and Bermuda—are major offshore financial services centres and as such have been condemned for facilitating global tax avoidance. Tourism, especially from North America, also plays a large role. Although it is impossible to forecast anything with certainty, as much of their trade is with states outside the EU, it is doubtful that Brexit will cause as much damage as it might to, say, the Falklands.

There is also concern about the potential loss of European Union development aid. Several of the territories, like Anguilla and the British Virgin Islands, have benefited considerably from European Union assistance. This is especially pressing since several of the Caribbean territories are still recovering from 2017’s devastating Hurricane Irma. Beyond the Americas, the isolated South Atlantic island of St Helena and the even more isolated Pitcairn Islands benefit substantially from EU funds. In theory there is nothing to stop any future British government guaranteeing the existing support. However, while Britain has promised to match any existing funded programme, there is no guarantee of what will happen once those programmes end. Because these territories have minimal political influence in Britain itself, this is worrying.

While one might expect Brexit-related disruption to fuel demands for independence, there has been little evidence for this so far. In truth, many of the overseas territories remain British for a reason. Some of the populations are very small. The Falklands, with only 3,000 inhabitants, could not survive on its own, even if Argentina should intervene. Volcano-devastated Montserrat, whose capital is buried under a blanket of ash, is likewise too economically weak to stand on its own.

While theoretically an independent Gibraltar might be economically feasible, the provisions of the Treaty of Utrecht make it impossible. If Britain wishes to give up sovereignty, it must offer the territory to Spain first. Because Spain never will accept an independent Gibraltar, the idea is therefore a non-starter. The most likely candidate for independence is Bermuda, which voted against the move in a 1995 referendum. Although there have been occasional murmurings, and one of the major parties on the island officially backs independence, Brexit does not seem to have influenced that particular debate.

With so much drama going on in the mother country, it is easy to forget that hundreds of thousands of people outside the British Isles are being affected in ways that even most informed observers would not even consider. While there is no need to cancel Brexit, there is a real need to plan for its impact beyond what will happen to the United Kingdom itself, and to reassure all those who are caught up in the Brexit process. Offering financial assistance to manage any disruption, guaranteeing continued development aid, and considering the territories’ needs in Brexit negotiations are all essential. Given the loyalty these territories have shown over so many years, it is the very least that Britain can do.