Above: the International Court of Justice
Under the standard legal doctrine of “pacta sunt servanda”, international treaties like the UK’s Withdrawal Agreement with the European Union are binding until and unless a formal process of withdrawal is completed.
A treaty may contain within itself the required processes for withdrawing, as with Article 50 of the Lisbon Treaty, setting out processes for withdrawing from the European Union.
If it doesn’t, the established standard is that withdrawal requires formal notice and a “reasonable time”. For example, the USA is withdrawing from the Paris Climate Agreement, but doesn’t actually get out of that treaty until November 2020.
What happens if a state breaks the rules? In another era, the aggrieved state would invade. The EU is not going to invade the UK, but the EU, or member states of the EU, could sue the UK.
The jurisdiction of the European Court of Justice is limited to EU member states. The EU, or EU member states, could bring a case against the UK at the International Court of Justice, though the UK could then refuse to accept the jurisdiction of the ICJ.
If the UK is found to have breached the Good Friday Agreement of 1998, it could be sued by Ireland, or by individual Irish citizens, in the European Court of Human Rights, whose jurisdiction covers all 47 member states of the Council of Europe, not just the EU.
Ireland won a case against the UK in the European Court of Human Rights in 1978, when Ireland sued over “inhuman and degrading” treatment of detainees in Northern Ireland.