Basically, a contract under seal can be pursued upon for 12 years, but the building societies have agreed among themselves not to pursue after 6 years.(posted 8082 days ago)If 12 years has now passed, and you have not acknowledged the debt to them during that time, you will be definitely OK. The problem is that phone call - which could have started a six year stretch all over again but I doubt it - you did not write back to them, they have no evidence that they successfully got you to admit the debt.
As you say, get on with your lives.