You'd better break off with him.
Some people can't bear it, so they choose to break off with them. They hold the idea: out of sight, out of mind.