If we no longer want a foreign server, we can drop it along with all dependent objects. This use case is probably the only one that will work, unless we simply never reference the foreign server at all. Use this SQL statement as a database superuser:
DROP SERVER primary_db CASCADE;