Trying to understand what DXCluster is, is a work in progress of ours. Some links first:

Dirk came into the PSYC welcome channel and said: 

I am particularly interested in how internode comms is done

In essence explained in Routing. 

the dxcluster system has something like 300 nodes worldwide with maybe 1000 or so users
during contests maybe 3-5000 users

PSYC has seen larger numbers with the symlynX events. See scalability. 

there are all sorts of scaling problems unless the user data is partitioned (as it currently is)
however I am looking for something that manages that better than I do currently

That sounds like, like with IRC, you have a dynamic user database which requires broadcasts for each change of IP address and suchlike. PSYC uses UNI to UNL resolution instead, which gives each node a static name and allows to fix routing on a per-node basis rather than broadcasting around. This implies having a part of the network infrastructure on static hosts, or at least with dynamic DNS, which may be unavailable in your constellation. 

in particular I want to know whether your inter node protocol can
handle looping efficiently
(ie multiple nod connections node a->b->c->a)
and more complicated networks

We have not implemented Packet Ids, thus we cannot provide redundant multicast networks yet, but we have not encountered network sizes or constellations where this actually became necessary.

Comments on snippets from your future protocol idea.

A Channel is a Group address that is not a Terminal. It is (unless qualified by a Terminal) broadcast on all a Nodes interfaces

Broadcasting without limiting data to nodes that actually want it is likely to be a scaling issue.

Any message that is directed to a particular node will be sent out on the best interface based on routing information gathered so far. If there is more than one possible route then, depending on network or local policy, the message may be duplicated and sent on other interfaces as well.

You are ahead in some areas. Would be nice to integrate your experience with PSYC. We'd like to use redundant topologies in some constellations. Our protocol syntax permits these things, but we have not needed to implement anything like that so far.

This is what we do which doesn't mean that we can solve any of your problems as we have to dig deeper to understand what your application really is and thus if our solutions work also with your requirements.

In fact after looking at http://www.dxcluster.org/main/ I get a clue of what we are really talking about. If you really cannot use the Internet with its DNS and static IP addresses, and all nodes are volatile as the weather hits the antenna, hmm.. yeah... could be you simply have a completely different class of problem, even if the intended application is similar.

Maybe http://en.wikipedia.org/wiki/Ad_hoc_routing_protocol_list is more helpful than we are..  ;)