Back to Strategy.
2009: <lynX> There are some tough decisions to be taken. The benevolent dictator rule model hasn't always worked out in the past, but I'd rather only do things that most of us agree to be right choices and work harder on the decision making process when we do not agree: Find more PROs and CONs why this or that strategy makes more sense, thus work on the transparency and 'scientificness' of decisions.
Contents |
Strategic Choices
We have a broad choice on where to go from here. We should choose a good balance between feasibility - that is, getting stuff actually done - deployability, being interesting for audiences to use this stuff - and vision, having an exciting number of great new features which make PSYC worthwhile digging into. lynX thinks, PSYC has enough vision to bring along anyway, so even if we focus on being fast and popular, this will still buy us the PSYC vision.
Start from scratch
May just be a bad idea. So much to be done. Choosing the weapon.. I mean.. language is dispute factor number one. Bad enough when we look at that with respect to existing work. Starting from scratch may appear to maximize the visionary aspect, but in fact it will of course make us do new or even old mistakes, so we'll end up with a software with strengths and weaknesses, like all other approaches, too.
Start from scratch with the protocol, too
keep things simple. What is important about psyc?
* mcast, group join and leave operations, including channels * state (is needed for mcast anyway) * reliable delivery (in the sense that if I have received messages n-1 and n+1 I MUST have received n, too) - required for state actually
Build that, add applications later. That project would be limited in scope and should be doable within a few months.
Starting from an existing base
We have very elaborate PSYC implementations especially in perl and pike. Pike is also the language that porting snippets of LPC code from is the least amount of work. Additionally we have a Java library that has been critized for being architected the wrong way around, .net code for psyconaut, a client in C# and python libraries that have been restarted from scratch at least five times by various authors and ultimately do not provide much of what is needed to write a new psyced.
Javascript comes with interesting potential: We need it for several web-oriented, web-based and Flash-based clients anyway. Extending it to the server side means we don't need to maintain implementations in each and every language - or at least, the one language would have the most feature-complete implementation also happens to be the language that can be used in most applications and is kind of familiar to the largest number of people, not just computer programmers. Also it isn't the slowest language with all the JIT magic that is being thrown at bytecode interpreters these days. Yet, PSYC implementations in Javascript have been comparatively simple and client-only as yet, but the one in psyctive is pretty well architected and object-oriented.
In other words, only pike and perl are platforms mighty enough to provide a jumpstart advantage in developing a feature complete client and interserver PSYC solution. All other languages are more like starting from scratch. Problem about pike is, it doesn't buy us credits in the popularity zone, it's just an improved LPC. At least most distros have it ready for download. Tough aspect about perl is, it takes discipline to write human readable code.
Starting from a popular platform
Hacking PSYC into a popular server app may give interesting synergy results and be a platform for popularity. Especially in the case of XMPP servers, one should expect the backend structures to be very similar. The problem is, an abstraction from the XML DOM tree approach used in most of those application needs to be done for the same code to be able to process messages in a flat PSYC way internally. And of course many query/retrieval type of Jabber concepts need to be extended by the push approaches within PSYC. This may even result in us proposing new XEPs about alternative push solutions to certain problems to the Jabber world.
Alternatively, VoIP-type of applications are a good fit for PSYC, like Mumble, Red5 or.. oh dear.. Asterisk, as symlynX has proven with its RTMP-enhanced PSYC servers: http://billyidle.com is essentially a PSYC client in ActionScript, tunneled through RTMP.
for Discussion, see the Discussionpage of the Article
C/C++ enhanced psyced
The most legitimate reason for a rewrite is that the current psyced implementation has a lot of problems. Towards that end, one alternative to scrapping a functional server application like psyced might be to incrementally reimplement its core functions as library code. Well-constructed C or C++ libraries could be implemented as efuns in LPC code of the original psyced implementation and incorporated in other PSYC implementations.
Funny, but C libraries may be what gets you added to existing applications the best. Even if we don't do the whole big thing in C/C++, providing a libpsyc C library that contains basic parsing, rendering, psyctext and textdb would be useful as a C-extension to psyclpc or whichever language we choose for the main server. Adding just minimal client stuff (UDP, no circuits) would be enough to have your average desktop app receive events from a PSYC daemon running locally or in the LAN. Submitting events is even easier and doesn't take any library at all.
Porting psyced to Pike
An approach that lynX has been undertaking half-heartedly and got him some 8% of psyced ported is to get psyced to run under pike rather than LPC. This will not cure bugs. Not at all. This will enhance speed a bit, it will give us access to other libraries (are the issues about pike and TLS settled?) but the most important point is: Once the whole thing is running in pike, we can choose to merge with ppp and radically refactor psyced to a lot more structured approach - objects, interfaces, APIs.
Fixing existing apps using a Test Suite
The list of bugs in psyced is big. A joint effort to fix them may be fruitful though. Sometimes bugs are correlated and evaporate quicker than expected when several eyes are looking at them together. psyced is a great tool - would be a shame to give up. Would we ever get a new software to the point of providing all of the cool things psyced can do? However, some new things like supporting multiple XMPP clients are hard to do as they require major refactoring. Still, it is always an option. psyced is so close to being at least stable and good at the things it does and kind of deserves its 1.0 publishing day (lynXthinks) - even if it wasn't to be the big client hub and platform for PSYC's future. A 1.0 release is also likely to bring new attention and new manpower to us. Don't you think, that a project fifteen years old getting to 1.0 would be worth a slashdot? ;)
<lynX> Btw, channels wouldn't be very hard to do in current psyced - it's just that I forced myself on trying to get a stable 1.0 out there before heading into new featurisms. A plan that doesn't always work out, because new features are more fun than fixing bugs.
<Kol_Panic> Development methodologies like Test Driven Development are great for keeping the bugs out, but are difficult to implement in a project already under development. A useful psychological trick might be to impose a self-discipline based on assigning a cost for a new feature defined in terms of bug fixes.
- <lynX> Sigh, yes. + on this one.
<lynX> Also, psyced has the bad notoriety of being too delicate: while you fix a bug you inadvertently break something else. I wonder if that can really be avoided when editing the core of big projects.
<Kol_Panic> That's what I was thinking of when I mentioned encapsulation at the beginning. Nobody is in as good a position as you are (lynX) to evaluate to what extent the perception reflects any real situation in the code. That is one of few defects that would actually make a rewrite preferable to less drastic measures like refactoring.
<lynX> Thinking again, I don't think encapsulation as such is a frequent source of problems. When you need to change something in the core of a system in order to achieve goal A, it is inevitable that you may no longer be serving goal B and C. That is you inadvertently introduced a bug. Formalizing every little aspect what that part of the core should do can in many places be too inflexible to get things done, and the real problem with this is: Unlike other software projects, goal B and C do not show up as being broken until somebody needs them. It's in the nature of chat systems that people do not /unfriend, /surf and write WAP messages right after you restart your test server. So it is likely the bugs will show up months later when you run them on the main server. Thus, what we need to get stability into any messaging application is to create seriously heavy test suites that check every functionality in order for you to be able to patch a core thing, then see if it breaks anything. el's perlpsyc has a solid test suite builtin. I wished we took the time to build something like that for psyced years ago. Ok, so let's consider Test Driven Development as an approach for developing several PSYC applications in parallel. I know that real Test Driven Development digs much deeper and should be applied to every single function, but I think a test suite that externally connects the protocol ports and runs a list of checks on the counterside would be of immense help. And the beauty of this approach is, it can be written in a completely different programming language, just suitable to the job. perl has a fine infrastructure for running tests and perlpsyc already has a bunch of scripts that do those kind of jobs. The aim is to develop the discipline to actually run all of those tests before you commit, and to have them assembled in an easy to use way.
Double-headed strategy
Additionally to fixing psyced we could head out developing a native PSYC server to serve clients better than psyced has been doing, and be a better platform for experiments. In fact that's exactly the aim that ppp started out with and partly fulfils. What it lacks is still a huge amount of functions that we have come to like from psyced - also, in order to allow for clients to connect to a new server and still make use of psyced's XMPP and IRC gatewaying strengths, the integration of gateways needs to be worked out. That is another large bunch of work that would need to be addressed in both psyced and a new server application. As it stands, a patched ppp and psyced only communicate superficially (ppp can route packets to psyced, basically), but we haven't been very successful at earning strength or even synergy from having two server applications with differing focus.
Meta-programming approach
What if we put utility functions into a libpsyc in C, which can be linked to most applications and languages, then write a meta language compiler that parses a meta language grammar and generates code in various languages such as perl, python, php, pike, maybe even LPC, using the basic tools provided in libpsyc.
Ketmar thinks Lisp is a natural pick for embedding domain-specific languages (eDSL).
What about the specs?
<lynX> I've always considered writing the specification as the fun and easy part of the process (actually it's a lot of work, too). My strategy so far has been to first think up something loosely in the wiki, then get it working, then finalize it simultaneously in code and spec. So far, we got stuck at the last step most of the time. Maybe a classic approach of writing the spec out in detail, then get angry at it while we implement it, is better after all. Still, pizza-box design frequently isn't implementation-friendly. I can imagine stepping back from detailed implementation work (although I will always try to help quality assurance by reading check-in diffs) and focus on getting those specs to a useful degree of completeness. I guess I'm the guy who has most of the big picture in his head at once, so it's not a job I could easily delegate. Progress we made when elmex took the spec initiative was great - it slowed down with me not being sure about things the way they are currently done and things we wrote into the spec that psyced can't actually do. I don't see a purpose in writing all those specs if an essential building block of the technology isn't up to speed. So once more, psyced/LPC is the blocker.
First new steps
From the current discussions it seems to make sense to start out with a strategy like this:
- have a format for specs that allow for automatic generation of test scripts
- discuss priorities of things that psyc is intended to do / is doing and write specs in a way at least sufficient to generate those tests (details in wording etc can be filled in later)
- apply test scripts to existing psyc applications to get an idea of the situation
- decide further from there which software projects to pursue
Generating Test Scripts
As PSYC packets already have semantic content, including a method for variable substitution, perhaps we should use what we have. PSYC packets should be distinguishable from surrounding text, but can be delimited in the specs by simple markup if that proves desirable. Textdb can be used to provide values from a configuration file to for outbound packets and to construct the expected response. The syntax is, by definition, exactly as rich as needed to define the language.
The execution engine would be a simple client capable of performing textdb expansion, making multiple connections to the target servers and comparing the packet received to the one anticipated. A direct lexical comparison may be allowable for the first generation, but second and subsequent generations of the test client should be comparing resultant state rather than raw text.
Priorities
We probably should start a list here so that we can perform triage.
Development Tools
- Spec-driven tests
- Library code
Features
- Presence
- Gateways
- Application hooks
- Content negotiation
Testing
Running the proposed test client on the existing server will generate the response packets that are to be indicated by the test.
Please volunteer
<lynX> Whatever choice we take, we would need people willing and able to put them into action. A few days ago I had this idea, that if ten people commit to work on this I would give this new approach a go and possibly give up psyced if that is the consensus. I realize that even if I, the former benevolent dictator, may be convinced of a different strategic choice than the majority, and I might even be right (as anyone for that matter), it doesn't get PSYC anywhere, if there aren't people working with me to do it. So I intend to accept on what may seem to me as taking a wrong turn in order to get anywhere rather than protect perfectionism from harm and get nowhere. Alright, nowhere is harsh. I wouldn't say we are nowhere, but it's a shame it wouldn't take so much to be a bigger dot on the map of Internet technologies. But we really need new pairs of hands to get serious about this. I guess you all have your preferences on how you like to do things and wouldn't be necessarily excited about doing something not in your language of choice and thus feel similarely about this as me, but we need to develop consensus to move this thing forward.
A team structure
<lynX> Yippie, we have about nine volunteers now. What next. We started out asking if we should rewrite everything from scratch, but everyone has his own ideas how to implement things the best, using the favorite languages and tools. Even on the big question of rewriting psyced several people seem to be convinced of the power of psyced and find repairing and debugging it worthwhile, more promising than getting a large team to redo everything anew. Indeed, getting all nine or ten volunteers to work on a single piece of software would be quite an organizational challenge and yes it is probably correct that we can fulfil our wishes using several platforms and projects at the same time.
Move on to Team structure or discuss on Talk:Strategy.