Related to Serialisation and SQL.
Contents |
1 Client Side Storage
1.1 Storage on a user's disk
How to store login data on a user's harddisk in a generic way for all PSYC apps: http://www.psyc.eu/storage
<Monkey 02:09, 1 July 2006 (CEST)> el suggests that we should have shared config files in the ~/.psyc directory. I add that we should have client specific files in ~/.psyc/clientName/*
On windows this would be kept in %sysroot%\Documents and Settings\Application Data\psyc\
With Mozilla applications the convention is to store files in the Mozilla profile which is already managed by the Mozilla application and is kept in conventionaly platform-dependant locations.
Suggested files / formats:
- ~/.psyc/me - simply contains "my uni" in plain-text
- ~/.psyc/auth - one per line: uni[tab]pass
- ~/.psyc/clientName/* overrides files in /.psyc/*
- Templates:
- ~/.psyc/templates/*
- For templates, please also consider the psyced textdb file format as in a PSYC 1.0 standard we may want to provide all standard templates with each software package, thus never transmit them on the wire. Then a common or at least convertible format would prove pragmatic.
1.2 Storage in a PSYC1 UNI
The client can leave data on the homeserver by sending _request_do_store and _request_do_retrieve. Here's the syntax for _request_do_store:
:_application_pypsyc_window_size 3000x2000 :_application_pypsyc_skin darkstar :_character_command % _request_do_store .
So whenever the client starts up it can connect the UNI and ask for its settings by issuing an empty _request_do_retrieve. one day we may specify subsets of the data pool.. using families i guess.
For now all variables of the _application family or any variable specified in PSYC profiles are accepted.
There is no explicit method do delete these stored _application variables as yet.
See also: Command.
1.3 Storage in a PSYC2 pubsub channel
Essentially persisting configuration data between devices on PSYC2 works the same way, only the variables are stored into the distributed state of a pubsub in conjunction with your ego and kept with the help of a few agnostic relays that agreed to host the pubsub.
2 psyced Storage
psyced (PSYC1) currently has the habit of storing user and place data in flat files, and to expect the harddisk to be fast enough for synchronous operation. In other words, loading a person's data and doing something to it is an atomic operation within psyced. That's in http://www.psyced.org/dist/world/net/storage.c
This is somewhat impractical if we were to provide native support for SQL or LDAP databases, even though ldmud provides APIs for that.
psyced makes some 30 references to summon_person() where it atomically expects to be able to access the user data even if the user hasn't been loaded yet. All of these places would have to be rewritten to deal with an 'ok-now-the-data-is-available' callback. In a multi-threading application you would simply put that thread to sleep, but psyced is an event-oriented application and generally draws a lot of advantages from that.
In fact, pulling information out of a database is a pull operation, and PSYC is about push, so here's a push approach to the problem.
2.1 Storage Synchronization
This protocol is by default not activated (it isn't even runtime-compiled into your psyced) unless you #define PSYC_SYNCHRONIZE.
You can #define PSYC_SYNCHRONIZE in local.h to contain a PSYC uniform that points to a process that will promote user data synchronization with any other application. Like psyc://192.0.2.44:12345/ for example. This instructs psyced to accept user data changes from a PSYC entity sending from TCP port 12345 of 192.0.2.44.
Also, once PSYC_SYNCHRONIZE is defined, a context called <yourServerUNI>/@sync will be incarnated. Your synchronization process should _request_enter this as this is the place where psyced will multicast all changes to its storage state to all entitled backup servers and services.
This is the sort of messages psyced will post there:
2.1.1 UNI Settings
. :_source psyc://localhost/~jack :_context psyc://localhost/@sync :_key_LDAP roomNumber :_value 101 :_nick jack :_key_set room :_key _address_room _notice_synchronize_set [_nick] has set "[_key_set]" to "[_value]". .
This is generated when jack types /set room 101, which is supposed to contain his office room number. Both the official PSYC key name and the /set command name setting are provided. Should the field have an equivalent LDAP or vCard name, they are provided for free. This information comes from profiles.gen.
2.1.2 Settings per contact
. :_source psyc://localhost/~jack :_context psyc://localhost/@sync :_value _none :_contact beeps :_setting _display :_nick jack _notice_synchronize_contact [_nick] has changed [_setting] for "[_contact]". .
Example of a synchronization notice to inform you that jack is ignoring a user called beeps.
. :_source psyc://localhost/~jack :_context psyc://localhost/@sync :_value _pending :_contact psyc://psyced.org/~el :_setting _notification :_nick jack _notice_synchronize_contact [_nick] has changed [_setting] for "[_contact]". .
Whereas this notice tells you that jack has attempted to make friendship with el on psyced.org, with no reaction from the other side yet. The list of possible values for _display and _notification is defined in the middle of http://www.psyced.org/dist/world/net/library/share.c
As you can see this protocol always only changes little attributes of people, not the entire set. This has the advantage of making collisions very unlikely and rather bearable, should the link between the synchronizing parties be broken for a period of time. If you guess this is not bearable (we should gain experience with it first, instead of guessing really), you can always add some timestamping strategy later on (see also below on replication strategies for databases).
On the incoming side, each psyced user object accepts _request_do_set as defined in Command. The object will accept the changes according to the principles of remote control.
Careful where you send this stuff to, as even your user's passwords will be exchanged in cleartext. Also careful who you are getting such messages from. You should ensure you are talking to your psyced, port and IP-wise, if you have allowed your users to modify their identity program (not available yet), then you would also have to make sure the _nick corresponds to the _source, or just simply ignore it.
Now, concerning the implementation of such a counterpart PSYC server: If there isn't a PSYC implementation suitable for your language or application, you can still easily parse and generate this one PSYC message type. Just take care on how to set up and shutdown a PSYC circuit, if you're using that.
There is a simple synchronization daemon written in perl available inside the perlpsyc distribution called psycsyncd (freshly baked from el industries). it gateways PSYC SYNC to SQL. Probably unfinished, but it's a starting point.
2.1.3 Presence synchronization
All presence changes in a psyced are also pushed to the special @sync place. The method is called _notice_presence_synchronize because it is syntactically identical to _notice_presence.
2.1.4 Sign-on and sign-off
Additionally to rich presence there is also a very technical form of presence: Do we have a link to a person or not? This is reported as _notice_synchronize_sign_on and _notice_synchronize_sign_off with some extras like user agent version.
2.1.5 Decentralized psyced community
This mechanism can be used to keep whole flocks of psyced servers in sync. Synchronization notices go to the @sync service, which is a multicast context you can subscribe to. All the servers should subscribe each other's @sync - and of course you shouldn't let anyone else in...
Not sure however what this is good for, as you still can't have the same hostname for all servers, thus the UNIs need to differ, thus there is no reason for one server to have the login data of a person on another server. I guess it serves the purpose of quickly taking over the IP and service of a crashed server in the LAN.
- <fippo> you're aware that this homegrown mechanism is absolutely inferior to the standard method of using a replicated database (pgsql, ldap), aren't you?
- <lynX> I'm not trying to do what they do. This is just an interface to applications running on the same host (typically) for now. But I am saying that this could be extended enough to enter the world of replication - by learning all the lessons learned from database replication protocols - and there is one thing they can learn from us: multicast. Also, you can still use a database replication mechanism on the WAN, if you are positive that it is a better idea: Just use a psycsyncd on all installations to get the data from the database into the psyced.
2.1.6 more discussion
<lynX> I bet if we were using async storage on an SQL database, we would now be thinking about using local object persistence as a way to cache user data and depend less on the speed of the database. So by doing the synchronization protocol we are going straight for the database + local cache strategy rather than losing the advantages of one for the other.
You may think the dot-oh file format to be quite funny but hell it's just an object persistence format. Editing such a file is just like editing the raw files of an OO or any database - not intended to be done. I always avoided doing tools to edit those files because the object class that loads the files is the right place to edit them.
If you want SQL as a data interface language to access your data, then use synchronization to make an SQL database. After that you can do any operations you want on it. Just make sure the database has the ability to signal to your controlling application whenever data is changed, so it can sync it back to psyced. I suppose these days it's a snap to add write event hooks to a database, although I haven't looked at that.
2.1.7 Implementation Hint
In LPMud and LDMud any object can get a list of its own variables using symbol_variable() and a while loop. An object can get a list of variables in a different object using the same routine and bind_lambda(). Of course, such a thing has to be very privileged to do this, and it's kind of a backwards way of doing it. It works fine though. You can also assign values to variables directly using the same technique, albeit with a bit more funkiness.
- says Matthew Julius.