Psi Whiteboarding
Latest Progress
If you're interested in learning about Psi Whiteboarding and/or Shared XML Editing (SXE), check out the latest documentation and the Psi svn repository. Simply uncomment "CONFIG += whiteboarding" in src/src.pro before running make to enable (experimental) whiteboarding in Psi.
Overview
This SoC 2006 project aimed to design a proposal for a XEP for whiteboarding and implement it for Psi. The Summer of Code is now finished and the code and the XEP draft I wrote can be found in this Media:Wbfiles.tar.gz file as they were at the time of the deadline. However, those files are completely outdated; check the "Latest Progress" section above for up-to-date information.
Progress Reports
Week 1 (6.6-12.6.2006)
General
These last seven days were hectic with all sorts of things happening as it was the first week in Finland. I had lots of forms from vaccines to housing to arrange an send out and I also had to be away for the weekend so I think I only managed to spend some 20 hours on the project. Mostly I spent the time thinking about the big issues of the protocol like synchronization and trying to put those ideas down. One big thing that I wasted time on in the first days was the fact that I didn't assume the integrity and seriality of packages which of course should be taken care of by underlying protocols. I contacted Mats (from Coccinella) which was good since it turns out he's prepared a paper describing a "neat" synchronization scheme which is very helpful. I didn't get any code done yet but I don't think that's such a bad thing; I did read some Qt coding material though. My life is starting to clear up from other things and I think I'm starting to be clear on what the complicated issues in the protocol are and I have an idea how to tackle them so I expect I'll be able to make good progress this coming week.
Things I did
- Prepared my desktop for development.
- Got a new PCI WLAN card because the old USB card seems to have burnt or something because it only worked for a few minutes at a time.
- Installed Dapper. The machine couldn't boot from the new install CD so had to do weird things to install Dapper using an old installation CD which took a bit of extra time.
- Installed Qt development tools
- Installed darcs and pulled Psi
- Installed Coccinella for reference
- Read again different existing material on the topic, like XEP-113, Mats' Memo, the whiteboarding mailing list, David Durand's Palimpsest model, some of the Tiny SVG spec and the XMPP specs.
- Defined subgoals for defining the protocol and wrote notes outlining what I think would be the principles, assumptions and basic format of the protocol in relation to these subgoals. Spent most of the first two days or so worrying about lost or corrupt packages which was of course redundant. Anyway, at least it came up quite soon.
- Got in contact with Mats
- Got some comments from him
- Read his draft version of a paper describing a synchronization scheme that doesn't rely on any third party in addition to the clients
- I think I wrote a nice and simple proof that the scheme works, but we'll see. I was quite tired and I haven't heard back from Mats yet.
Week 2 (13.6-19.6.2006)
General
This week I spent quite a lot of time on the project. I tried to note down the hours and all in all they amount to about 41 hours excluding all breaks. I spent amount quarter of the time on thinking about Mats' synchronization scheme and writing a proof for it for multiple clients. That's quite a lot of time but I think it was worth it because now it should be clear that the scheme works given some assumptions. This week I also finally got to write some code. I wrote two classes related to rendering changing SVG documents inherited from Qt's own QSvgRenderer and a test app to wrap the classes. This took more that half of my time this week because I had to learn some Qt, read the QSvgRenderer sources and remind myself of some of the C++ ways of coding because I've been recently using "higher level" languages like Perl or C#. I spent the remaining time starting to write the XEP, reading other XEPs and exploring the Psi code and thinking how to integrate my components there. All in all, I'm happy with the progress I made this week. The one question mark to me at the moment is how I'm going to implement the edit user interface. I can start from scratch but I'd be really surprised if there wasn't already a similar interface in some existing Qt based project that I just haven't found yet.
Things I did
- Wrote a proof for Mats' sync scheme
- Explored the Psi code
- Wrote SvgStreamRenderer and SvgStreamWidget classes (compare to QSvgRenderer and QSvgWidget)
- Read documentation again, mainly other XEPs and a few old mailing lists, especially about writing the XEP.
- Started writing the XEP
Week 3 (20.6-26.6.2006)
General
I managed to spend quite a lot of time on the project again, regardless of the midsummer's party here in Finland. I noted down 36.5 hours but I don't think that includes all of it. I think I spent the time more or less equally between working on the draft of the XEP and working on the code to process and render the elements according to the XEP. Besides that there was a useful groupchat with people interested in whiteboarding on Monday. I had some technical difficulties with receiving messages but otherwise that meeting was very useful. Also, I was in contact with Mats again and Dale who develops the Inkboard as his SoC project.
Things I did
- Improved the XEP
- Did some small polishing to the proof of the sync scheme with Mats.
- Wrote a WbWidget class that processes full <wb/> elements and renders them using SvgStreamRenderer
- Attended a meeting with other people involved with whiteboarding to start "harmonization" process
- Arranged a darcs repository[1] with Remko. Had to spend some time on that because I had to configure my postfix to use gmail smtp server which requires SASL and TLS.
Week 4 (27.6-3.7.2006)
General
I spent a fair amount of time on the project again though I didn't note down the hours. Even though I made progress I feel that I could have achieved more with a little more concentration. Well, next week... The concrete thing I managed to do was to start integrating my code to Psi. It can now do the basic things like open a new whiteboard dialog for a given contact and and handle the incoming <wb/> elements. There's still some work with proper popups and such and especially negotiation of new sessions. However, I haven't got the negotiation figured out on the protocol side yet so I'll leave that for later. What I realised is that I can't really use the QSvgRenderer that I've been using so far because it doesn't lend it self for allowing users to modify the drawn objects. I think in the end I will need to produce the code that converts the SVG elements to Qt's graphics items myself. I think getting started on that will be one of the big challenges of the coming week.
Things I did
- Discussed many protocol and some implementation issues with Mats (Coccinella) and Dale (Inkboard)
- Consequently improved the XEP again
- Wrote rough versions of WbManager and WbDlg classes for Psi
- Modified Iris (Psi) code to parse the <wb/> elements
- Modified Psi code to add user controls that open a new session
- Read some (KOffice) Karbon and QSvg Module sources
Week 5 (4.7-10.7.2006)
General
As I mentioned last week I realised that I can't really use the QSvgRenderer for rendering the elements because of the impossibility of letting the user modify the items later. Hence I started writing my own classes that can "import" SVG element to Qt graphics objects and vice versa. The situation with Qt is rather interesting at the moment because their QCanvas system is depreciated in Qt4 while their GraphicsView is only available in the preview version of Qt 4.2. I decided to install the preview version in any case and see if I can use it because I thought it was silly to write new code for Qt3.
Things I did
- Discussed many protocol and some implementation issues with Mats (Coccinella) and Dale (Inkboard)
- Consequently improved the XEP again
- Installed Qt 4.2 TP1
- Rewrote WbWidget and WbScene (which replaces SvgStreamRenderer)
- These classes are now able to interpret incoming messages, render <path/>s and lets the user move them and produces the corresponding <wb ...><configure .../></wb> element.
- Consequently found and reported some bugs/inconsistencies in the TP
- Came accross and tried to fix a weird bug that caused segfault in Psi with the TP. Didn't manage to do it properly yet :(
Week 6 (11.7-17.7.2006)
General
I think I made good progress this last week. I spent most of my time improving the WbWidget, WbScene, WbItem and WbPath classes. I think the next big goals will be finishing the integration of the classes to Psi, define and support session negotiation and how a new client can join an existing session. Also, the other basic elements like rectangles and ellipses should be supported, but that's secondary for now.
Things I did
- Discussed many protocol and some implementation issues with Mats (Coccinella) and Dale (Inkboard)
- Improved WbWidget, WbScene, WbItem and WbPath
- Added support for letting the user rotate, scale and remove items. The corresponding <wb/> elements are produced.
- Found and reported a bug in the TP
Week 7 (18.7-24.7.2006)
General
I made steady progress again towards reaching some of the goals I mentioned last week, specifically integrating the whiteboard classes to Psi instead of trying them out on test apps and impoving the session negotiation. Finally, I'm able to actually have a session in Psi.
Things I did
- Improved the all the whiteboard classes but especially WbManager and WbDlg that are responsible for session negotiatio and userinteraction.
- Started writing documentation for the class headers in Doxygen format
- Participated in discussion with Mats and on the list again
Week 8 & 9 (25.7-7.8.2006)
General
Week 8 was good and I managed to do most of the things mentioned below during that. On the other hand I was occupied by non SoC matters for almost all of Week 9.
Things I did
- Discussed many protocol and some implementation issues with Mats (Coccinella) and Dale (Inkboard) as well as someone called Bernhard who's doing XMPP whiteboarding with Javascript.
- Consequently, changed some of the syntax and structure of the <configure/> element
- Added support for changing properties (color, stroke width) of existing elements through a context menu.
- Added support for text
- Need to fix some issues with transformations
- Need to add support for different fonts at some point
- Started writing the code for sending the history to a client wishing to join an existing session
- Finished writing very basic documentation for each member of each class.
Week 10 (8.8.-14.8.2006)
General
I spent a lot of time coding multiuser session negotiation, fixing some issues related to rotating and scaling. I ran into some problems with the text item and I'll still have to solve them. I also added support for rendering the shapes included in SVG Tiny (ellipse, circle, rect, etc.). The basic functionality is starting to be there but of course there are lots of things to improve and polish
Things I achieved
- Added session negotiation for sessions in MUC rooms
- Fixed rotation and scaling so that it actually does them around the center of all items
- Added basic support for the shape module of SVG Tiny
Week 11 (15.8.-21.8.2006)
General
This week was probably the most productive week, at the cost of all other aspects of life :) I added a lot of new features and I'm quite with the available features now. There are still some issues with two important features: the text element and grouping of items. Also, a good amount of testing is still required. I shall work on these during rest of the summer and the fall.
In any case, this summer was a great experience and thanks for everyone involved.
Things I achieved
- Made small changes to the negotiation process
- A major update to the XEP draft so that it now accurately describes the protocol I use
- Restructuring of the classes
- Added support for adding the the shape elements (except polyline/polygon) of SVG
- Added support for adding and rendering images
- Added support for to handle remotely created groups (SVG Basic Structure module)
- Grouping/ungrouping locally still needs a little work in order to work sensibly
- Added support for changing the stacking order of items (bring to front, etc.)
- Added support for changing the size of the whiteboard
Notes
Goals
| Goal | Notes/Progress |
|---|---|
|
|
| Incremental addition and modification of elements for QSvgRenderer | |
|
100% |
| User interface to add and modify elements | |
|
0% |
|
0% |
Synchronization
Synchronization
Mats Bengtsson' memo[2] provides a nice scheme for synchronization. Here's my attempt to prove it for multiple clients. It adds an assumption that all edits pass through a single server and that there's seriality between the server and each client, which is the case with all XMPP chats at the moment as far as I know but it's important to realise that it's required.
Proof
Mats' has now updated he's memo to include a better formatted version of the proof that was described here. See the memo for details. I think the case for R in the proof is simpler though so I'm leaving the following statements here to provide an alternative to the proof that R rejects all crossing edits.
Appendices:
[1] Alternative for the proof that R rejects the crossing edit. See Mats' memo for context.
Let P8 be the point when R receives edit a and P9 be the point when R receives edit b. It cannot be that P8 != P9 so P8 < P9 or P9 < P8.
- If P8 < P9,
- Let j be the version (index) of a. As shown in A's case: i <= j.
- Hence edit a is revoked by R.
- If P9 < P8,
- Let s be the number of edits received by R before P8. If we impose seriality, s >= j + 1, where the 1 comes from edit b since P9 < P8. j < j+1 <= s. Hence R rejects edit a.
[2] Here's an example where the scheme breaks down if the seriality condition doesn't hold because "information" travels from B to A through C earlier than it does directly. Determine which edits get accepted and which rejected at each client and you notice that A rejects all edits whereas B, C and D accept B_1 (B's 1st edit). It happens because C had B_1 (i.e. "information" from B) before sending C_2 which arrived to A before B_1 (the "information" from B). This could happen if the edits are not transported through a central hub like a server.
A -------------*--------*------------------------*-----------------------
/1 2|\-- /3
--------/----------||--\-------------------
/ / | \ \
B -*--------|---*-------|--*--\------------------------------------------
1|\ | /2 | 3 \
| \ | / \ \
\ \ |/ \ \
C ---*-\----*--------------\------*--------------------------------------
1 \ 2\ \ 3
\ ---------------\---------------------------
\ \ \
D ---------*-------------------*--------------------------*--------------
1 2 3
References
I've used sources listed at [3] for information and particularly I've used XEP-0113 and Mats Bengtsson' memo [4].
The relevant parts of the original proposal
Introduction
I want to research and design a whiteboarding solution for the XMPP protocol. Clearly, there is interest for this[1] and some client specific implementations already exist[2]. However, in order for the feature to be adopted the clients need to inter operate. I think that the Google SoC is an excellent opportunity to have a person (i.e. me :) prepare a solid draft for a whiteboarding XEP based on the suggestions of different developers and the lessons of different existing solutions. Alongside with developing the protocol I would develop support for it for Psi. Obviously, I can not promise that the XEP will be final by the the end of the summer, in fact it's unlikely, because of the community based procedure. However, I will do my best to produce a solid framework.
Deliverables
- Initial draft for a XEP for Whiteboarding protocol.
- Implementation of the protocol for Psi (C/C++ source, documentation).
Milestones
- 0.0.0
- Collect more feedback from developers
- Familiarize Psi code base and QtSvg.
- 0.0.1
- Protocol: First draft with support for creation and deletion of any element and for editing lines.
- Psi: rendering any element and drawing and editing lines.
- 0.0.2
- Protocol & Psi: Support for conflict resolution.
- 0.0.3
- More feedback from developers about the proposed protocol basics
- Protocol: Modification of any element
- Psi: Creation, deletion and edition of basic elements
- 0.0.4
- Psi: Saving and loading of SVG
- Psi & protocol: Images embedded in SVG
- 0.0.5
- Multiuser sessions
- 0.0.6
- Protocol & Psi: Requesting and sending complete data or individual nodes -> support for lost packages and for joining during a session
- 0.1.0
- Features complete
- Testing and bug tracking
- Initial draft for XEP
- 0.1.1 - 0.9.9
- Fixing bugs, Getting feedback, Improving XEP
- 1.0.0
- Psi: Bug free
- XEP usable, even if not final