Well, it's been eight days since we released Pidgin 2.5.3. In that 8 days, we've had more duplicate tickets than we care to count over the all-too-common hang on exit on Windows systems. In the faint hope that people actually read my ramblings, I'm posting this here to give a brief synopsis of the problem and to discourage further duplicate tickets.
On Windows sytems, Pidgin uses threads for a few things, namely DNS lookups and Network Location Awareness (NLA) stuff. For the Linux-inclined, NLA is somewhat similar to NetworkManager. When we released Pidgin 2.5.3, we started getting a bunch of reports about hangs on exit (including a number of people who don't know the difference between a hang and a crash, but that's another post in itself). All the debug logs pointed to wpurple_cleanup(), a function we call on close to tie up some loose ends, or "clean up." One of the areas this function cleans up is NLA-related stuff.
This code hasn't changed meaningfully in quite some time, but magically it became a problem for users of 2.5.3. It doesn't really make any sense to me why this would suddenly stop working in the current release, but the previous release seems to be almost entirely problem-free in this regard. Confusion aside, however, we have a proposed fix that will be reviewed and possibly tweaked.
So in summary, we know what the problem is and we are working to fix it. Please, please, please don't open anymore duplicate tickets about it!
Wednesday, December 31, 2008
Sunday, December 21, 2008
Pidgin 2.5.3 (Finally!)
Well, we finally kicked Pidgin 2.5.3 out of the nest, two days late. It's spreading its wings and flying.
Noteworthy in this release are that 58 bullet points are in the ChangeLog (not counting the headers for libpurple, Pidgin, Finch, and the protocols) and 85 tickets were closed, 40 of which were marked in trac as patches. Also of note is the insane amount of work Mark Doliner put into our MSN and MySpace plugins, which should resolve a number of crashes and the MySpace "doesn't sign off" issue.
A few big items to summarize:
Noteworthy in this release are that 58 bullet points are in the ChangeLog (not counting the headers for libpurple, Pidgin, Finch, and the protocols) and 85 tickets were closed, 40 of which were marked in trac as patches. Also of note is the insane amount of work Mark Doliner put into our MSN and MySpace plugins, which should resolve a number of crashes and the MySpace "doesn't sign off" issue.
A few big items to summarize:
- Mark did massive code cleanup in MSN
- Mark fixed a number of shortcomings in the MySpace IM protocol as well as doing code cleanup.
- We accepted a ton of patches.
- ICQ typing notifications should work for some third-party clients now.
- XMPP resources now default to the empty string, causing modern servers to assign us a resource via a bind. In the event of an ancient server, such as the one DreamHost runs, we will detect the lack of this capability and enforce the default "Home" resource if a user hasn't set a resource.
- XMPP resources can now include __HOSTNAME__ as a special token that will be replaced with the hostname of the machine being used. For example, if I run Pidgin on my MacBook and configure the resource on an XMPP account to be __HOSTNAME__, the resource sent to the server will be "macbook", since this is the hostname of my MacBook.
- Some long-outstanding patches have been applied for Gadu-Gadu, implementing IM images.
- Apply some patches for Zephyr, enabling autoreply when away to emulate zaway and some bugs with the 'use tzc' option.
- We no longer get certificate errors for rsi.hotmail.com when logging into MSN and retrieving offline messages.
- Many, many other things best read about in the ChangeLog.
Sunday, November 30, 2008
Patches, Patches, and More Patches
On November 13, I mailed our development list about our open patches. At the time we had 93 patches open on our Trac. Since then, some work has gone on with respect to patches, a lot of it from me. In the last 17 days, there have been 23 patches committed. Not all of those tickets were marked as patches, however. There have also been a number of new patch tickets opened, bringing our total patch count down by only 11 to 82.
Now, out of those 82 patches, if we ignore the ones we've put on the Patches Needing Improvement milestone, we have 57 patches. Here is where we see a significant difference, as out of the former 93 patches, 75 were not on a milestone, indicating that the patches that have changed to the Patches Needing Improvement milestone had not been previously reviewed, or had been reviewed but not marked appropriately on Trac.
Among some notable patches that were accepted include MD5 authentication support for ICQ; fixes for Gadu-Gadu, Zephyr, and SIMPLE; ICQ X-status support (which can't be included until at least 2.6.0); and support for connecting to an XMPP server without specifying a resource, thus causing compliant servers to generate a resource for us. Aside from the X-status patch, all the ones I've just mentioned will be in 2.5.3.
But the work isn't done yet! We still need a lot of patch review from Pidgin developers. Also, a lot of patches need work, so there's still plenty to do if anyone wants to help! I, for one, certainly won't complain if someone comes along and fixes up a patch that needs work, and such fixing will make it much more likely that the patch will be accepted.
For reference, I also mailed the development list earlier today to discuss patches again, here.
Ah, how our work never ends...
Now, out of those 82 patches, if we ignore the ones we've put on the Patches Needing Improvement milestone, we have 57 patches. Here is where we see a significant difference, as out of the former 93 patches, 75 were not on a milestone, indicating that the patches that have changed to the Patches Needing Improvement milestone had not been previously reviewed, or had been reviewed but not marked appropriately on Trac.
Among some notable patches that were accepted include MD5 authentication support for ICQ; fixes for Gadu-Gadu, Zephyr, and SIMPLE; ICQ X-status support (which can't be included until at least 2.6.0); and support for connecting to an XMPP server without specifying a resource, thus causing compliant servers to generate a resource for us. Aside from the X-status patch, all the ones I've just mentioned will be in 2.5.3.
But the work isn't done yet! We still need a lot of patch review from Pidgin developers. Also, a lot of patches need work, so there's still plenty to do if anyone wants to help! I, for one, certainly won't complain if someone comes along and fixes up a patch that needs work, and such fixing will make it much more likely that the patch will be accepted.
For reference, I also mailed the development list earlier today to discuss patches again, here.
Ah, how our work never ends...
Tuesday, October 7, 2008
The Hosting Game
Well, by now probably everyone's aware that our (Pidgin's) sites had been down for several days. Here's the basic rundown of what's happened with our servers over the last couple years.
Prior to our going public with our rename, Luke Schierer secured a donation of a virtual server from DVLabs. This server was intended to enable us to migrate away from SourceForge's web platform and tracker. It allowed us to have our own domains, and thus host and control our own e-mail services, mailing lists, and Trac. This server did its job for us quite well, even though we did occasionally push the server to or beyond its limits. (Note that in these instances real hardware would actually not have made a noticable difference at all, as all the resources of the real hardware would have been utilized just as we had experienced in the virtual machine.)
Even though we were extremely grateful to DVLabs for providing us a server, we did realize that having only a single server presented us with some limitations, the most important of which is "graceful" failure of core services if the server goes down. It also presented a single point of failure in that the web server getting pounded (such as a posting to Slashdot causes) could potentially cause all our services, including monotone and mail, to grind to a halt. Because of this we had tossed around the idea of finding another host two or three times.
When we most recently thought about hosting, Evan Schoenberg from the Adium project (who also happens to be a libpurple developer) put us in contact with NetworkRedux, who generously provides Adium's hosting. Thanks to Evan's intervention, we spoke with NetworkRedux about our hosting, and they were willing to donate not just one, but two servers and a ton of bandwidth to our project. Thanks to NetworkRedux's generous offer, we're getting the potential to have a lot more raw computing power at our disposal (including more memory to help increase caches, thus hopefully improving speed), as well as the ability to spread our services out a little so that a heavy hit to the website or to trac won't have a detrimental effect on all our services.
We were still discussing details of migrating to these two new servers when the server at DVLabs unexpectedly went down. It turns out that some miscommunication and bad timing caused the guys at DVLabs, who were migrating their own stuff, to down the server hosting our virtual machine, thinking we were no longer using it. We were, of course, confused at first, but once we determined what was happening we were much more at ease. None of our data had been lost. DVLabs were kind enough to supply us with the raw VM image for our own use to recover our data and complete our migration.
From these events we have learned a lesson--redundancy is not only good, it's pretty much a must-have. In the interests of redundancy and graceful failover, we've taken some steps to help prevent future long-term outages. One of those steps is that Gary Kramlich and I decided that we would help out by providing some service redundancy on our server, guifications.org. The biggest, and most important, service we are helping with is monotone. If for some reason the monotone server is down, guifications.org can seamlessly stand in for the mtn.pidgin.im server through the magic of DNS. We're also looking at potentially acting as a backup for more services.
So all in all, the commotion about our sites being down was really just a lot of noise about nothing truly significant. Yes, our sites were down, but this wasn't the end of the world. Google's massive cache had pretty much all the critical information anyone could need from our wiki. Yes, monotone was down, but guifications.org was ready to help--I had been running a read-only mirror for nearly a year which was ready and able to stand in as a production service if needed. Our mailing lists were down as well, but again, it wasn't the end of the world. Overall, we have weathered a prolonged service outage pretty well and taken some lessons from it to help us in the future.
In closing, I would like to extend my deepest, most heartfelt thanks on behalf of all Pidgin developers to DVLabs for their hosting services over the past two years and to NetworkRedux for the services they are now providing for us. We truly appreciate these donations!
Prior to our going public with our rename, Luke Schierer secured a donation of a virtual server from DVLabs. This server was intended to enable us to migrate away from SourceForge's web platform and tracker. It allowed us to have our own domains, and thus host and control our own e-mail services, mailing lists, and Trac. This server did its job for us quite well, even though we did occasionally push the server to or beyond its limits. (Note that in these instances real hardware would actually not have made a noticable difference at all, as all the resources of the real hardware would have been utilized just as we had experienced in the virtual machine.)
Even though we were extremely grateful to DVLabs for providing us a server, we did realize that having only a single server presented us with some limitations, the most important of which is "graceful" failure of core services if the server goes down. It also presented a single point of failure in that the web server getting pounded (such as a posting to Slashdot causes) could potentially cause all our services, including monotone and mail, to grind to a halt. Because of this we had tossed around the idea of finding another host two or three times.
When we most recently thought about hosting, Evan Schoenberg from the Adium project (who also happens to be a libpurple developer) put us in contact with NetworkRedux, who generously provides Adium's hosting. Thanks to Evan's intervention, we spoke with NetworkRedux about our hosting, and they were willing to donate not just one, but two servers and a ton of bandwidth to our project. Thanks to NetworkRedux's generous offer, we're getting the potential to have a lot more raw computing power at our disposal (including more memory to help increase caches, thus hopefully improving speed), as well as the ability to spread our services out a little so that a heavy hit to the website or to trac won't have a detrimental effect on all our services.
We were still discussing details of migrating to these two new servers when the server at DVLabs unexpectedly went down. It turns out that some miscommunication and bad timing caused the guys at DVLabs, who were migrating their own stuff, to down the server hosting our virtual machine, thinking we were no longer using it. We were, of course, confused at first, but once we determined what was happening we were much more at ease. None of our data had been lost. DVLabs were kind enough to supply us with the raw VM image for our own use to recover our data and complete our migration.
From these events we have learned a lesson--redundancy is not only good, it's pretty much a must-have. In the interests of redundancy and graceful failover, we've taken some steps to help prevent future long-term outages. One of those steps is that Gary Kramlich and I decided that we would help out by providing some service redundancy on our server, guifications.org. The biggest, and most important, service we are helping with is monotone. If for some reason the monotone server is down, guifications.org can seamlessly stand in for the mtn.pidgin.im server through the magic of DNS. We're also looking at potentially acting as a backup for more services.
So all in all, the commotion about our sites being down was really just a lot of noise about nothing truly significant. Yes, our sites were down, but this wasn't the end of the world. Google's massive cache had pretty much all the critical information anyone could need from our wiki. Yes, monotone was down, but guifications.org was ready to help--I had been running a read-only mirror for nearly a year which was ready and able to stand in as a production service if needed. Our mailing lists were down as well, but again, it wasn't the end of the world. Overall, we have weathered a prolonged service outage pretty well and taken some lessons from it to help us in the future.
In closing, I would like to extend my deepest, most heartfelt thanks on behalf of all Pidgin developers to DVLabs for their hosting services over the past two years and to NetworkRedux for the services they are now providing for us. We truly appreciate these donations!
Sunday, August 24, 2008
It's a bit late...
It's a bit late for me to be posting this now, but we did recently release Pidgin 2.5.0. There are a few things about this release that I'd like to discuss:
MSNP15. Finally, we have a release which includes updated MSN protocol support. We now support the personal message, current media, and offline message features of recent MSN official clients. We do not, however, support fast file transfers. We still support only the MSN-server-proxied transfer method, which while slow is 100% reliable. Quite frankly, file transfer isn't a high priority. If someone wants to implement fast file transfers, feel free to submit patches to us.
Custom smileys. During the development cycle of 2.5.0, a patch was accepted that implemented custom smiley support on MSN and provided a framework within libpurple for other protocols supporting the feature to grow support. There is one issue with this support--we can't save an incoming animated GIF emoticon. This is a limitation in gdkpixbuf, which doesn't support saving the GIF format by default. There are perhaps some additional dependencies we could incur for this, but nothing has been done in this area yet. To set up your custom smileys, go to the "Tools" menu on the Buddy List window and select "Smiley."
AIM Tooltip Crash. We've had a number of duplicated reports of crashes when "mousing over" an AIM buddy. The crash happens when trying to display the tooltip, and only appears when Glib uses its internal vsnprintf() implementation (which happens on some non-glibc systems, such as Windows and the BSD flavors of UNIX). An updated liboscar.dll that fixes this problem is available on ticket #6627. Please don't open any more tickets about this bug!
There is one other AIM bug I'd like to mention, since we're aware it exists. Not too long ago it came to our attention that when a Pidgin user joins an AIM chatroom and tries to send messages, less than 25% of the messages actually make it to the other members of the room. We are aware of the bug, but I believe we're not yet fully certain of the cause. A bug report already exists (#6590), so please don't open any more tickets about this bug either!
Hopefully all you MSN users out there enjoy the new features. It's taken a long time to get the features out there, but I think in the end the wait is worth it.
- MSNP15 support
- Custom smiley support
- The Windows and *BSD AIM tooltip crash
MSNP15. Finally, we have a release which includes updated MSN protocol support. We now support the personal message, current media, and offline message features of recent MSN official clients. We do not, however, support fast file transfers. We still support only the MSN-server-proxied transfer method, which while slow is 100% reliable. Quite frankly, file transfer isn't a high priority. If someone wants to implement fast file transfers, feel free to submit patches to us.
Custom smileys. During the development cycle of 2.5.0, a patch was accepted that implemented custom smiley support on MSN and provided a framework within libpurple for other protocols supporting the feature to grow support. There is one issue with this support--we can't save an incoming animated GIF emoticon. This is a limitation in gdkpixbuf, which doesn't support saving the GIF format by default. There are perhaps some additional dependencies we could incur for this, but nothing has been done in this area yet. To set up your custom smileys, go to the "Tools" menu on the Buddy List window and select "Smiley."
AIM Tooltip Crash. We've had a number of duplicated reports of crashes when "mousing over" an AIM buddy. The crash happens when trying to display the tooltip, and only appears when Glib uses its internal vsnprintf() implementation (which happens on some non-glibc systems, such as Windows and the BSD flavors of UNIX). An updated liboscar.dll that fixes this problem is available on ticket #6627. Please don't open any more tickets about this bug!
There is one other AIM bug I'd like to mention, since we're aware it exists. Not too long ago it came to our attention that when a Pidgin user joins an AIM chatroom and tries to send messages, less than 25% of the messages actually make it to the other members of the room. We are aware of the bug, but I believe we're not yet fully certain of the cause. A bug report already exists (#6590), so please don't open any more tickets about this bug either!
Hopefully all you MSN users out there enjoy the new features. It's taken a long time to get the features out there, but I think in the end the wait is worth it.
Thursday, July 31, 2008
The Holy War of Tool Choice
Tools are vital to any open-source project. Because of that, tool choice is critical. You really don't want to be switching tools frequently, because doing so is more work than you'll ever be able to put into the project. By that same token, you don't want to pick tools no one in the project wants to use. This leads to the problem--tool choice is often a "Holy War", so to speak, because everyone has their pet tool and thinks everything else to be inferior.
Let's use Pidgin as a practical example. Prior to our most recent legal problems (which prompted our rename), we had been evaluating new version control software for some time. At that time, we were using CVS. Almost everyone who has developed on a project as large as Pidgin will agree that CVS sucks. You can't rename files unless you rename them in the repository itself, branching and merging are a pain, etc. I wasn't involved with the project at this point in time, but I do know that every distributed VCS in existence at the time was evaluated. After the evaluation, Monotone was the preferred choice. Keep in mind that this was over two years ago, more likely closer to three.
We continued to use CVS, and later made an ill-advised switch to Subversion. Sticking with these tools was mainly due to our complete reliance on SourceForge, who offered only CVS and SVN. Finally, when it came time to rename our project to Pidgin, we had a donated virtual server on which we could run (essentially) whatever we wanted. Ethan Blanton used tailor to convert our SVN repository into a monotone database.
We also set up Trac for bug, patch, and feature request tracking. The built-in wiki was a bonus. We ended up installing a number of plugins for Trac to customize it and provide some additional features, such as restricting some ticket fields from those who we feel should not be able to modify them.
All was well initially. We had a few complaints because we chose to use monotone instead of sticking with SVN, but these were users who had no intention of contributing patches, so we were (mostly) happy to see them angrily go back to using our "blessed" releases. We did, however, have a lot of confusion initially about what monotone was and how it related to Pidgin. A few hundred explanations later and those questions faded too.
Trac proved to have some scalability and performance issues for us on numerous occasions. Eventually we were able to narrow the biggest issues down to their causes and implement fixes. Some of this involved a LOT of poking and prodding, as well as some assistance from Trac developers, who we thank profusely for their time in resolving the issues. I am quite surprised, overall, that we haven't had any real complaints about Trac. For a while it was frequently completely unusable, and of course we saw a number of complaints related to that, but since the issues have been resolved we've seen no real complaints.
As for monotone, fast-forward a year from the announcement of the rename. We consistently get complaints about having to download a "huge" database just to get the latest development source for Pidgin. We do also get a few complaints that it's hard to use, but almost all of these are solved simply by pointing people at the UsingPidginMonotone wiki page. As for the huge database, we do acknowledge that it is inconvenient for some people, but "shallow-pull" support is coming to monotone. A shallow pull would be similar to an svn checkout.
I don't mind the database, myself. I have 11 working copies (checkouts) from my single pidgin database (8 distinct branches, plus duplicates of the last three branches I worked on or tested with). Each clean checkout (that is, a checkout prior to running autogen.sh and building) is approximately 61 MB. If this were SVN, each working copy would be approximately 122 MB due to svn keeping a pristine copy of every file to facilitate 'svn diff' and 'svn revert' without needing to contact the server the working copy was pulled from. Now, let's add that up. For SVN, I would have 11 times 122 MB, or 1342 MB, just in working copies. For monotone, I have 11 times 61 MB for the working copies (671 MB), plus 229 MB for the database, for a grand total of 900 MB. For me, this is an excellent bargain, as I save 442 MB of disk space thanks to the monotone model. For another compelling comparison that's sure to ruffle a few feathers, let's compare to git. If I clone the git mirror of our monotone repository, I find a checkout size of 148 MB after git-repack--running git-gc also increased the size by 2 MB, but I'll stick with the initial checkout size for fairness. If I multiply this by my 11 checkouts, I will have 1628 MB. This is even more compelling for me, as I now save 728 MB of disk space with monotone.
Richard Laager also took the time to point out an interesting, but as yet unexploited feature of monotone--the use of certs allow us to do a number of useful things while being backed by the cryptographic signing of certs. For one, we could implement something similar to the kernel's Signed-Off-By markers. We could also have an automated test suite run compiles of each revision and add a cert indicating whether or not the test succeeded. We could also include a test suite using check (we do already have some tests) and use more certs to indicate success or failure. The extensibility of monotone using lua and the certificates is truly mind-boggling.
I think I've picked on version control enough here, but before I move on, I'd like to say that I don't specifically hate any one version control system. Ok, maybe I hate CVS and Subversion, but I'm certainly not alone in that. I don't, however, hate git, bzr, hg, etc. In fact, I think hg looks quite promising--the Adium folks have decided to switch to it, and are currently in the process of trying to convert from SVN. However, given that for us monotone was the best choice when we made our decision, and given the extensibility of monotone, I think it would be foolish of us to choose another tool at this point. Perhaps in a few more years when all the DVCSes have had time to mature further it will make sense to revisit this decision.
Another tool choice involves communication between developers and users. Specifically, forums. We have elected not to have forums for Pidgin, although as a legacy of our continued involvement with Sourceforge for download hosting we are stuck with the forums they provided us. We are quite frequently belittled for our choice to forego widely visible forums. At least four of us that I am aware of have a strong distaste for forums. I find them to be worse than the ticket system for attracting duplicated questions which most users won't search beforehand. All the useless extra features, such as emoticons, formatting, etc., are a waste, as well.
It's also been my experience with forums that one or two people will start answering others' questions with misinformation that will sound legitimate to those receiving the bad answers. I've seen this numerous times, but it comes to mind so quickly as I've seen it quite recently on Ubuntu's Launchpad "Questions" forum or tracker or whatever you call it for Pidgin.
Instead of forums, we have chosen to use mailing lists. We receive complaints about this too. These complaints run the gamut from "mailing lists are hard to use" to "but you can't search the mailing list!" In reality, the mailing lists are searchable--Google indexes our mailing lists, as do a number of other search engines. Perhaps we could provide a nice search box or something that does the necessary google magic (make the search query "terms site:pidgin.im") to make it a bit friendlier, but searchability exists. Not that it matters much in my experience, but hey, I'm just a developer; what do I know?
I'd also argue that mailing lists are easier to use than forums, as all you have to do is, y'know, send an e-mail. With a forum, you have to register, log in, find the right forum if there is more than one, then post your question, and check back for answers. With the mailing list, those genuinely interested in providing assistance will helpfully use the "Reply All" feature of their mail client to provide the answer both to the user requesting help and the mailing list's archive, thus providing an answer that is both helpful and searchable.
Even so, all the choices made during the evolution of a project will result in people trying to start a crusade, holy war, revolt, ..., over the chosen tool. Unfortunately, it's a fact of life. The best we can do is take it in stride, justify our choices, and move on.
Note: It's come to my attention that I had missed the ability to share a git database across multiple working copies. In that scenario, the total size of the database and 11 working copies is slightly under 750 MB, and thus a space savings in the neighborhood of 150 MB over monotone. It had been my understanding that I needed a copy of the database per working copy. I stand corrected. I don't use git on a daily basis, as the projects I work with currently use CVS, SVN, or monotone, so I am bound to miss finer details of git here and there. There are other reasons I prefer to stick with monotone, but I won't get into them here, as they're not important to the point of this post.
Let's use Pidgin as a practical example. Prior to our most recent legal problems (which prompted our rename), we had been evaluating new version control software for some time. At that time, we were using CVS. Almost everyone who has developed on a project as large as Pidgin will agree that CVS sucks. You can't rename files unless you rename them in the repository itself, branching and merging are a pain, etc. I wasn't involved with the project at this point in time, but I do know that every distributed VCS in existence at the time was evaluated. After the evaluation, Monotone was the preferred choice. Keep in mind that this was over two years ago, more likely closer to three.
We continued to use CVS, and later made an ill-advised switch to Subversion. Sticking with these tools was mainly due to our complete reliance on SourceForge, who offered only CVS and SVN. Finally, when it came time to rename our project to Pidgin, we had a donated virtual server on which we could run (essentially) whatever we wanted. Ethan Blanton used tailor to convert our SVN repository into a monotone database.
We also set up Trac for bug, patch, and feature request tracking. The built-in wiki was a bonus. We ended up installing a number of plugins for Trac to customize it and provide some additional features, such as restricting some ticket fields from those who we feel should not be able to modify them.
All was well initially. We had a few complaints because we chose to use monotone instead of sticking with SVN, but these were users who had no intention of contributing patches, so we were (mostly) happy to see them angrily go back to using our "blessed" releases. We did, however, have a lot of confusion initially about what monotone was and how it related to Pidgin. A few hundred explanations later and those questions faded too.
Trac proved to have some scalability and performance issues for us on numerous occasions. Eventually we were able to narrow the biggest issues down to their causes and implement fixes. Some of this involved a LOT of poking and prodding, as well as some assistance from Trac developers, who we thank profusely for their time in resolving the issues. I am quite surprised, overall, that we haven't had any real complaints about Trac. For a while it was frequently completely unusable, and of course we saw a number of complaints related to that, but since the issues have been resolved we've seen no real complaints.
As for monotone, fast-forward a year from the announcement of the rename. We consistently get complaints about having to download a "huge" database just to get the latest development source for Pidgin. We do also get a few complaints that it's hard to use, but almost all of these are solved simply by pointing people at the UsingPidginMonotone wiki page. As for the huge database, we do acknowledge that it is inconvenient for some people, but "shallow-pull" support is coming to monotone. A shallow pull would be similar to an svn checkout.
I don't mind the database, myself. I have 11 working copies (checkouts) from my single pidgin database (8 distinct branches, plus duplicates of the last three branches I worked on or tested with). Each clean checkout (that is, a checkout prior to running autogen.sh and building) is approximately 61 MB. If this were SVN, each working copy would be approximately 122 MB due to svn keeping a pristine copy of every file to facilitate 'svn diff' and 'svn revert' without needing to contact the server the working copy was pulled from. Now, let's add that up. For SVN, I would have 11 times 122 MB, or 1342 MB, just in working copies. For monotone, I have 11 times 61 MB for the working copies (671 MB), plus 229 MB for the database, for a grand total of 900 MB. For me, this is an excellent bargain, as I save 442 MB of disk space thanks to the monotone model. For another compelling comparison that's sure to ruffle a few feathers, let's compare to git. If I clone the git mirror of our monotone repository, I find a checkout size of 148 MB after git-repack--running git-gc also increased the size by 2 MB, but I'll stick with the initial checkout size for fairness. If I multiply this by my 11 checkouts, I will have 1628 MB. This is even more compelling for me, as I now save 728 MB of disk space with monotone.
Richard Laager also took the time to point out an interesting, but as yet unexploited feature of monotone--the use of certs allow us to do a number of useful things while being backed by the cryptographic signing of certs. For one, we could implement something similar to the kernel's Signed-Off-By markers. We could also have an automated test suite run compiles of each revision and add a cert indicating whether or not the test succeeded. We could also include a test suite using check (we do already have some tests) and use more certs to indicate success or failure. The extensibility of monotone using lua and the certificates is truly mind-boggling.
I think I've picked on version control enough here, but before I move on, I'd like to say that I don't specifically hate any one version control system. Ok, maybe I hate CVS and Subversion, but I'm certainly not alone in that. I don't, however, hate git, bzr, hg, etc. In fact, I think hg looks quite promising--the Adium folks have decided to switch to it, and are currently in the process of trying to convert from SVN. However, given that for us monotone was the best choice when we made our decision, and given the extensibility of monotone, I think it would be foolish of us to choose another tool at this point. Perhaps in a few more years when all the DVCSes have had time to mature further it will make sense to revisit this decision.
Another tool choice involves communication between developers and users. Specifically, forums. We have elected not to have forums for Pidgin, although as a legacy of our continued involvement with Sourceforge for download hosting we are stuck with the forums they provided us. We are quite frequently belittled for our choice to forego widely visible forums. At least four of us that I am aware of have a strong distaste for forums. I find them to be worse than the ticket system for attracting duplicated questions which most users won't search beforehand. All the useless extra features, such as emoticons, formatting, etc., are a waste, as well.
It's also been my experience with forums that one or two people will start answering others' questions with misinformation that will sound legitimate to those receiving the bad answers. I've seen this numerous times, but it comes to mind so quickly as I've seen it quite recently on Ubuntu's Launchpad "Questions" forum or tracker or whatever you call it for Pidgin.
Instead of forums, we have chosen to use mailing lists. We receive complaints about this too. These complaints run the gamut from "mailing lists are hard to use" to "but you can't search the mailing list!" In reality, the mailing lists are searchable--Google indexes our mailing lists, as do a number of other search engines. Perhaps we could provide a nice search box or something that does the necessary google magic (make the search query "terms site:pidgin.im") to make it a bit friendlier, but searchability exists. Not that it matters much in my experience, but hey, I'm just a developer; what do I know?
I'd also argue that mailing lists are easier to use than forums, as all you have to do is, y'know, send an e-mail. With a forum, you have to register, log in, find the right forum if there is more than one, then post your question, and check back for answers. With the mailing list, those genuinely interested in providing assistance will helpfully use the "Reply All" feature of their mail client to provide the answer both to the user requesting help and the mailing list's archive, thus providing an answer that is both helpful and searchable.
Even so, all the choices made during the evolution of a project will result in people trying to start a crusade, holy war, revolt, ..., over the chosen tool. Unfortunately, it's a fact of life. The best we can do is take it in stride, justify our choices, and move on.
Note: It's come to my attention that I had missed the ability to share a git database across multiple working copies. In that scenario, the total size of the database and 11 working copies is slightly under 750 MB, and thus a space savings in the neighborhood of 150 MB over monotone. It had been my understanding that I needed a copy of the database per working copy. I stand corrected. I don't use git on a daily basis, as the projects I work with currently use CVS, SVN, or monotone, so I am bound to miss finer details of git here and there. There are other reasons I prefer to stick with monotone, but I won't get into them here, as they're not important to the point of this post.
Sunday, July 13, 2008
"State of the MSN Plugin Address"
Yes, the title is intended to be half joking. Just a few days ago I made a comment to a fellow developer that "someone needs to make a 'state of the msn plugin' address," meaning that someone needs to let people know what's happening. Well, looks like that someone's going to be me. So here's the outsider's summary of what's happened.
In current Pidgin releases, the default MSN plugin uses the old MSNP9 protocol. This protocol doesn't support Personal Status Messages (PSM, also called "personal messages"), nor does it support offline IM, current media (a field that shows what music you're listening to or what video you're watching), or messaging while invisible.
Two years ago, for the 2006 Summer of Code, we had a student whose project was to update our MSN code to MSNP13. MSNP13 supported the PSM and current media, although at the time libpurple didn't support the status attributes for "now playing" or "current media" or whatever you prefer to call it. MSNP13 may also have supported messaging while offline, but I can't recall for certain and I don't, at the moment, feel like looking it up in the documentation we have online. Later this was updated to MSNP14 for Yahoo! interoperability support. At the end of the summer, the code wasn't of a quality we were ready to release. The code sat.
Last year, we had another student take on MSN protocol updating. This time the student updated the code to support MSNP14, starting from the previous work. This supported offline messaging for sure, and there were a lot of commits that dealt with the OIM features. Again, at the end of the summer, the code wasn't release-ready in our opinion, so again the code sat.
Earlier this year, one of our developers started working on the MSNP14 code in a side branch, and eventually merged it to the im.pidgin.pidgin branch in our monotone repository. Because the code still wasn't release-ready, we disabled it and forced the MSNP9 plugin to build by default. The more clever users were still able to figure out how to get MSNP14 going, but there weren't too many until we introduced an argument to the configure script to enable the MSNP14 plugin.
A new "crazy patch writer" appeared and started to work on MSNP15 support. We eventually gave him write access to the central monotone repository at pidgin.im. This has paid off for us very well. Elliott has done a LOT of work, including some necessary cleanup, stability improvements, etc. MSNP15 was finally merged to im.pidgin.pidgin this morning. Currently it is also the default MSN plugin, and I hope it stays that way, but it might be changed if anyone objects to it.
This would not have been possible without the work of Elliott, the two Summer of Code students, one other crazy patch writer (known as Masca on #pidgin), our XMPP Voice and Video Summer of Code student for this year (known as Maiku on #pidgin), and Dimmuxx and other Adium users' testing. I'm sure there are a number of other people I've forgotten to mention who have made contributions to this progress, and I apologize. My memory isn't what it used to be. To all these people who have helped with the long road of updated MSN protocol support, I give well-deserved thanks.
So what does all this mean? Well, simply, a lot of the feature requests we've been getting are finally resolved. Ideally we also will stop seeing people asking why they don't receive offline MSN messages, why they can't message while invisible, and why they can't see personal messages. For these reasons alone, I can't wait for the release of Pidgin 2.5.0!
In current Pidgin releases, the default MSN plugin uses the old MSNP9 protocol. This protocol doesn't support Personal Status Messages (PSM, also called "personal messages"), nor does it support offline IM, current media (a field that shows what music you're listening to or what video you're watching), or messaging while invisible.
Two years ago, for the 2006 Summer of Code, we had a student whose project was to update our MSN code to MSNP13. MSNP13 supported the PSM and current media, although at the time libpurple didn't support the status attributes for "now playing" or "current media" or whatever you prefer to call it. MSNP13 may also have supported messaging while offline, but I can't recall for certain and I don't, at the moment, feel like looking it up in the documentation we have online. Later this was updated to MSNP14 for Yahoo! interoperability support. At the end of the summer, the code wasn't of a quality we were ready to release. The code sat.
Last year, we had another student take on MSN protocol updating. This time the student updated the code to support MSNP14, starting from the previous work. This supported offline messaging for sure, and there were a lot of commits that dealt with the OIM features. Again, at the end of the summer, the code wasn't release-ready in our opinion, so again the code sat.
Earlier this year, one of our developers started working on the MSNP14 code in a side branch, and eventually merged it to the im.pidgin.pidgin branch in our monotone repository. Because the code still wasn't release-ready, we disabled it and forced the MSNP9 plugin to build by default. The more clever users were still able to figure out how to get MSNP14 going, but there weren't too many until we introduced an argument to the configure script to enable the MSNP14 plugin.
A new "crazy patch writer" appeared and started to work on MSNP15 support. We eventually gave him write access to the central monotone repository at pidgin.im. This has paid off for us very well. Elliott has done a LOT of work, including some necessary cleanup, stability improvements, etc. MSNP15 was finally merged to im.pidgin.pidgin this morning. Currently it is also the default MSN plugin, and I hope it stays that way, but it might be changed if anyone objects to it.
This would not have been possible without the work of Elliott, the two Summer of Code students, one other crazy patch writer (known as Masca on #pidgin), our XMPP Voice and Video Summer of Code student for this year (known as Maiku on #pidgin), and Dimmuxx and other Adium users' testing. I'm sure there are a number of other people I've forgotten to mention who have made contributions to this progress, and I apologize. My memory isn't what it used to be. To all these people who have helped with the long road of updated MSN protocol support, I give well-deserved thanks.
So what does all this mean? Well, simply, a lot of the feature requests we've been getting are finally resolved. Ideally we also will stop seeing people asking why they don't receive offline MSN messages, why they can't message while invisible, and why they can't see personal messages. For these reasons alone, I can't wait for the release of Pidgin 2.5.0!
Friday, July 4, 2008
Pidgin 2.4.3 and ICQ
Well, this week has certainly been interesting with Pidgin development. On July 1, AOL made a small change to the OSCAR servers that affected our ICQ users. A lot of misinformation was spread in #pidgin about what really happened, so here's my best attempt to explain it.
Libpurple (and thus Pidgin and Finch) identified itself as ICQ basic 14.34.3000. Of course, we sent a version string that included our real identification ("Purple/2.4.2" for example). This has worked for us for years. We send the version bytes from official clients to the server because if we don't mimic an official client, we get locked out of some features. Which features, I'm not really sure, since I don't really know or care about the internals of the OSCAR protocol.
Suddenly we started seeing reports about ICQ no longer working in #pidgin. Upon investigation we discovered that apparently the client ID has been "retired", and any logins from that client ID are immediately met with a message from the server demanding an upgrade to ICQ6. The server then forcibly disconnects us. We catch this error and display a somewhat different message instructing our users to go to pidgin.im to get an update.
Stu Tomlinson found the version identification information for the most current official ICQ client and changed our identification. Magically logging in and holding a conversation works. This patch became the reason for the release of Pidgin 2.4.3.
So to summarize, this was NOT a protocol change, merely a tweak on the server side to disallow logins from "old" clients. I suspect this is to try to weed out older, buggier clients and get the users on current software.
Libpurple (and thus Pidgin and Finch) identified itself as ICQ basic 14.34.3000. Of course, we sent a version string that included our real identification ("Purple/2.4.2" for example). This has worked for us for years. We send the version bytes from official clients to the server because if we don't mimic an official client, we get locked out of some features. Which features, I'm not really sure, since I don't really know or care about the internals of the OSCAR protocol.
Suddenly we started seeing reports about ICQ no longer working in #pidgin. Upon investigation we discovered that apparently the client ID has been "retired", and any logins from that client ID are immediately met with a message from the server demanding an upgrade to ICQ6. The server then forcibly disconnects us. We catch this error and display a somewhat different message instructing our users to go to pidgin.im to get an update.
Stu Tomlinson found the version identification information for the most current official ICQ client and changed our identification. Magically logging in and holding a conversation works. This patch became the reason for the release of Pidgin 2.4.3.
So to summarize, this was NOT a protocol change, merely a tweak on the server side to disallow logins from "old" clients. I suspect this is to try to weed out older, buggier clients and get the users on current software.
Monday, April 21, 2008
Google Summer of Code, Pidgin, and Announcing Vulture
Well, as probably everyone reading this blog knows, yesterday was the day that accepted students were announced for the Google Summer of Code. I think I can safely say that for the most part we Pidgin developers are happy with the results. That said, I do wish we could have accepted two students who did not make the final six.
One of those two students applied for the native Windows user interface project. The student in question was actually our top-ranked Windows UI application, and was going to be accepted until we yielded to another organization when a conflict arose. He has been very good about all this and wants to work on his Windows UI project anyway, once the Summer of Code is complete. To that end, I have set up a project called Vulture (in keeping with the bird theme for libpurple UI's--Pidgin, Finch, Adium, Instantbird). I've also created a basic svn repository for the project, done because Trac integration with monotone really sucks at present. If it turns out that monotone would actually be more useful, we can always use tailor to convert later.
Since the student will be participating in another organization's Summer of Code, at the moment I'm basically flying solo, but I do welcome any help others can provide, as I'm not a Windows developer. I can dabble in a bit here and there, but I'm far from well-enough versed to be an effective developer on this project. I will do what I can, but it's always been my intention to be involved in a support capacity and also to help shape the progress of the application (in so much as the active developers are willing to put up with me and consider my opinions). I would, of course, also act as a representative within the Pidgin project to help ensure that any needs Vulture may have of libpurple are met as best they can be.
It's my hope that the buzz Summer of Code created around the native Windows user interface project remains and attracts people to actually work with us to produce a .NET libpurple client that can really fit the needs of Windows users better than Pidgin can. It's also my hope that it can be done while remaining as similar to Pidgin as possible. I think that these two hopes are completely compatible and doable--Pidgin represents a pretty reasonable UI as it stands. There are, of course, a few points of pretty hot contention, but we can deal with that in Vulture by developer consensus at first, and let user input shape the decisions beyond that if we want.
So, happy Summer of Code to our accepted students, and for anyone who cares to join us at Vulture, feel free to drop us a line! :)
One of those two students applied for the native Windows user interface project. The student in question was actually our top-ranked Windows UI application, and was going to be accepted until we yielded to another organization when a conflict arose. He has been very good about all this and wants to work on his Windows UI project anyway, once the Summer of Code is complete. To that end, I have set up a project called Vulture (in keeping with the bird theme for libpurple UI's--Pidgin, Finch, Adium, Instantbird). I've also created a basic svn repository for the project, done because Trac integration with monotone really sucks at present. If it turns out that monotone would actually be more useful, we can always use tailor to convert later.
Since the student will be participating in another organization's Summer of Code, at the moment I'm basically flying solo, but I do welcome any help others can provide, as I'm not a Windows developer. I can dabble in a bit here and there, but I'm far from well-enough versed to be an effective developer on this project. I will do what I can, but it's always been my intention to be involved in a support capacity and also to help shape the progress of the application (in so much as the active developers are willing to put up with me and consider my opinions). I would, of course, also act as a representative within the Pidgin project to help ensure that any needs Vulture may have of libpurple are met as best they can be.
It's my hope that the buzz Summer of Code created around the native Windows user interface project remains and attracts people to actually work with us to produce a .NET libpurple client that can really fit the needs of Windows users better than Pidgin can. It's also my hope that it can be done while remaining as similar to Pidgin as possible. I think that these two hopes are completely compatible and doable--Pidgin represents a pretty reasonable UI as it stands. There are, of course, a few points of pretty hot contention, but we can deal with that in Vulture by developer consensus at first, and let user input shape the decisions beyond that if we want.
So, happy Summer of Code to our accepted students, and for anyone who cares to join us at Vulture, feel free to drop us a line! :)
Monday, March 17, 2008
Purple Plugin Pack 2.3.0--Finally!
Yes, you read correctly. We finally quit being lazy and released a new Purple Plugin Pack. Ironically, this comes barely after our Debian packager uploaded a build of version 2.2.0 (our last release) for inclusion in unstable (cue evil laughter here).
New in this release is the inclusion of Andrew Pangborn's Enhanced History plugin that replaces the stock History plugin by adding several excellent features useful for the obsessive-compulsive who close their IM windows as often as possible. It is, of course, also immensely useful for the more normal people among us. :)
I also ripped out a plugin I never should have included in the first place--the broadcast plugin. Never again will it see the light of day.
The biggest thing in the release though is the major bugs we've fixed. We've been plagued with build system bugs that I couldn't fix. Gary finally got around to fixing them, so hopefully never again will we hear people having trouble because they have to build all the plugins all the time. We also fixed several plugin bugs, including the crash in the xchat-chats plugin that Pidgin 2.4.0 caused.
Enjoy!
New in this release is the inclusion of Andrew Pangborn's Enhanced History plugin that replaces the stock History plugin by adding several excellent features useful for the obsessive-compulsive who close their IM windows as often as possible. It is, of course, also immensely useful for the more normal people among us. :)
I also ripped out a plugin I never should have included in the first place--the broadcast plugin. Never again will it see the light of day.
The biggest thing in the release though is the major bugs we've fixed. We've been plagued with build system bugs that I couldn't fix. Gary finally got around to fixing them, so hopefully never again will we hear people having trouble because they have to build all the plugins all the time. We also fixed several plugin bugs, including the crash in the xchat-chats plugin that Pidgin 2.4.0 caused.
Enjoy!
Sunday, March 9, 2008
The Cost of Progress and Options
We recently released Pidgin 2.4.0 with a UI change that seemed inocuous to me. The change was actually in four parts.
We've had the idea for quite some time to put typing notifications in the conversation history area in Pidgin, just like Finch has always done and Adium has done for as long as I can remember. Code wizard that he is, Sadrul pounded this out in pretty short order. Sure it had some bugs, but given enough feedback, he was able to fix most of the issues with it. There are a few minor problems left, but I'm confident they can be worked out over time. This change proved controversial. At least one of our developers still doesn't like it. I do, but I'm a fan of a few different UI changes that have been made in the last year. I think it's safe to say we're going to stick with this change.
After the typing notification change, the old icon in the menutray was removed. The same developer who dislikes the history area notification also hates the removal of this icon. I personally don't care one way or the other--both notifications affect only the currently visible conversation, and other notification methods exist, such as the Guifications plugin. This change has also proven to be controversial, but I believe that's mostly because this change and the history area notification happened in the same release.
The next big change was the removal of the bar that allowed manual resizing of the conversation input area. This has caused far more uproar than I ever expected. The closely-related fourth change is that the auto-resizing of the input area is now a first-class citizen and cannot be disabled. Again, this caused much more of an uproar than I expected. The same developer I mentioned above also disliked these changes quite strongly.
These changes are each quite innocent on their own. The combination, however, has caused a flood of complaints over the changes in behavior. I'd like to detail some reasons for the changes and why there is a cost associated with restoring them.
The typing notifications in the history area have been requested numerous times over the years. To me it seemed reasonable, considering that libpurple's other two UI's have the same behavior. Granted, there is no real reason for the change to have happened, but it's not the end of the world. There is no loss of functionality and no "usability" issues as have been claimed. Anyone who wants the old typing icon back can fairly trivially write a plugin to restore it.
The input area is a whole new beast. We introduced an autoresizing input area sometime back in the 2.0.0 betas, before we renamed to Pidgin. Most people never saw this because we showed the buddy icon of the person we were conversing with to the left of the input area, and we forced the input area to be the larger of the minimum size (1 line, I think?) or the height of the buddy icon. This had some interesting bugs, too, as under some circumstances the size would be saved there because of the manual resizing option, so the input area would not autoresize. Another fun bug came with pasting text. Pasting sufficiently large quantities of text with a sufficiently small IM window would cause the autoresize to expand the IM window in very strange ways, sometimes resulting in conversation windows larger than the screen. There was also a bug in which dragging the manual resizing bar would not save the size of the input area correctly, so using it to work around the other bugs wasn't guaranteed to work either.
Sean decided to scrap the manual resizing because it significantly simplified fixing some of the autoresize bugs that he had fixed. Anyone thinking it's laziness may be justified in that opinion; I don't really know or care. I don't think it matters much, but apparently some users think it does. Yes, there are some bugs still, the most notable being the 1-pixel bug where a new conversation has an input area that's only about 1 pixel high. The changes also seem to have caused a crash in the xchat-chats plugin, which hijacks the history area of the conversation window and replaces it with the same widget xchat uses for irc channels. The plugin does this only for chats. There's also the fact that the lower and upper thresholds for resizing don't work well for rather large IM windows. On the mailing lists, we've already determined that what we have is not the optimal solution (thanks to a few helpful users--a very rare breed), and we are working towards making it better. We won't, however, bring back the manual resizing anytime soon.
The whole debate over manual resizing brings us to another debate--a ton of users are demanding that the automatic resizing vs manual resizing be a preference. We're not willing to do that, as it incurs a cost we're not willing to bear. Ethan Blanton said it best himself in a message to the support list:
Making both behaviors available will certainly cause additional bugs and stupid behaviors that would not appear if we were to leave well enough alone or even to revert to the previous behavior. Reverting to the previous behavior, however, again introduces complex interactions that we're not exactly excited about having, not to mention brings back the bugs that were successfully eliminated with its removal.
Aside from all this is the fact that while we could keep the same interface forever, we don't feel it's necessarily a good idea. Experimenting with the user interface is a way to achieve progress--we can't learn without experimentation, and what we learn from said experiments helps us to make Pidgin a better application. Yes, it causes friction with users and some developers, but it is a necessary process. That's the important thing to keep in mind--we're not taking features away in an attempt to punish users or see how far we can push users. Our changes are legitimate attempts to improve Pidgin's interface, reduce the number of bugs present, and make Pidgin the best we can for our own needs.
In that vein, we are fully aware that we cannot make everyone happy with our UI. It's simply impossible. There are numerous other messengers out there, many of which will certainly meet some users' needs better than we can. We also now have libpurple as a real, usable entity which anyone can use to get our protocol support while getting a UI completely different from ours. This raises my final point--there are options in the IM application space, even though they may be expensive for users to bear the consequences of.
We've had the idea for quite some time to put typing notifications in the conversation history area in Pidgin, just like Finch has always done and Adium has done for as long as I can remember. Code wizard that he is, Sadrul pounded this out in pretty short order. Sure it had some bugs, but given enough feedback, he was able to fix most of the issues with it. There are a few minor problems left, but I'm confident they can be worked out over time. This change proved controversial. At least one of our developers still doesn't like it. I do, but I'm a fan of a few different UI changes that have been made in the last year. I think it's safe to say we're going to stick with this change.
After the typing notification change, the old icon in the menutray was removed. The same developer who dislikes the history area notification also hates the removal of this icon. I personally don't care one way or the other--both notifications affect only the currently visible conversation, and other notification methods exist, such as the Guifications plugin. This change has also proven to be controversial, but I believe that's mostly because this change and the history area notification happened in the same release.
The next big change was the removal of the bar that allowed manual resizing of the conversation input area. This has caused far more uproar than I ever expected. The closely-related fourth change is that the auto-resizing of the input area is now a first-class citizen and cannot be disabled. Again, this caused much more of an uproar than I expected. The same developer I mentioned above also disliked these changes quite strongly.
These changes are each quite innocent on their own. The combination, however, has caused a flood of complaints over the changes in behavior. I'd like to detail some reasons for the changes and why there is a cost associated with restoring them.
The typing notifications in the history area have been requested numerous times over the years. To me it seemed reasonable, considering that libpurple's other two UI's have the same behavior. Granted, there is no real reason for the change to have happened, but it's not the end of the world. There is no loss of functionality and no "usability" issues as have been claimed. Anyone who wants the old typing icon back can fairly trivially write a plugin to restore it.
The input area is a whole new beast. We introduced an autoresizing input area sometime back in the 2.0.0 betas, before we renamed to Pidgin. Most people never saw this because we showed the buddy icon of the person we were conversing with to the left of the input area, and we forced the input area to be the larger of the minimum size (1 line, I think?) or the height of the buddy icon. This had some interesting bugs, too, as under some circumstances the size would be saved there because of the manual resizing option, so the input area would not autoresize. Another fun bug came with pasting text. Pasting sufficiently large quantities of text with a sufficiently small IM window would cause the autoresize to expand the IM window in very strange ways, sometimes resulting in conversation windows larger than the screen. There was also a bug in which dragging the manual resizing bar would not save the size of the input area correctly, so using it to work around the other bugs wasn't guaranteed to work either.
Sean decided to scrap the manual resizing because it significantly simplified fixing some of the autoresize bugs that he had fixed. Anyone thinking it's laziness may be justified in that opinion; I don't really know or care. I don't think it matters much, but apparently some users think it does. Yes, there are some bugs still, the most notable being the 1-pixel bug where a new conversation has an input area that's only about 1 pixel high. The changes also seem to have caused a crash in the xchat-chats plugin, which hijacks the history area of the conversation window and replaces it with the same widget xchat uses for irc channels. The plugin does this only for chats. There's also the fact that the lower and upper thresholds for resizing don't work well for rather large IM windows. On the mailing lists, we've already determined that what we have is not the optimal solution (thanks to a few helpful users--a very rare breed), and we are working towards making it better. We won't, however, bring back the manual resizing anytime soon.
The whole debate over manual resizing brings us to another debate--a ton of users are demanding that the automatic resizing vs manual resizing be a preference. We're not willing to do that, as it incurs a cost we're not willing to bear. Ethan Blanton said it best himself in a message to the support list:
Each individual option may be simple or trivial, but they add up. Options to do with UI behavior are _particularly_ expensive, due to their frequent interactions with the vagaries of UI toolkit behaviors, etc.
This expense takes the form, mostly, of subtle bugs and extra programmer effort. We have had numerous examples of options which appear to be simple on the surface causing numerous bugs and undesirable behaviors. A recent and memorable example of this was the option for persistent conversations -- I think we all agree that the ability to have conversations "open" without an open window is one which is appreciated by many users, but this seemingly simple feature has caused quite an avalanche of small and not-so-small bugs. In fact, there is speculation that the one pixel high input area which some people are seeing stems from this very option.
Anyone who tells you that options are not expensive is either not a programmer, or has never worked on a large program. It is true that orthogonal options in a clean code base can be implemented with little cost in *many* circumstances, but even the most forward-thinking and competent programmers will be blind-sided by subtle interactions on occasion, especially when throwing third-party libraries and code into the mix.
Making both behaviors available will certainly cause additional bugs and stupid behaviors that would not appear if we were to leave well enough alone or even to revert to the previous behavior. Reverting to the previous behavior, however, again introduces complex interactions that we're not exactly excited about having, not to mention brings back the bugs that were successfully eliminated with its removal.
Aside from all this is the fact that while we could keep the same interface forever, we don't feel it's necessarily a good idea. Experimenting with the user interface is a way to achieve progress--we can't learn without experimentation, and what we learn from said experiments helps us to make Pidgin a better application. Yes, it causes friction with users and some developers, but it is a necessary process. That's the important thing to keep in mind--we're not taking features away in an attempt to punish users or see how far we can push users. Our changes are legitimate attempts to improve Pidgin's interface, reduce the number of bugs present, and make Pidgin the best we can for our own needs.
In that vein, we are fully aware that we cannot make everyone happy with our UI. It's simply impossible. There are numerous other messengers out there, many of which will certainly meet some users' needs better than we can. We also now have libpurple as a real, usable entity which anyone can use to get our protocol support while getting a UI completely different from ours. This raises my final point--there are options in the IM application space, even though they may be expensive for users to bear the consequences of.
Friday, January 4, 2008
Why I Hate "Modern" IM
There are few things I dislike about "modern" instant messaging. I really do miss the days of old, when IM was text only and the most complicated thing we had to worry about was emoticon display and whether ICQ's or AIM's protocols were going to change while we slept. Yahoo was a joke, MSN was an afterthought, Pidgin was still known by its former name and still used GTK+ 1.2.x, and a few disgruntled people would decide to fork on occasion. Even emoticon display wasn't that important--the focus was communication by a faster method than e-mail.
I, like many of my generation, discovered "web mail" and then graduated into more advanced things that this magical "internet" had to offer. Quickly, I started using e-mail for communication purposes instead of random notifications that were overall meaningless. Thankfully, my school's internet service provider offered a way to have students and teachers each have their own e-mail address. When one of my classes required a school-sanctioned e-mail address, I was assigned the 354th e-mail address issued to my school. The ISP used an HP-UX machine for mail. With trusty old telnet and the MacOS Classic application BetterTelnet FAT as our new best friends, we would log into the mail host (omalp1; I'll let the obsessed among you see if you can figure out the rest of the hostname even though it doesn't exist anymore--good luck!) and be presented with a shell. None of us had any clue about UNIX in those days, so we knew of only two commands, pine and exit, and these were taught to us as the only "blessed" commands to use at the "dollar sign thingy."
My classmates and I started using e-mail to communicate both in and out of classes. At some point I finally got dialup internet access at home, albeit absolute trash, and joined in. We used this e-mail like it was IM--we would religiously check our e-mail and reply as soon as we received the other person's message. This worked pretty well until someone among us discovered ICQ. Soon we all used it. Ah, this brings back the memories. Mirabilis' official ICQ 98b client, as I recall, was my first ICQ client. I don't recall if emoticons worked at this stage or not; I don't believe I ever used them at that point. Some time later, an AOL user among us discovered that AIM and AOL's built-in messaging component could talk to each other. A few of us migrated to AIM, but most of us ran both AIM and ICQ together. All was still good with the world. AIM 3.x, with all the neat little back doors to get around AOL users being able to be invisible to AIM users, was the client of choice. Basic HTML formatting, emoticons, sounds that didn't drive us crazy when we got messages. Ah, these were the days. I miss it!
At some point later, IM started to go down hill. Someone decided that file transfer needed to be a first-class citizen in client features. Yahoo grew webcam support. MSN became a contender despite its stupid design (I'll come back to this). Third-party clients actually became useful and welcome, especially the multiprotocol variety. And the spiral continued.
Today, we have a disaster area of IM features that everyone demands of every client, even the ones without the resources to develop and provide such features. Yahoo has more file transfer methods than I care to count, and changes them about as frequently as necessary to irritate the crap out of third-party client developers. MSN has enough file transfer methods to make AIM's newly increased three-count look like a fledgling effort. I won't even speculate on how many methods other protocols have or bother to find out how many XEP's exist for file transfer over XMPP. The exact numbers here are all irrelevant--the point is that we have a landfill full of code to cope with these bajillion file transfer methods and scenarios.
Webcam support comes to mind next. Suddenly these USB camera devices exploded onto the market and now everyone had to be able to use them in their IM clients. Then came the wave of "You don't have a webcam? Why do you bother to be online?" from those so addicted to their webcams that textual communication seemed like the stone age. Every IM provider that implements this is hell-bent on world domination or something and decides to pick protocols that are completely different from the "competitors' products," eliminating any hope of sane interoperability between these features of the services. Some of these protocols provide features others don't. Users blindly used whatever was thrust in front of them. Pornographic use of the webcam appeared. Stupidity ensued, while valid uses of the webcam, such as family communication and distance learning, didn't become major factors until far later in the game.
As if webcam support wasn't bad enough, some morons in MSN's management and development ranks decided they needed to introduce custom emoticons, winks, and nudges. Custom emoticons, of course, allow any user to send any random image to replace any specified text. MSN being the world of frequent abuse (again I'll come back to the abomination that is MSN later), users quickly took advantage of this and soon we had users replacing literally all of their commonly-typed text with obscure, crappy multicolored flashing emoticon crap that should never have existed in the first place.
Enter the nudge, now. Nudge a user and cause whatever stupid effect in the official client. Suddenly it too was all the rage for no real reason. Somewhere along the line Yahoo introduced a similar feature called buzz that shakes the message window. This too was hotly demanded. And let's not forget about the winks. Flash objects, I believe they were. More crap that doesn't belong in IM. Seriously, take this stuff to poorly-designed personal web pages, or even MySpace, and keep it out of IM.
And MSN as a whole. What a wonderful disaster area. When MSNP2 was around, I suppose I could excuse the lack of a status message--after all, ICQ didn't have them for a good many years either. But I won't excuse it--AIM had user-definable away messages from the beginning of my recollection, although available messages and invisibility are relatively recent additions by comparison. To me this negates any valid reason for MSN not to have supported some sort of status message. Then we have the friendly name, which I swear was invented solely to give users something not to use, but to abuse in ways unimaginable to those who implemented it. The ability to have graphical emoticons render in the friendly name comes to mind. So does the use of the friendly name to compensate for the lack of status messages. By this point, the protocol's usage had devolved into a cesspool anyway, so who really cared about the addition of custom emoticons, winks, and nudges, right? Apparently someone thought so, because now we have all that crap on top of it.
Now it's too late in the game, but MSN decided to implement another new feature--Personal Status Messages. Wow, now MSN joins the ranks of sane IM protocols, right? Not quite. The PSM now gets abused almost as badly as the friendly name is. I submit that it's called a "personal status message" for a reason, and that the reason is not to provide another field for users to abuse. But, as I said, it's too late in the game now. Users don't change unless you force them to, and this feature-too-late-to-really-matter certainly doesn't give us any evidence with which to claim that users change.
If I haven't alienated everyone who would bother to read this post by now, those reading this far are probably wondering what relevance this has to Pidgin or the Purple Plugin Pack, given my repeated prior related postings about both. Well, it's relatively complicated.
A couple years ago, long before our renaming fiasco started, Pidgin had a contributor who was submitting pretty large chunks of code to our MSN support. I wasn't around much for this; I was mostly just a user at this point and hadn't even attempted to contribute yet. Somewhere along the line, communications broke down, frustrations came into play, like probably a million other things I don't know or care about right now did. The bottom line is that the patch sat and the contributor left.
Fast forward to now and look at Pidgin's MSN support. No one can look at it and say with any seriousness that we support MSN features well--granted, we do a decent job but we don't support status messages and still use MSNP9. After two Google Summer of Code projects, we have an MSNP14 protocol plugin, but it was merged into our main development line too soon and still isn't ready for general consumption. As a result of this, we still ship MSNP9 and disable MSNP14 by default. We have a new Crazy Patch Writer who has implemented some elements of MSNP15 support. If and when any of this new MSN code becomes release-ready is really anybody's guess. (Not trying to slight anyone here; stabilizing new protocol support is work.)
The same contributor I mentioned earlier has now returned with a fork of the MSN protocol plugin for libpurple. He implemented server-side alias storage and some parts of the MSN peer-to-peer file transfer methods. Near as I can tell from the mailing list discussions, this is all still our existing MSNP9 stuff with some modifications. All fine and well, I suppose, but really these patches should have been submitted to us for review before forking, in my opinion.
In the ensuing discussions over the forked protocol plugin, the Personal Status Message issue has come up again, and there seems to be a difference of opinion on what the intended use of such a status message is. One side of the debate insists that this message has nothing to do with status but is instead for advertisements that allegedly are completely unrelated to status. The other side, which I find myself firmly rooted in the ranks of, argue that this PSM is a status message and ought to be treated as such. It seems the battle lines are clearly drawn for a war over how to implement this stuff.
All in all, I come back to the simple statement--I hate "modern" IM with a passion. I want to return to the days when there were no graphical emoticons, no webcams, no nudges, nothing that caused this kind of crap to turn up. Amazingly, this comes from someone who still refuses to use a text-mode IM client for routine IM needs and instead runs Pidgin all the time; go figure.
In closing, why can't we all just use XMPP and forget about voice and video, nudges, winks, emoticons, etc. forever?
I, like many of my generation, discovered "web mail" and then graduated into more advanced things that this magical "internet" had to offer. Quickly, I started using e-mail for communication purposes instead of random notifications that were overall meaningless. Thankfully, my school's internet service provider offered a way to have students and teachers each have their own e-mail address. When one of my classes required a school-sanctioned e-mail address, I was assigned the 354th e-mail address issued to my school. The ISP used an HP-UX machine for mail. With trusty old telnet and the MacOS Classic application BetterTelnet FAT as our new best friends, we would log into the mail host (omalp1; I'll let the obsessed among you see if you can figure out the rest of the hostname even though it doesn't exist anymore--good luck!) and be presented with a shell. None of us had any clue about UNIX in those days, so we knew of only two commands, pine and exit, and these were taught to us as the only "blessed" commands to use at the "dollar sign thingy."
My classmates and I started using e-mail to communicate both in and out of classes. At some point I finally got dialup internet access at home, albeit absolute trash, and joined in. We used this e-mail like it was IM--we would religiously check our e-mail and reply as soon as we received the other person's message. This worked pretty well until someone among us discovered ICQ. Soon we all used it. Ah, this brings back the memories. Mirabilis' official ICQ 98b client, as I recall, was my first ICQ client. I don't recall if emoticons worked at this stage or not; I don't believe I ever used them at that point. Some time later, an AOL user among us discovered that AIM and AOL's built-in messaging component could talk to each other. A few of us migrated to AIM, but most of us ran both AIM and ICQ together. All was still good with the world. AIM 3.x, with all the neat little back doors to get around AOL users being able to be invisible to AIM users, was the client of choice. Basic HTML formatting, emoticons, sounds that didn't drive us crazy when we got messages. Ah, these were the days. I miss it!
At some point later, IM started to go down hill. Someone decided that file transfer needed to be a first-class citizen in client features. Yahoo grew webcam support. MSN became a contender despite its stupid design (I'll come back to this). Third-party clients actually became useful and welcome, especially the multiprotocol variety. And the spiral continued.
Today, we have a disaster area of IM features that everyone demands of every client, even the ones without the resources to develop and provide such features. Yahoo has more file transfer methods than I care to count, and changes them about as frequently as necessary to irritate the crap out of third-party client developers. MSN has enough file transfer methods to make AIM's newly increased three-count look like a fledgling effort. I won't even speculate on how many methods other protocols have or bother to find out how many XEP's exist for file transfer over XMPP. The exact numbers here are all irrelevant--the point is that we have a landfill full of code to cope with these bajillion file transfer methods and scenarios.
Webcam support comes to mind next. Suddenly these USB camera devices exploded onto the market and now everyone had to be able to use them in their IM clients. Then came the wave of "You don't have a webcam? Why do you bother to be online?" from those so addicted to their webcams that textual communication seemed like the stone age. Every IM provider that implements this is hell-bent on world domination or something and decides to pick protocols that are completely different from the "competitors' products," eliminating any hope of sane interoperability between these features of the services. Some of these protocols provide features others don't. Users blindly used whatever was thrust in front of them. Pornographic use of the webcam appeared. Stupidity ensued, while valid uses of the webcam, such as family communication and distance learning, didn't become major factors until far later in the game.
As if webcam support wasn't bad enough, some morons in MSN's management and development ranks decided they needed to introduce custom emoticons, winks, and nudges. Custom emoticons, of course, allow any user to send any random image to replace any specified text. MSN being the world of frequent abuse (again I'll come back to the abomination that is MSN later), users quickly took advantage of this and soon we had users replacing literally all of their commonly-typed text with obscure, crappy multicolored flashing emoticon crap that should never have existed in the first place.
Enter the nudge, now. Nudge a user and cause whatever stupid effect in the official client. Suddenly it too was all the rage for no real reason. Somewhere along the line Yahoo introduced a similar feature called buzz that shakes the message window. This too was hotly demanded. And let's not forget about the winks. Flash objects, I believe they were. More crap that doesn't belong in IM. Seriously, take this stuff to poorly-designed personal web pages, or even MySpace, and keep it out of IM.
And MSN as a whole. What a wonderful disaster area. When MSNP2 was around, I suppose I could excuse the lack of a status message--after all, ICQ didn't have them for a good many years either. But I won't excuse it--AIM had user-definable away messages from the beginning of my recollection, although available messages and invisibility are relatively recent additions by comparison. To me this negates any valid reason for MSN not to have supported some sort of status message. Then we have the friendly name, which I swear was invented solely to give users something not to use, but to abuse in ways unimaginable to those who implemented it. The ability to have graphical emoticons render in the friendly name comes to mind. So does the use of the friendly name to compensate for the lack of status messages. By this point, the protocol's usage had devolved into a cesspool anyway, so who really cared about the addition of custom emoticons, winks, and nudges, right? Apparently someone thought so, because now we have all that crap on top of it.
Now it's too late in the game, but MSN decided to implement another new feature--Personal Status Messages. Wow, now MSN joins the ranks of sane IM protocols, right? Not quite. The PSM now gets abused almost as badly as the friendly name is. I submit that it's called a "personal status message" for a reason, and that the reason is not to provide another field for users to abuse. But, as I said, it's too late in the game now. Users don't change unless you force them to, and this feature-too-late-to-really-matter certainly doesn't give us any evidence with which to claim that users change.
If I haven't alienated everyone who would bother to read this post by now, those reading this far are probably wondering what relevance this has to Pidgin or the Purple Plugin Pack, given my repeated prior related postings about both. Well, it's relatively complicated.
A couple years ago, long before our renaming fiasco started, Pidgin had a contributor who was submitting pretty large chunks of code to our MSN support. I wasn't around much for this; I was mostly just a user at this point and hadn't even attempted to contribute yet. Somewhere along the line, communications broke down, frustrations came into play, like probably a million other things I don't know or care about right now did. The bottom line is that the patch sat and the contributor left.
Fast forward to now and look at Pidgin's MSN support. No one can look at it and say with any seriousness that we support MSN features well--granted, we do a decent job but we don't support status messages and still use MSNP9. After two Google Summer of Code projects, we have an MSNP14 protocol plugin, but it was merged into our main development line too soon and still isn't ready for general consumption. As a result of this, we still ship MSNP9 and disable MSNP14 by default. We have a new Crazy Patch Writer who has implemented some elements of MSNP15 support. If and when any of this new MSN code becomes release-ready is really anybody's guess. (Not trying to slight anyone here; stabilizing new protocol support is work.)
The same contributor I mentioned earlier has now returned with a fork of the MSN protocol plugin for libpurple. He implemented server-side alias storage and some parts of the MSN peer-to-peer file transfer methods. Near as I can tell from the mailing list discussions, this is all still our existing MSNP9 stuff with some modifications. All fine and well, I suppose, but really these patches should have been submitted to us for review before forking, in my opinion.
In the ensuing discussions over the forked protocol plugin, the Personal Status Message issue has come up again, and there seems to be a difference of opinion on what the intended use of such a status message is. One side of the debate insists that this message has nothing to do with status but is instead for advertisements that allegedly are completely unrelated to status. The other side, which I find myself firmly rooted in the ranks of, argue that this PSM is a status message and ought to be treated as such. It seems the battle lines are clearly drawn for a war over how to implement this stuff.
All in all, I come back to the simple statement--I hate "modern" IM with a passion. I want to return to the days when there were no graphical emoticons, no webcams, no nudges, nothing that caused this kind of crap to turn up. Amazingly, this comes from someone who still refuses to use a text-mode IM client for routine IM needs and instead runs Pidgin all the time; go figure.
In closing, why can't we all just use XMPP and forget about voice and video, nudges, winks, emoticons, etc. forever?
Subscribe to:
Posts (Atom)