Pidgin and the Impending Shutdown of Windows Live Messenger

So, Microsoft recently announced that they’ll terminate the Windows Live Messenger service in favor of Skype in early 2013.  We’ve been getting a number of questions about what this means for Pidgin.  Quite honestly, we don’t know.  At this point, all we know is that China will still be able to use Windows Live Messenger.  That leads us to believe that the servers providing MSNP service will remain active and maintained for some period of time after the announced shutdown, but it’s not clear whether or not that will be the case.  It’s also not clear if the servers supporting China’s continued use of WLM will be accessible to non-Chinese IP space.  Even further, it’s not clear if the recently-launched XMPP interface to the WLM network will remain functional.  We don’t support that yet though, as it requires some authentication magic we don’t implement.  Even if we implement support for the authentication this XMPP gateway requires, it could end up being a waste of time, as it could get shut down at any time, either before or after the rest of WLM.

And before anyone goes there, we can’t support Skype.  There is no documentation of the protocol available to us, nor is there code we can borrow from a cleanly reverse-engineered alternative implementation.  All that exists is SkypeKit, whose license agreement explicitly forbids its use in open-source software.  The license also forbids use in “server applications” which precludes doing something like wrapping a simple closed-source XMPP daemon around SkypeKit.  It is not currently possible to legally support Skype, so we won’t try.

The bottom line is we have no idea what the announcement means for Pidgin or any other alternative clients yet.  We’ll all just have to wait and see.

Major Changes Afoot

Well, it's been about forever since I last bothered to post anything here. Since my last post, we've released several times, introducing and fixing a bunch of bugs. Now, however, we're shifting our focus to new development of a sort that we don't do often—compatibility breakage and big internal changes. This means that our main development effort is now in working toward Pidgin and libpurple 3.0.0. I'm going to try to explain some of the work going on for the benefit of anyone who reads my ramblings, so here goes.

Versioning

First of all, there seems to be some confusion about how Pidgin and libpurple version numbers work. I'd like to try to clear some of that up.

Pidgin, Finch, and libpurple use what's called “Semantic Versioning.” That is, each part of the version number has a particular meaning for users and developers. We chose this scheme to assist plugin developers in knowing when significant changes would require effort on their part to maintain compatibility with current Pidgin and/or libpurple versions. It also helps our users by letting them know when their existing plugins will stop working. So, how does this semantic versioning work? Let's look at the format of our version numbers and find out.

Pidgin, Finch, and libpurple version numbers have three components, separated by dots. At the time of this writing, 2.10.0 is the current version number. These components are called major (currently 2), minor (currently 10), and micro or patch (currently 0). Let's look at what each means.

• Major: the major version doesn't change often. The last time we changed it was in 2007 with the release of Pidgin 2.0.0, and before that was in 2004 with 1.0.0 when we started using semantic versioning under our previous name. Whenever this number changes, we've made changes to Pidgin, Finch, or libpurple that break compatibility with every UI and plugin that currently exists. Usually this means that we've removed something from the API exposed to plugins and UI's, or that we've changed something about a function (its name, arguments, return type, or the header file it's in). Sometimes plugins or UI's can just be recompiled when this happens; other times they need maintenance to become compatible with the new release. Additionally, the major version never decreases. It will always increase when it changes.
• Minor: the minor version changes more frequently than the major version, but generally less often than the micro or patch version. Whenever the minor number increases, we've added things to our API that do not break compatibility with existing plugins or UI's. A prime example of this is adding the voice and video support we added in 2.6.0. We added a bunch of stuff to the existing API, but didn't change anything that would cause a break in backward compatibility with existing plugins and UI's. When the minor version decreases (gets set back to 0), it means that the major version has changed, and the whole cycle starts over.
• Micro or patch: The patch version changes with almost every release. When this version increases, it means that we haven't touched API at all; instead we've done nothing but fix bugs or add small features that don't affect compatibility with plugins or UI's. When this version decreases (gets set back to 0), it means that the minor version has changed.

Related to all this, and important only for plugin and UI developers (so skip this paragraph if you're not a developer!), is the behavior of the PURPLE_VERSION_CHECK macro. Many developers expect PURPLE_VERSION_CHECK(2, 5, 0) or similar to expand to a statement that evaluates to 1 or TRUE in the case of building against libpurple 3.0.0. This, however, is not the case. Because our major versions are incompatible with each other, we have intentionally written PURPLE_VERSION_CHECK to fail if the major version is not an exact match. We understand this can be confusing and inconvenient, and we sincerely apologize for that, but we're not going to change it.

What all this means is that if the first number (major version) changes, you're going to need to update your plugins when you upgrade Pidgin, Finch, and libpurple. If the second (minor version) or third (micro version) numbers change, it means you need to upgrade Pidgin, Finch, and libpurple, but your existing plugins will still work. No matter what the Pidgin, Finch, and libpurple version numbers are, you should always be using the newest version.

Structure (“struct”) Hiding

This paragraph is for those who aren't programmers. Feel free to skip it if you don't care about it. Pidgin is written in the C language. C has a type system, which means that if you declare a variable (imagine this as a box somewhere in your computer's memory) you can store only one type of data in it. A structure, or struct in C parlance, is a type made up of other types arranged sequentially. This is pretty easy to picture if you think of Lego blocks--stack a red block, a blue block, a green block, a yellow block, and a white block on top of each other and connect them together and you now have a structure made out of Lego blocks. It's pretty similar in C, except that you're telling the compiler to assemble something out of sequential boxes of memory instead of little plastic blocks.

Pidgin uses structs everywhere. We use them to represent things like your buddies, conversations, accounts, etc. Currently in Pidgin 2.10.0, most of the structs are in the public API—that is, anyone can directly access the members of the structs and do whatever they like. This is all fine and well, but it means that if our code changes such that a particular struct needs to grow significantly by the addition of new members, we can't always do that without breaking backward compatibility. (Yes, we included padding in a number of our structs, but we've burned through the padding in several of them, and although there are ways to work around it, I and a few other developers don't like them.) It also means that if we discover, for example, that switching the order of members in a struct allows the compiler to improve its optimizations or if we think a different order makes more logical sense for those of us reading and maintaining the code, we absolutely can't do this without breaking compatibility. We also can't rename members of structs for the same reason—it breaks compatibility with existing plugins, UI's, etc.

Because having these structs in the public API limits us so much, we're striving to hide as many of them as possible. By hiding, we mean that we're removing the struct definitions from the header files and moving them to the .c files, thus making them private. Plugin and UI authors will still be able to reference the “objects” with pointers, passing them to functions and operating on them with the appropriate sections of our public API, but no longer will the members of the structs be directly accessible outside of the .c files that define the functions that interact with them. For example, we have a PurpleRoomlist struct in libpurple/roomlist.c and libpurple/roomlist.h. For 3.0.0, the struct definition is in roomlist.c; thus the individual members are not directly accessible outside roomlist.c, even in other parts of libpurple. This “hiding” of the strcuts allows us significant internal flexibility in each file to modify the struct as we see fit.

Clean-Slate API Documentation

Because we generally change so much each time we increase our major version number, the API documentation can get a bit confusing if we keep doing @since tags and whatnot in our doxygen documentation. Our general overall feeling is that we prefer just starting with a clean slate at each new major version. This means that each time we do a new major version, all existing @since tags will disappear, any functions marked as @deprecated will be removed, renamed, or replaced as described in the @deprecated statement, and so on. We realize this decision may make some things more difficult for some plugin and UI authors, and we apologize for that, but our aim is to have overall cleaner documentation for everyone.

Merging of Old Projects

Over the years that we've participated in Google's Summer of Code program, we've accumulated a number of branches that have been sitting for quite some time. Most of these need some form of TLC and integration work. We want to try to incorporate at least some of them into 3.0.0 so we can finally benefit from the fruits of the students' labor. Most notably, we've been talking about merging the webkit integration branch into what will become 3.0.0. Eventually, this would allow the support of Adium's message styles, although it may not happen right away.

Another project of notable interest is some of the logging changes that went on in a previous Summer of Code project. One of our new Crazy Patch Writers took some of that work and made some progress on it; we don't know if this will make it for 3.0.0 or not yet, but it would be nice to have some of the features, such as non-blocking log writing.

Other Changes As Wanted

We may decide to make other changes since a major version change gives us the opportunity to break so much. There have been a number of ideas floated, ranging from supporting that XDG directory spec that I can't stand to doing away with the xml files in .purple and replacing them with something else (with what, in particular, has not seriously been discussed). There is a whole range of possibilities of things we could do for 3.0.0; it's just a matter of one of us wanting it and sitting down to write it.

So everyone should stay tuned, as we'll be making more and more changes over the course of the 3.0.0 development cycle. We don't yet know when we'll be releasing 3.0.0—this is another one of our famous “when it's ready, and not a minute before” time frames. We do know, however, that it will change a lot!

Current AIM Issues

Over the last couple hours, we've had nearly everyone and his/her brother in #pidgin asking about connections to AIM causing a certificate prompt. The specific prompt is for bos.oscar.aol.com. The issue here appears to be that AOL has let the certificate expire. Because our certificate validation is more strict than some other applications, Pidgin users will get this prompt at every connection until AOL installs an updated certificate.

To resolve the prompt, you can make one of a couple choices. You can choose to trust that the connection is fine even though the certificate is expired and click Accept, or you can take the safe route and click Reject until AOL replaces the certificate.

To reiterate, this is not a Pidgin problem, but an AIM server problem.

Complex Transient Statuses for Quick Effect

In an earlier post, I discussed how to take advantage of Pidgin's "saved status" feature. After seeing some recent confusion in #pidgin about our status features, I decided it would be a good idea to give a quick overview on how to take advantage of a feature I never use--creating statuses (including complex statuses) from the status selector on the buddy list window.

The status selector feels like it's been around forever. Prior to this, we had a rather horrible interface to "status" whereby you could either globally set all accounts to "away" or go to each individual account and configure a given status (away, do not disturb, vacation, etc.). All this was done via a single menu. For those of you who don't remember it, let me just say that it sucked. Someone (I think it was Sean Egan) threw that whole UI out the window and came up with the current status selector that reminds me a lot of the old Windows ICQ 99 client that had a pretty similar status selector. Ours is a bit more sophisticated, though.

The simple use of the status selector is fairly obvious--pick a status and type a message. In fact, this is what the majority of our users do when they change statuses. This is what is called a transient status. What most people don't know, however, is that you can create complex statuses, such as having your MSN account set to "Busy" while your AIM account is set to "Invisible." Let's give a quick example.

I have a number of accounts in Pidgin. Let's say I want to have my pidgin.im XMPP account set to "Available" but the rest of my accounts set to "Away" to create a simplistic scenario that's really easy for me to snag a screenshot of. To do this, I'll go to the status selector and select "New status..." like in this picture:

Then I'll see this window, where I've already gone and entered a title for the status. This is the name you'll see in the status selector and in the middle section of the status selector's menu. If I wanted a message to go with the away status, I'd enter it here now, as well.

Since I want most of my accounts to be away, I'll leave the "Status" selection as "Away." As you can see, I have my mouse pointer over "Use a different status for some accounts," which when expanded will allow me to set statuses for individual accounts, like so:

I found my pidgin.im account in the list already, so I'll check the box in the "Different" collumn. That brings me to a new window:

Now I want my pidgin.im account to be available, so I'll accept what I see. If I wanted a message here, I'd type it in the "Message:" box. When I'm happy with what I have, I'll click OK. Notice that the "Status" column in the previous window changes to reflect what you selected for the individual account.

Since I'm happy with what I have, I'll now click "Use", which applies the status. If I wanted to make this a saved status that is saved permanently, I could click the "Save" button or the "Save & Use" button if I wanted to also immediately apply it when saving.

That's all there is to it!

Death of a thousand tickets

Well, by now it's obvious to the world that I kicked Pidgin 2.7.4 out of the nest last week. Although that release included some nice new features for ICQ users, lots of bug fixes, and some remote crash fixes, it's not without its share of problems. Those problems are producing a bunch of duplicate tickets for us to deal with, so I thought it might be a good idea to post about them here just in case anyone bothers to read my rambling.

The first problem is the AIM/ICQ chat bug. When using a multi-user chat on AIM or ICQ, no messages can be sent. You'll get an error stating your message was too long. This was an unintended side effect of merging the work of one of our Summer of Code students. Ivan accidentally removed a line of code that he shouldn't have. Yes, you read that right--a single line of code. Apparently it was pretty important! Ivan restored that line and things will be working as they should in 2.7.5 when we release it.

The second problem is an old one that's come back with a vengeance. This one is an ICQ issue where messages that contain <> disappear because they're treated as HTML tags. This seems to be specific to combinations of Pidgin and other non-official clients. Pidgin recently gained some new ICQ protocol-level features related to formatted messages; some non-official clients, including older versions of Pidgin, don't handle it gracefully. At this point, we've done all we can do about it; the other clients will have to step up and make fixes to handle the messages correctly now.

The next problem is another wonderful ICQ encoding bug. I'm not sure if this one is a side effect of Ivan's work or not, but Ivan did do a lot of work on encoding problems on ICQ. We all thought things were improving, but apparently there are still a few odd cases (and a crapton of stupid ICQ clients that should be purged from existence) in which our handling of encodings just isn't quite right. We're aiming to have this fixed for 2.7.5, but it's not quite done yet.

Is it obvious I wish ICQ would just disappear immediately and permanently?

The last "big" problem that seems to be cropping up is a crash related to MSN file transfers. When using direct-connect file transfers, somewhere along the line we do something stupid internally that causes a crash. We thought we had it fixed for 2.7.4, but alas it still exists. We're looking into it and hope to get it fixed soon. In the meantime, it's pretty easy to prevent the crash--edit your MSN account and turn off the direct-connect file transfers. It's an option on the advanced tab of the account editor.

At any rate, since we know about these bugs already, please, please, please DON'T open new tickets about them!

The Proxy Conundrum

As many of you Yahoo users out there know, Pidgin 2.7.0 and 2.7.1 introduced a challenge for you. In certain proxied environments, many of you can't connect. Quite frankly, (mostly) it's my fault. Let's start by explaining what I've done.

In Pidgin 2.7.0 I made some seemingly innocent changes to the Yahoo protocol plugin that I thought would magically make some problems go away. These changes make us mimic the official Yahoo client better. Very early in the connection process (in fact, it's the first step), the official clients request a specific URL from a Yahoo web server. The server's response tells the client what messaging server to connect to. The client reads that address and connects to it, then goes along its merry way. So I made our Yahoo plugins do this, once I determined the correct URL's for both Yahoo and Yahoo Japan. Everything was great when I tested.

But there's my mistake--I tested it. I don't run a proxy server--I'm an IT nerd, but I'm not that much of an IT nerd. So, I allowed this to be released, not knowing there were problems in environments with restrictive proxies. It came back to bite me hard, as I backported this stuff to Adium's build of libpurple 2.6.something for one of their beta releases and several users started complaining about being unable to connect. Then we got a couple tickets of our own. At first, I was stumped, until a user who was willing to almost bend over backwards to assist us came along.

This user worked in an environment where two separate proxy servers were in use. One handled HTTP and HTTPS traffic and the other handled IM protocols. It took me quite a while to finally wrap my head around his problem, but once I did (with the help of my fellow developer Etan Reisner), I was able to start solving the problem. It turns out I actually had two problems on my hands, but at first it seemed like none of it was the fault of our code.

The first problem is that Pidgin allows configuring proxies in two places--the "Proxy" tab shown when editing an account and the "Proxy" tab in Preferences--but does not allow specifying what ports or services are handled by each proxy. This ordinarily isn't such a big issue, except that I went to the extreme of creating an option for Yahoo accounts to help some users with weird proxies that don't handle SSL at all. This option is used to determine whether to send an HTTPS request (like Yahoo uses for login) to the account's configured proxy or to the global proxy configured in Preferences. When I did this for 2.6.0, I created a new utility function for URL requests that allowed specifying the account the request was for. If the magic value NULL is passed to this function, the request will use the global proxy instead of the account proxy. This allows our proxy code to route the request to the appropriate proxy. Being the idiot I can be on occasion, I set all URL requests in the Yahoo plugins to use the account proxy server no matter what. This is where the user's problem came in--his IM proxy doesn't do HTTP at all. Once Etan and I agreed on how to fix it, I changed the SSL option I mentioned earlier to act on both HTTP and HTTPS requests. I supplied our user the test build. It still didn't work, but the debug log showed some interesting new info.

Again, it took some head-bashing to understand what was going on, and I still wasn't the one who figured it out. In the meantime, I decided to implement a quick hack to make users at least be able to connect temporarily while we figured out what was really broken in our proxy code. The quick hack was to implement a worst-case, last-resort connection to a previously known messaging server. Unfortunately, I could implement this only for Yahoo, not for Yahoo Japan, as the Japan network appears to no longer has a single canonical hostname by which to refer to the messaging servers. I am, however, positive that even on the regular Yahoo network, this hack will eventually break, which is why I resisted it in the first place. After a couple quick bugfixes brought to light by the user's further testing, we had a working fallback mechanism.

Once this was done, Daniel Atallah, one of our co-lead developers, realized that the second problem our helpful user had was our own proxy code's fault--for some stupid reason, we would never authenticate to an HTTP proxy if we were making an HTTP request on the standard HTTP port (80). Daniel implemented some quick fixes to make our URL requesting code behave better in these proxy situations, thus (we hope) finally solving this problem completely.

All these fixes will end up in Pidgin 2.7.2. We're not sure yet when we're going to release that, but we're hoping it's soon, as we have some fixes for other annoying bugs, such as the famous MSN direct connection crash.

Now, after reading all this, some of you might think, "Why don't they just use libproxy?" Truthfully, that's probably a good idea. There are two reasons we haven't done this yet. First, we looked at this over two years ago. At that time, we were led to believe libproxy didn't handle all the proxy types we did. We were corrected on that, but I, at least, missed that correction being posted. (Sorry, sometimes I'm blind, especially if something like that is buried in a flood of other ticket mails.) I suspect I'm not the only developer who did. Secondly, we're all creatures of habit. As the old saying goes, "if it ain't broke, don't fix it." Well, quite honestly, it's not broken for us as developers! Mainly that's because either we don't use proxies or any proxies in our way don't do any of the crazy stuff we ran into in this case. If it's not broken for us, we don't have much incentive to go hacking stuff up to support new libraries that solve problems we don't experience. I know that sounds like a lame excuse to a lot of you, and I can understand how it seems that way. But if we don't experience the problem, it's hard to test that we've fixed it, and we could in fact be making matters far worse for everyone else.

All this said, we probably should support libproxy. I've been thinking about how we could do this in a quick and effective manner, and so far I've come up with a few ideas that don't feel like they work very well. I think this is a perfect case where code should talk, or at least provide an example. So, I invite people who would like to see us support libproxy to write patches. Actively engage us on our development mailing list (devel@pidgin.im), talk with us in #pidgin on irc.freenode.net, or drop by our XMPP MUC (devel@conference.pidgin.im). Or, do some combination of the three. I can't promise we'll be the most valuable resources during the process, but we do at least try.

Now supporting libproxy is one thing. Actually being able to do useful stuff with the information libproxy gives us is another thing entirely. Our actual proxy connection code is, shall we say, a bit sub-par. Quite frankly, it sucks, even ignoring the crappy configuration portions. While supporting libproxy is nice and may solve a lot of our proxy problems, I guarantee it won't solve them all. Ideas and patches to fix this are greatly appreciated too.

I look forward to hearing from those of you inclined to help!

Pidgin 2.7.0: Movin' on Up

Well, I finally managed to kick Pidgin 2.7.0 out the door, as most of you are surely aware of by now. There are several important things to note about this release:

• AIM and ICQ clientLogin issues have been resolved. You should be able to now use both clientLogin and SSL together.
• Our internal libgadu has been upgraded from something from the stone age to something much, much more current. This solves the issues that (primarily) Windows users had with not being able to communicate with new (> 17,000,000) Gadu-Gadu numbers.
• Our Yahoo and Yahoo Japan plugins now retrieve connect server information directly from Yahoo, just like the official clients do. This took me way too much time to implement, but I believe I've implemented a fairly stable and maintainable method to retrieve this information. During this process, I learned a lot about how Yahoo's authentication works and made some improvements there too that will make us look much more like the official client on the wire during authentication.
• ICQ now has X-Status support! It took far longer than it should have--I originally wanted this in for 2.6.0 but it was nowhere near ready.
• Minimum GTK+ and GLib requirements are now 2.10.0 and 2.12.0, respectively. This allowed us to drop some 3800 lines of code, much of which was the result of needing to bundle certain GTK+ widgets so Pidgin could function on platforms with ancient GTK+. This also makes use of many of glib's convenience functions much more convenient.
• The Windows installers have changed to be less obnoxious with respect to GTK+. We no longer install a system-wide copy of GTK+ and instead install GTK+ locally to the Pidgin installation. This may disrupt some GTK+ theming. The other big thing is that now, the default installer will download GTK+ and optionally the debug symbols (so there's no longer a special debug installer needed for crash reports!). There's also an offline installer that, while almost double the size of the default installer, I prefer to use. This offline installer includes GTK+ and the debug symbols. I prefer to call it the everything-but-the-kitchen-sink package (the kitchen sink in this case being spell checking support, which still must be downloaded).
  

Of course, any release I had a major role in is bound to have problems. I built the Windows installers this time around, and did not know that I needed to do extra work to make the online installer function. Thus, many of you complained (and complained, and complained some more!) that the installer didn't work. Even after Daniel fixed this for me, we still saw complaints. If the online installer does not work for you, try the offline installer. I tested the offline installer and it worked for me. I forgot, however, to test the online installer. I was more interested in getting the release out in order to (I thought) minimize complaints.

Additionally, after I cut the tarballs for 2.7.0, several bug fixes rolled in that will be delayed until 2.7.1. Among these are a couple minor annoyance fixes for MSN and XMPP. And today fixes for more bugs rolled in, including two for Bonjour and a fix for the taskbar entry (window) flashing on Windows. I plan to push out Pidgin 2.7.1 in a couple weeks to resolve these, and hopefully more, problems.

In the mean time, enjoy!