Some Clarification on Yahoo! Issues

From the recent activity on our support mailing list and in our IRC channel (#pidgin on irc.freenode.net), it's plain to see there is still a LOT of confusion about the recent troubles logging into Yahoo Messenger via Pidgin. I'm going to try to clear some of the confusion up. I apologize for the technical nature of this post. Feel free to skip ahead to the last paragraph of "Solving the Problem" below if you don't care what the problem is and just want to fix it. Or take the simple approach and just upgrade to Pidgin 2.5.7 and skip down to "But I already upgraded! It still doesn't work!" if you still have trouble.

The Problem

The specific problem that affected us is the authentication mechanism. Yahoo Messenger 6 used a spectacularly complicated password obfuscation method to "encrypt" the password as it was being sent over the wire to Yahoo's servers. Back in 2004 when we were preparing our 0.79 release, Cerulean Studios, the creators of the Trillian client, were kind enough to implement this authentication mechanism for us. As part of this change, we began to identify to the Yahoo servers as Yahoo Messenger 6. This was all fine and well.

The real problem came relatively recently. At some point in our recent history, which I honestly don't feel like tracking down now, we started identifying as Yahoo Messenger 7. At some point after that we again updated to identify as Yahoo Messenger 8. Both of these changes were related to file transfer and other feature enhancements. When we made these changes, however, we never updated our authentication code. This means we were claiming to speak Yahoo Protocol version 15 but we authenticated the same way protocol version 13 clients do. Even this worked for quite some time.

Yahoo began upgrading their servers at some point recently to phase out the old Yahoo Messenger 6 client. In effect, they want to force users of older software to update to the current client. Whether this is a good thing or not depends on your perspective. From Yahoo's point of view, I'm sure this is an excellent move, as it should make their server software simpler (speak one less protocol, one less auth scheme, etc.) and reduce their support load (fewer client versions to deal with). Where it became a problem for us is that at the same time, they started requiring protocol version 15 clients to speak the version 15 authentication scheme, which we never implemented. Since we still spoke version 13's authentication, this cut us off entirely.

[Note: Some users have discovered that not all of the Yahoo Messenger servers are rejecting the old authentication mechanism from Pidgin. The number of servers that still accept it appears to be shrinking, however. We advise upgrading to Pidgin 2.5.7 as soon as possible.]

Solving the Problem

A few months ago, a kind soul pointed out to us documentation he had assembled on how the current Yahoo Messenger client authenticated and communicated with the servers. It turns out that the convoluted authentication mechanism we had used for years was replaced with a significantly simpler one that uses a few simple HTTPS requests. The beauty here is the login data is encrypted in the same manner as your communications are encrypted whenever you purchase merchandise at a retailer like Amazon. This makes it significantly simpler for us, and probably allows Yahoo to simplify things on their end as well while increasing security during the login process for all their users.

As I mentioned in my previous post, two of our Summer of Code students, Sulabh Mahajan and Mike Ruprecht (keep an eye out for these names--they're responsible for chunks of Pidgin 2.6.0's changelog!), implemented this new authentication scheme. We eventually merged it into what will become Pidgin 2.6.0. This code has had some quality testing already, thanks to the guys over at Adium.

In response to the Yahoo server changes, I (quickly) yanked the most important changes for this new authentication scheme and slapped them onto Pidgin 2.5.6. After some testing and some other important bug fixes, we kicked Pidgin 2.5.7 out the door. In theory, upgrading to Pidgin 2.5.7 should make most people's Yahoo problems go away.

But I already upgraded! It still doesn't work!

Yeah, we have had a few people with this problem. Here are a few suggestions:

• Change your Yahoo account's Pager Server field (located on the Advanced tab when editing the account) to 'scsa.msg.yahoo.com', especially if you previously read this document
• If you're crashing when you connect your Yahoo account, again, set your Pager Server to 'scsa.msg.yahoo.com' and try to log in again. If you can't edit your account, you have two options:
• Run Pidgin from a shell (command prompt). 'pidgin -n' will cause Pidgin to start up in an Offline status, thus allowing you to edit your accounts. Set the Pager Server as described above.
  
• If you know the name of the server that you entered in the Pager Server field, you can change this manually. Locate your .purple directory (covered in the FAQ!). In this directory is a file called accounts.xml. Open this file with an editor (Wordpad on Windows; any text editor will do on Linux and other Unix-like systems) and find the server name you entered. Change this value and save the file. Start Pidgin and you should be fine.
  
• If you upgraded by compiling Pidgin yourself, make sure you removed your distribution's existing libpurple package (for Ubuntu and Debian users this is libpurple0; for RedHat/Fedora users it should be libpurple). Then run 'sudo ldconfig' (or 'ldconfig' if you have a root shell already) and try again.
• If you're behind a proxy server that proxies HTTP connections but not HTTPS connections, and you have to explicitly configure the proxy in Pidgin, you're probably not going to have any luck. Our current proxy code can't handle this kind of configuration. Sorry.
• If you're on Windows and you get errors about a corrupted installer, clear your browser's cache, then restart your browser and try to download again.

But I can't upgrade!

If you really can't upgrade, then try setting the pager server to either 'scsc.msg.yahoo.com' or 'cs101.msg.mud.yahoo.com' and try again. But you really, really, really, really should upgrade. If you do eventually upgrade, you'll have to change this again, as I describe above.

I upgraded but now ______________ doesn't work!

We've had some reports that after upgrading there are some issues such as missing buddy icons and buddies never being shown as signed off. We are looking into these issues and hope to have them fixed for Pidgin 2.6.0. These issues didn't show up in my testing before the 2.5.7 release.

Tracking Yahoo Problems

Please don't open any new tickets about these issues or about not being able to log in. There are already open tickets to track them. Please also don't comment with "Me too!" or similar on these tickets--it just creates more noise for us to sort through when we're working on Pidgin. If you want to express a "Me too" on one of the tickets, click the arrow near the top of the ticket page that's pointing up. This will cast a vote for the ticket.

I hope this information is helpful to people.

The Great Yahoo Debacle

By now it's no great secret that Yahoo has decided to break clients that use outdated authentication code. The result of that change is that Pidgin 2.5.6 and older no longer work unless you connect to one of a very few servers left that still speak the old authentication mechanism. We knew this change was upcoming, but all indications we saw pointed to us having almost another two months to push a release with updated authentication code.

All that aside, we've had some updated authentication code for a while now, and were originally waiting to include it in the Pidgin 2.6.0 release. However, since Yahoo made the authentication changes earlier than we expected, we had to react as quickly as possible to stop the flow of incoming complaints. As a result, we backported the Yahoo authentication changes so we could cut a Pidgin 2.5.7 release fairly quickly.

Pidgin 2.5.7 is now released and does work to connect to Yahoo.

Just for the record, I'd also like to point out to those who complained about waiting three days to get a working release that the last time Yahoo screwed with authentication, it took over a week to even get code to fix the problem, let alone prepare a release and actually get it out to users.

Additionally, we fixed the annoying MSN disconnect-on-block issue and changed AIM's behavior such that the "could not retrieve your buddy list" error appears only once per AIM account.

Enjoy the new release!

Random Monotone Hacks

I imagine most people who bother to read this blog know that I am rather fond of Monotone, the distributed version control system (a.k.a. DVCS) that we use to maintain Pidgin's code. One of the most convenient features of monotone is that it's extensible via the Lua scripting language. Over our time using monotone, some of us have come across (or written!) some useful snippets of lua code that can be tossed into ~/.monotone/monotonerc to make things easier on us.

The first such snippet is pretty simple. I have three monotone keys that I use for various purposes. One is for Pidgin development, another is for my work on the Guifications project and the Purple Plugin Pack. The third is for my private projects. Instead of needing to remember to specify which key I want to use in specific circumstances, a small snippet of lua can take care of this for me (of course, anyone wanting to use this will need to edit branch patterns and key names appropriately):

function get_branch_key (branch)
    d = { ["im.pidgin"]="rekkanoryo@pidgin.im",
          ["org.guifications"]="rekkanoryo@guifications.org",
          ["org.rekkanoryo"]="rekkanoryo@rekkanoryo.org"
    }

    for k, v in pairs(d) do
        if string.find(branch, k) then
            return v
        end
    end
end

Monotone also has the ability to cherry-pick revisions from one branch to transplant into another. (Of course, I recognize that other DVCS tools have this capability too.) To do this, ordinarily we would issue a command like mtn pluck some_revision_id within a working copy of the branch we want to transplant the revision to. In our experience, the default log messages for a pluck leave some to be desired. My fellow developer Sadrul wrote this cool bit to add a pluck-log command to Monotone.

The pluck-log command takes a series of individual revision ID's as arguments. For each revision ID in the list of arguments, the command will execute mtn pluck $REV to grab and apply the changes to the workspace. Additionally, the plucked revision's original changelog entry will be inserted into the changelog for the new commit on the current branch. This is particularly useful if you're creating a new release branch by branching from a previous tag, then grabbing individual revisions from your main development branch. This command became popular among us Pidgin developers while we were preparing the Pidgin 2.5.6 release.

Here's the code:

--[[
  Pluck a revision with the log and authors filled in for the next commit log.

  @author: Sadrul Habib Chowdhury (updated for mtn 0.43 by John Bailey)
--]]

-- pluck-log command
function pluck_log(...)
  local revs = {...}
  local result
  local topsrcdir
  local logfile
  local log

  -- mtn_automate() returns a pair of a boolean and a string; we don't really
  -- care about the boolean here, but we need to do something with it.
  result, topsrcdir = mtn_automate("get_workspace_root")
  topsrcdir = string.gsub(topsrcdir, "\n", "")
  logfile = io.open(topsrcdir .. "/_MTN/log", "r")
  log = ""

  if logfile then
      log = logfile:read("*all")
      logfile:close()
  end

  table.foreach(revs,
      function (index, rev)
          r, sel = mtn_automate("select", rev)

          if r == false then return end

          for rev in sel:gmatch("%S+") do
              r, certs = mtn_automate("certs", rev)

              certs:gsub("%s+key \"(.-)\"\n%s*signature \"(.-)\"\n%s*name \"(.-)\"\n%s*value \"(.-)\"\n%s*trust \"(.-)\"",
                      function(key, sig, name, value, trust)
                          if name == "changelog" then
                              log = log .. "*** Plucked rev " .. rev .. " (" .. key .. "):\n" .. value .. "\n"
                          end
                      end
                  )
              execute("mtn", "pluck", "-r", rev)
          end
      end
  )

  logfile = io.open(topsrcdir .. "/_MTN/log", "w")
  logfile:write(log)
  logfile:close()
end

register_command("pluck-log", "REVISION1 [REVISION2 [...]]", "Pluck a revision with a good log",
      "This plucks a list of revisions, each individually, and adds the changelog of each revision for the next commit log." ..
      "\nEXAMPLE:\tmtn pluck-log h:im.pidgin.pidgin deadbeef\n",
      "pluck_log")

The resulting log entry will look something like this:

*** Plucked rev 074c5aedf9bbc512331f0d3130f076190b290676 (rekkanoryo@pidgin.im):
Set the default pager host to scsa.msg.yahoo.com; this seems to be what the
official client uses.

Originally, Sadrul wrote this code for mtn 0.42 and earlier, which did not have the mtn automate get_workspace_root functionality. I updated the code to call mtn_automate("get_workspace_root") instead of finding the workspace's root with successive checks of parent directories. The revision ID printed in the log was also truncated to the first 8 digits. Ordinarily, this is fine; however, it's possible that in the future a new revision will be created that has the same first 8 digits and thus the short ID will be ambiguous. I wanted to avoid this situation, so I removed the truncation. I also rearranged a few things to make it easier for me to read. I also have to credit my fellow Pidgin developer Elliott with figuring out that I needed to kill the trailing newline in the string returned from mtn_automate("get_workspace_root").

Hopefully we're not the only ones who find this stuff useful. As I make frequent use of other useful monotone lua hacks, I'll probably post about them here too.

Pidgin 2.5.6

Well, as most people have noticed by now, we recently released Pidgin 2.5.6. This release was simply a bug and security fix release. Hopefully it will hold everyone over until we can kick 2.6.0 out the door!

Linux Journal Readers' Choice Award

Yesterday, Linux Journal issued a press release announcing the winners of its annual Readers' Choice Awards. It's an annual event that attracts a lot of attention in the magazine. In their effors to "take the pulse of the Linux community," they run this poll and announce the winners and grant honorable mentions to "strong contenders" in the pool of runners-up.

This year, it's been announced that Pidgin has won a Readers' Choice award for Favorite Communications Tool, a category that we're no stranger to winning--we won this same category last year as well. The landscape is similar as well. This year, we won with 42% of the votes, beating out honorable mention Skype, which received 18% of the votes. Last year, we also garnered 42% of the votes, with Skype and Kopete earning honorable mentions at 17.8% and 12.8% respectively.

Winning this award for the second year in a row reminds us that our work is quite well appreciated, something that is often easy to forget. It also reminds us that every once in a while we need to stand back and thank everyone involved in making Pidgin such a popular project. In this case, we need to thank the 2,000+ people who voted for Pidgin in these Readers' Choice Awards, as well as all our developers, Crazy Patch Writers, drive-by contributors who spot a simple bug and fix it, and our users. All of these people in our community come together to make Pidgin a success as a project, and while we'd be just as happy to work on Pidgin if we had only a few hundred users, we certainly appreciate all of the contributions to our success. Thanks!

And the Summer of Code is set!

Well, Google finally announced the list of accepted students for this year's Summer of Code. You can read about our accepted proposals here. Of course, we're all very excited about the projects we've chosen for this year. We have two students returning to us this year, both with projects that will be hugely important moving forward. Those students are Eric Polino and Sulabh Mahajan.

Eric has chosen to work on GObjectification, which will move libpurple from using our own object system to taking advantage of the GObject capabilities of GLib, one of our existing dependencies. As a result of this project, a massive amount of libpurple infrastructure will change, allowing us to leverage a tried and true typing system and giving plugins an incredible amount of increased flexibility by allowing them to subclass objects and providing interfaces they can implement. This will, of course, have huge rammifications for everyone using libpurple--us, plugin authors, Adium's authors, and even the Instantbird authors. We're confident the result will be worth the effort, though.

Sulabh has volunteered to take on the Privacy Rewrite. What makes this such a daunting task is that the protocols we support provide a very wide range of privacy options, some of which are unique. For example, ICQ supports an "invisible" list and a "visible" list in addition to a "block" list and probably a number of other things that I'm forgetting. Sulabh will have to look at what all our protocols support and provide a foundation in libpurple for all the protocol plugins to provide the privacy options their protocol allows for. Additionally, he'll have to smooth over the differences between protocols to make things consistent.

In addition to these projects, we've accepted two proposals for native Windows user interfaces. Each project will take a different approach to achieve the same goal--providing a user interface that fits better into a Windows desktop than Pidgin currently does. I've already seen a few complaints that we accepted two projects for this. Quite frankly, I don't understand the complaints. We chose two projects for two reasons--first, we want to have the highest possible chance to better serve our Windows users, and second, it's entirely possible that a single Windows UI project won't be able to do enough to satisfy all our users. Having two Windows UIs will allow each to focus on whichever set of users it is more appropriate for.

We've also accepted projects that will make XMPP transports based on libpurple, allow libpurple to use telepathy protocol support in addition to or instead of our own, and using webkit for our message displays. Each of these projects is going to be challenging in its own right. Of all the projects, the XMPP transports will certainly be the least user-visible, as the transports will run on an XMPP server. Conversely, the most visible will probably be the webkit message view, as this will completely replace what we currently use to display messages. By using webkit, it's possible that we'll eventually be able to support Adium Message Styles (no, I'm not promising this support will happen, but it will be possible). The Telepathy plugin will provide some interesting functionality by allowing us to use nothing but telepathy for protocol support if the user so chooses.

There are other deeper details in these projects, but I've already wasted too much time obsessing over this post and discussing the larger projects in detail. I'll leave further discussion of the technical merits and details of projects to the students and mentors. Needless to say, this post could have gone on just about forever. ;)

Overall, this Summer of Code looks like it will be one of our best. Good luck to all our students!

It's that time again...

Well, once again we've come to the student application phase of Google's Summer of Code program. We're several days into the process, and we've seen a marked decrease in our intake of student applications compared to prior years. Why is anybody's guess, but I'm personally hoping it means that potential applicants are spending more time polishing their applications before submitting.

Based on the applications we've seen so far, I'd like to publicize a few notes for applicants:

• Provide a detailed timeline for your project. "I expect to complete work in 10 weeks" doesn't cut it here. Tell us what milestones you expect to achieve and how far along. You could instead estimate how long a specific goal will take to implement. Yes, your estimate could ultimately prove wrong, but that's not necessarily the end of the world. We really want to see that you have a concept of time management and prioritization.
  
• Don't plagiarize. If we know you're plagiarizing, we'll invalidate your application. If you can't deliver an application without plagiarizing, you shouldn't be applying. We don't mind if you use some of the text on our ideas page to help with the abstract, but keep in mind that a serious application won't rely on simply copying and reformatting what we've already said.
  
• Don't be afraid to come up with a unique idea! The ideas page are not the only ideas we'll entertain. We love to see well thought-out, original ideas, especially ones that make us wonder why no one proposed it before.
• Make sure your proposal is for a specific project idea. A general "I want to work with Pidgin for the summer" is a sure ticket to the reject bin.
• You don't have to write an encyclopedia for the application, but you do have to give us something to work with. There's an old adage that "less is more." Sometimes that's true. Sometimes the opposite is true. The point here is that you should be verbose enough to explain your idea, but don't ramble. I know this can be difficult to judge. Just re-read the application, make sure it makes sense, and make sure it doesn't drag on needlessly.
• You have to apply using the SoC webapp, not our mailing lists. We have to have applications submitted via the SoC webapp in order to review them and have them in the running for our student slots. This is a Google thing, and it makes everyone's lives a lot easier.
• The application isn't final until submissions close. We will provide feedback on your application if it needs work. You can modify your application to address our feedback until applications close. Use this to your advantage!
• We don't yet know who will mentor any given project. We don't assign final mentors and backup mentors to projects until we have decided which projects we want to accept. Asking us about this is just wasting your time.
• We expect you to treat the project as a full-time job. That means at least 35 hours per week. We want to make sure everyone gets the most out of Google's money, and this is one way to do that. Since you'll technically be a contract worker, we expect the contracted work (your project) to take priority.
  
• We expect you to remain actively involved with us after the Summer of Code finishes. Quite frankly, if you're going to collect the checks and run, you're missing the entire point. The Summer of Code is intended to attract new contributors and get them involved in open source software, turning them into long-term contributors. Participating in the Summer of Code and then disappearing isn't serving that intention, and leaving us with code we have to maintain ourselves is quite rude. If that's your intention, please don't apply. Let the potential for acceptance go to someone who will stay with us.
  

Keep these in mind, and happy applying!