Facebook support in KDEPIM

27 02 2011

I am at the  annual KDEPIM meeting in Osnabrück for the weekend. It is awesome meeting old friends again and seeing new faces as well. Even better is that it is a welcome break from writing my thesis which is due next week, one weekend of hacking in discussing without worries.

I took the opportunity to hack on Facebook support for KDEPIM. The result is an Akonadi resource that provides access to your Facebook friends and events.
Screenshots say more than thousand words, so below are two screenshots, showing my Facebook friends in KAddressbook and showing my Facebook events in KOrganizer. Notice how birthdays from Facebook are automatically integrated (although birthdays without the year display the wrong age at the moment).
Sorry for the ugly widget style in the screenshots, for some reason the Oxygen style doesn’t work for me.

Facebook in KAddressbook

Facebook in KAddressbook

Facebook in KOrganizer

Facebook in KOrganizer

That is not all! Since events also show in the Plasma clock calendar, the events from Facebook are automatically shown there as well. The Kontact summary will of course show upcoming birthdays and events from Facebook. The data is also cached when you are offline, thanks to the Akonadi framework. Much more is planned for the future:

  • Show the Facebook news feed in the Kontact summary and in a plasma widget
  • Show Facebook notes in KJots
  • Support for Facebook messages, either in KMail or in a custom Kontact plugin
  • Support for creating and editing events

The Facebook resource is still very young, thus there is no official release yet. Work is underway for a 0.1 release, just a bit of polishing is needed.
Please try out the resource for yourself and help with testing. Since it is a developer preview, you need to build the resource from source code. Do the following:

  1. Make sure to have an up-to-date kdepimlibs from Git master, the resource needs new API that was added this weekend
  2. Get the source code with git clone git://anongit.kde.org/akonadi-facebook
  3. Build and install
  4. Open KAddressbook and chose File->New->Add Address Book.
    1. Select Facebook. A settings dialog will open.
    2. In the settings dialog, click the Authenticate button
    3. A webpage will open and ask you to give the KDE Connector access to the Facebook data. Confirm that. You may need to login to Facebook first.
    4. Close the settings dialog. The address book and calendar should automatically synchronize with Facebook. If not, you can always do that manually by  right-clicking on the Facebook address book or calendar folder.
  5. Enjoy!

Please contact me in case you have any problems or feedback. Bugs are best reported on the KDE bugtracker, there is a Facebook component for the Akonadi product.

For version 0.1, I plan to move the repository to projects.kde.org and add support for translations. More importantly, 0.1 will be the first real release, which will hopefully be picked up by packagers, so stay tuned!

Oh, and since we’re talking about Facebook, there is also a page for the Akonadi resource here, click the Like button if you like it 🙂

Writing the resource was easy and lots of fun, and hopefully many of you will find it useful. I am certainly looking forward adding more cool features to it.

Update: The Facebook resource should now compile against KDE SC 4.6.0. Please comment or file a bug report if you still have a problem compiling.

Update: Some people report having trouble authenticating with Facebook, sometimes the links in the embedded web view don’t seem to work. I would appreciate help from a KDE webkit developer here. To workaround that problem, first log in to Facebook with Konqueror. You may also need to grab the authentication URL from the debug output of the resource and view that URL in Konqueror, you should see something like “https://graph.facebook.com/oauth/authorize?…” in the debug output. Update: The bug in the authentication was caused by a bug in kdelibs master, which has already been fixed.

Advertisements




Akonadi Meeting and the KDE SC 4.5 release

14 05 2010

The meeting

We are at the Akonadi meeting at the KDAB offices in Berlin right now, which was quite nice so far. We had the first round of API review of new methods in KDEPIMLIBS for 4.5, and already cleaned up quite a bit. Having multiple eyes look at the API is a nice way to improve the overall quality of the API. We met with Andrey Moiseenko and Alvaro Manera of Nokia, who work on calendaring for the next Meego phone from Nokia. They use our KCal library, which they have forked/extended for some special requirements they have. We’re now making plans with them to integrate their changes back to our version of KCal, so that both sides will profit from changes and have a single point of maintenance.

The release

Many of you are interested in the state of KMail 2, which is the port of KMail to the Akonadi PIM framework. We haven’t communicated much about that lately, as most of us aren’t keen bloggers¹. Finally, here is an update for you now. Yesterday, we had an intensive discussion about the state of KMail 2 and our release plans for it. While we made some good progress on it in the last few weeks, we came to the conclusion that KMail is not yet in a state that is suitable for a beta release. With the KDE SC 4.5 beta 1 release approaching fast, we had to go for some plan B. KDEPIM will not be included in KDE SC 4.5 beta 1. Instead, we will concentrate on making KMail more stable and usable, and release the first beta version of KDEPIM one month later, at the time when KDE SC 4.5 RC1 will get released. That gives us some time to finish important bits and bring KMail 2 to a state in which it can be used.
This means that there will be no fresh KDEPIM packages in KDE SC 4.5.0, and distributions should ship the KDE 4.4.x version of KDEPIM instead. The final KDEPIM release will then follow a bit after the SC 4.5.0 release, at the same time as KDE SC 4.5.1.
KDEPIMLIBS is not affected, it will be released together with KDE SC 4.5.0 as planned. The rest of KDEPIM progressed very nicely: Tobias fixed a lot of problems in KAddressbook and brought back some missed features. The Akonadi port of KOrganizer is also working quite well. However, releasing KDEPIM without KMail is not an option, as it is a quite important part of the PIM suite. We briefly pondered replacing KMail with an older version for the release, but that would involve ripping apart and moving around the dependencies, which is a lot of work and has a high risk of regressions. I think delaying the release is the best option here, as it gives us some more time for stabilizing without too many downsides.
If you’re a KDE developer, please try out the beta version of KDEPIM when it gets released and help us making it work for your usecases. I hope the revised release schedule will work out, keep your fingers crossed 🙂
So basically the release of KDEPIM will happen a bit after the KDE SC 4.5.0 release. We’re not sure about the release name yet, maybe we should call it “Preview” or something like that, just to prevent the endless discussion and flames about naming that happened with the KDE SC 4.0 release.

Features

KMail 2 is a major rewrite of KMail: We throw out the ancient brittle storage layer and replaced it with Akonadi. On top of that, we ported it from the mimelib library to KMime. Finally, we factored out much of KMail into separate libraries, such as Akonadi itself, the messageviewer component, the messagecomposer library and more. That allows us to re-use those components in other applications, such as the mobile version of Kontact, and makes it possible to unit-test the components, which was not possible before, as everything was tightly glued together in KMail.
Such a major rewrite is not without consequences: On the one hand, it will finally modernize the complete architecture and make it maintainable. On the other hand, such rewrites always introduce regressions and remove features. This is unavoidable, and it is always a compromise between releasing and between stabilizing. Software would never get released if the requirement would be to make it regression-free and feature-complete.
What does that mean for KMail 2? With the first release, some features will be missing:

  • There will be no on-server filtering for POP3. I guess most people weren’t using this anyway, as it was actually quite hard to configure this correctly in KMail 1. Normal client-side filtering will still work, which is what everyone uses.
  • There will be no attachment loading on demand for IMAP. In KMail 1, with online IMAP, it was possible to activate loading attachments on demand, which means attachment will not be downloaded when viewing a message, but only when the attachment is actually saved to disk. This had some problems in KMail 1, like attachments having a size of 0 bytes, but it also sometimes worked. While the Akonadi architecture supports retrieving only parts of items, we don’t use that yet in the message viewer.
  • The most hated KDE bug will not get fixed. The bug is that KMail’s UI will freeze when the incoming mails are filtered, which is especially annoying for slow filters, like the filters that pipe the mail through spamassassin. That is not a regression from KMail 1, but we had hoped that the bug would be fixed with Akonadi. The idea is to use a filtering system which runs in a separate process in the background, instead of doing the filtering inside of KMail. Such a filtering system was created in a summer of code project last year, but due to lack of time, it is not yet integrated in KDEPIM. That means filtering will still be done on the client-side in KMail, with the consequence of freezing UI. Once we get a release out of the door and have some time, we will integrate the new filtering framework that will finally solve this bug.
  • We’re not sure if we manage to be able to migrate the IMAP cache if you’re using disconnected IMAP. That would mean re-downloading the mail again.
  • Probably IMAP server-side searching will not make it either

As said earlier, KMail is not yet ready for being released as a beta version. One major blockers right now is the lack of data migration, which includes reading the old KMail 1 index files and storing those flags in Akonadi, and also migrating over the account configuration. This is being worked on at the moment by Kevin Krammer, so we’ll have data migration before the first KDEPIM beta. The other blocker is the lack of client-side filtering, which Volker Krause is working on. As said above, that will at first be almost the same filtering system as we had in KMail 1, which allows us to have a working client-side filtering system for the first KMail 2 release. After those two blockers are fixed, which is before the first KDEPIM beta release, KMail 2 should be usable for other developers and testers, and we can concentrate on making KMail 2 usable for daily usage.

So the major changes in KMail 2 happen in the background, mainly not visible to the user, as the user interface largely stays the same. We have talked about the advantages of the Akonadi framework in other places, so I won’t repeat that here in detail. Overall, the porting to Akonadi is an important step in long-term stability and maintainability, and also fixes the old architectural issues we had with our 10-year-old system. I’m looking forward to the release of KMail 2 and of the ported version of KOrganizer and Kontact, which finally gives as a new architecture that meets modern requirements.

¹: No wonder I’m not blogging much, writing this blog took a long time 🙂





Akonadi, Nepomuk and Strigi explained

3 10 2009

Tobias recently blogged about Nepomuk, and from the comments it seems that people are a bit in the dark about what Akonadi, Nepomuk and Strigi actually do and how they interact with each other. So if you want to understand those technologies, read on! This blog post is an attempt to clear things up a bit.

Soprano

Let me start with Soprano. Soprano is a Qt library for accessing semantic storage (RDF). In many ways, Soprano can be compared to the QtSql module, the key difference is that QtSql accesses relational data with SQL as the query language, whereas Soprano accesses semantic data with SPARQL as the query language.

Semantic Data

But what is semantic data, and what is the difference to relational data? You probably all know relational databases, which use tables to store data. Semantic databases, on the other hand, use statements, also sometime referred to as sentences, to store data. Statements consist, just like real-world sentences, of a subject (noun), a predicate (verb) and an object. By storing many sentences in a database, one can create a big network of data. The best way to make this clear is probably by examples, so here are some:

.

  • “image.jpg” “has the width” “800 pixels”
  • “image.jpg” “is tagged with” “example-tag”
  • “image.jpg” “was photographed by” “Max Mustermann”
  • “example-tag” “has the title” “Holidays”
  • “example-tag” “has the icon” “beach.svg”
  • “Max Mustermann” “lives in” “London”
  • “Max Mustermann” “was born on” “01.01.1970”

As you can see in those examples, it is possible to link together totally different topics, such as information about a file, a tag and a person. Those examples are of course a bit fabricated, real data would look different. However, you can see the basic sentence structure consisting of subjects, predicates and objects. One can add an arbitrary number of statements about things to a semantic database. The powerful aspect about semantic data is the way it is linked together. With the above statements, you could for example search for all images tagged with “example-tag”. You could also do much more interesting searches, like searching for all pictures taken by people living in London, or searching for all files that were sent as e-mail attachments from your boss.

Read the RDF Primer from the W3C for an in-depth introduction about RDF and semantic data (RDF is one way to describe semantic data, but not the only way).

Backends

Like QtSql, which supports different backends such as SQLite or MySQL, Soprano also supports different backends. Currently, there are three backends for Soprano: Redland, Sesame2 and Virtuoso. Redland is C++ based and orders of magnitudes too slow for what we need in Akonadi, so basically Redland should never be used at all. Sesame2 is Java based and performs well. This currently seems to be the only usable backend for Soprano. Read Tobias’ blog entry for the details about this. The third backend is Virtuoso, which combines the strength of the other two backends: It is C++ based and performs well. To my knowledge, this backend is currently in development and therefore not usable, but it will certainly be an interesting choice in the future.

Nepomuk

Ok, now you should have a basic understanding about what semantic data is and what Soprano is. Read on to find out what Nepomuk is!

Nepomuk is the KDE library for accessing semantic data. It uses Soprano for storage access. Nepomuk provides a KDE API for many high-level functions such as tagging and annotating. An important point is that Nepomuk also provides a set of standard ontologies, and convenience classes to use them.

Let me explain what an ontology is. Although you can store arbitrary statements in a semantic database, that rarely is useful. Consider the case that you store the sentence “Laura lives in Leeds” and the sentence “Ralf resides in Leeds”. Notice that the predicate, i.e. the verb, is different in those sentences, once it is “resides in”, and once it is “lives in”. Now, if you attempt to do a semantic search for all people living in Leeds, you will not find Ralf, since the statement about him uses a different predicate. Therefore it is a good idea to have a set of standard predicates and other terms, to have a clearly defined vocabulary about things. This is what ontologies are. Nepomuk comes with a set of standard ontologies which define vocabulary for talking about annotations, files, contacts, mails, calendars, music and more. These ontologies are now also a freedesktop standard, GNOME’s Tracker uses them as well.

Now, Nepomuk would be useless without any data. There are basically two ways of getting data into Nepomuk: One way is by manual user action, for example when a user tags a file in Dolphin. The other way is automatic indexing, which is done by both Strigi and Akonadi. Read the next sections for details.

Strigi

Strigi is the file indexer for KDE. It looks at every file on your hard disk, extracts semantic data out of the file, and then feeds the data into Nepomuk. When saying that Nepomuk uses a lot of CPU or IO, that is usually because Strigi is indexing files in the background. There are however many settings to improve this, for example indexing is disabled while on battery, and the IO niceness is set to a low level. Also, Strigi indexing can be disabled completely, without disabling any other parts of Nepomuk, because the file indexing is just one way to get data into Nepomuk.

What currently is badly missing, in my opinion, is a good GUI client to actually search for all the data that has been indexed. It seems that there was some very nice progress during this year’s summer of code with that, so I am sure the situation will get a lot better in the future


Akonadi

Now, on to the last technology of this blog post! Akonadi is a framework to access PIM data like mails, contacts and calendar events. Think of it as a cache or a proxy to your PIM data: The real data is still stored in local Maildir folders, local vCard files, IMAP servers or in your Google address book. Akonadi provides an easy API to access that PIM data in an uniform way. Additionally, it can act as a cache, for things like disconnected IMAP or for offline access to your Google address book. Another advantage is that the PIM data can easily be shared between applications. Now not only KMail can access your mail, but also LionMail or Mailody. Additionally, there is no need to have KMail running to access your calendars and contacts on your Kolab server. Akonadi furthermore replaces the brittle system of index files in KMail, and the new Akonadi IMAP resource is already much faster than KMail’s old IMAP code.

So as you can see, Akonadi will bring many advantages to the end user, once the applications are ported to use Akonadi. For KDE 4.4, only the new KAddressbook and KPilot will use Akonadi natively. Ports of KOrganizer, Akregator and KMail are in progress and (hopefully) expected to be released with KDE 4.5.

Akonadi of course needs to store information about the PIM items and folders it knows about somewhere. For this, we use a classical SQL database. For now, we support only MySQL, but there is work done on PostgreSQL and SQLite support. Those two database backends are both work in progress, help there would be very welcome.

Now, how is Akonadi related to Nepomuk? Applications which use Akonadi require a fast search and good support for virtual folders. Now, we didn’t want to code our own search support into Akonadi. It is quite a lot of work and difficult to get right. The virtual folders in KMail 1 are for example too slow to be useful for larger volumes of mail. What we did for searching instead was to use a technology that is actually good at finding stuff: Nepomuk.

We use Akonadi agents to feed information about contacts, events and mails into Nepomuk. So just like Strigi, those Akonadi agents will put data into Nepomuk. We use the standard ontologies, like the NMO (Nepomuk Mail Ontology) to store the mails. This data is then used for searches and virtual folders. By using Nepomuk, we hope to overcome many of the KMail 1 shortcomings, like the slow virtual folders mentioned earlier or the inability to search in base64-encoded attachments. It already works quite good, for example we have a working tag resource to show all your mails tagged with specific tags, and searches with SPARQL are also working (although there is no GUI for it, yet, for now you need to use the development tool akonadiconsole to see them).

That’s it, folks. I hope I made some things clearer to you. If anything is unclear, please ask in the comments section.
My next blog post will have screenshots again, I promise 🙂