GSoC Timeline in KOrganizer

25 04 2011

Google just announced the list of accepted summer of code projects for this year. Together with Torgny Nyblom, I’ll be mentoring Sudhendu Roy in his quest to bring support for HTML replies and forwards to KMail. Looking forward to this project!

If you are a student or a mentor, you need to know the timeline of this year’s summer of code. Adding it as a calendar to KOrganizer (here for KDEPIM 4.4) is very easy:

  1. Right-click in the calendar area to add a new calendar
  2. Choose “Calendar in Remote File
  3. Enter the correct URL and give it a nice name
  4. Done, the timeline is displayed in your calendar




Facebook support in KDEPIM

27 02 2011

I am at the  annual KDEPIM meeting in Osnabrück for the weekend. It is awesome meeting old friends again and seeing new faces as well. Even better is that it is a welcome break from writing my thesis which is due next week, one weekend of hacking in discussing without worries.

I took the opportunity to hack on Facebook support for KDEPIM. The result is an Akonadi resource that provides access to your Facebook friends and events.
Screenshots say more than thousand words, so below are two screenshots, showing my Facebook friends in KAddressbook and showing my Facebook events in KOrganizer. Notice how birthdays from Facebook are automatically integrated (although birthdays without the year display the wrong age at the moment).
Sorry for the ugly widget style in the screenshots, for some reason the Oxygen style doesn’t work for me.

Facebook in KAddressbook

Facebook in KAddressbook

Facebook in KOrganizer

Facebook in KOrganizer

That is not all! Since events also show in the Plasma clock calendar, the events from Facebook are automatically shown there as well. The Kontact summary will of course show upcoming birthdays and events from Facebook. The data is also cached when you are offline, thanks to the Akonadi framework. Much more is planned for the future:

  • Show the Facebook news feed in the Kontact summary and in a plasma widget
  • Show Facebook notes in KJots
  • Support for Facebook messages, either in KMail or in a custom Kontact plugin
  • Support for creating and editing events

The Facebook resource is still very young, thus there is no official release yet. Work is underway for a 0.1 release, just a bit of polishing is needed.
Please try out the resource for yourself and help with testing. Since it is a developer preview, you need to build the resource from source code. Do the following:

  1. Make sure to have an up-to-date kdepimlibs from Git master, the resource needs new API that was added this weekend
  2. Get the source code with git clone git://anongit.kde.org/akonadi-facebook
  3. Build and install
  4. Open KAddressbook and chose File->New->Add Address Book.
    1. Select Facebook. A settings dialog will open.
    2. In the settings dialog, click the Authenticate button
    3. A webpage will open and ask you to give the KDE Connector access to the Facebook data. Confirm that. You may need to login to Facebook first.
    4. Close the settings dialog. The address book and calendar should automatically synchronize with Facebook. If not, you can always do that manually by  right-clicking on the Facebook address book or calendar folder.
  5. Enjoy!

Please contact me in case you have any problems or feedback. Bugs are best reported on the KDE bugtracker, there is a Facebook component for the Akonadi product.

For version 0.1, I plan to move the repository to projects.kde.org and add support for translations. More importantly, 0.1 will be the first real release, which will hopefully be picked up by packagers, so stay tuned!

Oh, and since we’re talking about Facebook, there is also a page for the Akonadi resource here, click the Like button if you like it :)

Writing the resource was easy and lots of fun, and hopefully many of you will find it useful. I am certainly looking forward adding more cool features to it.

Update: The Facebook resource should now compile against KDE SC 4.6.0. Please comment or file a bug report if you still have a problem compiling.

Update: Some people report having trouble authenticating with Facebook, sometimes the links in the embedded web view don’t seem to work. I would appreciate help from a KDE webkit developer here. To workaround that problem, first log in to Facebook with Konqueror. You may also need to grab the authentication URL from the debug output of the resource and view that URL in Konqueror, you should see something like “https://graph.facebook.com/oauth/authorize?…” in the debug output. Update: The bug in the authentication was caused by a bug in kdelibs master, which has already been fixed.





Akademy 2010

1 07 2010

I'm going to Akademy 2010

I’m going to this year’s Akademy in Tampere, Finland! This is my third Akademy now, and I think it will be the best! Looking forward to meet everyone there again, and also looking forward to escape the heat here for a week.

I’m now preparing the last bits and pieces for my presentation, which I’ll hold on Saturday. I’ll present various tips, tricks and tools which are useful for KDE development, such as:

  • Advanced debugging with conditional breakpoints and reverse debugging
  • The valgrind suite for memory error detection, performance profiling, heap usage profiling and thread checking
  • Various smallish Git tips
  • No evil monkeys
  • More misc stuff

So much stuff to present in so little time! Come and attend my talk, you’ll learn something useful.





Akonadi Meeting and the KDE SC 4.5 release

14 05 2010

The meeting

We are at the Akonadi meeting at the KDAB offices in Berlin right now, which was quite nice so far. We had the first round of API review of new methods in KDEPIMLIBS for 4.5, and already cleaned up quite a bit. Having multiple eyes look at the API is a nice way to improve the overall quality of the API. We met with Andrey Moiseenko and Alvaro Manera of Nokia, who work on calendaring for the next Meego phone from Nokia. They use our KCal library, which they have forked/extended for some special requirements they have. We’re now making plans with them to integrate their changes back to our version of KCal, so that both sides will profit from changes and have a single point of maintenance.

The release

Many of you are interested in the state of KMail 2, which is the port of KMail to the Akonadi PIM framework. We haven’t communicated much about that lately, as most of us aren’t keen bloggers¹. Finally, here is an update for you now. Yesterday, we had an intensive discussion about the state of KMail 2 and our release plans for it. While we made some good progress on it in the last few weeks, we came to the conclusion that KMail is not yet in a state that is suitable for a beta release. With the KDE SC 4.5 beta 1 release approaching fast, we had to go for some plan B. KDEPIM will not be included in KDE SC 4.5 beta 1. Instead, we will concentrate on making KMail more stable and usable, and release the first beta version of KDEPIM one month later, at the time when KDE SC 4.5 RC1 will get released. That gives us some time to finish important bits and bring KMail 2 to a state in which it can be used.
This means that there will be no fresh KDEPIM packages in KDE SC 4.5.0, and distributions should ship the KDE 4.4.x version of KDEPIM instead. The final KDEPIM release will then follow a bit after the SC 4.5.0 release, at the same time as KDE SC 4.5.1.
KDEPIMLIBS is not affected, it will be released together with KDE SC 4.5.0 as planned. The rest of KDEPIM progressed very nicely: Tobias fixed a lot of problems in KAddressbook and brought back some missed features. The Akonadi port of KOrganizer is also working quite well. However, releasing KDEPIM without KMail is not an option, as it is a quite important part of the PIM suite. We briefly pondered replacing KMail with an older version for the release, but that would involve ripping apart and moving around the dependencies, which is a lot of work and has a high risk of regressions. I think delaying the release is the best option here, as it gives us some more time for stabilizing without too many downsides.
If you’re a KDE developer, please try out the beta version of KDEPIM when it gets released and help us making it work for your usecases. I hope the revised release schedule will work out, keep your fingers crossed :)
So basically the release of KDEPIM will happen a bit after the KDE SC 4.5.0 release. We’re not sure about the release name yet, maybe we should call it “Preview” or something like that, just to prevent the endless discussion and flames about naming that happened with the KDE SC 4.0 release.

Features

KMail 2 is a major rewrite of KMail: We throw out the ancient brittle storage layer and replaced it with Akonadi. On top of that, we ported it from the mimelib library to KMime. Finally, we factored out much of KMail into separate libraries, such as Akonadi itself, the messageviewer component, the messagecomposer library and more. That allows us to re-use those components in other applications, such as the mobile version of Kontact, and makes it possible to unit-test the components, which was not possible before, as everything was tightly glued together in KMail.
Such a major rewrite is not without consequences: On the one hand, it will finally modernize the complete architecture and make it maintainable. On the other hand, such rewrites always introduce regressions and remove features. This is unavoidable, and it is always a compromise between releasing and between stabilizing. Software would never get released if the requirement would be to make it regression-free and feature-complete.
What does that mean for KMail 2? With the first release, some features will be missing:

  • There will be no on-server filtering for POP3. I guess most people weren’t using this anyway, as it was actually quite hard to configure this correctly in KMail 1. Normal client-side filtering will still work, which is what everyone uses.
  • There will be no attachment loading on demand for IMAP. In KMail 1, with online IMAP, it was possible to activate loading attachments on demand, which means attachment will not be downloaded when viewing a message, but only when the attachment is actually saved to disk. This had some problems in KMail 1, like attachments having a size of 0 bytes, but it also sometimes worked. While the Akonadi architecture supports retrieving only parts of items, we don’t use that yet in the message viewer.
  • The most hated KDE bug will not get fixed. The bug is that KMail’s UI will freeze when the incoming mails are filtered, which is especially annoying for slow filters, like the filters that pipe the mail through spamassassin. That is not a regression from KMail 1, but we had hoped that the bug would be fixed with Akonadi. The idea is to use a filtering system which runs in a separate process in the background, instead of doing the filtering inside of KMail. Such a filtering system was created in a summer of code project last year, but due to lack of time, it is not yet integrated in KDEPIM. That means filtering will still be done on the client-side in KMail, with the consequence of freezing UI. Once we get a release out of the door and have some time, we will integrate the new filtering framework that will finally solve this bug.
  • We’re not sure if we manage to be able to migrate the IMAP cache if you’re using disconnected IMAP. That would mean re-downloading the mail again.
  • Probably IMAP server-side searching will not make it either

As said earlier, KMail is not yet ready for being released as a beta version. One major blockers right now is the lack of data migration, which includes reading the old KMail 1 index files and storing those flags in Akonadi, and also migrating over the account configuration. This is being worked on at the moment by Kevin Krammer, so we’ll have data migration before the first KDEPIM beta. The other blocker is the lack of client-side filtering, which Volker Krause is working on. As said above, that will at first be almost the same filtering system as we had in KMail 1, which allows us to have a working client-side filtering system for the first KMail 2 release. After those two blockers are fixed, which is before the first KDEPIM beta release, KMail 2 should be usable for other developers and testers, and we can concentrate on making KMail 2 usable for daily usage.

So the major changes in KMail 2 happen in the background, mainly not visible to the user, as the user interface largely stays the same. We have talked about the advantages of the Akonadi framework in other places, so I won’t repeat that here in detail. Overall, the porting to Akonadi is an important step in long-term stability and maintainability, and also fixes the old architectural issues we had with our 10-year-old system. I’m looking forward to the release of KMail 2 and of the ported version of KOrganizer and Kontact, which finally gives as a new architecture that meets modern requirements.

¹: No wonder I’m not blogging much, writing this blog took a long time :)





Nepomuk in KMail 2

18 01 2010

As you might now, we are currently porting KMail to Akonadi. The Akonadi-based KMail will be called KMail 2 and released together with KDE SC 4.5 if everything goes well.

Just a quick summary of what Akonadi is, for those who don’t know: Akonadi is an abstraction layer/proxy and a cache for PIM data.
PIM data can by anything like mails, contacts or calendar entries, and they can come from different sources, like an IMAP server, a
local vCard file or an Exchange server. Akonadi provides an easy API for the client application developer to access that PIM data
in a transparent way.

This post is about Nepomuk, not Akonadi. Akonadi uses Nepomuk to index the mails. Basically this means every mail that was seen by the Akonadi cache is also indexed by Nepomuk. This makes some great features possible, which I’ll describe below. With screenshots!

Powerful searches

The main benefit of Nepomuk is that we can do very powerful searches. KMail 1 had a search function as well, and even virtual folders. Those however were so slow that they were basically unusable.
With Nepomuk-powered searches, this is different: The searches are quite fast, and because they run in another process, the KMail GUI is not slowed down by searches anymore. Now it is for example possible to have a virtual folder with all messages marked as to-do or important.
Speed is not the only thing that is improved: The search results are now much, much better. We can now search in attachments and also optionally in encrypted mails. We can even do powerful semantic searches, for example linking mails with contacts. One example is “search all mails sent by persons who are tagged with ‘boss’”, since you can now tag contacts in KAddressbook.
Because Nepomuk indexes the mails, they will now show up in KRunner when you type something there. The amount of mails it finds for a given search term is so powerful that it is downright scary: when typing your name, it finds mails sent by you, sent to you, mentioning you or even mails in which you are mentioned in some attachment.
To conclude, searches in KMail 2 will rock!

Tagging

In KMail 2, we now use Nepomuk for all our tagging needs, instead of relying on some home-brewn system. The big advantage of course is integration with the rest of KDE, for example an application showing you every thing that is tagged with a certain tag can now also show the tagged mails. And KRunner will also display mails when you search for a tag.
We also have a Nepomuk tag resource, which shows one folder for each tag, with the tagged mails inside. When you drag a mail into such a folder, it will even get tagged automatically. Again this is powered by Nepomuk searches behind the scenes.

Below two screenshots showing how to tag a mail, and how the tag folders look. Please ignore the horrible colors, I was just testing the function that tags can change the text and background color.

Annotations

Having annotations for mails is a much requested feature. Often, people want to attach a little note to a mail where they can put some text in. Now, which technology could we use for annotations? Right, Nepomuk! I implemented annotation support today, and I have to say it was pleasantly easy to do so.
In KMail 2, you can now add a note to a mail. The mail list shows a little icon when a mail has a note attached, and you can edit the note by clicking it, or by using the appropriate action from the main menu.

Below are some screenshots which shows how to use annotations.

Did you notice the tooltip in the last screenshot? The note you have added to the mail also shows up in the tooltip. Even better: The tooltip shows a short preview of the mail text now, for mails that are indexed by Nepomuk. It was again really simply to get that information out of Nepomuk.

Closing

I hope I could give you a glimpse at the nice improvements related to Nepomuk which will be available in KMail 2.

So when can you use that cool stuff?
KMail 2 is currently under heavy development, right now it should not be used at all :) In the past, we have been working in a separate branch, the akonadi-ports branch. That branch will now be merged back to trunk during the next days. If you are a user who follows trunk, you should switch to the 4.4 branch. Otherwise you’ll see a big fat warning when starting KMail 2.

Right now there are quite many bugs and regressions in KMail, the Akonadi port is probably comparable with the KDE3 to KDE4 port, if not worse. We hope to stabilize KMail 2 so that it becomes usable soon, and if everything goes well, it can be released together with KDE SC 4.5.
Helping hands for the work on KMail 2 are very welcome, there is a huge amount of stuff to fix where you could help us. Find us in #kontact or #akonadi on IRC, or on the kde-pim mailing list.





Akonadi, Nepomuk and Strigi explained

3 10 2009

Tobias recently blogged about Nepomuk, and from the comments it seems that people are a bit in the dark about what Akonadi, Nepomuk and Strigi actually do and how they interact with each other. So if you want to understand those technologies, read on! This blog post is an attempt to clear things up a bit.

Soprano

Let me start with Soprano. Soprano is a Qt library for accessing semantic storage (RDF). In many ways, Soprano can be compared to the QtSql module, the key difference is that QtSql accesses relational data with SQL as the query language, whereas Soprano accesses semantic data with SPARQL as the query language.

Semantic Data

But what is semantic data, and what is the difference to relational data? You probably all know relational databases, which use tables to store data. Semantic databases, on the other hand, use statements, also sometime referred to as sentences, to store data. Statements consist, just like real-world sentences, of a subject (noun), a predicate (verb) and an object. By storing many sentences in a database, one can create a big network of data. The best way to make this clear is probably by examples, so here are some:

.

  • “image.jpg” “has the width” “800 pixels”
  • “image.jpg” “is tagged with” “example-tag”
  • “image.jpg” “was photographed by” “Max Mustermann”
  • “example-tag” “has the title” “Holidays”
  • “example-tag” “has the icon” “beach.svg”
  • “Max Mustermann” “lives in” “London”
  • “Max Mustermann” “was born on” “01.01.1970″

As you can see in those examples, it is possible to link together totally different topics, such as information about a file, a tag and a person. Those examples are of course a bit fabricated, real data would look different. However, you can see the basic sentence structure consisting of subjects, predicates and objects. One can add an arbitrary number of statements about things to a semantic database. The powerful aspect about semantic data is the way it is linked together. With the above statements, you could for example search for all images tagged with “example-tag”. You could also do much more interesting searches, like searching for all pictures taken by people living in London, or searching for all files that were sent as e-mail attachments from your boss.

Read the RDF Primer from the W3C for an in-depth introduction about RDF and semantic data (RDF is one way to describe semantic data, but not the only way).

Backends

Like QtSql, which supports different backends such as SQLite or MySQL, Soprano also supports different backends. Currently, there are three backends for Soprano: Redland, Sesame2 and Virtuoso. Redland is C++ based and orders of magnitudes too slow for what we need in Akonadi, so basically Redland should never be used at all. Sesame2 is Java based and performs well. This currently seems to be the only usable backend for Soprano. Read Tobias’ blog entry for the details about this. The third backend is Virtuoso, which combines the strength of the other two backends: It is C++ based and performs well. To my knowledge, this backend is currently in development and therefore not usable, but it will certainly be an interesting choice in the future.

Nepomuk

Ok, now you should have a basic understanding about what semantic data is and what Soprano is. Read on to find out what Nepomuk is!

Nepomuk is the KDE library for accessing semantic data. It uses Soprano for storage access. Nepomuk provides a KDE API for many high-level functions such as tagging and annotating. An important point is that Nepomuk also provides a set of standard ontologies, and convenience classes to use them.

Let me explain what an ontology is. Although you can store arbitrary statements in a semantic database, that rarely is useful. Consider the case that you store the sentence “Laura lives in Leeds” and the sentence “Ralf resides in Leeds”. Notice that the predicate, i.e. the verb, is different in those sentences, once it is “resides in”, and once it is “lives in”. Now, if you attempt to do a semantic search for all people living in Leeds, you will not find Ralf, since the statement about him uses a different predicate. Therefore it is a good idea to have a set of standard predicates and other terms, to have a clearly defined vocabulary about things. This is what ontologies are. Nepomuk comes with a set of standard ontologies which define vocabulary for talking about annotations, files, contacts, mails, calendars, music and more. These ontologies are now also a freedesktop standard, GNOME’s Tracker uses them as well.

Now, Nepomuk would be useless without any data. There are basically two ways of getting data into Nepomuk: One way is by manual user action, for example when a user tags a file in Dolphin. The other way is automatic indexing, which is done by both Strigi and Akonadi. Read the next sections for details.

Strigi

Strigi is the file indexer for KDE. It looks at every file on your hard disk, extracts semantic data out of the file, and then feeds the data into Nepomuk. When saying that Nepomuk uses a lot of CPU or IO, that is usually because Strigi is indexing files in the background. There are however many settings to improve this, for example indexing is disabled while on battery, and the IO niceness is set to a low level. Also, Strigi indexing can be disabled completely, without disabling any other parts of Nepomuk, because the file indexing is just one way to get data into Nepomuk.

What currently is badly missing, in my opinion, is a good GUI client to actually search for all the data that has been indexed. It seems that there was some very nice progress during this year’s summer of code with that, so I am sure the situation will get a lot better in the future


Akonadi

Now, on to the last technology of this blog post! Akonadi is a framework to access PIM data like mails, contacts and calendar events. Think of it as a cache or a proxy to your PIM data: The real data is still stored in local Maildir folders, local vCard files, IMAP servers or in your Google address book. Akonadi provides an easy API to access that PIM data in an uniform way. Additionally, it can act as a cache, for things like disconnected IMAP or for offline access to your Google address book. Another advantage is that the PIM data can easily be shared between applications. Now not only KMail can access your mail, but also LionMail or Mailody. Additionally, there is no need to have KMail running to access your calendars and contacts on your Kolab server. Akonadi furthermore replaces the brittle system of index files in KMail, and the new Akonadi IMAP resource is already much faster than KMail’s old IMAP code.

So as you can see, Akonadi will bring many advantages to the end user, once the applications are ported to use Akonadi. For KDE 4.4, only the new KAddressbook and KPilot will use Akonadi natively. Ports of KOrganizer, Akregator and KMail are in progress and (hopefully) expected to be released with KDE 4.5.

Akonadi of course needs to store information about the PIM items and folders it knows about somewhere. For this, we use a classical SQL database. For now, we support only MySQL, but there is work done on PostgreSQL and SQLite support. Those two database backends are both work in progress, help there would be very welcome.

Now, how is Akonadi related to Nepomuk? Applications which use Akonadi require a fast search and good support for virtual folders. Now, we didn’t want to code our own search support into Akonadi. It is quite a lot of work and difficult to get right. The virtual folders in KMail 1 are for example too slow to be useful for larger volumes of mail. What we did for searching instead was to use a technology that is actually good at finding stuff: Nepomuk.

We use Akonadi agents to feed information about contacts, events and mails into Nepomuk. So just like Strigi, those Akonadi agents will put data into Nepomuk. We use the standard ontologies, like the NMO (Nepomuk Mail Ontology) to store the mails. This data is then used for searches and virtual folders. By using Nepomuk, we hope to overcome many of the KMail 1 shortcomings, like the slow virtual folders mentioned earlier or the inability to search in base64-encoded attachments. It already works quite good, for example we have a working tag resource to show all your mails tagged with specific tags, and searches with SPARQL are also working (although there is no GUI for it, yet, for now you need to use the development tool akonadiconsole to see them).

That’s it, folks. I hope I made some things clearer to you. If anything is unclear, please ask in the comments section.
My next blog post will have screenshots again, I promise :-)





Junior Job Achievements

24 08 2009

I my last post, I talked about Junior Jobs in KMail a bit. Now, I want to write about the progress of those Junior Jobs that was made during the last few weeks.

I think the Junior Jobs program was a success, the wiki page where I listed ideas was almost empty at one point. Some of the developers who started with Junior Jobs later picked something else up by themselves, which is the way it should go. Especially since it is not easy to actually come up with ideas :)

Now, without too much further talking, let me present you the progress, including pretty screenshots. Note that the order here is totally random, and also I probably forgot many things here, so don’t feel left out when you committed something that isn’t listed here.

James Bendig improved the usability of the options of the new message list. There is now an unified way to configure tooltips for the folder list and the message list. Remember all the buttons next to the quick search field that appeared in KDE 4.2? Those buttons were confusing to new users, who often didn’t discover how to change the theme or the aggregation. Also, those buttons cluttered the UI a bit. This is how it looked like:

Quick search line before the changes

This is how it looks now:

Quick search after the changes

As you can see, the buttons have been removed. Vincent Dupont helped with converting the filter to a combo box again. But where are the options to change the theme, aggregation and sort order now? They are now in two places: The View menu, and in the context menu of the header. The global theme and aggregation now can also be changed in the config dialog. Setting a per-folder theme or aggregation is now also easier, it can be done in the folder properties dialog now:

Folder Properties Dialog

Overall the options for the new message list are now much more user-friendly.

Bruno Bigras ported over some long forgotten features from the old kdepim 3.5.5+ branch, for example an improved recipients picker that shows the alternative mail addresses as children of the contact and has more grouping capabilities like grouping per address book category.

New recipient picker

Bruno also added a new filter action that can add people to the addressbook automatically. Also, you can now filter messages in KMail before they are sent.

Torgny Nyblom again converted one hardcoded dialog into an UI file. I remember a year ago or so, there were no UI files in KMail, but now those keep increasing.

Jonathan Armond brought back searching by status and added searching by tag. Tags can now also be added by filters. Switching the identity in the composer now switches the template as well, if you have not modified the message already. For some people, adding the signature at the beginning or the end is not enough, so now the %SIGNATURE command is supported in templates.

Jaime Torres, whom you probably know as a member of the bugsquad, also contributed a couple of bugfixes.

Apart from the Junior Jobs listed above, there were of course more commits in KMail, but I don’t want to talk about those now. One person deserves special mention though: Martin Koller. He is by no way a “junior”, since he was listed in the KMail about dialog long before I was added there.  But recently he started coding for KMail again and fixed a lot of bugs, over 30 I think. He also went through bugzilla and closed a lot of bugs there as well.

A big thank you to all the people who contributed to KMail and help making it better!

Last but not least, there has been much progress in the akonadi-ports branch. Kevin blogged about progress with the ports of the message list and the reader widget, you should read that if you haven’t already. Constantin made good progress with the port of the composer, which does not sound exciting, but it is a very important step. His work will eventually make it easier to implement HTML replies, share the composer library with other applications, make it easier to support native Exchange sending methods instead of SMTP and much more. But there is so much stuff in Akonadi-land that I really should do a separate post about this.








Follow

Get every new post delivered to your Inbox.