Internet lifestream with Django

By Nuno Mariz, on 4 April 2008 @ 01:30
My goal was to archive and display my internet lifestream. My first approach was writing a client for each API of the social networks that I'm in.
This turned out to be a complete waste of time and effort. All that I needed after all was a FriendFeed account that would centralize all my feeds.

Archiving and displaying your entries with Django is quite simple.
First of all, you need to download the Python FriendFeed API client. Then start a new application in your project, lets call it lifestream:

./manage.py startapp lifestream

On the settings.py add the lifestream project to the INSTALLED_APPS and a variable to store your FriendFeed username:

FRIENDFEED_USERNAME = 'your_username'

In the models.py add a model named Entry:

from django.db import models

class Entry(models.Model):
    id = models.CharField(max_length=255, primary_key=True)
    service_id = models.CharField(max_length=50, null=True, blank=True)
    service_name = models.CharField(max_length=50, null=True, blank=True)
    service_icon = models.URLField(max_length=255, verify_exists=False, null=True, blank=True)
    service_profile = models.URLField(max_length=255, verify_exists=False, null=True, blank=True)
    title = models.CharField(max_length=255, null=True, blank=True)
    link = models.URLField(max_length=255, verify_exists=False, null=True, blank=True)
    updated = models.DateTimeField(null=True, blank=True)
    published = models.DateTimeField(null=True, blank=True)
    media_title = models.CharField(max_length=255, null=True, blank=True)
    media_link = models.URLField(max_length=255, verify_exists=False, null=True, blank=True)
    media_thumbnail = models.URLField(max_length=255, verify_exists=False, null=True, blank=True)
    created = models.DateTimeField(auto_now_add=True)

    def __unicode__(self):
        return self.title

    class Meta:
        ordering = ['-published']
        verbose_name = 'Entry'
        verbose_name_plural = 'Entries'

    class Admin:
        list_display = ['title', 'service_name', 'published']
        list_filter = ['service_name']
        date_hierarchy = 'published'

Create an url.py on the lifestream folder:

from django.conf.urls.defaults import *
from lifestream.models import Entry
 
entry_list_dict = {
    'queryset' : Entry.objects.all(),
    'paginate_by' : 30,
}

urlpatterns = patterns('',   
    (r'^$', 'django.views.generic.list_detail.object_list', entry_list_dict),
)

As you can see, I've used a generic view. You can also use the date based generic views and pagination to build an archive like mine.

Add to your project root urls.py:

(r'^lifestream/', include('lifestream.urls'))

Create a template lifestream/entry_list.html:

{% for entry in object_list %}
<div class="source">
  <a href="{{ entry.service_profile }}" title="{{ entry.service_name }}"><img src="{{ entry.service_icon }}" alt="{{ entry.service_name }}" alt="{{ entry.service_name }}" /></a>
</div>
<div class="details">
  <ul>
    <li><a href="{{ entry.link }}">{{ entry.title }}</a></li>
    <li>{{ entry.published|timesince }} ago</li>
    {% if entry.media_thumbnail %}<li><a href="{{ entry.media_link }}"><img src="{{ entry.media_thumbnail }}" alt="{{ entry.media_title }}" /></a></li>{% endif %}
  </ul>
</div>
{% endfor %}

Finally, create a script to synchronize your feeds:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys
import os

ROOT_PATH = os.path.realpath(os.path.dirname(__file__))
PROJECT_PATH, PROJECT_DIR = os.path.split(ROOT_PATH)

sys.path.insert(0, ROOT_PATH)
sys.path.insert(1, PROJECT_PATH)

os.environ['DJANGO_SETTINGS_MODULE'] = '%s.settings' % PROJECT_DIR

from friendfeed import FriendFeed
from django.conf import settings
from lifestream.models import Entry

ff = FriendFeed()
feed = ff.fetch_user_feed(settings.FRIENDFEED_USERNAME)

for e in feed.get('entries'):
    entry, created = Entry.objects.get_or_create(id=e.get('id'))
    if created:
        service = e.get('service')
        entry.service_id = service.get('id')
        entry.service_name = service.get('name')
        entry.service_icon = service.get('iconUrl')
        entry.service_profile = service.get('profileUrl')
        entry.title = e.get('title')
        entry.link = e.get('link')
        entry.updated = e.get('updated')
        entry.published = e.get('published')
        media = e.get('media')
        if media:
            entry.media_title = media[0].get('title')
            entry.media_link = media[0].get('player') or entry.link
            thumbnails = media[0].get('thumbnails')
            entry.media_thumbnail = thumbnails[0].get('url')
        entry.save()

If you want, you can add a job in your crontab:

# synchronize every 15 mins
*/15 * * * *   root   /path/to/your/application/lifestream_cron.py

See my lifestream as the working example.

UPDATE: Friendfeed sends the time in UTC, if you want to use your timezone you have do some hacking:

Install pytz:

easy_install pytz

Import and assign your timezone to a variable:

import pytz
tz = pytz.timezone(settings.TIME_ZONE)

And replace entry.updated and entry.published with:

updated = e.get('updated')
updated = updated.replace(tzinfo=pytz.utc).astimezone(tz)
published = e.get('published')
published = published.replace(tzinfo=pytz.utc).astimezone(tz)
if settings.DATABASE_ENGINE == 'mysql': # http://code.djangoproject.com/ticket/5304
    updated = updated.replace(tzinfo=None)
    published = published.replace(tzinfo=None)
entry.updated = updated
entry.published = published

Thanks to Chris Kelly that send me an email reporting this.

Comments

  • #1 By Justin Lilly on 4 April 2008 @ 01:39
    It would be pretty nifty if I could click the links in your twitter feed of your lifestream and be taken to the link instead of the status.

    <plaintext>I met <link>@justinlilly!</link> at my blog. <link>tinyurl.com/393kd</link></plaintext> or something similar.
  • #2 By Nuno Mariz on 4 April 2008 @ 01:46
    Hi Justin,
    I've made it generic for all kind of sources(services), but I can easily write a templatetag that formats the title.
    Thanks for the suggestion.
  • #3 By Peter Baumgartner on 4 April 2008 @ 02:36
    Jacob built an app called Jellyroll that has similar functionality without using FriendFeed. Out of curiousity, why not split media and services into different models?
  • #4 By kevin on 4 April 2008 @ 05:24
    i was going to exactly the same thing - but super psyched to see someone else doing it already. nice job.
  • #5 By Nuno Mariz on 4 April 2008 @ 09:38
    @Peter My idea was to simplify all the process, so I've made a generic table for the job.
    If I add a new feed to my FriendFeed account, I don't need to change a thing in my lifestream app.
  • #6 By André Luís on 4 April 2008 @ 09:46
    Whoa! Very nice Nuno. I'm not a django guy myself, but I like the end result. Similar to what I wrote in php, only I didn't go with FriendFeed. I went straight to the sources.

    I do think one very important aspect in lifestreams is the feed pace. Some feeds have 1 post per week, others (like last.fm) can achieve hundreds per day. It should be balanced to forbid one source to takeover the stream.

    Also clicking on the item and showing the item description would be extra cool. I hate that friendfeed only tells I posted a post but not the contents of the post.

    Still, I hope you see these as tips and not criticism. ;) Very well done dude. Rock on. :)
  • #7 By Nuno Mariz on 4 April 2008 @ 10:41
    @André Thanks for the tips, I will review all of them.
  • #8 By Chris Kelly on 8 April 2008 @ 12:49
    I'm in the same place as Kevin - I was actually sitting down to write this (even have the FriendFeed.py file open!) and you've gone and done it! Great Job!
  • #9 By Stu on 9 April 2008 @ 14:14
    This is v.cool... kinda like mugshot
  • #11 By Nuno Mariz on 17 April 2008 @ 16:51
    @Rui Ferreira: Cool. Would be nice to have the archive functionality on it.
  • #12 By Rui Ferreira on 18 April 2008 @ 05:35
    @Nuno: Junta-te ao projecto no google code. Para dizer a verdade o código foi "descolado" à pressa da minha página e ainda precisa de algum trabalho.
  • #13 By Nuno Mariz on 18 April 2008 @ 19:31
    @Rui Ferreira: Ok, assim que passe a próxima semana infernal junto-me.
  • #14 By ilteris kaplan on 12 May 2008 @ 06:44
    Hey Nuno, Thanks a lot for this tutorial. About when you are starting the application : python startproject lifestream. this gave me an error on the terminal. Is there a typo in there? I could able to start the app by doing ./django-admin.py startapp lifestream though. Maybe it's the different OS.

    My main question is different. I couldn't for the life of me make the synchronization py file able to call the FRIENDFEED_USER var from my settings file. Are you going to make the code available for us to peek through it?

    Thanks a lot for your efforts.
  • #15 By Nuno Mariz on 13 May 2008 @ 01:31
    @ilteris Thanks for the correction, it was a typo.
    About the question, I don't understand what you mean. Could you be more explicit?
    The code that I use is all in this entry.
  • #16 By André Gonçalves on 17 May 2008 @ 02:39
    uber cool

    I've been working hard on a PHP framework which draws some inspiration from what i've learned from learning a little python and reading some django ideas, by the way

    anyway, the fact is that writing a framework abstracts me from thinkin about concrete use cases...

    I'm so gonna port this to my framework's library sometime soon...
  • #17 By Peter on 4 August 2008 @ 22:12
    Create tutorial on creating a lifestream. I'm currently looking at what I want to use as a lifestream (or more lifecache), because being able to look back into your past to see what you did a year ago is the main reason I want a lifestream. Hopefully I have some time next weekend to play around with your code. For now this page goes into my bookmarks.
    Thanks for the great post.
  • #18 By Idan Gazit on 2 September 2008 @ 10:55
    Hey there,

    I'm working on a BSD-licensed aggregation library that doesn't use the friendfeed approach, but I do use the code in your "sync" script as a starting point, and I'd like your permission to redistribute it.

    If you could contact me via email, I'd be happy to show you the code + your credit in the readme.

    Thanks!

    -Idan
  • #19 By Nuno Mariz on 3 September 2008 @ 00:04
    Hi Idan,
    Feel free to use that code.
Comments are closed.