Morrie the Toupee Salesman

Like I can’t think of a good simile… Anyway based on seeing a python toolkit for the digg api, and more importantly a comment about how a generator-based toolkit would be really useful, I’ve started to build the latter thing. And it’s true, the use of generators makes the need to get multiple pages from the api so effortless it’s a real concern making sure that you don’t accidentally suck down way more data than you need and get blocked by the people who monitor digg api usage. I’ve currently got a nice class to retrieve stories, I’m going to do the rest of the datatypes and add some bells and whistles before I release. But I thought I would just show how nice the code is to call the api.

    d = Digg()
    s = d.stories()
    i = 0
    try:
        for j in range(40):
            story = s.next()
            i += 1
            print i, story
    except StopIteration:
        pass

    print 'Read %d out of %d items' % (i, d.total)
    # using keyword arguments                                                               

    start = int(time.mktime(datetime.datetime(2008, 4, 30, 0, 0, 0).timetuple()))
    stop = int(time.mktime(datetime.datetime(2008, 4, 30, 1, 0, 0).timetuple()))
    # endpoint is /stories/ + what's specified below
    s = d.stories(endpoint='popular', kwargs={'min_submit_date': start, 'max_submit_date': stop })
    try:
        for j in range(10):
            story = s.next()
            print j, story
    except StopIteration:
        pass

    print 'Read %d out of %d items' % (j, d.total)

This is 2 examples actually, first retrieving from the /stories endpoint, and then from the /stories/popular endpoint using 2 keyword arguments (min_submit_date and max_submit_date). s.next() gets the data, and hides all the complexity of getting data from the api page-by-page. If instead of “for j in range(40):” you had “while (1):” then that little bit of code would proceed to suck down the 382,682 stories currently available at the /stories endpoint. And that’s cool…

Disclaimer