These days the internet of things (IoT) is virtually everywhere you look. And as it is often with things approaching the peak of the hype cycle, many seem to struggle to explain what the heck IoT really is. A common denominator seems to be that in the future our fridge will talk to our phone to remind us to buy milk once we are in the supermarket and your house will turn the heat up/AC on when you’re about to get home. Only that the future is now and that some of those technologies are here today (let’s leave out the discussion that companies really struggle to actually sell them to customers).

Still, it’s high time we start exploring all the wonderful real-time streaming technologies out there and how to harness them in order to build cool stuff (like RESTful APIs, which are awesome).

Flask

The amazing flask framework allows you to build web applications in a matter of minutes. I highly recommend you check it out. The mandatory “Hello, World” example is literally under ten lines of code.

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()

Kafka

Apache Kafka is a distributed publish-subscribe messaging system. Originally developed by Linkedin, it is commonly used for high-throughput messaging applications such as collecting user clicks on webpages, aggregating logs, stream processing, and the like. Since the IoT premise is that virtually every device will be linked up to the internet and talking to whatever services are relevant, Kafka seems like a solid choice. It’s also really easy to set up and use. Once you’ve downloaded it, all you need to do (provided you have a JVM installed) is issue these commands.

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

Putting it together

Let’s start simple. Assume you want to provide an API to list all topics, it would be as easy as that (thanks to pykafka):

from pykafka import KafkaClient

def get_kafka_client():
    return KafkaClient(hosts='127.0.0.1:9092')

@app.route('/topics/')
def topics():
    client = get_kafka_client()
    return jsonify([topic for topic in client.topics])

Don’t forget to import

jsonify

from

flask

.

Let’s do something more exciting. Like writing to a topic. It’s also really easy, just a few lines of code including parsing.

@app.route('/post/<topic>', methods=['POST'])
def write_to_topic(topic):
    # make sure we have valid json
    json = request.get_json() 
    client = get_kafka_client()
    topic = client.topics[topic.encode('ascii')]
    producer = topic.get_sync_producer()
    producer.produce(jsondump(json))
    return "OK"

I hope you enjoyed this post and stay tuned for the next episode in which we will consume some messages. As usual, if you have any questions or want me to talk about a certain topic that interests you, shoot me a message!