I’ve been interested in computer vision for quite some time, and the opportunity to tinker with face detection (in an ongoing project to weaponise our beer fridge) was just the the thing to get to grips with the rather awesome bit of technology that is OpenCV.

What is OpenCV?

OpenCV is an open source computer vision project (it’s in the name!) that’s been around for quite some time and is one of the most powerful (and easy to get started with) libraries to start working with computer vision without paying for it.

What makes OpenCV even more powerful is that it has Python bindings, that means you can use almost all of its functionality seamlessly in Python – the syntax and API is one-to-one, so getting to grips with the documentation is pretty straight forward, and there are plenty of examples around to get started with simple object detection.

What is Face.com?

As part of the PyOfSauron project, we make use of the face.com API – Face.com is a free-to-use face recognition API that (quite handily) has an excellent Python wrapper to get started with using it.

The Project: PyOfSauron

As mentioned earlier, the project PyOfSauron was initially meant to help us count how many beer cans were left in the beer fridge – and although that project is still on going – a much more fun side-project emerged to protect the beer fridge with an impromptu security system.

So what does the system do? The main module runs off of watch_me.py, it initiates a loop that starts grabbing images from the first webcam on the system and runs it through one of OpenCV’s detection algorithms, these are pre-set and can be changed in the command line.

The easiest way to detect a face (or any other body part) in OpenCV is to use a haar detector. These are handily encapsulated in easy-to-transport XML files that will do all the work for you. It’s even possible to develop your own Haar profiles with enough sample imagery and patience.

As it passes the images through this filter, the script will trigger a callback event with the details of the detected objects as parameters.

The callbacks are modular, so you can develop your own callback modules to do with the image and data what you like. In this case, we’ve developed the FaceClipper module the most.

The FaceClipper module will take the detected coordinates, create an identity for each (so it can be tracked over time) and draw a crop square around the detected object to feed back to the screen.

In the background, if a face comes close enough to the screen, a capture will be taken and uploaded to Amazon S3 for processing by the Face.com API – if a successful detection has happened, the script will then assign the identity of that face to the tracked object in memory.

Highlights and cool bits

1. Giving a detected object a lifespan – inventing visual memory

One of the key problems with object detection is that the computer is completely ignorant as to what is detected – so when the callback module gets passed a list of the detected objects and their coordinates, it has no way of knowing if item in position 1 is the same as it was in the previous frame – the detected objects have no identity.

To solve this, @errkk and I (mainly @errkk) came up with a simple solution to identify (or assume) whether a detected object is the same as a previous one.

To do this, the system determines a ‘maximum area’ that the face can move in before it is determined to be new – using some simple geometry (an actual use for Pythagoras!) we detect the distance the face has moved from the centre and compare it to any previous objects from the last frame – if there’s a match (within a certain maximum distance) then the face is assigned an ID and will (hopefully) be identified next time.

Each of the identified objects (remembered objects) has a memory decay time so that it is erased from the ‘memory’ list if it hasn’t been renewed (re-identified) within a second (this can be changed, but this seems to work, mainly for those occasions when a face moves out of frame for a second or looks sideways – stopping detection for a frame or two).

2. Recognising a face

Identifying that there is an object in the frame is one thing, but if you are looking for Faces, and then want to have those faces identified, you’re looking at a wwhole new hyper-complicated ballgame, it can be done using something called eigenfaces, but I’d rather use someone else’s tweaked API then try to bend my head around that.

This is where Face.com comes in – Face.com has an excellent face recognition system that is very easy to implement (and fast as well).

The FaceClipper module will take a cropped snapshot of a detected face once it reaches a certain size, this file is then sent to Amazon S3 for online storage and then Face.com is used to try to identify the face.

To ensure a smooth frame rate, this whole process happens in a thread that writes it’s results back to a queue that is checked on every frame, intitially, face recognition took ~5 seconds per face – which is good, but not ideal given how much people move about in frame.

By keeping the http connection open to S3 when the app starts, we managed to shave the recognition time down to ~1.5 seconds – which is much better. Naturally when the connection times out you’ll be back to a ~5 second recognition for one frame.

Getting your hands on the code

The whole project is hosted on BitBucket here – have a go, make sure all the pre-requisites are installed, and if you happen to make a module that works with PyOfSauron, drop me a line on twitter for some show and tell.

happy hacking,
Martin

It all started off with such a simple idea – we had a client, their site was not performing well, so we started load testing. Now we didn’t have LoadRunner or any other expensive load testing solution to hand, so we opted for a web-based system instead.

The system ran really well, in fact it did exactly what we wanted (albeit for a little chunky interface), and an intellectual challenge was born – surely it must be easy as pie to write a script that will zombie a bunch of servers in the cloud and point them at a target… an ethical DDoS

So Loadzen was born – as a python shell script and some cobbled together RPC code.

It was only after it actually worked and was surprisingly effective that we thought about taking it to market, so the long road started to making it market worthy.

As this is a technical blog for technical people, let’s talk about what it does under the hood…

The Architecture

The whole system runs of a three-tier architecture:

This separation is essentially so that the website acts as a client to the job server, ensuring the machinery that manages and generates tests is fully separated and isolated from the business end. We can bring down the site and the job server will continue running (and retain your results).

The job server will spawn generators as needed to meet the specific load requirement as required by the test being run at that moment.

But that’s not all, the system actually uses a single thread for each “virtual user”, given we know how many threads any specific generator can support we simply meter out accordingly, this way we can load-share multiple tests on the same load generators, with some processes running one set of test scenarios while the others are running a completely different test. This ensures maximum utilisation of the systems running at that moment (they’re expensive!) and also ensures we’re not just spawning a ton of new servers for each test.

This is the basic architecture of the system at the highest level, but there are a few cool little tricks in the overall architecture that we’ll get into later as we discuss the feature set.

The workflow

The standard workflow for a load test is:

1. Identify your use cases
2. Create scenarios for those use cases
3. Determine the ‘mix’ of the use cases (e.g. 20% of visitors will buy something, 50% will bounce and 30% will just browse or search)
4. Set up the test and the load maximum
5. Run the test

Loadzen does all of the above, the load generators will automatically scale out the ‘mix’ of tests based on the growth rate of the test curve, they will act in complete lock-step to ensure that each ‘wave’ of users starts at the same time and they will strive to introduce some realistic behaviour by running the virtual users at various stages of ‘drunkeness’, varying their step-rate through a scenario randomly so that we simulate more realistic user loads.

The kit

The Load generators will then record and average out each wave and report the data back to the job server, which stores it and makes it available to a client.

The load generators and the job server both work with Python running Pyro RPC, the reason for this choice? Complete object transparency and interropability between client and server, so that load generators have access to jobserver functions and jobservers can pass test objects to load generators with a single function call with no translation layer. This is a little fiddly, but in the end offers us the ability to just code without worrying about data types and formatting errors.

Both the job server and the load generators run as instances in Amazon EC2, and are controlled using the rather awesome Boto library for AWS.

The Website

The website is written in Django with MySQL and a shiny fat server provided by the good folks at Media Temple.

Probably the most interesting part of the website is the real-time results and control feed that is induced every time a test is started. This is actually a real-time push feed form the job server that uses a bastardisation of Socket.IO and an EventJS clone for Python called Tornado (from those good folks at live journal) all backed by an infrastructure queue powered using RabbitMQ and Pika.

The actual infrastructure for the site looks like this when we introduce these systems back in (and to think, all of this effort just so you get some shiny animations and a graph on a screen!):

Can I just say this now: I love RabbitMQ, I have fallen in love with real-time systems thanks to setting this up – it’s amazing how your viewpoint changes when you start thinking in terms of queues and channels and processors. When this feed was set up we briefly considered completely re-tooling the system to have a full-blown RabbitMQ back-end to power ALL of the things.

Pragmatism (thankfully) won out.

We chose Tornado / TornadIO because the Socket.IO JavaScript client is so versatile and offers a bunch of compatibility, taking care of back-ends, transports and browser compliance for you without having to mess around with concepts such as long-polling, STOMP or Comet.

By running a seperate Tornado server to handle the push feed we again managed to decouple everything, this way the thing that manages the transport is decoupled form the web client is decoupled form the work generator, ensuring we can work on each independently and not have a monolithic code base.

Making it easy to use

The day before launch (I really shouldn’t be admitting this), I wrote the Chrome extension client that made Scenario creation MUCH easier than was originally built into the website (although the manual wizard still had to be built to pave the way for other clients).

Chrome extensions really are quite simple – using a messaging infrastructure to make all the bits work together nicely, and all in JavaScript and HTML, so it’s not much different from building a website.

Conclusion

It’s a massive bit of kit, that works but could always be made better, one of the key things learnt from this exercise is making a decision of when to make something work and when to make something beautiful, we all want to code stunning software and have great code that is well architected, but if you want to get something out the door, you need to make pragmatic choices of when to say “earmark it for the next build, iterate and improve as you go.

At the same time, we learnt to ry to identify those bugs that seem niggling that you know will turn into cancerous, nasty, evil blobs that you have to work around because you were too lazy to tackle that nasty problem head on.

Anyway, I hope you guys enjoy load testing with Loadzen!

Happy coding (and testing),

Martin

It’s very likely that you’ve heard of node.js, the event-based JavaScript powered server – and to go hand in hand with it, SocketIO: the cross-browser websockets implementation that plugs right into it.

If not, I’d recommend you go look that up, then come back here so we can talk…

Ready? Good.

What makes SocketIO so appealing is it’s client-side implementation – it is completely cross-browser compatible and basically makes it very simple for you to read and write directly to web clients from your server implementation and making that real-time app you’ve always wanted to make without implementing something wierd like COMET, Twisted, Orbited or STOMP.

The only drawback is that in order to use SocketIO you need to use Node.js, which means you have to use a JS stack server-side… and that just doesn’t feel right.

Lucky for us, there’s a python event-based server very similar to node.js that can be implemented instead, and even better, it has a SocketIO plugin that speaks the same language as the SocketIO client-side library. So now you can implement SocketIO with a python stack.

With this server implementation you can develop a SocketIO based site that runs entirely in python, which is great.

Unfortunately, if you’re trying to add real-time to your existing site the last thing you want to have to do is re-write everything in another stack (that would just be silly) or be forced to use the Tornado server to build your app, ideally you will want to just be able to bolt it on.

That’s where RabbitMQ comes in – RabbitMQ is an AMPQ queue implementation – you can set it up to pipe information throughout your application and transport messages, commands or data to different clients and functions in your app. What RabbitMQ will do for you is serve as a backbone for you to be able to pipe events out of your traditional website implementation, hold onto those messages and pipe them into your Tornado server.

Pythons AMPQ client is called Pika, so, if like me you are using Django you can bridge the two using Django’s eventing system.

In order to get SocketIO on your front-end to talk to TornadIO and in turn receive messages from your webapp you need to pipe those messages via RabbitMQ into a TornadIO listener that is implementing Pika to recieve them.

The traditional implementation for one of these apps would be a chat server, but as this could easily be done with tornado alone and would have few tie-ins with your webapp, so for this how-to we’ll use the example of sending notifications to your users when they receive a notification that is specific to their UserID (like an email, or one-on-one chat invitation).

Setting up your front end:

To get things going, let’s set up your front end. Here all we need to do is link to the SocketIO library on the SocketIO CDN and implement a few functions when the page loads that will initiate your connection to your TornadIO server.

The code above is slightly adapted boilerplate from the SocketIO documentation. First, we set up a connection to the SocketIO server, connect to the server and finally set up some event handlers to fire when things happen. In our case those handlers simply populate an un-ordered list so we can capture the messages on screen.

Tornado and TornadIO

First of all you will need to get your hands on Tornado, TornadIO and Pika – all of which can be installed via easy_install, so I won’t go into it here.

To get the server side part working, we need to set up Tornado with a Tornadio implementation, and have TornadIO listen to RabbitMQ to bridge the queue down to the browser-level elements.

The code we will be working with is this:

The code above, when put into a file called socket_server.py will start your SocketIO server. Some interesting aprts to note are that MyConnection hasn’t been declared yet, this is a class that will manage the various events that happen to the tornadIO implementation, so running this will fail at the moment.

The TornadIO setup is pretty straightforward and is actually very well documented at the TornadIO git repository, all the real action happens in MyRouter, which is documented below:

This class is instantiated whenever a SocketIO connection is made, in the traditional implementation of a chat server the connection is stored in a list in the class, which we can’t implement as we are running a server to client bridge and not inter-client communication and need to keep a connection open.

It is because of this bridging aspect that we instantiate our RabbitMQ listener as a thread – store it in a variable (this should be in a list really) so that we can kill it when the connection drops and save some memory on your server.

Setting up your RabbitMQ listener

This part of the code will define the thisRabbit() class, this class will listen to your RabbitMQ queue and keep the connection open, piping data back down to the client vie TornadIO when needed:

This is a lot of boring code, but in order to understand it, think of each of the various methods as a waterfall of functions that are callbacks instantiated as the events happen. So open connection -> define what to do when connection is made -> connection made -> call callback for connection made -> declare queue -> define function to declare queue etc. etc.

The one to look at here is the handle_delivery() which will do something with the message received from the queue, in our case we parse the message to check if it is a commadn to kill the connection, or, alternatively – to use the t_client property – the TornadIO connection class – to send a response down the wire to the client.

As the thread is tied to the connection, this is a server -> individual client call, strictly speaking this could be mediated here with an arbitrary protocol definition in the queue messages as one queue could individually manage individual messages for clients, although it would be more efficient to have a queue for each if that were really what you wanted to set up.

Sending a Signal

Sending a message to RabbitMQ is quite simple, the class below sshould make it easy to send a message:

Testing the implementation

When yo use the Rabbit Producer, you will be opeining a connection, passing a message to the queue and quitting, there are better ways of doing this and this is a little quick and dirty, however for one off pulses this should suffice.

Some thoughts

What makes this so interesting is that RabbitMQ will enable you to build streams of messages that can be scaled and attached to whatever data consumer you need, be that another Tornado service or streaming messages directly to your users, this implementation offers a scalable and robust way to manage data streams without having to use NodeJS and SocketIO.

This implementation will also enable you to decouple your real-time data from the rest of your site, by using RabbitMQ as a mediator, you can connect as many services together as needed, make them event based or publishers into the queue without having to change or be tied to a platform (RabbitMQ has a wide set of client libraries).

Good luck!

M.

As you may or may not know, the EU is introducing some new legislation that comes into effect on the 25th of May 2011 that will force website owners to expressly ask permission of their users to store a cookie on your site.

Now this is a little troubling, since anyone that is using Google Analytics will be storing a cookie on your machine just to know how well their site is performing! Unfortunately, a simple act as tracking your user behavior via a free, awesome service like GA will probably fall afoul of the ‘absolutely necessary’ clause that comes as part of the directive.

So how do you, as a site owner or manager of a site go about making the site compatible with these new regulations? Well, put simply it’s a mess, as a site owner you will need to:

1. Get permission form the user prior to storing any cookies
2. Clearly explain to the user about how you use cookies

Now that will be quite difficult for most site owners as if you use a framework, third party plugin, tracking tool etc. You might not even be aware of how many different cookies are being stored on your users machine, or what they do for that matter.

But that’s a different matter that can most likely be covered by some boilerplate legalese on a page somewhere (and I am sure there are lawyers out there already drafting the documentation). What’s more important is how you can stop your carefully crafted site from using cookies without expending too much time and effort.

One solution would be to simply ask the user before any JavaScript is executed on your page that could put a cookie on the machine. To that end, I put together a little script that will:

1. Stop the browser from loading the page
2. Ask for permission
3. Offer to redirect the user to a page where you explain how the site uses cookies
4. Will redirect the user to a custom page if they say ‘no’ where you can tell them very politely to grow a pair
5. Only asks once, so once they say yes it’s all good they will never be asked again

The source code for the script is available here – do with it what you will, there’s no warranty from me. I’ve also put the script on a site called IDidItAllForTheCookie.com – so you can just add a short code snippet to your site and the script will take care of the rest.

Enjoy!

Martin

I recently came into posesion of an Advent Vega android tablet – it’s a 2.2 tablet with a nice big screen and a very sweet price spot (£200). Having already installed a custom ROM and tweaked the settings to  no end to get the tablet to behave how I like, I’ve been more and more interested in actually building mobile apps (actually, the fact that to flash a new ROM onto a virgin device requires you to set up the android SDK gets well on your way!).

This piost will describe how to go about building a fledgling andriod app without having to learn Java, but instead build the entire thing in HTML, CSS and JavaScript. What we won’t go though is all the intial stuff such as getting the JDK, installing the SDK and installing Eclipse – that is all nicely covered in Phonegap’s quick-start documentation.

What we will cover is how to get started with Phonegap and Sencha Touch – the two frameworks that I feel are the best options out there for someone who isn’t versed in Java or Objective-C and wants to rapidly prototype applications that can get up and go.

In this post we’ll walk through building a simple Reddit client for Tablet PC’s – at the moment only for android as I don’t have the spare cash to get a Mac as a build machine.

Our app will be simple – it will pull the posts from the /r/pics subreddit, and display the images in a right hand panel of the app, the application will also enable forward and backward movement through the posts list.

What is Phonegap?

Phonegap is a set of libraries that have been created to build a bridge between a mobile platforms’ native API and a JavaScript ‘Hook’ – what all mobile platform API’s have in common is that they all have a web view – some way of rendering a web page in a discrete window. What PhoneGap does is help you set up a basic natural-API environment that will host your app, and then let you focus on actually building the app in languages more familliar to you such as HTML/CSS for layout and Javascript for functionality.

What is Sencha Touch?

Strictly speaking you don’t need this library, you can start building your apps straight off with Phonegap – however, as all smartphone users can attest to, there’s something about the interface of a mobile app, the momentum-asssisted scrolling, pinch to zoom and swipe gestures to navgate between screens. Sencha Touch (along with XUI and JQuery Mobile) are pre-built frameworks that make building simple User Interfaces easy and pre-styled.

I selected Sencha touch for this post becuase it is by far the most advanced of the lot – jQuery mobile is still in Alpha and XUI doesn’t quite cover what I want out of my layout language.

Above and beyond this is also the fact that Sencha Touch is based on ExtJS, which is a framework for web-app creation and follows a (loosely) MVC approach. It’s a bit of a paradigm shift if you are used to jQuery, as Sencha Touch can generate most of your HTML for you without you ever writing a line yourself, while easily adding hooks to actions that hapen to your on-screen items. At some ppoint in the future I may repeat this process with jQuery mobile, however as it is still in alpha I feel this isn’t for practical use in production scenarios.

Getting Started

First off – the best thing to do to debug and test your app is to set up Xamp or WampServer on your dev machine and allow it to be accessed from anyone in your network.

The reason being that you will be able to use your tablet / phone device to access the web version of the app with the native mobile browser – which would pretty much be exactly the same as the phonegap application – so you can test without deploying via eclipse.

Once you’re happy with everything – go ahead and download Sencha Touch and unzip it to a folder in your www directory so you can access it like this: http://localhost/sencha. This will give you access to the demos, source code and most importantly the API documentation.

Setting up your first project:

First, create a folder in your www directory for your project, in this case we’ll call it andreddit – the exact structure is laid out below:

• www
• – andreddit
• —- lib
• —— touch
• —— andreddit
• —— phonegap
• —- imgs
• —- index.html

Pretty straight-forward so far. Your app will be running in index.html, and it will reference your application logic in the libs/andreddit folder, while the sencha files will live under the /touch folder.

Your first job is to copy (or symlink) the sencha folder into the /touch folder, this will make those files available to your application.

Writing some code

Open up your favourit editor (I use Notepad++) and edit index.html, copy and paste the code below into the file:

The most important bits here are the script and link tags:

• sencha-touch-debug.js – this is the sencha touch library in a friendly debug version that will give meaningful error messages
• sencha-touch.css – all they styling info for your app, not to be used for any custom styles!
• phonegap.0.9.5.js – Phonegap, you won’t actually be using this
• application.js – your sencha-touch application
• models.js – your sencha touch models, more on this later

The models file is a little odd, and it may be worthwhile explaining here how Sencha touch is organised in terms of handling data sources…

Models and Data Stores

Sencha uses Models to represent the structure of a data source – that can be XML or JSON data that s comming in via a web service and the beauty of the model is that it follows a very simple layout – essentially you define the ‘model’ of the data to create a programmatic representation of it’s structure – this is a fairly detailed topic, I’d recommend reading up on it in the main API documentation to find out more – for now, suffice  it to say that you’ll be using models to represent the structure of the data you will be accessing from the Reddit feed.

Having a model is all well and good, however it is irelevant if you don’t have a place to store your data, this is where Sencha has the concept of the ‘Data Stores’, these are essentially the machinery that do all the calling, processing and filtering of your data according to the rules you set out in your models.

To recap – you plug your models into the data store, you plug the data store into a web service / XML feed and then you access the data store from your application logic. This way, if the data changes, you can just modify the store to make that data available throughout your application.

Setting up your model

The reddit API works by sticking a ‘.json’ parameter at the end of a URL, so to get the structure, you could pull the front page and run the returned text through JSONLint – it will give you a much better picture of what the structure of your JSON looks like (instead of the gunk you get back normally!):

Loking at the JSON data – you can see that the root of the data we really want is under data -> children, so we will set up our model respectively, paste the folling into your models.js file:

So what is going on here? First, we create a new Model instance – this is called the Ext.regModel, we then give the model a name (‘RedditModel’), and then go on to define the fields that will be found in the data set. You can explicitly define field types using the common identifiers such as boolean, string, long, int, float etc. However in the case of the ‘data’ field, since it is actually further nested JSON, we don’t assign anything and let Javascript figure out that it’s a JSON object on its own.

Models are complicated, and can be nested for related data sets (for example if you have a list of customers and their related orders) – the documentation in Sencha is very good for this, so I’d recommend taking a peek at that if you are applying this to another data set.

Now that we’ve created the model, we can create the data store – this will handle the calls to the service as well as manage the ‘active’ data as you change the parameters of the store programmatically later. Paste the folling underneath your model code:

The Data Store does four things:

1. Assign the model to the store
2. Set up a proxy – this is the thing that will access the service, since this is a mobile app, you are 95% guaranteed that you will be accessing data from third-party domains, which means you will need to make JSONP calls instead of standard Ajax JSON requessts. The proxy will take care of this for you by using a ‘scripttag’ type instead of a ‘httpproxy’, the ‘scripttag’ type will manage all the crap around making JSONP calls (HttpProxy will juist perform a standard ‘GET”) so you wonl;t need to worry about the ins and outs, however if you want to know more, please see this page which explains it quite nicely.
3. Set up a reader – the reader is the thing that interprets the data t put into the store – you can see here that we are using the ‘json’ type (there is also an XML type) and have set a ‘root’ parameter – remember how we stated earlier that all the interesting bits live under data -> children? Well, by setting up the root parameter we can skip all the boring parsing and tell the store to focus only on this one node of the JSON data.
4. Autoload – this will re-load the store when it is created, you can also call the load() method against the model to manually load the store programatically later (we actually dot his later on, so hold your horses).

So far so boring – we still havenlt made any pretty things! Strictly speaking we’re building this app the wrong way round – starting with the data and then moving to the UI, most processes will focus on UI first. However, in terms of ‘building up’ the application layers, this approach makes the most sense in terms of a step-by-step approach as the display layers consume the data stores in order to work correctly, so having them set up first will save you a lot of time and debugging.

Setting up your Viewport and User Interface

On to the more exciting stuff – Sencha makes it very easy to start prototyping a UI programatically without having to directly manuipulate the HTML yourself – this is done mainly through objects called Panels.

But we’re getting ahead of ourselves here – first you need to set up your application – this is pretty straightforward:

Here we do two things – first we create a namespace for the application (you’ll notice we used this namespace in the models.js to define our data store), we then run the setup function to create our main Panel. The most interesting bit here is the dockedItems property – this is where you can plug in a bunch of new Panels and UI elements.

So lets do just that – paste the following into the docketItems list:

Also, add this above the Ext.Setup() function:

What have we done here? We’ve built our User Interface! I realise it looks like a horrible mess of nested curly brackets – but essentially what we have done is added two items to the main panel:

1. A toolbar
2. A left hand panel
3. A right hand panel (this is just called ViewDock, this is because we want to be able to manipulate it later directly you can use the Cmp() function to call it by name, however being old-school I just assigned it a full blown variable name)

Lets look at the more interesting bits:

First of all, xtype and ui – these are properties that tell Sencha Touch what the item you are creating is – we are using ‘toolbar’ and ‘panel’ in this example.

Next, you’ll notice how we are fitting objects – there are properties we are using a percentage to handle widths as this way we can be sure the UI will display on variable-width screens.

Most importantly, you will notice in the left hand panel that there is another dockedItems[] array, we will be using this to set up our post list and forward and back buttons as follows:

In this code snippet we are:

1. Adding a toolbar to the left hand panel
2. Adding a list to the left hand panel
3. Adding a forward and back button tot he left hand toolbar

Furthermore, we are attaching functionality to the ‘handler’ property of the buttons to move forward and back and onItemTap event of the post list object.

As you can see the post list is plugged into our Data Store using the store property, using the temTpl proeprty we can define how the list item will look and what elements from the data store we want to plug in. We then add the onItemTap event handler, which essentially pulls the record from the data store and modifies the html property of the viewDock, followed by an update() call to make the change visible.

The buttons on the other hand do something more interesting, we use the handlers to modify the extraParams property of the store to modify the web service endpoint to pull the next 25 records from the web service. Now this is specifically for reddit but you can see how just modifying these params and calling load() on the store will also update the list itself – so you won’t need to update display objects that are bound to data stores as updating one will trigger an update in the other.

You’ll have noticed we haven’t written any HTML yet – if you browse to you rindex.html file now though, you should see something that looks like this:

What Sencha does is create the HTML for you based on your docking options and how you nest your panels, to create the above we had an overall framework of top bar, left bar and right panel, then sub-divided the left panel into the back/forward toolbar and the post list – Sencha was clever enough to interpret this properly and build up the interface appropriately.

Using the models/stores approach, the list was populated form a web service with relatively little fuss and the loading of new or old posts only required a minor change to the store paramters to make possible.

Testing this on your tablet / Phonegap

Now you may say this is all well and good – but if you’ve set up your HelloWorld app as advised in the Phonegap quick start then you should be able to re-map that to a new AndReddit application. If you have your basic project set up – simply put your lib folder and the index.html file into the assets/www directory of your phonegap project (copy them, do not link them).

Once this is done, plugging in your tablet via USB (make sure debug mode is on and you’ve installed the coreect drivers for your tablet) and right clicking the project and selecting Run As -> Android Application, your tablet should display on the list of available devices, clicking ‘Run’ will then install and run the app on your device.

Round-Up

You’ll have noticed we didn’t actually use any of Phonegap’s functionality to do anything with the device – more on that later, this tutorial was to get you started building a UI that seems familliar using one of the more popular frameworks and how to interface that with the Phonegap method of building Android apps.

However, had you set up your UI the same way and used a list of locations you could activate the mapping API of phonegap and display it in one of your panels, enabling the nice UI interfaces as well as interfacing directly with the device.

I’llbe making the full project avaialble soon, ion the meantime if you want a copy leave a comment

Happy hacking,

Martin

I’ve done quite a bit of work doing web development with Django, and one thing that doesn’t really come up when doing standard CRUD work is that mostly all web sites are linear – take a request, process it and spit out a response. Sure you’re underlying architecture uses threading, but that’s invisible to the humble web developer, and most of the time is completely unused.

Recently I’ve been working on an application, a genuine client/server system that didn’t HTTP, that didn’t use SQL databases and relied solely on good ol’ memory to function – continuously – and respond to multiple requests quickly and speedily without locking up a client.

This is where I learned that threading is awesome, and learned to love the Queue.

What is threading?

Threading is when you take a function or a process that takes a long time and is quite independent form the rest of your application and let it run in it’s own memory space, or ‘thread’ – the function will split off from your main execution loop and finish its business all of its own accord. The benefits? It frees up your application to stay responsive, and not ignore other user inputs while it’s executing.

Take for example a web server – if it were a linear application it would not acept requests while it was processing a page request from a user until it was done. In the case of a web server – it will take the request, launch a thread and immediately return to a listening state to handle the next one, without having to actually manage the request.

A threading example in Python

Let’s walk through the code:

1. First we import threading and time – threading is the base class that allows you to run your function/object as a thread
2. Since we want to pass values to intitialise the thread, we override the __init__() function – remember to call the base class init function afterward otherwise you will get an error
3. The run() function is the real meat of your class – this is where you put the code that will kick off the thread – in this case we take a number, increment it – if it equals our end value we break otherwise, keep going and sleep 5 seconds

Pretty simple right?

The kicker is that you might want to do something with the output of your calculation separately in another part of your application – this is where a Queue comes in.

What is a Queue?

A Queue is a memory structure that is thread-safe, it will not cause collisions in your application or memory space (for example, if you started 5 threads of the above and wrote to a text file it could cause an error as two or more threads try to write to the file at the same time). Queues work on three principles: LIFO, FIFO and Sorted.

So, how do we write and pull data from a queue?

Let’s walk through the differences here:

1. First we import Queue
2. We create a Queue object
3. We modify the __init__() function to take a new parameter – here we pass it the queue object for modification
4. When the thread runs, it will put its output into the queue using the put() method
5. In our main execution section we have added a loop that will poll the queue and pull out data – Queue’s are sensitive things and can behave in complex ways, so I’d recommend reading the docs to get more detail (after all this is just an intro)

The most interesting part of this code is the last section where we poll the queue – we are using a loop to continually poll the queue – naturally a queue could be polled by other threads in your application to allow for message passing between running processes.

Queue’s are everywhere – most notably (and a great example) is in Hadoop and AWS Simple Messaging Service – both have simple messaging systems that replicate the functionality of a Queue on a cluster level, which is awesome.

Some really handy links:

• Dev Shed article on threading in python
• Practical threaded programming in Python

Enjoy,
Martin

While working on the initial version of DjangoSVN, I realised that one of the coolest things I could do with the DjangoSVN command interpreter was to make it pluggable.

The idea was simple: You can set up a DjangoSVN server anywhere, then write your own custom plug-ins that you can just drop into the plug-in folder AND make use of the Django ORM should you want to give it database backing and an admin screen.

So I had to figure out how to make that work, and it was surprisingly easy, so this will be a short post. Credit where it is due, I found a lot of help over at StackOverflow – so never underestimate the true power of Google

So, how do you write a plug-in framework?

The first step is planning – in order to use a plug-in, it should have a set, fixed format that can be written in. In the case of DjangoSVN, I wrote a ‘master’ plug-in class that could be inherited from any new plug-in to add functionality – this would ensure that the interface between your app and any new plug-in will always work the same way. It also gives you a chance to think about how you’ll pass values to it and return them in a standardised way.

Here’s the interface for DjangoSVN plug-ins:

It’s pretty straightforward – essentially you have a class, with an init() function you can override. You can put whatever you like in here that’s specific to your app.

It get’s interesting when you look at how a plug-in gets called in the app itself – here’s the command loader from DjangoSVN that loads up a plug-in:

What is happening here is this:

1. Get a list of the plug-ins that you have created in a directory – the plug-in name is the name of the directory (they are basically python modules)
2. If the plug-in we’re looking for is in the commands[] dictionary, set up a load statement as a string. It’s important to get the parent module here – it might be worth it dynamically generating this if you don;t have your app in you PYTHONPATH
3. Run exec hook_module – this will actually run the string as if it were typed in the interpreter
4. Try / Catch that thing to ensure that you can shoot out an error if it doesn’t work
5. Run the hooks main function – in the case of our interface, we use execute_hook()

That’s it – essentially this is all that is required to make your app plugable.

To make it more Django-specific, we include the following lines in the actual hook to load up the ORM machinery:

Here we load up the django machinery in the init function and can essentially bolt-on a django app as a manager to your plug-in using your normal settings.py file!

How would you improve this model? Do you have ideas on making a plug-in framework that can be re-used in python easily?

Recently I have been contemplating client-server applications, how these can eventually be tied into a web app and how best to write all of this in Python. This post will outline the usage of one of the more awesome XMLRPC libraries and also discuss why it’s better than what else is out there…

So, the first place that good ol’ Google too me was the XMLRPC library for Python, however – may I recommend to steer clear of this library as it suffers from the following most immediate problems:

1. Complex python objects cannot be passed easily, not without writing your own picking
2. It presents errors when handling types it can;t marshal – things such as lists and dictionaries, which, out of pure interest are possibly some of the more useful elements in python
3. No traceback for errors – this is a biggie, debugging an RPC client/server app is a pain in a** when you can;t clearly identify which part of the server code is failing since all your errors are coming from the XMLRPC library.

So, enough about that. If you are interested in torturing yourself with XMLRPC, you can find some really nice tutorials from IBM here on building a simple client / server.

Having given up on XMLRPC, I swiftly moved to the rather wonderful PYRO library. This library has great documentation, a clear syntax and some very handy features. Particularly, I would really enjoy:

1. When you pass objects from client to server, they are full-blown python objects that retain all properties, not some poxy proxy
2. It can handle all those un-marshalable types, so passing a really complex custom class down the wire isn’t a problem
3. Full blown exception handling – you can trap the traceback of the error to your server app with a really nice error handler

So, without further ado – here’s the simplest way to use the PYRO library (this is pretty much a copy of the main example – I’ve just expanded with some additional notes underneath to save everyone some frustration):

The server:

As you can see from the code above, it’s very similar to the XMLRPC implementation of a service – essentially, all you need to do is create a class that represents your server, add some methods that do fancy things and return some kind of object. Unlike XMLRPC, PYRO will not crash if your method returns None.

Once you have created the server, the startServer() function is only used to kick off the daemon serving loop, an interesting thing to note here is the URI that the daemon spits out, because it will look horrendous and changes each time the server boots up. Now this might seem like quite silly behaviour, but it only occurs because there are different ways of accessing PYRO objects from a client, and the difference in the access URI’s left me with quite a bit of confusion for a while. But all will become clear.

Part 2 – The client:

Create a new client.py file and stick this code into it:

Simples! So, what’s happening in this piece of code?

After importing the library, we create a proxy object – you will never call an object directly on the server – all calls are handled through a proxy. In pythonic terms, this is essentially the same as creating a local object based on the aService() class with all calls being called remotely (thanks to inheriting the server class from the Pyro ObjBase class).

You then treat the methods of the server class as if they were local – in this case we call whatsMyName() and it will return the string “Your name is Jively, Hi Jively!” and print this out.

What is important to note and to include is the structure of the try/except exception handling – this wonderful bit of code will make it super simple to debug your server app.

If you’ve worked with Pyro or think there’s any really useful resources out there – let me know in the comments!

Resources:

1. Pyro3
2. IBM Developerworks XMLRPC with Python
3. Why I ditched XMLRPC in favor of Pyro

PyTranscode is a set of classes that control and report from FFMPEG. They also make using FFMPEG from python very very simple.

Things you can do with PyTranscode:

• Transcode videos from format a to b using presets stored in another file
• Extract splash images from a video file (actually extracts a series of splash images at user-defined intervals)
• Objectify video files – create a VideoObject form a file and it will provide you with all the details of the file from framerate to duration to formats used in both audio and video
• Runner – run FFMPEG command and trap the output, reporting back percentage complete of any action

This software is being released under the GPL v3 because I think it should be out there and I think it’s important to get a handle on video in python as there just isn’t any up-to-date code out there that truly wraps FFMPEG’s functionality up (not in any modern way anyway).

There is an overarching strategy to this, PyTranscode will hopefully form a cornerstone of my next Django project – unnamed at the time of writing, that will essentially offer:

• A storage engine for local or cloud-based storage
• A queue manager for scaling, this again would have vairous back-ends for local queue management or something like Amazon Queue Service
• A transcode server that can be run on multpile machines and be managed via an API OR be run in a mode to interact with the queue service
• An API wrapper to control the transcode-server for common tasks

Anyone interested in contributing please contact me on Twitter @lonelycode or leave a message on the blog – I’ll be glad to get back to you.

The project is hosted on google code and can be found here

Sample usage:

Enjoy!,

M

Amazon has recently released a long-awaited feature addition to it’s Web Services infrastructure: Relational Databases. Having for a long time only had SimpleDB on offer (non-relational and weird to implement), it’s always been something of a struggle to get MySQL running on EC2.

Traditionally one would have to set up a block storage device, mount it, hack the mySQL  configuration to use the mounted EBS volume for storage (not to mention messing around with XFS locking and snapshotting the DB when the instance goes down). All in all a pain to implement and maintain.

Now with Amazon’s Relational Database Service, you get a MySQL database, hosted in the cloud – auto backed-up, patched and running on a scalable infrastructure (you can re-size it to your hearts content) and at the same time independent of your instances – meaning no more messing around with block storage for your database.

Now RDS is actually completely compatible with all mySQL tools, so interfacing it with django is stupid simple – but, for those that want a quick walkthrough, I’ve prepared one below, from getting the service running to getting django to use it:

1. Sign up for the RDS account

I’m going to assume you’re already using amazon AWS, or at least have an account – if not, sign up now. Once done, go tot he rds page and sign up for the RDS service.

2. Get the tools

You’ll need the command line tools, if you’re familiar with EC2 this should be easy, but if not, you will need to download Java and install it.

Next, download the command line tools and unzip them to a friendly directory.

3. Set up your paths

I’ve put together a simple batch script (for windows) that will set all the paths up for you, it is helpfully included below:

setuprds.bat

make sure to set the JAVA_HOME path to your java installation

4. Set up your credentials template

If you look at the above path, I’ve put it into the command line tools  home, name it aws_credentials and it should look something like this:

For those that don’t know where to get these ID’s, you can get them from your AWS account page under ‘Your Account’ -> Security Credentials

5.  Set up your paths

Run the batch file you’ve set up – that should get all your paths sorted out – if you want to make the changes permanent, you will need to set the paths up in your control panel to make them permanent.

To test if everything is running type in:

6. Let’s get started – setting up your first instance

I’ve based these instructions on the Amazon quick start guide, so if you want to see the originals just head over there, here’s how you would start up your first instance:

Change the names of the instance, username, password, DB Size and DB name in the command line before running it, you should get output showing that the instance is starting.

7. checking your instances

Make sure that your instance is started – and get the address of your new DB, simply type in:

If your instance has started, you’ll see something along the lines of:

In the output, that’s your db host name – write it down.

8. Set up the security settings.

For simplicity (and testing) you can grant access to the instance using an IP range, it’s easy – just enter the range with the following command

If you want to grant access to an EC2 security group, you will need your account ID (this is different from your access ID) – you will find it under ‘Your Account’ on the top right, it will say something along the lines of:

Account Number XXXX-XXXX-XXXX

remove the dashes and place the account number into the following command:

9. Try it out with django

I set up a standard app and ran it from localhost, the most important thing is to set up your settings.py file:

In your app models.py create your model the same way you would normally, go to the command prompt and run syncdb. Your model will be created and you can go on ahead and use it to your hearts content the same way you would any other MySQL back-end.

I’d recommend reading up on the RDS command line tools – the documentation can be found here

Hopefully Amazon will add RDS to the AWS management console like they did EBS, which would circumvent this whole set up malarkey.

What are your thoughts on this new service? Is it too pricey?