Reading the article critically (i.e. properly)

Internet news sources are all fighting for readership, the more readers you have, the more advertising revenue you can generate. And therefore, so many of them now resort to linkbaiting tactics and misleading representation of data in order to "scare" people into reading their supposedly factual news article. The problem is also, that so many people just cannot seem to read between the lines to really see what a lot of these news reports are saying.

Lets take an example. I have just read an article by Juniper Networks about the rise of mobile malware. The title is the usual link-baiting, scare tactic fare; Mobile Malware Development Continues To Rise, Android Leads The Way. The article is here if you want to give it a quick once over. I'm going to cover the title first. Take the word "malware" out of that title (Mobile Development Continues To Rise, Android Leads The Way). Now we can all probably surmise thats true as well. More and more development is happening in the mobile space every day. More and more companies and software developers are jumping on the mobile bandwagon. And, in fact, Android is also leading the way there too. Android Market is having more new apps being developed for it than any other mobile application store.

And this is my first point. If you say X phenomenon is on the increase, you firstly need to determine if thats an anomaly based on the trend in the area it operates in. I don't have figures for this (because, wouldn't you know it, that article didn't give any), but this discussion isn't about proving the article, its about discussing the misleading way things are being reported.

For example, if all mobile application development is seeing an increase of 100%, and at the same time we are also seeing malware development increasing by the same 100%, well, that's not pointing to a unexpected increase in malware. All mobile apps are increasing by that amount. Its just a part of the curve.

What a lot of articles tend to do as well is throw "data" at a reader that make things look worse than they really are, and the use of "% increase from last year" is another popular tactic used. A lot of readers don't really seem to understand how that works. Using the linked Juniper article as an example, they make the claim:

A 472% increase in Android malware samples since July 2011.

Ok. Thanks. But what was the actual number in July, 2011? If there were only 10 malware apps in July, that means now there are only  57.2. A tiny number compared to the vast number of non-malware applications. Why wouldn't they give you the real figures? I'm not privy to what they were thinking, but my first assumption is that the actual numbers are so low that it wouldn't be compelling reading, so they rather use the scarier 472%.

And this practice continues throughout the article. All accurate data, just completely out of context and therefore ... useless!

But along with the use of out-of-context data, writers of these articles like to mix in other data which looks related but actually isn't. Here's an example from the piece:

October showed a 110% increase in malware sample collection over the previous month and a striking 171% increase from what had been collected up to July 2011.

So, the way they collect data about malware is by collecting the actual malware. Kind of like anti-virus software on a computer does it. What they don't mention in the article is that the increase in malware "collection" isn't only a factor of there actually being more malware. Its also a factor of improved methods of detecting and "collecting" them. Those are two very different trends which the article tries to bundle into one in order to help paint a scarier picture.

Readers should (must) read articles far more critically as there are innumerable tactics writers will use to find ways to make you click that link, become outraged at the content, and share it. When you eventually break it down, they are usually full of half truths and data taken out of context.

They find a way to lie by telling you the truth...

(And for the record, with regard to the example article itself, I don't think malware is not an issue on mobile devices, I just don't think its anywhere near as bad as Juniper paints it)

Thinking about your data model

Web applications are so much more than they used to be these days. With integrations into other web applications through exposed API's, the shifts to Single Sign On mechanisms, data sources that vary from the traditional database backends, no-sql solutions such as the Cassandra's out there and even flat files, the amount of data an application needs to process and be aware of is pretty intense.

And yet most web applications treat every data source except the local database as a second-class citizen. Even though those alternate data sources are critical to the running of the application, its only the database itself that is treated with abstraction within the application's model layer.

Model Layer? Well, any web developer attempting to build a web application in this day and age without the structure of some form of MVC (Model-View-Controller) architecture behind it is asking for a difficult time ahead. MVC imparts a fixed structure to a project with a very sensible separation of concerns in order to make your web application a more maintainable as well as extensible product. If you still work in the days of single files with HTML, business logic and data access all scrunched together, then you are woefully behind the best practices at the moment.

Unfortunately, a lot of the power of the MVC design pattern is diluted by misuse. Hell, I have even caught myself doing it at times. The one aspect I am discussing here is the model (or data) layer, which exists for the sole reason of being a central mechanism to allow you to grab the data you need for your application without having to worry about how that data is implemented, where it is stored, what the database architecture is, or even if its a database at all. And that last point is where things fall short.

A number of web apps I have seen (and BrandFu is not exempt from this unfortunately) will use the model layer exclusively for the applications own database. Any other data source is accessed ad hoc, and in varying ways, all throughout the application's controller and view layers and occasionally within the model but only to extend the abilities of grabbing rows out of the database. The problem with this method is that, if you ever want to decouple from a specific data source, such as a web service for example, and want to switch from consuming that web service to storing and managing that data on your own database, it will be a nightmare.

I am not saying I am not to blame either. I do get caught out with this myself. Developing BrandFu, we found ourselves occasionally making calls to external web service from outside the model layer. And a few weeks ago, we had some interest from a company who would like to have the service installed as a seperate instance over their own network to be able to provide BrandFu services to their own clients but on their own managed servers.

Sounds great but theres one problem. At SYNAQ we have an internally used "API" and Single Sign On (SSO) system called SASY. The BrandFu application itself relies quite heavily on SASY as a data source, but unfortunately for us, the web service requests are scattered around the code in the controller layer. Not all, but a fair number of them.

The solution? Replicate the object model returned from these existing API calls as pseudo-database tables in our symfony schema.yml file. Essentially, map the data returned from these API calls as if they were tables in our local database. symfony can then auto-generate the model classes for these API calls, exactly as it would in the more traditional database model, except we can then go ahead and create methods within these model classes that, instead of resulting in calls to our database, will make the API request to SASY, hydrate the object and send that back.

The result is that any chunk of code that needs that data doesn't know where it came from. It doesn't care. As long as it gets what it wants and can continue processing, why should it? This also encourages re-use a lot more, reduces code complexity, and makes maintenance even easier.

The other benefit, is that if we ever need to move away from an API-based data source for those "tables", well, their schema has already been defined and adding the bit of additional code to make a database query instead of a REST request is a lot simpler. You could even have support for both an API data source or a local database and switch between the two via config.

In fact, that's exactly I will be doing now. BrandFu is going to be transitioned to as clean a data model as possible over the next few weeks. This will simply make the application easier to maintain, easier to extend and easier to implement over a variety of systems and networks.

BrandFu: Distilling the lessons learnt

Over the last few months, yours truly has been a little busy. We recently built and launched a product in the US called BrandFu. I'll let you go look at the site for details on what it actually does.

Along the way, I have also been doing a bunch of reading up on different aspects of our Internet economy. Jeff Jarvis' books was one source. What Would Google Do? is actually a fantastic  look at the way the Internet (and not just Google) has changed the face of our economies and how to leverage the same techniques as the big players in that space to excel at what you want to accomplish in your business. I also just recently bought another book of his as soon as I heard it was available on the Kindle; Public Parts. I haven't read it yet, only just started the intro, but again, it seems like a must read for anyone involved in online economies.

Another addition to the reading list is The Lean Startup by IMVU co-founder, Eric Ries. Again, not finished it, but another good read that does echo a lot of the same thoughts as the other two books.

So with the references out of the way up front, I just wanted to highlight some of the big lessons learnt from our own experiment launching BrandFu into an unknown market, as well as what these books point out.

1. Be prepared to move out of your comfort zone

The first thing to be aware of is that the modern age requires that people can multi-task. These days, if you want to be considered a valuable asset to your organisation, you need to be able to do more than just be an engineer. More than just a designer. You need to be able to dip your hand into marketing and customer service as much as you do actual coding.

Sounds like this has been said before but its surprising how many people, including myself, struggle to move out of that comfort zone. I wanted to develop a product. Dealing with customers was someone else's problem. What I didn't realise was that without customers (via marketing), we could get no feedback (via customer service) which meant we could not develop an application that best suited their needs. That customer interaction was vital to the engineering.

2. Release early. Even if you think you're not ready!

This is one of those scary aspects for developers. We want perfection. We don't want to push code out that might be buggy and feature-poor. But as I said above, you need customer input. You could spend 12 months in a silo developing what you believe is an awesome application, only to have customers come to you afterwards and tell you that it doesn't do what they need.

For BrandFu, we did an experiment first. We went from a very simple Proof of Concept just to see what our stumbling blocks would be, to a rapid 3 month development cycle. And then we released the product to a South African customer-base, leveraging off of SYNAQ's existing clientele. Sure, it was buggy, had features missing we thought would be awesome, but by releasing as early as possible to an admittedly smaller audience, we learnt a ton.

First of all, we learnt that the stuff we thought would be the most popular feature, banner campaigns, actually was secondary to the signature management aspect of the product. If we had siloed ourselves we would have made the banner and campaign management portion of the application absolutely kick ass, but people wanted to use the signature management stuff more. Releasing as early as we did pointed this out to us. We were able to shift focus rapidly and early.

3. Stay Agile! Especially in the first few months

One of the key things we tried to do was to remain as agile as possible. We have a ticket tracking system, JIRA, which is an awesome piece of kit, but we found it slowed us down. While developing our Minimum Viable Product for BrandFu, we were still getting constant feedback from our South African user base. This meant that when new information came in, we had to analyse it, determine what we would do it about it, and then implement. JIRA was slowing us down.

We ended up with the team sitting in one room, around one table with a big whiteboard. Ideas were hashed out immediately, details were scrawled onto the whiteboard and eventually erased when implemented. It meant our turn around time was hours instead of weeks. We could keep on top of changes we needed to make for our launch rapidly.

Now that things have launched and we don't have a crazy deadline, we can switch back to the more staggered process of log ticket, allocate to sprint, execute and so on.

4. Don't get attached to your code

Your customers will feedback (if you let them to of course, and why wouldn't you), and they will tell you things you don't want to hear. That feature you thought would be awesome and then people tell you they don't want it? Say good bye to it. Never be afraid of throwing stuff away. The Lean Startup even has an example of IMVU, where the product had to be altered so dramatically after launch, that it is now an almost entirely different product serving an entirely different need than it started with.

Things will change, things will be added, and things will need to get thrown away. It happens.

5. Anything else?

Sure! There was tons of things learnt. But a blog post is really not a good way to point them out here. The books I mentioned I would consider invaluable reading for anyone looking to develop a commercial application on the web. Hell, if your looking to create ANY business in this era of Internet transactions and communications, you would find good use out of the material in those books.

World of Warcraft: Cataclysm - First thoughts

If you have been living under a rock, you may not have heard of the release of the next expansion to the World of Warcraft franchise called Cataclysm. To summarise what this expansion adds:

• Two new races. Worgen for the Alliance (a shape shifting, werewolf race coming from the Kingdom of Gilneas which, through WoW's 6 year past, were hiding behind the Greymane Wall in Silverpine Forest) and the Goblin's for the Horde.
• Level cap increase to 85 with all the associated questing zones for those 5 levels as well as dungeons and starting raids.

Now that seems par for the course when it comes to a new expansion for an MMO. But what Blizzard did in addition is that before the expansion was released they "shattered" all the old, 6-year old content. Essentially, they gave themselves licence to go back and change, well, everything of the old world that was released with the World of Warcraft 6 years ago. This specific change has me so excited, and I have delved deeply into playing the rebooted content.

My first impression of it? Holy crap its awesome! They have taken every lesson learnt from the last two expansions (Burning Crusade and Wrath of the Lich King) as well as the technologies employed in Wrath like the phasing technology to make questing in starting zones feel more persistent. One of the oddities of questing was when you did a quest such as fight off a horde of invading forces, go hand in the quest where the quest giver thanks you profusely for helping them and it will make life so much easier and yet you go back to where you did the quest and things are still the same. Now, with the phasing tech, they can actually make "permanent" changes to the world for your character.

One of my favourite moments was levelling my new Night Elf Mage (yes, Night Elves can now be mages), and arriving at Astranaar, the town is under attack by the Horde. Your first quests are to kill the fires burning all over the town and then hop into a glaive thrower and kill off a bunch of Horde flyers attacking the town. When you hand the quest in, the fires are gone and the Horde are no longer attacking. Definitely makes it feel like your doing quests actually has an impact on the game world.

The other thing that Blizzard has changed is reducing (and in some zones, eliminating) the stupid quests where you need to kill some bunnies or a few pigs that the guy asking you to do it could probably get it done himself. Now, starter quests make you feel a lot more epic. Every starting zone I have tried so far has that "epic" quest line just before you move onto the next zone. This is important because Blizzard has said they have a large number of new accounts that get created and then never progress past level 10. Making a new player, still getting used to his or her class, feel like they are important to helping save Azeroth will make them want to continue on with their adventure.

There is one exception to this however that I encountered. I created a Human Hunter, just to see what the newer starting area for humans felt like, and unfortunately you still have those rather silly quests in Elwynn Forest where you need to help two love struck farmer's children meet up in secret because their families are feuding. Not exactly inspiring for whats to come but I guess Blizzard had a few limitations as to what changes they could squeeze in before release.

All in all, this is by far the best expansion Blizzard has produced for a game I enjoy playing (as well as, apparently, 12 million other people) and they deserve the record breaking sales they are getting. With all the changes to all the starting content it is far less intimidating than it used to be to get started in WoW as a new player and old players can find some extra fun checking out what was ripped to shreds and replaced.

FIX for the ATI/Kubuntu bug for Window resizing

So for a long time now, since I have had this machine, I have had an annoying bug. It really is only an annoyance but one I am glad I have finally gotten rid of!

The problem? If you have Kubuntu window compositing turned on and you have an ATI graphics card, resizing a window lags by a few seconds. In effect it means that you drag a window border to resize it, and while your mouse cursor moves to the new destination it takes a few seconds for the window to actually move to the new resized location. This also affected maximising a window from the task bar and even getting the Kmenu to display.

The cause? After a bit of reading, it seems to have something to do with a "fix" added to the Xorg server (one of the components of Kubuntu that handles creating the GUI display) to prevent "garbage" graphical anomalies that displayed for certain Intel graphics cards. The side-effect was that unfortunately, ATI's driver support being what it is (read abysmal), this fix caused resize effects to become incredibly laggy.

The solution? Well, if only there was some way to remove that fix from the Xorg server so that it doesn't affect ATI users (we don't need it after all)? The good news is that you can and it really is not that difficult. We need to replace the existing Xorg server packages with one that does not have this fix in it. Thankfully there is a handy repository that contains just that replacement. Below are the steps to replace the Xorg server with the "unfixed" version that should totally remove the resize issue:

1. Use Alt + F2 to open the run dialog, and type in konsole.

2. Press enter to open the Konsole terminal application.

3. To add the new location for these replacement Xorg files we need to paste the following into the terminal:

sudo add-apt-repository ppa:launchpad-weyland/xserver-nobackfill

4. Once it completes type in:

sudo aptitude update

5. This will reload all the available packages you can install by adding those packages from the new repository we added above. Now type in:

sudo aptitude upgrade

6. You should now get a list of upgrades that will be installed, amongst them being x-server upgrades. Enter Y to prompt asking if you are sure and the downloads will begin.

7. Once things are download and installed, close the Konsole and then logout (you don't need to shutdown but you can if you really want). Once back on the login screen press Alt+E, which forces the Xorg server to restart.

8. Log back in and enjoy the compositing effects as they were meant to be enjoyed.