Gareth McCumskey ... a PHP Developer

Thinking Critically #1: Homeopathy

2011-11-28T02:42:00.001-08:00

This is the first in a series of articles I've decided to write that takes current controversial topics and looks at them with an unbiased, critical point of view by weighing up the claims and evidence (if any) that may exist out there. At the end, I will also give a run down of my personal views of the topics I talk about in these articles.

As I am sure you could have gathered from the title, this particular post is about Homeopathy.

What is Homeopathy exactly?

There seems to be a lot of misunderstanding as to what homeopathy actually is. People I have spoken to (family and friends) seem to misunderstand it, thinking it means natural or herbal remedies. While a lot of homeopathic remedies do purport to contain natural ingredients, a lot do not. So if homeopathy isn't about natural/herbal treatments, what is it? The easiest way to describe it is to go through the process a homeopathic remedy goes through to be prepared:

1. Identify the condition or affliction you wish to cure. Such as insomnia for example (of which there are many homeopathic remedies).
2. Find an active ingredient that would replicate the symptoms; in the case of insomnia, caffeine would replicate the symptom. This called the mother tincture (undiluted).
3. Dilute the active ingredient at a ratio of 1ml per 100ml. Thus creating a 1C solution as per homeopathic nomenclature.
4. Perform a process, known as succussion by homeopathic practitioners, which is supposed to activate the vital energy of the active ingredient. It involves a sequence of shaking and strikes against a surface.
5. Dilute this diluted, succussed liquid into another 100ml of water at the same ratio as the first dilution and perform succussion again. We now have a 1:1000 solution. In other words for every 1 molecule of the original active ingredient we have 1000 parts of water. This gives us a 2C solution to follow homeopathic nomenclature.
6. Continue this process again another 28 times until we arrive at a 30C solution, the recommended dilution according to the homeopathic founder Samuel Hahnemann. We now end up with a solution that is 10⁻⁶⁰. Or in other words 1 part of active ingredient per 10⁶⁰ of water.

To put that into perspective, "... a patient would need to consume 1041 pills (a billion times the mass of the Earth), or 1034 gallons of liquid remedy (10 billion times the volume of the Earth) to consume a single molecule of the original substance ..." as quoted from Wikipedia.

Why is it purported to work?

So with such a massive amount of dilution why are there so many people that believe it works? Putting aside the erroneous belief that homeopathy is about natural and/or herbal products, the following is how homeopathy is explained to work:

1. The act of succussion releases the active ingredients active energy into the water molecules which have the capacity to "remember" that energy, and the potency of that energy stored within the water molecules is enhanced with further dilutions and succussion.
2. That the electron makeup of the water molecules vibrate at the same frequency as the energy of the water molecules and therefore impart the benefits even with no active ingredient remaining.

There is also a lot of incidental evidence based on the reports from specific individuals that have claimed that they have suffered from a specific affliction that conventional medicines have been unable to cure but after using homeopathic remedies the symptoms and affliction have stopped. You can see people espousing such events on YouTube for example, and these isolated testimonies are often used as an indicator of its efficacy by homeopathic practitioners and those who believe in it.

The testimony of individual, isolated cases is not, however, a valid way to prove the efficacy of any medical treatment. There are just too many unaccounted for and uncontrolled variables to be able to truly determine if it was the homeopathic remedy that cured/alleviated the persons affliction or some other cause. Other possible causes that could be unaccounted for (and this is not a complete list, just what comes off the top of my head):

1. Environmental: Perhaps some aspect of the persons environment changed such as change of season (pollen, temperature, winds, etc) or they are no longer subjected to an external cause such as heavy metals, fungus', spores, etc.
2. Placebo Effect: A known medical phenomenon whereby a person given a treatment and told the treatment will help them feels an improvement even though the treatment was faked (sugar pill, fake surgery, etc). Essentially an anomaly produced by the mind that replicates the expected effects of the treatment had it been real.
3. Daily Habits: Perhaps the person started eating a healthier diet at the same time? Maybe they started an exercise regimen? Perhaps they shower more regularly, use a different route to work, or any number of possible alternatives...

It would also be good to mention that there are many individual, isolated cases where people have shown that homeopathic remedies do not work. These individuals will, for example, take an entire bottle full of homeopathic sleeping pills and show no effect.

What undeniable evidence exists to prove the claims?

I have hunted high and low for any studies by those for and against (and neutral) towards homeopathy. While there seem to be many studies that show homeopathy has no benefit besides that which a placebo would give you (such as this one, this one, this one, this one or even this one), I have yet to find any definitive studies that prove the efficacy of homeopathy greater than a placebo.

While I have read a number of studies that claim to provide undeniable evidence for the efficacy of homeopathy there have been numerous problems with these:

1. The study is not done in an isolated, lab environment and therefore cannot control for other possible effects correctly within statistical norms.
2. Population group of study participants are not randomised validly thereby biasing the end result (one study allowed the participants to choose whether they wanted to be treated with the homeopathic vs the conventional medicine which would bias the placebo effect).
3. The participants aren't limited in the affliction they suffer from, thereby introducing many unknowns into the study that cannot adequately be accounted for (one study used a group of cancer sufferers as the basis of its investigation but included around 10-15 different types of cancer sufferers).
4. Cherry picking of data. Some studies blatantly ignore any results that would show against homeopathy and only choose the results that show for homeopathy.
5. Comparisons of homeopathic treatments to invalid data sets. Some studies will take the results of the treatments and compare them to conventional medicine results from other sources such as other hospitals, or even results in of global averages instead of comparing with data retrieved from similar situations.
6. Separate groups are treated differently. One study had the homeopathic group also receiving additional care from medical staff where the "modern medicine" group did not receive this additional care. The improvement in quality of life of the participants would therefore be biased.

What would be undeniable evidence?

There is a very simple and logical method to attempt to prove the efficacy of any drug. A technique used all the time by pharmaceutical companies. It goes something like this:

1. Get together a large group of people. The more the better, but no less than say a couple hundred. These people need to all be suffering from the same affliction that your treatment/remedy is supposed to cure in relatively the same intensity if possible.
2. Split these people into two groups with roughly the same number in each group.
3. Tell both groups they will be given a remedy to help cure or alleviate the condition they suffer from.
4. Give only ONE group the actual remedy you want to test while giving the other group a placebo.
5. While the test is underway both groups are treated identically to not bring any bias into the results.
6. Measure the changes in participants over time from both groups by asking the same questions and doing the same tests.

After such a study you will now be able to compare both groups to each other. Seems reasonable to assume that if a remedy/treatment actually worked then the test group should show much better results than the control group. This setup also helps negate the placebo effect to some degree.

But that's not quite enough. This test would need to be done multiple times with different people. Preferably by different scientists as well to help negate any bias the experimenters may have.

To date, there have not been any such tests that prove the efficacy of homeopathy as a valid alternative treatment to modern medicine.

My personal view

I remain completely unconvinced. The first step, I feel, to proving homeopathy's efficacy as a viable medical treatment is to perform the kind of tests and studies as I have detailed above. Every other such study has proven it does no better than the placebo group but homeopathic practitioners cry foul of these studies.

The problem with the homeopathic "culture" is that it is also tends to make conspiratorial accusations. Claims against "Big Pharma" (whatever that is supposed to mean) try to denounce any studies that disprove homeopathy's effectiveness as attempts by the big pharmaceutical companies to protect their own profits. What doesn't seem to be factored in is that, if homeopathic remedies really did work, then the pharmaceutical companies would be the first ones to leap on it as a new revenue producer.

Speaking of big money, there are homeopathic suppliers which charge outrageous sums of money for these alternative and more "effective" remedies. Homeopathy is a big business, making a lot of money, out of peoples ignorance.

There are also the scientific issues to deal with. Those dilutions I talked about would essentially wipe out all possibility for even one molecule of the active ingredient to be present in a remedy. And once the dilution gets far enough, there will even be water molecules that have never came into contact with the original active ingredient. Apparently, it will inherit the "energy" or vibration of the ingredient from the water molecules carried over.

Homeopathic practitioners also will tell you that homeopathic remedies should be used in conjunction with conventional treatments. This serves very well to hide the lack of effect.

And lastly, the biggest reason I try to educate people on homeopathy, is that, unlike other pseudo-sciences like Astrology or the existence of Psychics, we are dealing with peoples health and well being here! People are using homeopathic remedies expecting to be cured and only end up delaying treatments that DO work or dying because there is an industry of quacks, peddling water, that claim it cures you when there is NO evidence it does!

Reading the article critically (i.e. properly)

2011-11-23T21:56:00.001-08:00

Internet news sources are all fighting for readership, the more readers you have, the more advertising revenue you can generate. And therefore, so many of them now resort to linkbaiting tactics and misleading representation of data in order to "scare" people into reading their supposedly factual news article. The problem is also, that so many people just cannot seem to read between the lines to really see what a lot of these news reports are saying.

Lets take an example. I have just read an article by Juniper Networks about the rise of mobile malware. The title is the usual link-baiting, scare tactic fare; Mobile Malware Development Continues To Rise, Android Leads The Way. The article is here if you want to give it a quick once over. I'm going to cover the title first. Take the word "malware" out of that title (Mobile Development Continues To Rise, Android Leads The Way). Now we can all probably surmise thats true as well. More and more development is happening in the mobile space every day. More and more companies and software developers are jumping on the mobile bandwagon. And, in fact, Android is also leading the way there too. Android Market is having more new apps being developed for it than any other mobile application store.

And this is my first point. If you say X phenomenon is on the increase, you firstly need to determine if thats an anomaly based on the trend in the area it operates in. I don't have figures for this (because, wouldn't you know it, that article didn't give any), but this discussion isn't about proving the article, its about discussing the misleading way things are being reported.

For example, if all mobile application development is seeing an increase of 100%, and at the same time we are also seeing malware development increasing by the same 100%, well, that's not pointing to a unexpected increase in malware. All mobile apps are increasing by that amount. Its just a part of the curve.

What a lot of articles tend to do as well is throw "data" at a reader that make things look worse than they really are, and the use of "% increase from last year" is another popular tactic used. A lot of readers don't really seem to understand how that works. Using the linked Juniper article as an example, they make the claim:

A 472% increase in Android malware samples since July 2011.

Ok. Thanks. But what was the actual number in July, 2011? If there were only 10 malware apps in July, that means now there are only 57.2. A tiny number compared to the vast number of non-malware applications. Why wouldn't they give you the real figures? I'm not privy to what they were thinking, but my first assumption is that the actual numbers are so low that it wouldn't be compelling reading, so they rather use the scarier 472%.

And this practice continues throughout the article. All accurate data, just completely out of context and therefore ... useless!

But along with the use of out-of-context data, writers of these articles like to mix in other data which looks related but actually isn't. Here's an example from the piece:

October showed a 110% increase in malware sample collection over the previous month and a striking 171% increase from what had been collected up to July 2011.

So, the way they collect data about malware is by collecting the actual malware. Kind of like anti-virus software on a computer does it. What they don't mention in the article is that the increase in malware "collection" isn't only a factor of there actually being more malware. Its also a factor of improved methods of detecting and "collecting" them. Those are two very different trends which the article tries to bundle into one in order to help paint a scarier picture.

Readers should (must) read articles far more critically as there are innumerable tactics writers will use to find ways to make you click that link, become outraged at the content, and share it. When you eventually break it down, they are usually full of half truths and data taken out of context.

They find a way to lie by telling you the truth...

(And for the record, with regard to the example article itself, I don't think malware is not an issue on mobile devices, I just don't think its anywhere near as bad as Juniper paints it)

Thinking about your data model

2011-11-20T22:03:00.001-08:00

Web applications are so much more than they used to be these days. With integrations into other web applications through exposed API's, the shifts to Single Sign On mechanisms, data sources that vary from the traditional database backends, no-sql solutions such as the Cassandra's out there and even flat files, the amount of data an application needs to process and be aware of is pretty intense.

And yet most web applications treat every data source except the local database as a second-class citizen. Even though those alternate data sources are critical to the running of the application, its only the database itself that is treated with abstraction within the application's model layer.

Model Layer? Well, any web developer attempting to build a web application in this day and age without the structure of some form of MVC (Model-View-Controller) architecture behind it is asking for a difficult time ahead. MVC imparts a fixed structure to a project with a very sensible separation of concerns in order to make your web application a more maintainable as well as extensible product. If you still work in the days of single files with HTML, business logic and data access all scrunched together, then you are woefully behind the best practices at the moment.

Unfortunately, a lot of the power of the MVC design pattern is diluted by misuse. Hell, I have even caught myself doing it at times. The one aspect I am discussing here is the model (or data) layer, which exists for the sole reason of being a central mechanism to allow you to grab the data you need for your application without having to worry about how that data is implemented, where it is stored, what the database architecture is, or even if its a database at all. And that last point is where things fall short.

A number of web apps I have seen (and BrandFu is not exempt from this unfortunately) will use the model layer exclusively for the applications own database. Any other data source is accessed ad hoc, and in varying ways, all throughout the application's controller and view layers and occasionally within the model but only to extend the abilities of grabbing rows out of the database. The problem with this method is that, if you ever want to decouple from a specific data source, such as a web service for example, and want to switch from consuming that web service to storing and managing that data on your own database, it will be a nightmare.

I am not saying I am not to blame either. I do get caught out with this myself. Developing BrandFu, we found ourselves occasionally making calls to external web service from outside the model layer. And a few weeks ago, we had some interest from a company who would like to have the service installed as a seperate instance over their own network to be able to provide BrandFu services to their own clients but on their own managed servers.

Sounds great but theres one problem. At SYNAQ we have an internally used "API" and Single Sign On (SSO) system called SASY. The BrandFu application itself relies quite heavily on SASY as a data source, but unfortunately for us, the web service requests are scattered around the code in the controller layer. Not all, but a fair number of them.

The solution? Replicate the object model returned from these existing API calls as pseudo-database tables in our symfony schema.yml file. Essentially, map the data returned from these API calls as if they were tables in our local database. symfony can then auto-generate the model classes for these API calls, exactly as it would in the more traditional database model, except we can then go ahead and create methods within these model classes that, instead of resulting in calls to our database, will make the API request to SASY, hydrate the object and send that back.

The result is that any chunk of code that needs that data doesn't know where it came from. It doesn't care. As long as it gets what it wants and can continue processing, why should it? This also encourages re-use a lot more, reduces code complexity, and makes maintenance even easier.

The other benefit, is that if we ever need to move away from an API-based data source for those "tables", well, their schema has already been defined and adding the bit of additional code to make a database query instead of a REST request is a lot simpler. You could even have support for both an API data source or a local database and switch between the two via config.

In fact, that's exactly I will be doing now. BrandFu is going to be transitioned to as clean a data model as possible over the next few weeks. This will simply make the application easier to maintain, easier to extend and easier to implement over a variety of systems and networks.

BrandFu: Distilling the lessons learnt

2011-10-03T03:00:00.000-07:00

Over the last few months, yours truly has been a little busy. We recently built and launched a product in the US called BrandFu. I'll let you go look at the site for details on what it actually does.

Along the way, I have also been doing a bunch of reading up on different aspects of our Internet economy. Jeff Jarvis' books was one source. What Would Google Do? is actually a fantastic look at the way the Internet (and not just Google) has changed the face of our economies and how to leverage the same techniques as the big players in that space to excel at what you want to accomplish in your business. I also just recently bought another book of his as soon as I heard it was available on the Kindle; Public Parts. I haven't read it yet, only just started the intro, but again, it seems like a must read for anyone involved in online economies.

Another addition to the reading list is The Lean Startup by IMVU co-founder, Eric Ries. Again, not finished it, but another good read that does echo a lot of the same thoughts as the other two books.

So with the references out of the way up front, I just wanted to highlight some of the big lessons learnt from our own experiment launching BrandFu into an unknown market, as well as what these books point out.

1. Be prepared to move out of your comfort zone

The first thing to be aware of is that the modern age requires that people can multi-task. These days, if you want to be considered a valuable asset to your organisation, you need to be able to do more than just be an engineer. More than just a designer. You need to be able to dip your hand into marketing and customer service as much as you do actual coding.

Sounds like this has been said before but its surprising how many people, including myself, struggle to move out of that comfort zone. I wanted to develop a product. Dealing with customers was someone else's problem. What I didn't realise was that without customers (via marketing), we could get no feedback (via customer service) which meant we could not develop an application that best suited their needs. That customer interaction was vital to the engineering.

2. Release early. Even if you think you're not ready!

This is one of those scary aspects for developers. We want perfection. We don't want to push code out that might be buggy and feature-poor. But as I said above, you need customer input. You could spend 12 months in a silo developing what you believe is an awesome application, only to have customers come to you afterwards and tell you that it doesn't do what they need.

For BrandFu, we did an experiment first. We went from a very simple Proof of Concept just to see what our stumbling blocks would be, to a rapid 3 month development cycle. And then we released the product to a South African customer-base, leveraging off of SYNAQ's existing clientele. Sure, it was buggy, had features missing we thought would be awesome, but by releasing as early as possible to an admittedly smaller audience, we learnt a ton.

First of all, we learnt that the stuff we thought would be the most popular feature, banner campaigns, actually was secondary to the signature management aspect of the product. If we had siloed ourselves we would have made the banner and campaign management portion of the application absolutely kick ass, but people wanted to use the signature management stuff more. Releasing as early as we did pointed this out to us. We were able to shift focus rapidly and early.

3. Stay Agile! Especially in the first few months

One of the key things we tried to do was to remain as agile as possible. We have a ticket tracking system, JIRA, which is an awesome piece of kit, but we found it slowed us down. While developing our Minimum Viable Product for BrandFu, we were still getting constant feedback from our South African user base. This meant that when new information came in, we had to analyse it, determine what we would do it about it, and then implement. JIRA was slowing us down.

We ended up with the team sitting in one room, around one table with a big whiteboard. Ideas were hashed out immediately, details were scrawled onto the whiteboard and eventually erased when implemented. It meant our turn around time was hours instead of weeks. We could keep on top of changes we needed to make for our launch rapidly.

Now that things have launched and we don't have a crazy deadline, we can switch back to the more staggered process of log ticket, allocate to sprint, execute and so on.

4. Don't get attached to your code

Your customers will feedback (if you let them to of course, and why wouldn't you), and they will tell you things you don't want to hear. That feature you thought would be awesome and then people tell you they don't want it? Say good bye to it. Never be afraid of throwing stuff away. The Lean Startup even has an example of IMVU, where the product had to be altered so dramatically after launch, that it is now an almost entirely different product serving an entirely different need than it started with.

Things will change, things will be added, and things will need to get thrown away. It happens.

5. Anything else?

Sure! There was tons of things learnt. But a blog post is really not a good way to point them out here. The books I mentioned I would consider invaluable reading for anyone looking to develop a commercial application on the web. Hell, if your looking to create ANY business in this era of Internet transactions and communications, you would find good use out of the material in those books.

World of Warcraft: Cataclysm - First thoughts

2011-01-11T02:24:00.001-08:00

If you have been living under a rock, you may not have heard of the release of the next expansion to the World of Warcraft franchise called Cataclysm. To summarise what this expansion adds:

Two new races. Worgen for the Alliance (a shape shifting, werewolf race coming from the Kingdom of Gilneas which, through WoW's 6 year past, were hiding behind the Greymane Wall in Silverpine Forest) and the Goblin's for the Horde.
Level cap increase to 85 with all the associated questing zones for those 5 levels as well as dungeons and starting raids.

Now that seems par for the course when it comes to a new expansion for an MMO. But what Blizzard did in addition is that before the expansion was released they "shattered" all the old, 6-year old content. Essentially, they gave themselves licence to go back and change, well, everything of the old world that was released with the World of Warcraft 6 years ago. This specific change has me so excited, and I have delved deeply into playing the rebooted content.

My first impression of it? Holy crap its awesome! They have taken every lesson learnt from the last two expansions (Burning Crusade and Wrath of the Lich King) as well as the technologies employed in Wrath like the phasing technology to make questing in starting zones feel more persistent. One of the oddities of questing was when you did a quest such as fight off a horde of invading forces, go hand in the quest where the quest giver thanks you profusely for helping them and it will make life so much easier and yet you go back to where you did the quest and things are still the same. Now, with the phasing tech, they can actually make "permanent" changes to the world for your character.

One of my favourite moments was levelling my new Night Elf Mage (yes, Night Elves can now be mages), and arriving at Astranaar, the town is under attack by the Horde. Your first quests are to kill the fires burning all over the town and then hop into a glaive thrower and kill off a bunch of Horde flyers attacking the town. When you hand the quest in, the fires are gone and the Horde are no longer attacking. Definitely makes it feel like your doing quests actually has an impact on the game world.

The other thing that Blizzard has changed is reducing (and in some zones, eliminating) the stupid quests where you need to kill some bunnies or a few pigs that the guy asking you to do it could probably get it done himself. Now, starter quests make you feel a lot more epic. Every starting zone I have tried so far has that "epic" quest line just before you move onto the next zone. This is important because Blizzard has said they have a large number of new accounts that get created and then never progress past level 10. Making a new player, still getting used to his or her class, feel like they are important to helping save Azeroth will make them want to continue on with their adventure.

There is one exception to this however that I encountered. I created a Human Hunter, just to see what the newer starting area for humans felt like, and unfortunately you still have those rather silly quests in Elwynn Forest where you need to help two love struck farmer's children meet up in secret because their families are feuding. Not exactly inspiring for whats to come but I guess Blizzard had a few limitations as to what changes they could squeeze in before release.

All in all, this is by far the best expansion Blizzard has produced for a game I enjoy playing (as well as, apparently, 12 million other people) and they deserve the record breaking sales they are getting. With all the changes to all the starting content it is far less intimidating than it used to be to get started in WoW as a new player and old players can find some extra fun checking out what was ripped to shreds and replaced.

FIX for the ATI/Kubuntu bug for Window resizing

2010-03-19T06:08:00.000-07:00

So for a long time now, since I have had this machine, I have had an annoying bug. It really is only an annoyance but one I am glad I have finally gotten rid of!

The problem? If you have Kubuntu window compositing turned on and you have an ATI graphics card, resizing a window lags by a few seconds. In effect it means that you drag a window border to resize it, and while your mouse cursor moves to the new destination it takes a few seconds for the window to actually move to the new resized location. This also affected maximising a window from the task bar and even getting the Kmenu to display.

The cause? After a bit of reading, it seems to have something to do with a "fix" added to the Xorg server (one of the components of Kubuntu that handles creating the GUI display) to prevent "garbage" graphical anomalies that displayed for certain Intel graphics cards. The side-effect was that unfortunately, ATI's driver support being what it is (read abysmal), this fix caused resize effects to become incredibly laggy.

The solution? Well, if only there was some way to remove that fix from the Xorg server so that it doesn't affect ATI users (we don't need it after all)? The good news is that you can and it really is not that difficult. We need to replace the existing Xorg server packages with one that does not have this fix in it. Thankfully there is a handy repository that contains just that replacement. Below are the steps to replace the Xorg server with the "unfixed" version that should totally remove the resize issue:

1. Use Alt + F2 to open the run dialog, and type in konsole.

2. Press enter to open the Konsole terminal application.

3. To add the new location for these replacement Xorg files we need to paste the following into the terminal:

sudo add-apt-repository ppa:launchpad-weyland/xserver-nobackfill

4. Once it completes type in:

sudo aptitude update

5. This will reload all the available packages you can install by adding those packages from the new repository we added above. Now type in:

sudo aptitude upgrade

6. You should now get a list of upgrades that will be installed, amongst them being x-server upgrades. Enter Y to prompt asking if you are sure and the downloads will begin.

7. Once things are download and installed, close the Konsole and then logout (you don't need to shutdown but you can if you really want). Once back on the login screen press Alt+E, which forces the Xorg server to restart.

8. Log back in and enjoy the compositing effects as they were meant to be enjoyed.

Movies do well while music suffers - How to prevent piracy

2010-01-25T01:59:00.000-08:00

Last year (2009) saw the movie industry breaking records at the box office left, right and center. From the smash-hit Avatar really making a big splash, to record-number turn outs at the cinemas making the movie industry a bucket load of cash. And yet movie piracy is still on the increase.

2009 also saw sales on the traditional music CD dropping and profits dipping for record labels. While artists revenue has increased some have commented that this is simply because artists have been working harder with schedules more stuffed with live performances than ever before.

So why is it that the movie industry is still making huge profits, even though both entertainment industries are suffering from piracy? To me, I see this as a perfect example of the value proposition that each industry's products represent to the average Joe.

When you pirate an album what do you lose out on compared to buying the original CD? Well, you lose the cover art but you can download that online too. The lyrics potentially but again, easily downloaded. Ummmmm.... You do however, depending on the record label, GAIN from piracy because you no longer need to deal with potentially draconian DRM measures that make your life difficult.

Lets look at the movie industry. What do you lose when pirating a movie as opposed to going to the cinema? Well, in short ... you don't get to go to the cinema! Sure, you can download the movie, then gather your friends at your place to watch it together. But you don't get to "go out" to the cinema! You don't get to share the mob excitement of watching this new movie. Shuffling into your seat in this vast space with other like-minded people. Have your eyes bedazzled by the enormous big screen in front of you and even experience 3D. Have a top of the range surround sound system that would cause the police to come knocking at your door due to noise complaints if you tried that volume at home. In essence, the experience of going to the movies is something substantial and meaningful to miss by just pirating to watch at home. Sure, its a LOT cheaper to just copy that movie over peer-to-peer networks than for each person to have to pay for a seat, but we pay it anyway because the experience is just not possible to duplicate. And besides, the guys who will download and not go the movies probably cannot afford the ticket prices. However, they get to watch it anyway, spread the word on awesome they thought it was, and potentially push more people to go to the cinema.

A pirated MP3 on my MP3 player sounds just the same as the original CD. So why should I shell out a lot of money for something I can essentially get for free. There is some good news, however, on the music front. A new technology called MusicDNA has been touted as the new replacement for MP3 and aims to add more to the experience of a genuinely purchased music file from valid commercial sources by embedding new content into a music file as well as meta-information that can get updated over time. This would be the right thinking amongst the record labels.

The only irreplaceable aspect of the music industry right now for customers is the live performance. You cannot download the excitement and feeling of connection with your favorite band or singer a live performance gives you by downloading it. This, in my opinion, is why artists are making more from live performances. And as I said before, people are attributing this to the fact that artists are now working harder doing more live performances. Well, of course, the only way to fulfill demand for more live shows (apart from doubling the sizes of concert venues) is to have more shows. If you have a lot more people wanting to go to the live show, you're going to schedule more of them. Totally logical. And why are there suddenly more people wanting to go to live shows? Perhaps because piracy of music has meant artists are actually getting more exposure than before because their music is being so widely distributed. An audience is being generated where previously there was no traction due to the relatively high cost of buying genuine media.

The movie industry is still breaking box office records because the cinema-going experience cannot be replicated (at least not all that easily) by piracy, whereas the music industry is essentially stuck offering the exact same thing people can get for nothing. MusicDNA is a good idea that has some merit in trying to do what the movie industry has done; create an experience people are willing to pay for that can't be replaced that easily. In addition, the music industry needs to re-invent itself around the live performance which is the closest corollary to "going to the movies" they have, as this is what people are willing to pay for.

Chrome OS: The big misunderstanding

2009-11-23T22:34:00.000-08:00

For some reason people are mistakenly calling Chrome OS a Windows killer. The more I read the comments that come with articles on Google's newly open-sourced Chrome OS, the more I realise people don't actually read the articles. Some people will comment that they believe Chrome OS will be a replacement and actually beat other desktop operating systems like Windows or the Linux operating systems. If people actually read what Chrome OS is and what it does they would realise this is simply not the case.

Google stated pretty blatantly that Chrome OS is basically a web browser built as an operating system. It is being designed to be installed on netbook computers and will only give access to the Internet and the associated applications. Files will not be stored on the local file system as we know it today. The only local storage will be as a cache. All files are permanently stored on the cloud.

The devices that will have Chrome OS installed on them will be, according to Google, companion devices. Most people will have their traditional, full-blown desktop machine or laptop and a Chrome OS netbook. The netbook will be useful for the times you are out and about and want to quickly access your email, Twitter, Google Apps, etc.

There is a good reason for this, again according to Google. They want to create an experience with the device whereby if you happen to lose it, it breaks or gets stolen you can just pick up a new device, login and all that data is still available. Also, with no local applications they can also make the OS more secure. If there is an application running that shouldn't be there, and by default there should be no apps apart from what is included on purchase, then Chrome OS can reset the entire device. Again, all your data is stored remotely, so you lose nothing with a reset, just a local cached copy.

These features, while a great idea, mean that Chrome OS is by no means a replacement for your current desktop operating system. There are just too many things that need a desktop application that the cloud does not yet provide for. But if you're an online connected kind of person you will find use for it.

What Chrome OS may do, however, is start to push people toward the cloud a bit more. If you use a Chrome OS device, you will need to make sure your documents are stored somewhere that would be available to you over the Internet. And, if you do start storing and even working on all your documents and work over the Internet from your desktop machine, only then will Chrome OS look like a viable alternative to your current desktop OS.

This is a very clever tactic by Google, because, while the world isn't ready for running everything off the Internet right now, Google's business is the Internet, and by releasing an operating system that starts pushing people towards trusting and utilising the cloud for everything, they are inexorably increasing their reach and possibility for providing web based services to .... well everyone. Extend that a little further and this means big money potentially for the Internet giant.

Jaunty was Canonical's Vista

2009-10-26T22:46:00.000-07:00

Jaunty Jackelope, compared to the Karmic Koala Beta I recently installed, seems to have been to Canonical what Windows Vista was to Microsoft. And both Jaunty and Vista suffered problems for the same reason; the OS's included too many new and "improved" features that hardware and software vendors just weren't ready for.

What really was the major issue with Jaunty was a newer version of the X server that eschewed the proven track record of using xorg.conf files for the auto-generated X configurations. Great idea but didn't work so well, especially for folks forced to use older graphics drivers or even anyone using ATI graphics cards and ATI's proprietary drivers. For myself, with an ATI card installed at home, it meant I had to do some rather tedious work arounds to get my graphics card working on Jaunty.

Another issue was the switch to Pulse audio being the default sound sub-system in Jaunty. Again, Pulse is great technology but people just weren't fully prepared for it. To this day I still have sound issues with my USB sound card "dongle" for my headphones, as well as issues with sound from Flash video in Firefox.

With Karmic, it looks as if the Ubuntu developers have not messed around with systems so dependant on outside support such as the graphics and sound systems. Massive improvements to the startup systems though have meant I have boot times of around 5 to 10 seconds (yes you read that right).

Jaunty was, however never an out and out disaster like Vista was for Microsoft. It had its issues, but with a 6 month release cycle, as opposed to Microsoft's years of development, things can get back on track very quickly, and Karmic sure seems to show that they are.

One of the big advantages of agile, open source development that Canonical has over Microsoft.

PHP Session write locking and how to deal with it in symfony

2009-10-14T00:10:00.000-07:00

It has been a while since I last posted to my blog. My personal life has seen a lot of upheaval recently with a house move, a hiatus from Synaq, then back to Synaq, and the release of Pinpoint 2 to Synaq customers all playing a major role in eating into my personal time.

So as my comeback article I thought I would write on an issue that was plaguing us in development of Pinpoint and how, thanks to help from the great symfony community at the symfony users mailing list, we got it resolved.

The problem

With Pinpoint 2, a symfony based application we have developed here at Synaq, we employ quite a lot of Ajax requests in order to make the interface more responsive and less bandwidth hungry; why reload an entire page when you really only want a small sub-set of that page to change? A problem came about when we had an Ajax request running, and while this request was waiting for a response from the server, if a user clicked another link, that link would not "process" until the previous Ajax request had completed. What this meant to us was that it seemed that our requests were "queuing" instead of working asynchronously as they should.

On one particular section of Pinpoint, we have a number of Ajax requests loading at once, each one interrogating the database for data. Each of those requests "queued" behind each other, and any attempt by the user to go to another module resulted in waiting for each of these queued requests to complete before the browser would process the users interaction.

The cause

After going through all sorts of different possible fixes, none of which worked, I eventually submitted the above problem to the symfony users mailing list. The response that came back was that it probably had something to do with PHP session-write locking.

PHP manages sessions, this anyone who codes in PHP knows, and in order to ensure that session data does not become corrupted between requests, PHP will lock write access to the session files for a user while it is processing a request. This results in the following process if you have multiple requests coming through:

1. Request comes into server, and PHP locks session files.
2. Another request comes in but cannot access the session files because they are locked.
3. The first request processes, running all SQL, processing results, etc.
4. Yet another request comes in but cannot process because session files are locked.
5. The first response is finally finished, sends its output back to the calling function and unlocks session files.
6. The second request begins processing, locks session files and continues to do what it needs to.
7. Request three is still waiting for session access.
8. Yet another request comes in but ..... I think you get the picture.

The solution

The only way to resolve this issue is to force the requests to unlock the session files as soon as possible. Thankfully symfony has its own user session storage classes that make this incredibly easy.

The one problem is that you cannot release the session lock until after you have saved data into session that needs to be saved. Our solution was that for each action that processes an Ajax request, write everything as soon as possible to session that needs it and then unlock session to allow any other request to begin processing.

We hit a roadbump. Using symfony's $this->getUser()->setAttribute() command to store session data, we then used PHP's session_write_close() to force PHP to let go of the lock and let the next request begin work. This did unlock session but we noticed that all the data allocated to session using $this->getUser()->setAttribute() was not saved.

After a little exploration of the symfony classes we noticed that when the setAttribute() method is used, in order to speed up processing, symfony does not immediately write to the global PHP $_SESSION variable. Instead it keeps those values in an array until the end of script execution and only then writes to session. Using PHP's session_write_lock() we pretty much made it impossible for symfony to do this because to prevent session data from losing concurrency, PHP does not allow a script to write to session if the session was unlocked.

We did, however, find another method: $this->getUser()->shutdown(). This forces symfony, when the shutdown() method is called, to write session data into $_SESSION and then it also runs session_write_close() itself.

The end result

We now have actions that process Ajax requests and once all data has been sent to session using $this->getUser()->setAttribute() we run the $this->getUser()->shutdown() method. The difference was incredible and has actually speeded up our entire application a ton.

One thing to be careful of however. You do need to be sure that you call that shutdown() command at the right time, because if you call it too early, session data will not get saved and PHP will just ignore it. We had to reshuffle some code so that all the database calls and data processing functions were run after shutdown() as well.

Thanks again to the symfony community for helping to point this out and hope this helps others who may have the same issue as well.

Warcraft Movie is going to ROCK!

2009-07-22T03:01:00.000-07:00

I am sorry, but I absolutely HAD to blog about this. It is now confirmed that Sam Raimi will be directing the upcoming Warcraft movie based on Blizzard's Warcraft game franchise. For those not in the know, Warcraft has been around since 1994 in the form of a real-time strategy game with two sequels and a number of expansions. It then transformed into the now massive World of Warcraft, the worlds largest Massively Multiplayer Online game, with over 11 million subscribers.

How do I know its going to be amazing?

1. The franchise lore

The storyline that accompanies Warcraft and the events that play out in its plot all the way from the 1994 RTS game to the current WoW are a move makers delight. Full of action, intrigue, drama, wars, treachery and love stories.

2. Sam Raimi

When Blizzard first publicly expressed an interest back in 2007 about making a Warcraft movie, Uwe Boll approached Blizzard saying he could direct for them. With this directors serious lack of success at game to movie conversions (anyone remember Street Fighter?), Blizzard pretty much laughed him out and told him where he could stick his offer. They then went and selected, in my opinion, one of the industries finest action/fantasy directors. Pretty much anything Sam has directed has ended up golden.

3. Blizzards neurotic caution with its IP

Blizzard has a very big reputation for ensuring that the quality of anything related to its IP is stellar. For evidence, look at Starcraft Ghost, a game that was supposed to be an FPS based version of the Starcraft universe that was never launched because Blizzard thought it wasn't good enough. The expansions for World of Warcraft were also supposed to only take a year each and notoriously took a lot longer because Blizzard wanted to ensure a quality product. Starcraft 2 as well was supposed to be released last year but was delayed because again, Blizzard wanted it perfect.

I have no doubt they will employ the same tactic with the movie and not allow it to be released if it ends up being a load of crap.

There have also been numerous comments around by people who are not as in love with the Warcraft universe as I am, or the 11 million other players of WoW, and believe that only die-hard fans will watch it. Personally I don't think this is the case. The same was said about Harry Potter and yet more people have watched the movies than have read the books or even knew about the books. The lore and storyline that exists within the Warcraft universe is so broad and appealing that it should be a great movie to watch regardless of if you have ever seen or taken part in the games.

So come release day I will be at the front of the ticket buying queue and happily settle down in front of the big screen with popcorn and drink to be absorbed in a universe I have come to love.

Lok'tar Ogar!!!

RIAA needs to follow the example of other industries

2009-07-21T22:40:00.001-07:00

By now, the RIAA's actions in its attempts to combat piracy is infamous. From suing individuals that they think have illegally downloaded content (even if it is blatantly obvious they never did) to trying to push ISP's into policing the Internet on their behalf. However, there are other IP industries that have moved on and followed the times, innovating to keep themselves current and up-to-date with the latest information age trends that demand content online. I am referring specifically to the movie and book publishing industries.

One of my fondest memories here in South Africa, was writing an article to a local IT publication, discussing how the latest movie titles can be bought at almost any street corner for about half what they cost in the stores, and that the best way that the movie industry could combat these pirates was to drop pricing and so make it unprofitable and too risky for these operators to continue. A few months later, genuine DVD titles in shops dropped from R300 each to a paltry R150 or thereabouts. Now, even if you went looking, the street-side vendors selling their pirated versions have all but vanished.

Combine this kind of pro-active approach to keeping your physical stock moving with online content-delivery platforms, and the amount of news I read about the movie industry losing out to pirates is so minimal its more like background noise.

Moving onto book publishers, ebooks could have been a real threat and could have resulted in publishers employing tactics similar to the RIAA. Instead, you can now buy ebooks online for a fraction of the price of physical books themselves. And developments like the iPhone and iPod Touch (amongst others) make consuming an ebook that much more pleasant. While devices like the Kindle by Amazon and now Barnes & Nobles own ebook reader also point to the book publishing worlds flowering digital transformation, I still think these large, expensive devices are ridiculous, but that is just a personal opinion.

The RIAA on the other hand seems convinced that the only way to continue in a world where people want their content digitally, is to deny them that and enforce draconian and, to be honest, ridiculous measures to preserve a business model that is so behind the times its laughable. People are already buying music online on a per track basis. The RIAA, and the companies it apparently attempts to protect, need to realise that the days of the "album" are nearly at an end. People want songs not epics. People do not want to be forced to buy a physical storage medium like a CD that contains 12 songs when they are only interested in 1.

Lastly, if the RIAA is not careful, they may find themselves defunct, along with the record companies they represent. With the explosion in online presence by artists and the ease with which anyone can publish content online, even music, recording studios that sign up artists as has traditionally been done, may no longer be necessary. Artists generate the majority of their own income by touring and royalties, not record sales. Record sales only help market the artist to a prospective audience and line the recording studios coffers. But when an artist has access to an all pervasive medium like the Internet as well as the multitude of portals to market their own content (YouTube, etc), they may realise they don't actually need a recording studio anymore. Artists may dictate to marketing agencies how they want their content promoted instead of the other way round.

PHP Security Tips

2009-07-20T01:44:00.001-07:00

Having worked with PHP for a number of years I am still at times shocked at how little some developers know about the security implications of their code or how they may be inadvertently compromising their applications by coding a certain way. This articles focusses on some aspects of PHP coding and what to look out for while you are developing to help make your application as secure as possible.

The following tips I have arranged in order of what I feel can be the most dangerous. If anyone has any comments please feel free to post them up and share with everyone else.

1. Command line commands

PHP has a great amount of power, and that power extends to the servers command line itself, allowing you to execute shell commands to the server directly from within the script you are coding. I have seen some coders liberally pepper their code with the shell_exec() function or equivalent PHP functions without a seeming care for what they may be opening up.

My first tip here ... don't use these functions! That's right, if you have no critical need to actually run a command on the servers shell then just don't do it. Rather take the time to figure out another way, rather than potentially open up your server to an attack through this vector.

Now granted that there are numerous ways that the server itself can be setup, as far as PHP's rights as a user to access certain server functions, and these should be enabled anyway. These are topics for a system admin and so are out of scope of this article, but why run the risk anyways.

Secondly, if all other options are exhausted and you absolutely have to use the shell, then please remember to clean up all data being run. As an easy way to clean, you can use PHP's own escapeshellarg() function which can clean some obviously shell-like stuff from any input, but this is also not good on its own.

Another tactic is to avoid using user submitted data as part of the shell argument. For example you can let a user choose from a few options and then run a corresponding argument in shell that was pre-written, rather than add a users submitted data in the argument.

2. Variable variables

Like shell commands above, just don't use them! They are a very big potential risk security-wise and there is almost always an alternate method to using a variable variable. For those who are not sure what I mean, PHP has the ability to create variables based on the value of another variable, for example:


$foo = bar;
$$foo = "Hello";
echo $foo.'\n';
echo $bar.'\n';
echo $$foo.'\n';

What would be echoed is:


bar
Hello
Hello

Now imagine a hacker figures this out and is able to submit user data to overwrite one of your own variables with his value and so compromise your system. The consequences, depending on where its been used, can be very dangerous.

Other ways to make using this more secure? There aren't any. Just don't use them!

3. Clean, clean, clean those submitted values

We all know the $_SERVER, $_COOKIE, $_GET and $_POST server variables. Those oh so convenient variables that store all that lovely URL and form data that a user submits. But it is shocking how many developers go straight from using that user input as is! Values submitted in URL's and forms by hackers is the number ONE vector of attacks on web applications and therefore should get the biggest attention when you are developing applications. Great PHP frameworks like symfony will escape arguments from forms and URL's as a matter of course and are therefore great, but this doesn't mean you are still safe.

SQL injection is the biggest attack through form submission and is really quite preventable. Functions such as mysqli_real_escape_string() are great for ensuring that posted data is escaped properly before it hits the database query level.

Another to check for, and this applies to framework users as well, is to properly validate your data. Frameworks can do some basic checking on data but are not complete as they tend to be very generic, validating for the sake of "is it a string", "is it x characters long", "is it an email address" etc. Fields like First Name in a form for example can be validated further. Does the name have any spaces? If so, it is incorrect. Does it have any numbers? If so it is invalid. Again, these just help prevent any possible vectors of attack.

These are just a few of the very large security risks to keep a look-out for while developing. While this isn't a complete guide by any stretch of the imagination on PHP application security, hopefully people may find the information here useful.

If you have any comments or extra tips to share, feel free to add your comment.

Mouse gestures with KDE 4.3

2009-07-14T02:10:00.000-07:00

For a few months now I have been using KDE 4.3 which, as of this post, is in RC 2 stage and will be the default on Kubuntu Karmic Koala (9.10). One of the greatest features I have found with this new version of KDE is the ability to use mouse gestures!

Sure, mouse gestures themselves are nothing new and have been used in the likes of the Opera browser for years now, but to have an operating system supporting this feature is a great step as it just increases the usefulness of the feature ten fold. This post is just a quick demo to show how to setup and use mouse gestures. Remarkably very easy.

First of all, you need to get to the System Settings panel. Press Alt+F2 and type "System Settings" or go to the Application Launcher >> Computer >> System Settings until you see this window:

Now open the Input Actions applet and you should see:

Along the left side there is a tree menu. Just right-click on the blank space there and navigate to New >> Mouse Gesture Action and click on Command/URL. You will then see:

Enter the name Firefox. On the right side, click the tab near the top labelled Trigger so that we can set what mouse action will open Firefox. Click the Edit button at the bottom and a new window pops open. Hold the left mouse button down and draw whatever mouse movement you want to use to open Firefox. In my case I did an upside-down L or rather an F without the little line. Generally you want to keep the gestures as uncomplicated as possible. You should now see:

Click the Action tab at the top and in the Command field type in "firefox" (minus the quotes of course). Hit the Apply button and its setup.

By default, mouse gestures work while holding the right mouse button down. So hold your right mouse button and draw the gesture you decided on anywhere on the screen. In my case I will draw my upside-down L. Firefox should now open. You can of course change which mouse button needs to be used; especially useful if you have a multi-button mouse.

Under the tree-menu click the Settings button. You will see it says that mouse button 3 is selected. 1 is the left button, 2 is the wheel and three is the right mouse button. Change the number, hit apply, test your gesture with the mouse button you want use and if it doesn't work change it again. Eventually you will get the correct number that corresponds to the mouse button you want to use.

There are other options as well for mouse gestures. For example, by exploring the DBus option, I was able to figure out how to use a mouse gesture to allow me to switch desktops easily. This is a very advanced feature however and I'll put together a little demo on how to use DBus and mouse gestures in another post if I have time. For now, enjoy the ease-of-use that this advanced new feature of KDE 4.3 gives you and have fun. :)

Why Linux will take over from Windows

2009-06-11T22:26:00.001-07:00

Oh no, not another one of these "Linux ruls, Windoze sucks and must die!!1!1" posts. Thank god, therefore, that I am not writing one. No, this is a long term prediction based on the two very different development methodologies employed in Windows vs Linux development.

To sum it up briefly, Linux's open source, distributed development by a group of passionate developers has a far greater longevity than Microsoft's hierarchical, closed-source development driven by money. But why does this make such a big difference? First lets look at the two groups drivers and what makes them do what they do.

Microsoft develops Windows as a closed-source proprietary and, most importantly, commercial product. Microsoft is a corporation and therefore needs to answer to share-holders, governments, anti-trust law makers, and other businesses. The biggest reason Microsoft develops Windows is to generate revenue. They employ developers to work on the project, and so monetary cost affects the skills and numbers of developers they can acquire. who are instructed as to what they should develop with Microsoft management guiding the direction and mission (in a holistic sense) of the design and development. They are under pressure to develop products with strict timelines in place and investors to satisfy.

Linux is developed by an organisation of non-profit developers. The developers who work on the project volunteer (for the most part) and so are driven not by the pay-check but by a passion for what they do. Generally speaking, people can select what they want to work on and what best fits their own skill sets. In addition, being open-source, anyone can get involved, and in fact they have their entire user-base available as potential contributors. Naturally not all can be developers but contributions can be made many ways. Linux does not need to answer to shareholders to justify its releases. Goals for a specific distributions release are decided upon communally so features get completed that the users actually want. Linux does not want to make money from sales of the operating system so there are no problems with anti-trust. Commercial backing, while helpful, is not necessary so Linux cannot go bankrupt or suffer cashflow problems (very applicable in the current uncertain economy) .

Its pretty much because of these differences that Linux distributions have an edge over their commercial counterparts. Microsoft, because of its reliance on being commercial, could go bang at any time. We don't know what goes on behind the closed doors of the Redmond giant. Who knows if Microsoft will be the next Exxon? Perhaps the money troubles at Microsoft are larger than they let on? Or it is possible that some new form of anti-monopoly legislation shoots them in the foot and makes it difficult for them to continue developing Windows the way they are. Who knows? We can't tell because everything at Microsoft is closed off to us and they rely on dollars, not contributors, to keep ticking. If Microsoft suddenly shut its doors, all those Windows users would be left high and dry because Windows is closed source. No one can pick it up and carry on.

Even if the financial world collapses totally, Linux wouldn't die. Being developed by passionate contributors they do not rely on a pay check to keep working. Even if a single Linux distribution nabbed 90% market share, anti-trust is not an issue because it is open-source and freely available, there isn't any direct income generated from sales of the distribution. And because of its open source nature, even if every single core developer for a Linux distribution left the project, the community behind can still keep developing it. Anyone can pick up the source code and continue development. Users wouldn't be shafted!

A timeline? I have no idea. Like I said, this is a long term prediction. It may take 2 years, may take 10. But eventually the time will come where Microsoft can no longer compete development-wise and the crown will start slipping, perhaps dramatically, perhaps over a slow, accretionary process of user disgust. Only time will tell.

Microsoft's new babies don't play well together

2009-06-05T05:21:00.000-07:00

Microsoft has released Windows 7 as a Release Candidate. In other words this is pretty much what we can expect to hit shelves in October of this year, bar a few bug fixes and maybe a small interface fix here and there. Microsoft also released their new search engine, Bing, aimed at competing head on with the giant Google. But, however, it seems Bing doesn't like Windows 7.

I have Windows 7 installed on my home machine and have been giving the RC a spin over the last few months (before that the Beta as well). And when I attempted to use Bing on Firefox in Windows 7, I noticed that most of its flash (literally) and pizaz were missing. I didn't get the little mouse-over effects on the Bing homepage nor did I get a link to search for Video in the options. Thinking that this was probably a quirk of Firefox on the Windows 7 RC, I opened Internet Explorer 8. Low and behold, I had the same problem.

My next port of call was the Adobe Flash plugin, as Bing seems to use that for its homepage effects as well as for the video search pages. I checked and the plugin was actually installed both on Firefox and IE 8. To make sure that I hadn't missed something obvious I uninstalled the plugins for both browsers and reinstalled. Still no joy. It seems that Bing is unable to detect that IE 8 or Firefox on Windows 7 actually have Flash installed. Either that, or it checks for a supported operating system and Bing just hasn't got Windows 7 on the list yet.

Microsoft's newest baby Bing, seems to not play well with its other baby, Windows 7. So they cannot even make sure their own products work well together.

One consolation. I have been using Bing on Kubuntu 9.04 with Firefox 3.0 with absolutely no problem. At least they are supporting the rival OS......

Microsoft's Bing is cool but....

2009-06-02T23:21:00.000-07:00

After reading a lot about Microsoft's attempt at improving search and trying to steal market share I decided to try it myself. No, this is not a bashing post about a monopoly trying to monopolise. Microsoft do not have a monopoly on search, Google do, and any decent competition in search would be a good thing as it would drive improvements in the field including at Google if Microsoft were successful in pulling a large enough market share. In fact, Microsoft are now so confident in Bing as a search portal that they have made it the default search engine for Internet Explorer 6 (whethere you want that or not).

Bing is actually really good. It eschews Google's philosophy of simple and clean by making the search interface more attractive. It keeps a search history so you can always refer back to what you searched for before which in itself attracts me as I constantly need to re-search for things. One of its greatest features is the mini-preview of a site you can get by hovering on a link to the right of the search result which gives you a quick run down of what that page is about and relevant links on that page that may be useful too making getting to what you want on that site relatively easy. See the pic showing the search results for Synaq and the preview of the first result.

So why the but in the title? Bing will probably get blocked at pretty much any organisations firewall level, perhaps even at home. The simple reason for this is because Bing also allows you to preview video results by just hovering over a video thumbnail. The video itself actually plays in the search results window as you hover over it, which is a great way to preview video but can allow people to bypass firewall settings that are supposed to block things like porn.

Techcrunch have already written about this and it can mean bad things for Microsoft's goal to grab market share. Hard to do that if schools, corporations and any other organisation providing people Internet access over its network block access to Bing. The problem is because most organisations filter on a per site basis. Bing circumvents that with its video preview feature acting as a kind of proxy to these not-so-safe sites. The feature is great and makes finding that video you are after even easier, but human nature will abuse it and already has started doing so.

Sure there are other ways to filter at a firewall like if the url contained certain search strings. But then someone has to maintain a growing list of potential search terms that people can use to try and get results from Bing to satisfy their craving for the hardcore. Its a lot easier to just add www.bing.com to a block list and will probably end up being the norm unless Microsoft can come up with a better way to do this.

Bing does have a safe search setting, like Google, but its a matter of two clicks to disable it. One way Microsoft can help alleviate this issue is to include the safe search setting (full, moderate, off) into the url as well with each web request. A firewall can then filter based on that, allowing people onto Bing if their safe search setting is on full.

Who knows though what the big Redmond will end up doing. I'd actually hate to have that video preview ability removed because it is really useful, especially in a place like South Africa where bandwidth is still at a premium and being able to preview a video quickly for a few seconds before loading up the entire host site is advantageous to the bottom line.

Symfony ... Good for ALL your projects

2009-05-24T22:28:00.001-07:00

A while ago now, Fabien Potencier and Francois Zaninotto were interviewed for the Zend Developer Zone blog and Fabien commented on how symfony is really better suited to larger web applications. I am here to counter that statement and show that symfony is good for all your web projects.

I can understand why Fabien would make a comment like that. To get going with symfony can be a little bit time consuming and to get to grips with its architecture and how to "code for symfony" can again take some time. But if you have already used symfony and learnt how to use it or plan to use it for all your projects, then those disadvantages fall away.

There is another reason why I feel symfony is great for even the small projects. How many "small" projects actually stay small? How many times have you started work on a project that is supposed to take only a few weeks at most to finish and it ends up still in active development months later? The problem with starting any project with the mind set that its only a small one is when it suddenly grows to be a rather large application, extensability and maintenance starts to become, well, a little nightmarish.

If you start a new project, even a so-called small one, with symfony, the abstraction required for good extensibility and maintainability is enforced on you. If this project suddenly grows its not a problem because everything is already setup to allow it be expanded.

An example is here at Synaq, one of our Senior Linux Technicians, a guy who usually works on setting up new servers, was asked to create a simple little interface for a Small Business Firewall product we are developing. This application was only really supposed to pull basic info into a simple interface for a customer to read. The problem is that now, more functionality than was originally planned needs to be integrated into this little app and it now needs a database backend to accomplish that. If the project had been started with symfony, it would have been a simple case of creating the database itself, sending a couple of symfony commands to generate an ORM model to interact with that database and 90% of the work would have been done.

After chatting with Jason, the System admin developing this application that used to be considered small, and explaining symfony, he is thinking of migrating the application to it. Ajax was another example. Some of the functionality that was added to the requirements of this little application was that data be updated on a few pages every few seconds. This meant that Jason now had to learn the Protoype library. With symfony he could have just used the built in helper functions to accomplish the same thing.

The biggest problems with "small" projects is what people don't forsee. A lot of the time these projects end up growing in requirements and suddenly turn into large, unwieldy projects that are difficult to maintain and extend with new functionality. By using symfony you may have a little slow down (perhaps) in the beggining, but you will end up with a far more robust and useful framework around which you can almost infinitely extend into the near future.

Why Blizzard rocks!

2009-05-12T01:45:00.000-07:00

I really do love Blizzard. They are a group of incredibly intelligent and insightful games developers. They only have 4 game franchises to their name and yet they are infinitely succesful. From the strategy genre with the likes of Warcraft and Starcraft to RPG's and MMO's such as World of Warcraft.

The real reason I love them is that while most games development houses are so focussed on the latest and greatest graphical punishment for your PC's hardware, Blizzard will work out the lowest common-denominator when it comes to hardware specs and then focus on that level of graphical grunt. No need for the latest price busting graphics card to play World of Warcraft or Warcraft 3. And the good news is that their upcoming releases, Diablo 3 and Starcraft 2 will follow the same trend.

Instead of graphics, Blizzard focus on what is, to me at least, a more important aspect; gameplay. A game can be absolutely drop-dead gorgeous in the eye-candy department and yet still totally suck as far as enjoyment is concerned. I played Crysis. Gorgeous game, with everything looking so incredibly realistic it was jaw dropping. But after the jaw was picked up off the desk and glitz fades away what was left? A game that was, well, average in my humble opinion. I never finished Crysis because it bored me eventually.

Blizzard's focus on aspects such as playability and plot means that its games far outlast those that focus primarily on visuals. Look at Starcaft as an example. There are professional (yes, people get paid) Starcraft leagues and tournaments in Korea with huge attendances. Not to mention television rights as well. This is from a game that was released in 1998 for goodness sake. World of Warcraft is another example. Graphically WoW is slightly aging, and yet it still manages to enthral a massive audience (well over 11 million subscribers now).

What this also means is good news for Linux gamers. I play WoW on Kubuntu (my Linux distribution of choice) and it runs better in Linux with Wine than it does in Windows XP. Recently I found my Starcraft CD's from back in the day and they played too. Diablo 2 also does as well as Warcraft 3.

This is absolute brilliance by Blizzard. Not only are they ensuring that the largest possible group of hardware configurations on PC can play their games, but they are turning every Linux user into a fan at the same time by their decisions to not be "cutting-edge" in hardware needs.

I just hope that with the soon to be released Starcraft 2 and Diablo 3 that they haven't forgotten that and that us Linux users will be able to help contribute to Blizzard's bottom-line by spending our money on their great games ... and then play them for the next 10 years. From what I have seen of the video's and screenshots, this does happily, seem to be the case. Only time will tell.

The Kindle: Whats the point

2009-05-11T06:18:00.000-07:00

Amazon have released the Kindle. A hand held device to ... read books. When I eventually heard the details, including the price-tag of $485, my first thought was "What's the point?"

I certainly am not going to pay nearly $500 for something that will let me read an e-book. In South Africa that's close to R 5000, enough to get yourself a reasonable desktop PC. If I really had to have a device to read e-books on the move I would rather get myself an iPod Touch or even go the whole hog and get an iPhone. the i devices can also allow you to read books but adds music player, wifi access, 3G Internet and a whole lot more.

But even the iPhone. Do we really need a device like that? Or many of the other gadgets and paraphernalia that clutter our lives and the shelves of electronics stores. Sure it looks gorgeous, and if you're a travelling salesman who needs access to your documents, emails and the web while on the move, then I can see it being a good tool. But for most people who go to work and work in an office; you have a PC on your desk. Then you go home inthe evenings; you have a PC (or laptop) there as well usually.

I am not one of those gadget freaks who must have the latest and greatest just because everyone else does. I will not buy an iPhone (even though I am enamoured by its capabilities and sexyness) because for me the vast majority of its features would just be a toy. I don't own a PS3 or XBox 360 because my PC at home has enough gaming potential to keep me more than happy.

Why do we allow our money-spending decisions to be dictated by fashion? Who cares what the Jones' have or want? Is our society really so driven by the materialism-as-status mindset that, like someone I know, they will put themselves into debt just to keep up with "everyone else"? Questions I don't have the answer to, unfortunately.

A fantastic I was pointed at a while discusses our materialistically driven society and how that affects the world we live in. The Story of Stuff is a fantastic narrative about what our society's impact is on the world we live in and how we cannot keep wanting wanting wanting. Take a look and tell me that its not at least a little eye-opening.

Next time you watch that advert about the latest sexy gadget that really adds no value to your life apart from you being able to say you have one, ask yourself if buying it is really what you need in your life. If in owning it it will change your life for the better or if it will merely be an ornament.

The Amazon Kindle? Last I heard books were available on this fantastic technology called paper for a 10th of that price. Perhaps we should buy more of those and fill book shelves. Or maybe I'm the odd one. If thats the case ... then I guess just ignore this blog post....

Don't forget your Bitwise Operators

2009-04-29T01:23:00.000-07:00

Edit After getting some comments about this post I realised some people might want a little intro into what Bitwise operators are. A great tutorial on it for PHP can be found here

I have had discussions before with other PHP developers, and in fact with developers in general, geeking out about ways to get things done in our respective languages etc. One thing I noted from these chats is that the knowledge of Bitwise operations, and how they can be used to create cleaner, more efficient applications, seems to be lacking. So I thought I would take the opportunity to point out one way that we are using Bitwise operators to make our jobs a little easier here at Synaq in developing Pinpoint 2.

A little bit of a history. Pinpoint 2 is our own development to replace the aging Pinpoint 1 interface which is based on the widely used, open source Mailwatch PHP application. Essentially it is a front end interface for the Mail Security service we provide; scanning companies mail on our servers for viruses, spam, etc, before forwarding the clean mail onto the clients own network. One thing that the old system (and of course the new one) needs to do is store classifications of mail. Some of the types they get classified as are Low Scoring Spam (i.e. probably spam but a chance that it could be clean), High Scoring Spam (i.e. definitely spam with a very slim chance that is clean), Virus, Bad Content (eg. the client blocks all mail with movie attachments), etc, etc. The old Pinpoint 1 based on Mailwatch uses a database schema that stores a 1 or 0 flag for that specific type. As a simplified example:

is_high_scoring: 0 or 1
is_low_scoring: 0 or 1
is_virus : 0 or 1
is_bad_content: 0 or 1

As you can see this gets rather limiting because what if, for example, you wanted to add another classification type? You then need to go ahead and alter the table schema in order to accomodate adding another is_* column to the table which is really kludgy and not that easy to implement.

So for Pinpoint 2 we decided to reduce all those classification columns into one and assign each classification a bit value. For example:

if clean: classification = 0
if low scoring: classification = classification + 1
if high scoring: classification = classification + 2
if virus: classification = classification + 4
if bad content: classification = classification + 8
if something else: classification = classification + 16
if another something else: classification = classification + 32

So if we had a mail that was classified as high scoring spam with a virus attached and would you know it, the content is also bad its classification value would be :

2 + 4 + 8 = 14

So in our classification column a value of 14 is stored. If we now want to in our interface check the type we do not have to access multiple columns and determine if it contains a 1 or 0 but instead retrieve one value and work our bitwise operators on them. For example with Propel in symfony, if we wanted all messages that were viruses:


$mail_detail_c = new Criteria();
$mail_detail_c->add(MailDetailsPeer::CLASSIFICATION,  4 , Criteria::BINARY_AND);
$virus_mail_obj_array = MailDetailsPeer::doSelect($mail_detail_c);

We now have an array of results with all messages that are viruses. If we wanted all messages that were viruses AND high scoring spam:


$mail_detail_c = new Criteria();
$mail_detail_c->getNewCriterion(MailDetailsPeer::CLASSIFICATION,  4 , Criteria::BINARY_AND);
$classification_criterion = $mail_detail_c->getNewCriterion(MailDetailsPeer::CLASSIFICATION,  4 , Criteria::BINARY_AND);
$classification_criterion->addAnd($mail_detail_c->getNewCriterion(MailDetailsPeer::CLASSIFICATION,  8, Criteria::BINARY_AND);

You can see from all this it is a lot easier to write dynamic queries using bitwise operators than it is to try and add new columns to a schema everytime you add a new classification type.

Matt Kohut. A man in need of an education

2009-04-22T10:05:00.000-07:00

A scathing title? I know. And to be honest I don't really care. Matt Kohut is Lenovo’s Worldwide Competitive Analyst, and he is sorely in need of an education in operating systems and specifically Linux. In a recent blog on tech.blorge, Mr Kohut is quoted as saying a few hilarious things. Blatantly incorrect statements and remarks about the state of Linux development that is about 5 years behind the times makes for some entertaining reading. The reason I am upset by this, and not just amused, is that it was said by a high level representative for a major player in the hardware industry, someone that the average user looks up to, listens to. Someone in that position shouldn't be allowed to spout his mouth off without at least verifying the most basic of facts. Lets go through a few.

“There were a lot of netbooks loaded with Linux, which saves $50 or $100 or whatever, but from an industry standpoint, there were a lot of returns because people didn’t know what to do with it,” he said.

There is no way to verify whether this is true or not but lets assume it is. The simple reason why no one knows what to do with it is because the world is so ingrained into using Windows that they have no idea that there is something else. They see something different and they think it is immediatly inferior just because it is not familiar. This is, of course, speculation on my part so lets move on for now.

“Linux, even if you’ve got a great distribution and you can argue which one is better or not, still requires a lot more hands-on than somebody who is using Windows.

“You have to know how to decompile codes and upload data, stuff that the average person, well, they just want a computer.

“So, we’ve seen overwhelmingly people wanting to stay with Windows because it just makes more sense: you just take it out of the box and it’s ready to go.”

4 months ago my fiance moved in with me. Her computer was flaky as hell because Windows XP did not like her hardware for whatever reason. The problem is that because no one except Microsoft can see source code there were no guides to help fix her problem on the web so it was either spend a ton of cash or try another Operating System.

She did still want Windows for the familiarity and a "safety net" so we started off by reformatting her drive into two chucks for a dual boot configuration. We installed Windows. Four hours and three restarts later, Windows was up. But this was a pre-SP1 disk she had with her machine so we had to install our own firewall, antivirus and a trove of other "security software" before we went online to install Windows Updates.

Whew! That done we decided to install Ubuntu on the other partition. 45 minutes later she was looking at her Gnome desktop. Her 4 year old printer worked out the box, scanner, the lot.

Then the other day she wanted to get her favourite old game Dungeon Keeper 2 installed so that she could play for a bit. I suggested she just boot into Windows XP cos it was more likely to run. I suggested this simply because I was busy at the time and didn't want to have to go through the hassle of trying to make a game designed for Windows ONLY to run on Linux.

She pouted at me. She actually dropped her lip in a sullen pout and then, and I will never forget, uttered the words "I don't like Windows". I felt so elatedly happy that I got up and got Dungeon Keeper 2 working. And as a side note, there was no problem getting it working. It installed and ran with no fuss whatsoever.

My point? My fiance uses a PC in her job. She is by no means a computer geek or ultra-savvy. She had to ask how to watch a DVD in Ubuntu. I told her to put the disk in. It loaded and she watched with no problem. Decompile codes (whatever that means anyway) and upload files? I beg your pardon?

Kohut argues that for Linux to be successful on netbooks (or notebooks or desktops for that matter), the open source operating system needs to catch up with where Windows is now.

“Linus needs to get to the point where if you want to plug something in, Linux loads the driver and it just works.

“If I need to go to a website and download another piece of code or if I need to reconfigure it for internet, it’s just too hard.

“I’ve played around with Linux enough to know that there are some that are better at this than others. But, there are some that are just plain difficult.”

Ubuntu, as far as an interface goes, exceeded Windows XP and Vista even a year before Vista was released. The combination of Gnome and Compiz or KDE 4.2 blows away anything Microsoft has been able to get Windows to do visually. Stability? Linux has been the predominant server technology keeping hugely complicated web presences and sites running for decades now, so stability is not something to worry about.

A few years ago I tried running a Fedora desktop but I struggled to get my USB DSL modem working to get online. These days? I plug it in. It asks me for username and password, I am online. Thats it. Could it get any easier, Mr Kohut?

“From a vendor perspective, Linux is very hard to support because there are so many different versions out there: do we have Eudora, do we have SUSE, do we have Turbo Max?

This is just evidence of Mr Kohut's lack of expertise in the field. Eudora is a mail client (you know, like Outlook?), Turbo Max has nothing to do with software as far as Google tells me. And no .. you don't have to support every distribution of Linux. Pick one or two (Ubuntu and Fedora are two good ones) and support only them. In fact, charge for Ubuntu and Fedora like you do for Windows installations but instead of it being payment for the software make it payment for support that people can actually get. They get the OS for free, you charge for support, and customers actually do get support on their OS.

I am shocked, angry and a little sad too that someone in that position of influence and power can be so dense, clueless and down-right imbecilic. How can you make remarks on a topic that you obviously know nothing about? What also saddens me is that these comments, my own and those of all the other outraged Linux users, will not by read by the majority of users.

Mac Ads seem to be a little presumptuous

2009-04-20T22:45:00.000-07:00

I was checking out the Mac adverts that Apple post on their web site and had to shake my head in amazement once or twice. Admittedly they are funny but for some reason it seems that Apple think all PC's run Windows. I know that the vast majority do run Windows but not all. And the fact that on the PC I have a choice of operating system seems lost on the marketing guys at Apple. Perhaps the IT guys briefing the marketing firm didn't fill them in on the fact that there is another player in the Desktop Operating System arena, namely Ubuntu. With the new Ubuntu Jaunty Jackelope (9.04) about to be released in a couple of days from this post, Ubuntu (and its derivatives such as Kubuntu) are a massive improvement over Windows.

And it got me thinking, what reason do people have to switch to Mac? For the stability of the OS? The built in nature of all the applications? The one downside (and the ONLY reason that I keep a Windows XP installation on my home PC) is that gaming, with some exceptions, does not work on Mac or Ubuntu. But then, when you can buy two PC's with the same hardware specs as a Mac and then install Ubuntu on both for nothing and get all the same benefits as a Mac, I still don't understand why people are so drawn to it. I understand that the Mac is prettier to look at but not everyone has R12 000 for the bottom range Macintosh just for eye candy.

The Mac ads seem to point out that PC's suffer viruses. That they have no applications for producing cool movies, pictures, etc. Crash constantly. And now, need massive hardware upgrades because of the operating system they use. Erm. None of that is true if Ubuntu is used. Ubuntu has access to safe collections of applications. In fact, since my switch to Ubuntu as my primary OS, I have never had to worry about finding an application to do what I want. I needed to find a book cataloguing system because I do have a rather large collection and to keep track of the books I still want can be a little hard. Alexandria is a freely available application that took me 5 minutes to locate on one of these collections. For those already using Ubuntu just do a search in your package manager for Alexandria or on terminal

sudo apt-get install alexandria.

I load up my Windows machine and want to find an application to use and its a few Google searches to find an application written by somebody. I then have to hope that this person is on the level and that its safe to use (i.e. contains no viruses) and that the application will actually work properly on my PC and not slow things down too much.

The funniest thing? I play World of Warcraft (I know, seriously geeky, but thats another discussion). WoW actually plays faster in Ubuntu using Wine (Wine is another application to try and make Windows programs work on Linux) than in Windows XP! I couldn't believe it myself but its true. A game made for Windows runs faster on a competing operating system that has to use a translation layer like Wine.

So no .. I won't be buying a Mac because I am not afraid of malware or viruses, I do not have problems finding applications I need and my operating system is not unstable. In fact... I think I can go get another PC. You stick to your one Apple Mac then.

What Trac really needs

2009-04-08T02:16:00.000-07:00

Anyone who has read this blog before might have found my previous post where I mentioned how I started getting into setting up my own web server on Linode.com and that I was also installing Trac, which according to the Trac documentation:

"Trac is a minimalistic approach to web-based management of software projects. Its goal is to simplify effective tracking and handling of software issues, enhancements and overall progress."

Well, I have got Trac up and running. Trac actually relies quite heavily on its command-line client which, to be honest, I have no problem in using. Anyone that develops and/or works on and for a *nix environment is probably more than comfortable using the command-line and probably, like me, finds it far more useful and efficient than any GUI could probably be. There were a few issues however in setting up Trac that I thought I would share here for anyone reading and interested in setting up their own Trac installation.

1. Command Line requires a learning curve

This may seem counter to what I said above but the one advantage a GUI interface has over a command-line is that it is intuitive. With a GUI you can see buttons and prompts beckoning you to use them. With a command line you need to know the commands or ... well you can do nothing. This means that anyone looking to install and run Trac as of now will have to spend extra time learning the, albeit rather basic, commands.

This is alleviated somewhat with a very useful help system as well as fantastic online documentation for Trac, but the fact still remains that that learning curve might put people off.

2. Root access needed

Trac is not a simplistic web application, even though the documentation calls it minimalistic. It requires the person installing to have root access to the machine and is one of the reasons why I am moving to running my own server as opposed to continuing with a shared, managed service as I have done in the past. While it is understandable to some degree because of the SVN integration, again, this requirement will limit the available user-base to those who know how to setup and maintain web servers or have enough dosh to fling around to get their server management company to install it for them.

3. No built-in authentication

Thats right. Unfortuantely Trac does not include its own authentication system, so managing multiple projects for different clients who should not have access to one anothers projects can be a little nightmarish. If you want authentication then Trac expects you to use Apache's own built-in authentication system's (or whichever web server you happen to have installed). This means that anyone installing this also needs to know how to setup Apache in order to authenticate users based on encrypted, password files stored on the server itself and referenced to using Virtual Host settings.

Again, this limits the potential users of Trac to those that are sys-admins or have the money lying around to get someone to do it for them.

4. Lets give Trac a break

I mentioned a few issues I had but lets cut the developers some slack. Why? Well, Trac is only at release 0.11. Yup! They haven't even reached a full release version yet and pretty much what is available is beta-ish. Once you know that Trac development is still steaming ahead and that the "issues" I described above will probably have solutions to make things easier once the development team do incremement that version counter to 1.0, it doesn't seem such a bad deal for a free development management and bug-tracking application. I am pretty sure that in the next few weeks to months we will see Trac become a feature complete system and I cannot wait for that day. So far I am very impressed with what I can do with it and am so glad I stumbled across it a few months ago.

National Skirt Extension Project is a WHAT?!

2009-04-06T21:49:00.000-07:00

Yes thats right. The National Skirt Extension Program. This morning sitting in my car listening to a popular, local radio station and I hear an advert for the National Skirt Extension Program which aims, get this, "To increase the displayed length of a ladies skirt on restroom signs". Feel free to go check it out yourself here. There is also a telephone number to call for more information. There are some fears that calling that number will result in large call fees as if you had called one of those premium number sex lines, but the 0860 number is NOT a premium call number so out of curiosity I called it. After listening to a recording about the NSEP I hung up.

I honestly cannot think there is anything real about this. If our government really has initiated a project as frivolous as this, then we really do need to be careful who we vote into power in the next election. I can only see this as some form of prank. One person commented on another blog that he actually called the agency responsible who apprently says its for real and their client is the NSEP but that they couldn't go into more details.

I hope this is only some delayed April Fool's joke or an attempt by someone to garner attention and then use that attention to launch some other project or initiative. If any one has anymore information please feel free to share because this is a little scary if it's real....

EDIT: Apparently there is speculation that this advertising is being arranged by UniLever, you know, the household products company. If this really is a viral marketing strategy by a company then I must say they have done good work here. I am actually surprised there is nothing from the government yet to denounce the adverts as nothing to do with them.

Memory caching can be a saviour

2009-04-06T02:31:00.000-07:00

At Synaq we are busy working on a pretty complex application. Essentially its a frontend interface to a system that scans and processes customers emails for spam then records the results of the scans in a MySQL database. Without going into too much senseless detail, the backend processes a few million items per day and suffice it say that is one helluva database to search through when you need to extract useful data.

Because of the sheer quantity of data we have had to use numerous techniques to try and make the frontend still act at least reasonably responsive when it needs to query the database. Then one day I asked myself "Does the interface really need to query that database so often for data that in essence hardly ever changes?". The scenario is that the interface does not really make many alterations to the data extracted and a lot of the data used is repeated per page for a specific users session. One security feature we have for example is that every user is defined as belonging to a specific Organisation (or Organisational Unit to be technically correct) and every page load requires retrieving this list of Organisations that the current user is allowed to see. This is not likely to change that often and so we came up with an idea.

We use APC, a memory caching facility for PHP scripts, and it also allows you to store your own values through your code into memory explicitly. Thankfully, symfony provides a class that can manage that for us as well, the SfAPCCache Class, that makes using the cache a doddle. Our problem? We need to ensure that the data we store is totally unique.

The solution was to store the results of a database query for our OrganisationalUnits model class into the APC Cache memory. The way we did this was to use the Criteria object for the Propel query as the name of the item to be stored. It stands to reason that if the Criteria object for a specific query is unique then the result will be unique. If the same Criteria object is passed again then the results from the database will be the same as the same Criteria object we passed before. Why query the database a second time?

The APC Cache though cannot take an object type as a name only a string. Easily enough done with PHP's serialize() function. But that string is excessively long (a few thousand characters sometimes) so we need to find a way to shorten and yet keep the uniqueness. So we get the MD5 hash of that serialized Criteria object. There we go. But due to our own paranoia and the need to be 110% sure that we wont by some ridiculous stroke of bad luck create another Criteria object later that against all the statistics of MD5 creates the same hash, we also make an SHA1 hash and concatenate the two hashes. There! Now the chances of any Criteria objects having the same name are so remote as to be nigh-on impossible.

But it doesn't end there. This doesn't help us if we don't know a way to actually add this to the cache and remove etc. For this we go to our OrganisationalUnitsPeer class and overwrite the doSelect method that recieves all calls to run a query onthe database as such:

public static function doSelect(Criteria $criteria, $con = null)
   {
       $data_cache = new sfAPCCache();
     
       $serialised = serialize($criteria);
       $md5_hash = md5($serialised);
       $sha1_hash = sha1($serialised);
     
       $complete_name = "organisational_units_doSelect_".$md5_hash.$sha1_hash;
     
       if ($data_cache->has($complete_name))
       {
         
         
           return unserialize($data_cache->get($complete_name));
       }
       else
       {
           $query_result = parent::doSelect($criteria);
           $data_cache->set($complete_name, serialize($query_result), 3600);
         
           return $query_result;
       }
     
   }

Rather simple I thought. We also wanted to be sure that if the user added, updated or removed a new Organisation that the cache would not give the incorrect listing so we added to OrganisationalUnits class (not Peer):

public function save($con = null)
   {
       $data_cache = new sfAPCCache();
     
       $data_cache->removePattern("organisational_units**");
     
       $return = parent::save();
     
       return $return;
   }
 
   public function delete($con = null)
   {
       $data_cache = new sfAPCCache();
     
       $data_cache->removePattern("organisational_units**");
     
       $return = parent::delete();
     
       return $return;
   }

Just doing this to the one set of data has increased our page loads speeds dramatically as well as reducing the load on the server itself as well when we do intense performance testing. We hope to employ this further along with other items that similarly load for each page etc and will never change.

Our background in symfony

2009-04-03T01:59:00.000-07:00

PHP is a great language, in my humble opinion, to program in because of its flexibility and pervasiveness. It has its odd quirks, which you get used to, but generally speaking coding in PHP has always been fun for me. One problem that the development world has had is using programming languages to build large and complex applications. While the efforts of Object Oriented Design have done a great deal to help push the mantra of making code re-usable, extensible and maintainable, it can still be a daunting task to build some of those projects out there.

One very important design principle I came across years ago was something called MVC; Model View Controller. Essentially what that entails is instead of bunging all your PHP code (and MVC does not only refer to PHP, its a design concept used in lots of other languages) into one file to represent a page, like database connection, running a query on the database, formatting that data and manipulating it, followed by echo'ed HTML to display that data in tables or whatever format is desired, MVC seeks to seperate all the different parts of a web application to make managing them easier.

Model refers to the actual object classes that describe the database schema your data is stored in. Instead of writing your own SQL queries by hand and hard-coding things like database, table and column names, the model is the intermediary. The model is responsible for connecting to the database, generating a query based on parameters you have passed to it, manipulating that returned data and then sending the end result back to whatever called it ready for use.

View refers the actual presentation on screen that an end user would see. The view doesn't care what the database looks like or even if one exists at all as long as it has the data it needs to create the presentation it is supposed to. It will generate the HTML needed for that data the model extracted to be displayed in a way that makes sense to the user.

Controller is the intermediary. It will take the events generated from the View, such as mouse clicks, page loads, etc, analyse what the view has done, decide what the next step will be, such as load another view or ask the model to return more data and then send that data to another view, etc. The controller can be thought of as the glue that binds the model and views together.

Whew! Ok, enough of that lecture. There is one problem with this seemingly clever seperation of tasks. Coding an MVC framework can be a nightmarish task and the complexity of making an MVC alone work can be more effort than its worth. This is where symfony comes in. Symfony is an already pre-built MVC framework for PHP, and while setting up your own MVC structure would be laborious, symfony's is ready to go and using the framework as opposed to writing your own PHP code from scratch actually makes the job even faster than using no MVC at all.

So why is symfony so great? Well, feel free to try it yourself. Symfony's philosophy is convention over configuration which means that, instead of explictly defining the relationships between classes and database schema, for example, that there is an implied relationship. For example, if you had a table called "sales_history", the model class that deals with interacting with that table is called "SalesHistory". Its a convention, we agree to use it this way. It is only if you decide to not use this convention and name your class "SalesMade" that you need to worry about reconfiguring aspects of your code to do that.

Because of this convention scenario you can do the following steps, after having installed symfony, to have a fully working set of database-agnostic model classes ready to use in your application:

Go to a terminal and enter:

mkdir project_name;
cd project_name;
/path/to/symfony generate:project project_name

Then go to "/path/to/project_name/config/schema.yml" and define your database structure in the easy to use YAML syntax
Go to terminal and enter "symfony propel:build-model;"

And voila! There you have it. You now have classes that match your schema that you can use anywhere in your symfony application.

At Synaq, we have been using symfony for over a year now on a specific, large, and complex project and it has proved invaluable. There has been a lot of learning and experimentation involved in getting to know and use the framework to its best, but the experience has been well worth it seeing how quickly, even with the learning curve, we have been able to produce results.

There is far too much involved with symfony for me to be able to go into great detail here, and I will be giving more information in future on tricks and tips we have learnt while using it. Suffice it to say, if you want to simplify the way you develop large projects, feel free to go give symfony a look.

A non sysadmin trying sysadmin

2009-04-02T00:56:00.001-07:00

Well I couldn't resist waiting till tomorrow but I have over the last few days been attempting something I have never done before. But a wee little bit of background first.

In the past as a PHP developer I have usually used servers to host the work I have done that have already been pre-setup, using a shared/managed hosted service where all the necessary server software (Apache, PHP, MySQL etc) is already up and running. This has served me well in the past because I could then focus on making the applications and leave the server administration to those more qualified.

However, over the last few days, I purchased myself a Linode, a VPS, which allows me to install a Linux distro of my choice as well as then go ahead and install my own applications onto it with the eventual aim of being the web and mail server for my domain (which is currently hosted at an afore-mentioned shared hosting company). Doing this has helped me learn an absolute ton about what goes into setting up a Linux, production web server as well as helping hone my Google search skills even further.

The reasons I decided to do this was simple. I needed a server that would allow me root access to install applications such as Trac, an SVN repository and issue tracking, web based application. Also, I wanted more control over things such as the modules of PHP I can install myself and even which version of PHP as well.

For this little project I chose to use the Ubuntu 8.10 (Intrepid Ibex) server distribution. One reason was simply I am used to using Ubuntu as it is my desktop environment as well, and also because one of my colleagues at Synaq (a systems administrator by profession) suggested it, citing that for my limited needs I needn't worry about LTS versions and so on. Included in my little server are the usual applications:

Apache 2 for webserver
PHP 5.2. I needed 5.2 because I want to use the new Propel 1.3 ORM in my PHP applications and 5.2 is a minimum requirement.
MySQL 5.0 as my primary database application predominantly because I am used to it. I plan to also install and experiment with PostgreSQL, amongst others.
Python, perl and Ruby. All because I plan to play with them too :)
SVN as my code store and code backup
Trac (which runs off Python/perl) to use as my Project management and SVN interface tool for a few projects I'd love to make.

A few of the sites I visited to help teach me how to set all this up, seeing as I had never really done this before were:

http://www.howtoforge.com/perfect-server-ubuntu8.04-lts-p6
https://help.ubuntu.com/8.10/serverguide/C/index.html
https://help.ubuntu.com/community/SubversionInstall
https://help.ubuntu.com/community/Trac
http://trac.edgewall.org/wiki/TracGuide
A whole heap of others I used to help fix small problems here and there that I came up against that the more "official" docs above had not covered.

What I loved about this experience was that I was now able to better appreciate the work the sysadmins here do, as well as be amazed once again how little time I spent picking their brains because of the wealth of knowledge the Internet provides. I love WWW :D

The best part of this whole experience is going to my new server IP address (I haven't tranferred the domain yet, DNS is the next issue I need to tackle) and seeing it all happily load in my browser....

Its all about me.

2009-04-01T23:04:00.000-07:00

Well, no, I am not that pretentious. This blog is all about me and my day-to-day activities related to web development, primarily with that revolutionary, server-side scripting language known as PHP, which started out standing for Personal Home Page and now represents Hypertext Pre-Processor ... pretty much a "hacked" acronym.

But enough about the boring history of PHP (I assume boring as when I discuss it with most people the glazed look is a dead give-away), and more about me. I am Gareth ... Oh, you want a bit more? Alrighty then. I am as of the date of this post a 28 year-old, engaged (should earn me some kudos with the little lady), South African guy, currently employed by Synaq, a company that specialises in providing Managed Linux Services to corporations using Open Source technologies. Pretty much were a bunch of Open Source geeks having a blast playing with really powerful machines that handle millions of processes per day. Well, thats what the System Admin's do at least. I am the geek that makes some of the software my fellow colleagues use and even some of our clients. I am the Web Developer. Or rather a web developer because with my compatriot Scott we are the team of two that try our best to write, hack, squeeze, prod, improve, maintain and otherwise maul code into some semblance of what the company wants. Its a nice job ... keeps me kinda busy .. and we get free coffee .. which is a nice perk..

And I am waffling so ONWARD! The reason I created this blog is simply because in my day-to-day work I often find myself sitting with a Great Discovery in my hands, either conjured alone or with Scott, and no one to share it with. While there are sites I could post this stuff onto it somehow feels more like just chucking stuff at the world in general than it does making a contribution. So, I hope to use this blog to share my web development woes and I have a few ideas I hope to get up and running on here as well. One of which is a complete A-Z tutorial of becoming a PHP web developer. All the way from understanding the client-server chain, to installing a testing server on your machine, choosing an IDE and getting stuck into the coding.

Well enough waffle for today. Tomorrow I shall make my first post (or later if I can't contain myself). Thanks for taking the time to read my first piece of drivel and hope you come back