Prototype It!

So I saw this post today – it’s by Paul Buchheit a former Googler (he is one of the founders of FriendFeed) and the lead developer of one of my all time favorite software applications – GMail (it was his 20% project at Google) and it’s about the concept of Communicating with code.

Paul writes (in his post) on the concept of  using prototypes  to communicate ideas and concepts. He talks about his work with GMail and how he threw together a prototype in order to show the idea of targeted ads in GMail – targeted ads was not a priority until the prototype showed how useful and interesting it could be. The post ends with a similar exercise that he has done using the Friendfeed API (it’s pretty cool – check it out :-)) The reason I read this post and decided to blog about it is that it talks about developer communication – a topic that I wrote about in another post .

Communicating ideas through prototypes is a great idea – I have always noticed that people get more excited about something they can play with and try out. In fact this is old news in other industries – the auto industry, for example, spends millions to make concept cars to introduce new ideas to the public and solicit feedback Architects likewise – build scale models of their ideas to present to clients. So why don’t we adopt these ideas ? After all we are always talking about “software architecture” and “software construction” and other civil engineering analogies when we talk of software development ;-)

My experience is that when the term “prototype” comes up in software development projects most people are thinking of mock-ups. This is especially true in the web-development shops where there is a separate team of graphics designers creating HTML and image pictures of the user interface while a separate team of developers get to “build” the application from the pictures :-) I think this is a very limiting thing.Prototypes should not be limited to the user experience or to presenting and communicating new ideas. I like to make prototypes of my technical solutions to software problems. For example, if you are trying out this great new idea you had on caching data – write a prototype application – the absolute simplest application you can use to exercise your idea.  This would provide you with feedback on whether your idea is valid as well as show up gotchas or limitations in your design.  The pragmatic programmers in their seminal book – The Pragmatic Programmer refereed to these applications as “Tracer bullets”.

Sometimes the concepts or ideas themselves are large and need a lot of programming to even build the prototype(tracer bullet). In these cases I still believe one must prototype the concept – so the question remains – how do you do this ?
Well one way would be to take the approach Paul took in building the initial GMail prototype – modify some existing code. I find sitting in front of a blank file makes the task ahead seem even bigger than it is – so I start with a piece of code to modify even if it is something as simple as a “Hello World” application.  Another good place to look is in the open source forums (though this might be a problem if you are working in a  company that does not allow open source software) – usually there is some variation on your concept that you could work with there :-). A third option is to accumulate code, links and other resources over time that you can use to jump start your coding.

So – Happy Prototyping :-)

Continuous Integration – more than just automated builds

This post is derived from an article by Martin Fowler on Continuous Integration. It my take on Continuous Integration and it’s use.

‘Continuous Integration‘ is a software development practice where members of a team integrate their work frequently; usually each person integrates at least daily – leading to multiple integrations per day. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.

Software development is full of best practices that are talked about at great length and detail but rarely actually done. Continuous integration is one such best practice that has been around under various guises for a long time. Anywhere you have heard or discussed the virtue of having a common build that is done in a regular manner – you are hearing about continuous integration. In fact Microsoft has been doing automated daily builds for years.

The term ‘Continuous Integration (CI)‘  is one that originates in the Agile Programming discipline known as ‘Extreme Programming (XP)‘ as one of its twelve original practices. I believe CI has enough merit that it should be implemented even in projects that are not following XP or other agile development practices.

So how does one go about achieving CI within your software development process?
There are two critical aspects to this effort – evangelizing, educating and mentoring the team in CI practices and creating a repeatable process that allows the team to integrate their code quickly, preferably in an automated fashion.

One of the hardest things to express about CI is how it makes a fundamental shift to the whole development pattern; one that isn’t easy to see if you’ve never worked in an environment that practices it. For many people team development just comes with certain problems that are part of the territory. CI reduces or eliminates these problems, in exchange for a certain amount of discipline. The fundamental benefit of CI is that it removes sessions where people spend time hunting bugs where one person’s work has stepped on someone else’s work without either person realizing what happened. These bugs are hard to find because the problem isn’t in one person’s area, it is in the interaction between two pieces of work. This problem is exacerbated by time. Often integration bugs can be inserted weeks or months before they first manifest themselves. As a result they take a lot of finding. With CI the vast majority of such bugs manifest themselves the same day they were introduced. Furthermore it’s immediately obvious where at least half of the interaction lies. This greatly reduces the scope of the search for the bug. And if you can’t find the bug, you can avoid putting the offending code into the product, so the worst that happens is that you don’t add the feature that also adds the bug. (Of course you may want the feature more than you hate the bug, but at least this way it’s you have the choice.)

Though CI is a practice and so does not need any particular tooling to deploy, it is generally found that automation is a key factor in order to make CI. In order to automate the CI process generally it is recommended that the following is done –

  1. Keep a single place where all the source code lives and where anyone can obtain the current sources from (and previous versions)
  2. Automate the build process so that anyone can use a single command to build the system from the sources
  3. Automate the testing so that you can run a good suite of tests on the system at any time with a single command. (The build does a self test)
  4. Make sure anyone can get a current executable which you are confident is the best executable so far. (The build creates a deployable version of the software)

All of this takes a certain amount of discipline and consistency in the approach. It is difficult to introduce it in a project but once introduced it does not take that much effort to keep it up. So let us examine the points given above in turn.

  1. Keep a single place where all the source code lives and where anyone can obtain the current sources from (and previous versions)
    This is a basic requirement in the software development. Regardless of the size of the team or the complexity of the project all the source code must be available in a single easily accessible location. The versioning requirement means that one must invest in a proper source control system. All the artifacts needed to run the application should, ideally, be located in the source control repository. This would allow the build process to obtain all the resources needed to build the project from a single location.
  2. Automate the build process so that anyone can use a single command to build the system from the sources
    Automated builds are essential to create a repeatable process that can be run on demand. This prevents the occurrence of human error in typically complex systems where in addition to building the source code one must set configuration value, etc in order to produce a deployable application. For complicated projects building the source code itself can be a complicated process involving multiple projects and dependencies. Most major software platforms have build tools that help make these tasks simple.
  3. Automate the testing so that you can run a good suite of tests on the system at any time with a single command. – make your build self testing.
    Traditionally building involves compiling, linking, etc – all the stuff required to make your program run. However simply because your program loads does not mean that it is working correctly. One very effective way to catch bugs is to run a test suite against your program. These test suites should be built so that they can run automatically against the latest build of the software.
  4. Make sure anyone can get a current executable which you are confident is the best executable so far.
    This can be achieved by getting everyone to commit their code to the build often (every day) and ensuring that these commits trigger an automated build every time. This helps ensure that anyone can get the latest workable version of the code at all times – one that passes all the tests. This allows one to be more confident about the code that one is developing will work when merged back in. It also allows testing of the code to be done on the latest version of the code ensuring that bugs being tested are not due to version conflicts or integration issues. It helps testers focus on the changes to the code and identify issues faster as a result. Overall the confidence in the quality of the code is improved.

While CI is a great process – it is like any process only as good as its implementation. There is a abundance of tools out there that do CI or have it as part of their feature set. I will list some that I have dabbled in and heard nice things about –

  • CruiseControl .NET This is a open source tool from ThoughtsWorks .
  • CruiseControl This is the original java version of CruiseControl.NET
  • CI Factory Based in CruiseControl.NET but with a lot of the other agile tools also integrated as well as templates on how to set up your repository, etc :-)
  • Hudson Java tool I have seen nice things written about it’s simplicity.

A good book to read is Continuous Integration: Improving Software Quality and Reducing Risk

Update: A Stack Overflow user was asking about Continuous Integration tools today and I pointed him here – I found another useful link while going through all the answers.

ThoughtsWorks have created a feature matrix for CI tools that seems to have all of the major tools covered – link

Programmer – Express thyself

I was in one of my pondering moods the other day and it struck me that software development has less to do with pure mathematical prowess or engineering skill (though having that is certainly a benefit) and more to do with expressing yourself. This is no coincidence – it is because programming software is nothing but an exercise of making ones wishes understood. Indulge me for a minute here while I put to you my thought process…

Ever since we invented a computer, we have been evolving ever more sophisticated ways of communicating with it. As the computer grew more powerful and capable of understanding more complex thoughts we moved from machine code, to assembly instructions to software languages. We have reached a place now that computers are capable of processing huge amounts of data and instructions, executing extremely complex instructions and passing information between each other as needed. In response to these capabilities computers have been programmed using ever larger programs to solve ever bigger problems.

Programming solutions to today’s problems have gone beyond the capabilities of a single individual. Most software projects today require teams of programmers working in concert. This has lead to an interesting dynamic – developing software in groups. Except, software development has moved beyond geography – the ephemeral nature of software makes it an ideal candidate to be developed in a virtual construct that transcends physical proximity. Almost every non-trivial software program developed today is a collaboration of individuals, many of whom are literally at opposite ends of the earth and may have never met face to face.

This leads to a big problem – software programming is a pretty creative and complex process. Add to that, the fact that your collaborators are probably sleeping while you are coding and vice-versa, pile on the disparities in culture and language and you can see why I am saying that software development is more about expressing oneself. One of the bonuses we have being programmers is that, despite all these differences, we are trying to talk to computers and computers have standard ways of communicating with people. So, it can be argued that as long as we all are coding in the same programming language, and we are using the same tools we should be fine. Unfortunately, coding is a only a part of a software project – requirements, design, marketing, users, clients, quality assurance, maintenance, provisioning, etc – all are factors to be considered. All these aspects may be handled by people who are not programmers but still need to know what is going on. Besides, coding itself is much more than simply writing code, it also involves commenting, code reviews, design decisions, trade-offs and alternative approaches, all of which need to be discussed and communicated.

Given all the interactions that I described above, it is no surprise that programmers spend a significant amount of their time talking and writing about the stuff that are doing and are planning to do. Given this dynamic, in my experience, I have noticed that, successful programmers have learnt to express themselves clearly and concisely. I have seen this so consistently, that I would even go so far as to state that, a programmer that cannot clearly articulate a problem either in writing or in conversation has either not understood the problem or is incapable of solving the problem.

Expressing a thought clearly and concisely is a skill that must be practiced. Personally, I am using this blog as much to exercise this skill as I am for anything else. I would encourage you, dear reader, to give thought to this as well. Spend a few minutes over your next email or your next requirements document, proof reading it and making sure you have conveyed you point clearly and succinctly. Reading is another great way to improve your communication skills – look for books outside your normal fare, look at newspaper articles – I have found them to be great sources, since journalist have specific windows of reader-attention and space to convey their points.

Well, until next time… Programmer – Express thyself !!

The Point of N-Tier Architectures

For the longest time whenever the question of software architectures in web applications came up, the N-Tier architecture was the front runner. And why not? It is flexible ,scalable,very easy to model and more importantly – easy to communicate. In fact it is so common that there are code generation tools, and frameworks galore for it. I like N-Tier architectures – for the reasons I mentioned before – however, I have seen many occasions where this particular approach is used as a panacea for all ills. Every application – regardless of its size or the business problem is given the N-Tier treatment :-)
In most cases this is overkill – especially in departmental intranets that service a finite number of people. Sure, the solution would be scalable and flexible but that comes at a cost – performance – N-Tier architectures – in my experience, are actually slower when compared to single tier architectures – more scalable but slower…
This means that if your web application has an audience that is fairly stable and specific business requirements you would be better off designing a simpler application that meets the need. By lowering the number of objects that need to be instantiated and simplifying the logic needed, a significant improvement in performance, as well as a marked reduction in the complexity of the code could be achieved.
The biggest argument against this approach is of course that of ever changing requirements and scope creep and the need to obviate that by introducing flexibility in the code. The argument is a valid one and compelling one. However, this risk can be mitigated by insisting on making clear the goal(s) of the application. Once that is clear, it is easier to distinguish between requirements that support the goal(s) and ones that change them completely :-) In my experience, there have been very few occasions where major changes in the goal(s) did not require a major re-design of the application – regardless of how modular and layered you make the application. In fact, in these cases it is far easier to refactor or even throw away and start again, if the application was not very complex to begin with.
All this being said – I must point out that N-Tier applications have their place – all I am saying is that using them on every software problem is like using a bulldozer to drive in a nail – you could do it far more easily and a lot cheaper and easier with a hammer …

Yahoo’s Web 2.0 Makeover

Yahoo! has always prided itself as being… well, a Yahoo.
Starting out as a hobby project in the early days when the Internet was still an uncool geek hobby, Yahoo transformed into the most visited website in the world . From its beginnings as a directory for websites Yahoo has grown with the Internet to become a whole on-line ecosystem.
However, with the meteoric rise of Google, Yahoo has competition. It is no longer the coolest website on the block. One of the things Google has been leveraging have been their geek credentials. The company (Google) has positioned itself as a programmer friendly, innovative company and one of the cornerstones of this effort is the opening up of its API.
In response Yahoo has gone on a major makeover effort. It changed its entire look including the Yahoo home page, Yahoo email, Yahoo maps and Yahoo search. It has added new services like Yahoo Music, Yahoo Video and Yahoo 360, and acquired great sites like Flickr and Del.icio.us.
But the biggest thing in my opinion is the Yahoo Developer site. What I really liked about this is not so much the fact that they opened it up as the approach they took with it.
Instead of simply publishing their API and a few examples, the Yahoo development team took the effort to create good documentation, including patterns, best practices etc. They went ahead and added a blog, allowing their developers to post on the site, which gives it the accessible image. Yahoo went the extra mile and that shows through. The latest API they have opened up is Yahoo authentication. Lets see how the developer community responds to this – I can only see more web 2.0 goodness from this development. Let the pretenders to the throne watch out for the Yahoo! is ready…

Error messages in Haiku…

OK it’s not strictly Haiku but you know what I mean :-) –

The Web site you seek
Cannot be located, but
Countless more exist.

Chaos reigns within.
Reflect, repent, and reboot.
Order shall return.

Program aborting:
Close all that you have worked on.
You ask far too much.

Windows NT crashed.
I am the Blue Screen of Death.
No one hears your screams.

Yesterday it worked.
Today it is not working.
Windows is like that.

Your file was so big.
It might be very useful.
But now it is gone.

Stay the patient course.
Of little worth is your ire.
The network is down.

A crash reduces
Your expensive computer
To a simple stone.

Three things are certain:
Death, taxes and lost data.
Guess which has occurred.

You step in the stream,
But the water has moved on.
This page is not here.

Out of memory.
We wish to hold the whole sky,
But we never will.

Having been erased,
The document you’re seeking
Must now be retyped.

Serious error.
All shortcuts have disappeared.
Screen. Mind. Both are blank.

Installing Apache, MySQL and PHP on a Windows machine

A few months back, I resolved to learn an open source langauage. I did some looking around, and sure enough PHP was everywhere on the internet. But everyone talking about PHP, also talked about Apache and MySQL in the same breath. The more partisan amongst them also threw Linux into the mix and called the whole shebang – LAMP. As luck would have it, I found a PHP fan a few cubicles away that raved about PHP.

Now while I was willing to tackle PHP, I was more circumespect about adding Apache and MySQL. I was especially doubtful about Linux because that would involve me either partioning up my computer or buying another, both options I wasnt willing to do. Even if I was to ditch the Linux portion and simply have Apache, MySQL and PHP on Windows (WAMP), I was hesitant over the installation and management of these services.

But then my friend pointed me to the appropriately named WAMP. It is a distributable bundle that installs and configures Apache, MySQL and PHP on Windows as services. It even has a dashboard from which you can start any or all the services as well as modify the key configuration files.

I love it and I highly recommend it to anyone who wishes to install Apache, MySQL and PHP on Windows with a minimum of fuss and bother (It does not even require a reboot).

Sample Job Listing with translation

Do you want to join an exciting new company and make a difference? (We’re a startup, we suck, we need help)

We have an immediate need for a Senior (They have no idea what this is) Architect / Developer / Engineer / DBA / Analyst for a fun (Insane) 3-6 month project (They don’t know how long stuff should take) as a contractor (Because for some reason they don’t see a long term need for this bargain guru)

Must have a Bachelors Degree in Computer Science, have 12 years of professional experience in the IT industry, 5 years experience in our specialized industry, know all the web languages, all the scripting languages, know all of the C++, Alphabet++, Be an expert in Unix, Windows, Linux, and Mac OSX, 5 years as a manager. (Someone who perfectly fits the position that we can’t define, who works for a company that is virtually a clone of ours even thought that would mean they too are bankrupt.)

Salary: $20 – $30 / hour DOE (Depending on your experience you’ll get paid like a trucker. Despite our requirements for a gates / wozniak / jobs in their early days hybrid)

(Maybe we aren’t willing to pay what it will take to get our company out of the ditch. Lets just continue to treat our IT personnel like they are a bunch of high school geeks with no need for compensation because they do this stuff for fun anyways. Besides, our company is built on __Fill_in_the_blank_here___ our web prescence / network / database / email / security isn’t our top priority.)

——————-

You wouldn’t believe how true this is. Of course it has sarcasm and exageration, but people are seriously kidding themselves. IF they ever fill some of these positions they will get someone who is either lying like a politician, or someone who is desperate for some reason. Why would you do this to yourself? Has anyone learned from Googles example yet? Do these people still reject the VCR? I want to run a company someday… one the main purposes being so that I can implement the hiring methodologies that I believe in so firmly…

Office Banter

One of the nice things about working here is the lunch. I go out to lunch everyday, with some of my colleagues in office to different places around the Atlanta midtown area. More than the food I think what attracts us is the rants and raves and the passionate discourse. For I started a thread on my blog about a topic and here is how it went :-

Me (starting a topic):
Everyone has heard of the GNU Public License which is used to keep software free of the greedy corporates and their patents.
I read in an article in Slash-dot – a couple of guys have created a license that makes their software free for use except for military use. Do you think this violates the spirit of “free” software?

Riz (the contrary one in the bunch):
Uhhm, OK let me be the first to say that I am not comfortable about discussing my brass tacks. Don’t get me wrong, my brass tacks have/are served me extremely well, but I don’t want to discuss this with you guys.

Anthony:
Riz: I don’t know what you own that you refer to as brass tacks, or why you are uncomfortable talking about them. But it seems like you should be more embarrassed at the fact that you have this awkward nickname for such an unsaid set of objects.

Rishi (bringing the discussion back into focus):
What happened to the discussion about GNU public license and the software talk?
Riz always has to bring up something that is totally out of context and like Bilal would say “gayish”.

Anthony:
As for the article, I like it. I believe in open source / free… it’s almost like volunteer or “Community work” for geeks. You want to advance technology etc. in a public interest… but you don’t want to volunteer for a destructive purpose. Kind of like running a Non-Profit for the Mob, or something to that degree. I understand the concept and the thought behind it, the problem is the precedent. If you want to generalize it and say, “I want this to be free to everyone except for those whose cause or belief I don’t stand behind” then everything would be too complicated and eventually go back to licensing for specific interests… usually profit. I think a better approach is to say that even if the military / government or anyone else without exception makes a modification to the code they need to post the modifications to it and without doing so they violate the license. This would either force the military not to use it for legal purposes (if their government cares) or the military will contribute to the project as well. We can’t forget that there have been a lot of practical inventions born from the military.

Me:
What I find particularly interesting is the way they have phrased their modification to the license “the program and its derivative work will neither be modified or executed to harm any human being nor through inaction permit any human being to be harmed.”. They have said they picked it up from Isaac Asimov’s – Three Laws of Robotics – “A robot may not harm a human being, or, through inaction, allow a human being to come to harm.”

Anthony:
Even that is flawed, and easily tainted by today’s standards. “I wasn’t harming, I was defending” or “There would have been more deaths due to inaction so action was necessary” all of this is easily spun into politics, we need to try and abstract the technologies from politics… the open source licenses were designed in the spirit of advancing technology as a community, it should stay that way… it’s inevitable that some of it will be used maliciously, sometimes to an extreme, but if the intent is there, they would have accomplished the goal regardless of a license.
Can you imagine this? “The US Government is being sued today for breaking the licensing model of an open source application they installed into their upgrade to the cruise missile navigational system. The case has been put on hold indefinitely for reasons of national security.” Good luck with that… I call this philosophy “Geeks without borders” :)

Riz:
THE 3 LAWS, by The Riz (all bow your heads)
Law 1: The law of paper bag: Just bag it!
Law 2: The law of light: Beauty is a light switch away!
Law 3: The law of boos: When in doubt drink more, she’ll get prettier.