On Distributed Development

 So, I have been doing distributed development at Bang the Table, for the past year. It’s a company with a very strong work from home culture. We even had a virtual birthday party for one of our directors over Skype video chat –

Birthday party at Bang the Table

Telecommuting / working remotely / working from home – these are various terms for an increasingly popular mode of working. In this model (of working) people use the amazing ability of the internet to communicate over vast distances instantly and modern collaborative software tools to effectively work together without actually being in the same physical room, office or even city.

There are some pretty powerful advantages to such an arrangement –

1. No need for any commuting, one can stay close to one’s family (a huge factor if you have children) and flexible working hours. It is a more relaxed way to work. I have been working in such an environment for about a year now, and with a new baby, the lack of commute and the freedom to pop downstairs and help out in case my wife needs an extra pair of hands has been awesome!
2. Another major advantage (at least for the company), is the reduced office infrastructure costs. In places where getting an office building thats a reasonably close to where your employees live is difficult, this can be a major boon. Also, not forcing your employees into having to commute is a major plus both for employee morale and the environment.
3. One has the opportunity to hire and work with people from a larger pool rather than being limited by a single geographical location, like a city or state or even a country. This can be very powerful advantage, sometimes finding all the right people you need in a single location can be a major problem.
4. An “advantage” that is often touted is that the geographical separation allows for better efficiency, since, theoretically, one could have employees coming online and starting their workday as others end theirs – making use of the entire 24 hours on the clock for work. I believe, at least for software development, this is a fallacy and attempts to leverage this so called “advantage” is a big reason for the failure of certain companies to successfully leverage this model of working – but more on this later…

In the software development field we have been working in such a distributed model for a long while now and I have seen this model evolve, influenced both by the advances in the technology and tools available for collaboration as well as the evolution of the software development methodology itself. I’d like to take a few lines to write about this evolution and distinguish between what I consider to be “distributed development” model and what I call “outsourced development”.

Image via Wikipedia

In the outsourced development model  – a software project is broken up into separate well defined phases, each phase having definite outcomes with concrete artifacts (usually in the form of documents). This means that (in theory) one can easily move around parts of a project between various teams who would then rely on the artifacts produced in the previous phase to execute the next phase of the project. Other than a short period between phases, where one team handed over the project to the other, there would be very little actual interaction between teams. This form of software development follows the waterfall methodology and while it is a good fit for outsourcing, it is, in practice, prone to problems, that, for the most part arise due to a lack of communication between the various stakeholders of the project and the members of the team(s).

What I call “distributed development” (there is a more generic definition on wikipedia that is not limited to software development) is a refinement on the outsourced development approach that attempts to leverage technological advances in internet connectivity and better communication tools to try and minimise the disconnect introduced by working remotely in software teams. Distributed development places emphasis on communication and interaction between all the members in the virtual team. The idea is to reduce the feeling that we are working in different physical spaces and that everyone is off working on some discrete task on their own. This kind of thinking is in-line with the agile philosophy which emphasizes individuals and interactions over processes and tools.

At Bang the Table, we get pretty close to what I consider to be an ideal distributed development model and I’ve come up with a few guidelines are a result of my experiences working here and from what I learned listening to a Hanselminutes podcast about working remotely –

1. One can use many tools and technologies to do your job. But regardless of what you use, patience and trust are the biggest success factors in this kind of working. I initially put this point lower down in the list but on consideration decided that this is the most important tip. Without trust the whole thing is an exercise in futility. One needs to learn to trust in the skills of the developer to do his or her bit and the patience to listen and accept that problems and delays might happen. The best way to build trust is by having open and rich channels of communication and meeting and interacting with everyone in person.
2. Reliable and fast internet is an absolute must and its not that easy to get – at least where I come from. Ask if the person you are working has a backup power supply and a good internet connection. In terms of power supply, the need for one depends a lot on the place the person is working from – if it’s a big city then usually a UPS that provides an hour of backup should be fine, unless there is chronic power problem. In smaller towns and such, they might need to have something more substantial like an inverter or a portable generator.  Modern laptops with their longer battery life and portability are a major help since they allow one to ignore smaller outages and coupled with the appropriate wireless or cellular data connection can even allow you to simply shift your base of operations to other places with power and internet.
3. This guideline, I think is going to be a bit controversial and is the reason I think that 24 hours work day idea I have mentioned is not really a good idea. If you have a team that spans countries and (in my case) continents, it is very important to have good overlap between your timezones – at least 4 hours of overlap is needed in my opinion, since this will allow you to actually collaborate rather than be limited to a short meeting at the beginning or end of your day. After all, developing software in a team is, in my experience, a collaborative activity, requiring lots of discussion (and argument)  and having your stakeholder at hand while you work through a problem is very useful. No amount of documentation and pretty pictures can substitute someone  being there, clarifying and providing context for you.
4. You need to have an open channel of communication during the time of overlap while you are working – this can be as simple as an open IRC (Internet Relay Chat) channel. We use Skype at work and I think the ability to call or video chat on demand make it far superior to IRC. Another option to explore is to use something like a Google+ Hangout while you are working – obviously this will mean the both you need to have reliable high bandwidth internet. Paid options for software tools are there as well like Microsoft Lync which may be a more integrated solution if you have a Microsoft based IT infrastructure with Microsoft Exchange (I got this from the podcast – haven’t used this myself since we don’t use MS Exchange). There are variety of communicators and tools out there to tryout – have a look at this link where Scott provides a deep dive into the communications tools available for remote workers.
5. Schedule regular meetings and get togethers in the real world. If this means that someone needs to travel somewhere from time to time, consider that the cost of doing distributed development. I would consider this an investment in the team’s productivity and efficiency. When everyone has met in person and worked together on issues, you build up mutual trust and respect. In my year of working at Bang the Table,  I think we have had get togethers at least once every quarter. Sometimes we just got together and spent a few days simply working together with no other agenda, sometimes it was a conference or a training.
6. Distributed development tends to work well with relatively flat and simple organizational structure. The idea is that everyone should feel equally invested and responsible for the software project. The challenge of course is that this particular setup does not scale very well and can be problematic for large organizations.
7. A good online project management tool is a must to ensure you are heading in the right direction. We are constantly trying out new tools for this part of our development. I have found JIRA to be good for support and task based work while, I am liking Trello for more open ended new development.
8. At work we use a Rackspace server as a development server and for testing. This is a machine on which we stage our commits from our local machines and have our testing done on. Having a machine to deploy your code on also forces you to test it in the right environment and also brings deployment considerations into your design and development schedule.

It’s been a fascinating experience for me – working from home and I think I have learnt a lot about the vagaries of distributed development. In fact, I have been experimenting with a more distributed approach to pair programming with a couple of friends of mine.

We used Vim in a GNU Screen session to set up a development environment that we could share between us and then while one person typed we used Skype to chat with each other. We followed the classic test driven mechanic – one person writes a test and the next person passes the test cycle. The experience was remarkably powerful, since I could see when the editor was being manipulated by the other person, the thought process going on as the code was being typed in. We built a small Python program – I came from that experience with a real appreciation of what can be done with some of the simple (and powerful) unix tools already out there.

I came across another tool Tmux which is similar to screen and is used a lot in the Mac when I was listening to this podcast on the Changelog. There is a very nice screencast showing how to use Tmux I found recently, you can checkout.  I also came across this awesome website called pair.io that spins up a server for you to use in your pair programming sessions.

Related articles

• Hanselman Blog – Remote Work
• Hanselminutes – The Plight of the Remote Worker with Pete Brown
• The Changelog – Tmux with Brian Hogan and Josh Clayton
• Pair Programming with Tmux – a screencast
• Bang the Table
• On working remotely
• Wemux – Tmux enhancement for remote pairing

Open source writing using Git

Of late I have noticed a trend – writers, especially technical writers, are embracing the software development process and incorporating it into their writing process. There’s a great podcast about this on Herding Code where some technical writers talk about this trend.

Writers are embracing the agile philosophy of software development, one of the core tenets of this is to publish often and get feedback so you can improve/adjust. Authors are posting the content for their book projects online to get feedback – to the extent that publishers like Manning and O’Reilly offer programs that let people access to the book content as soon as the author creates it, for a discounted amount. The pioneers of this practice were the Pragmatic Programmers who offer “beta” versions of their books to customers at discounted prices. The idea behind this of course is that it lets authors get valuable feedback as to what is important, what to focus on. It also provides, authors with an incentive to keep from procrastinating about their writing.

Another major aspect of this trend is the embrace of source control in the writing process. I think this is a great development, and it makes a lot of sense if you think about it.  The book writing process, like software development, is subject to a lot of chopping and changing as the book evolves. Also writing collaboratively on something is I imagine similar to writing code in a team and probably have the same issues –  one needs ways for multiple people to be able to work on the same document without worrying if they are overwriting someone else’s hard work. Finally, a version control provides you with a fine-grained backup that allows you to rewind and replay the evolution of the book, which should be invaluable during the editing process. If there is a conflict somewhere, version control also offers tools to detect and fix it in a safe manner.

Some authors have posted their content on Github, which I think is an awesome idea.    Leveraging, the capabilities of a version control, especially a powerful and distributed one like Git and Github, is well suited for writing projects. Authors can leverage the tools  and capabilities of Github  –

• On Github, feedback can easily be provided in the form of  commit notes, line notes, issues – heck you can even fork the project make the corrections and submit a pull request back to the author.
• You can use Git as a fine grained distributed backup for your writing project that allows you to rollback specific portions of you project independently.
• You can use the social features of Github to collaborate on the project and even drum up interest by distributing the link online.
• You can maintain your code samples and other resources separately from your content and correct and upgrade them as needed. This can be very useful if the book is about specific technical frameworks that get upgraded after the book is completed and published.
• Github even has a basic editor so you can work on your project online

While browsing through Github for examples of books that use Github as a version control system for writers – I came across this tool which is  in my mind a good first step, but I think there is a lot more that we can do to improve this experience. There are a whole lot of tools being built around this concept which is quite encouraging.

Writing and programming, have a lot in common. Both involve text, both are creative activities and both are ways to express ideas (one to a machine and the other to an audience) . There is scope for a lot of cross pollination of ideas between these fields. I think, currently the main challenge is a lack of knowledge and a fear of the technical nature of a tool like Git.

Related links

• Git Scribe
• On writing technical books
• Arcturo ebooks on Github
• Mordern Perl book on Github
• Backbone fundamentals on Github
• Node Beginner on Github

A Ruby on Rails glossary.

So it’s been about a year since I started working on the Ruby language in the context of the Rails framework. It’s been awesome,  the combination of Ruby’s developer friendly language style and the incredible productivity of the Rails framework makes this a killer framework for web applications. Building the basic code is simply a matter of typing in some commands that generate all the scaffolding of code needed for a basic website. The rest is usually a matter of tweaking and adding to the generated scaffold.

Image by dullhunk via Flickr

Like every web technology the Ruby on Rails framework has its own set of idioms, that one needs to get familiar with and since I am new to Ruby, for me this includes some Ruby idioms as well. I saw a post today that lists a glossary of Rails terms and I thought I’d do my own. So here goes –

1. Application Servers – Thin, Mongrel, Passenger, Unicorn, Pow – These are application servers that run the Ruby web application and respond to the requests. They are generally integrated with a web-server like Apache or Nginx, though Mongrel is technically capable of running on its own as a web-server (though since it is not multi-threaded this configuration is not useful for anything except for very light loads or on a development machine).
2. Active Record – Active record is the ORM tool used in the Rails framework to abstract the database handling. It is an implementation of the active record architectural pattern described by Martin Fowler thus – “An object that wraps a row in a database table or view, encapsulates the database access, and adds domain logic on that data.” The Active Record code is available as a separate gem that is downloaded as a dependency to Rails.
3. Active Resource – A simple way to think of active resource is that it’s like active record but for REST based web-services. Active record provides a wrapper around REST actions providing a transparent proxy between a business entity and REST based web-services.
4. Active Support – This is a set of utility classes and standard library extensions that are useful in Rails application development. It includes classes for dealing with caching, dates, time zones, text, etc. Like active record and active resource, active support is available as a separate gem that is installed as a dependency of the Rails gem.
5. Bundler – Bundler manages the gem dependencies of a ruby application in a repeatable way across multiple machines. It automatically looks for and installs / updates all the gems needed by the application when it is deployed on a machine. It can do this is a repeatable way across multiple machines and deployments.
6. Capistrano – Capistrano is a tool for executing command on groups of servers over SSH. It is used mainly in deploying web applications and is capable recognizing different types of server environment (like staging and production, etc).
7. Capybara – This is an advanced version of the Webrat tool. It lets you do acceptance testing of applications using various drivers (Selenium and Rack are built-in) by simulating a real user hitting your web application.
8. CoffeeScript – Coffee-script is a little language that compiles into java-script. It has rapidly become very popular and has been included in Rails 3.1 as the default method of writing JavaScript code in a Rails application. I am personally a bit on the fence here but I like its terse syntax.
9. Cucumber – Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable DSL and serves as documentation, automated tests and development-aid – all rolled into one format. Cucumber along with RSpec are main tools used for the BDD methodology favored in many Rails projects.
10. DSL(Domain Specific Language) – A domain-specific language is a language dedicated to a particular domain or the representation of a particular problem or solution. DSL’s are more common than you think, some good examples in the Ruby being – Cucumber, Rake, Haml etc.
11. ERB (Embedded Ruby) – ERB is a simple and powerful templating system for Ruby. It is part of the Ruby core and it allows one to place ruby within other files (any text document).
12. FactoryGirl – This is a gem that can be used for creating object factories in an automated fashion. This is useful in testing or even for populating an application when deployed
13. Gem – At its most basic form, a Ruby gem is a package. It has the necessary files and information for being installed on the system. There is an online repository for Ruby gems at http://rubygems.org/ For me this is the biggest find and literally the “hidden gem” of my Ruby learning experience has been – Ruby Gems.  It’s like having an app-store for your development needs, where most of the best gems are freely available to you!  The number of awesome, useful gems actually available to ruby developers means that we have a huge advantage over other frameworks. A testament to the power of Ruby gems has been that Microsoft has started its own package management library – Nuget which, seems to be simply a straight lift of the Ruby Gems concept, except of course they have incorporated it into the Visual Studio as an extension. Nuget was released by the same team that created the Microsoft version of Rails – ASP.Net MVC so you can see the Rails influence…
14. Gherkin – Gherkin is a DSL  that emerged from the Cucumber project. It was designed to succinctly describe behavior in a form that can be acted upon by a tool like Cucumber. It is described as business readable DSL and can be used in any project that has need of such a tool. The lexer and parser for the language has been create and the library is available in several frameworks including Ruby, JVM, Javascript (Node.js) and as a .NET dll.
15. Git – Git is a free and open source, distributed version control system. It was written by Linus Torvalds to replace the old source control system for Linux and has become one of the tools of choice for a lot of Ruby projects. Git is typically used along with Github which is an online source control repository that leverages Git.
16. Haml – Haml is an HTML abstraction language. Haml functions as a replacement for inline page templating systems such as PHP, ASP, and ERB, the templating language used in most Ruby on Rails applications.
17. Heroku – Heroku is a cloud based platform as a service (PaaS) which was one of the first to have Ruby and Rails support. One could build a Rails application and directly deploy it on Heroku without having to set up a web-server. The platform has evolved to support multiple languages including Java, Scala, Node.js, Clojure and Python.
18. jQuery – jQuery has become the JavaScript framework of choice for web developers everywhere. It is a fast, framework that leverages CSS style selectors to allow developers to traverse, animate and handle events in the HTML DOM while abstracting browser specific quirks as far as possible. It has become so popular that it displaces Prototype as the default JavaScript framework used in Rails 3 applications.
19. Memcached – A distributed open source caching system, memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering. It is a generic store and can be used in any situation where caching is needed and is available in multiple languages and technologies. It is used in a lot of Rails applications even though today, other object stores like Redis are getting popular.
20. Merb – Like Ruby on Rails, Merb is an MVC framework. But, it a lighter and more barebones version. I haven’t used Merb – I’d love it if someone could give a more apt description of what Merb is and how it compares to Rails and Sinatra.
21. Open classes / Monkey Patching – Open classes is a feature of the Ruby language where you can simply open and add methods to existing classes. Active support library uses this feature to add a lot of its helper functions on the Ruby core classes like string. A popular name for this particular practice is “Monkey Patching”.
22. Paperclip – Its a complete (and easy) file attachment library for active record. It provides validation features based on the properties of the file like size, type, etc. The files themselves are saved in the filesystem. It has some basic image manipulation features as well, like creating thumbnails etc. There is a dependency on the ImageMagick library (an awesome image manipulation library) in order to use Paperclip.
23. Prototype – Prototype is a JavaScript Framework that aims to ease development of dynamic web applications. It was originally the default JavaScript framework bundled with Rails.
24. Rack –  Rack is a gem that provides a simple interface between a web-server and the Ruby web framework. Rack is used in various Ruby web frameworks, like Rails, Sinatra and Merb.
25. Rake – Rake is the ruby version of make but much more developer friendly. It provides capabilities that are similar to make but it uses the Ruby syntax instead of XML files or makefile syntax to define tasks.
26. REST(Representational State Transfer) – REST is a set of principles that define how Web standards, such as HTTP and URIs, are supposed to be used. Well, actually REST as a concept is more abstract than that – its architectural style that can be applied to distributed systems. REST in the context of Rails, mainly is used to refer to the way Rails exploits REST concepts in the way that Rails web applications are structured.
27. RJS(Ruby JavaScript) – It is a ruby helper used to generate JavaScript that are used to decorate HTML pages. It is used mainly in AJAX calls, and in situations where you wanted to conditionally emit JavaScript. The general opinion though was that it sucked and was avoided in favor of the unobtrusive style of jQuery.
28. Routing – Routing is the aspect of the Rails web application that deals with the translation of a URL request to a specific action in the application. In Rails, it is the job of the router to recognize URLs and dispatch them to a controller’s action. It can also generate paths and URLs, avoiding the need for hardcoded strings in your views.
29. RSpec – RSpec is a Behaviour-Driven Development tool for Ruby programmers. RSpec helps you do the TDD part of that equation, focusing on the documentation and design aspects of TDD. Like Cucumber RSpec uses a DSL to simplify design and implementation tests.
30. Ruby Debug – Just like it says on the tin – it is a debugger for ruby applications. Like the other tools it is gem that can be installed and can run your ruby application in debug mode.
31. RVM(Ruby Version Manager) – This is an awesome tool from Wayne Seguin that is used to install, manage and work with multiple versions of the Ruby programming language on your machine. It even allows you to segregate your gems so you could work with multiple versions of the same gem for different applications.
32. SASS & SCSS – These are CSS abstraction languages (I believe SCSS is a reference to the version that supports CSS3). They are compiled down to CSS. They allow you use variables, modules and include other files. I am not very familiar with them and would welcome more information from readers.
33. Sinatra – From the website – “Sinatra is a DSL for quickly creating web applications in Ruby with minimal effort.” Like Merb, Sinatra is a minimalist web framework. Again, like Merb, I haven’t used Sinatra at all and would welcome some comments to flesh out this entry.
34. Test::Unit – This is the default testing library available with Ruby. It is from the xUnit family of testing frameworks and is a part of the core Ruby libraries. It is mostly not used in favor of more powerful BDD style tools like RSpec.
35. UJS(Unobtrusive Javascript)– “Unobtrusive JavaScript” is a general approach to the use of JavaScript in web pages. The general principles of this approach includes –
    1. Separation of behavior from the markup – The HTML in a document is meant to describe the structure of the document and should not contain programmatic elements in it. This is similar to the principle of separation of the structure from the presentation of the document when talking about the separation of CSS and HTML.
    2. Should not pollute the global namespace – UJS code should add as little as possible to the global object or global namespace of the environment in which it runs.
    3. Degrade gracefully – If the browser rendering the document does not have the capability needed the code should be able to degrade gracefully without breaking the rest of the application.
36. Vim – IMO Vim, the text editor that just celebrated its 20th anniversary in 2011 has experienced a resurgence of sorts with the rise of Ruby and Rails. In fact there is a large number of Vim plugins that have been created to make developers even more productive and happy on this text editor. Indeed after Textmate (the other favored tool for Rails developers) I think Vim has the biggest following among Rails developers and the fact that its available even on Linux and Windows only makes it cooler…
37. Web Servers – Apache, nginx, Lighthttpd , Thin – In the Rails world web servers options include Apache, nginx and lighty(lighthttpd) of which Apache is by far the most popular. I have included Thin in this list as well since it includes both a web-server and an application server. With advent of PaaS providers like Heroku, this particular aspect of web application development is becoming abstracted out, just as the database is getting abstracted on the other end.
38. Webrat – This is a headless browser that is used in web application testing. It lets you run in browser test without the associated overhead of actually instantiating a browser. This does not execute any JavaScript but is great for doing tests of simple websites without much JS interactions.
39. Will_Paginate – This is the most popular pagination library out there for Ruby on Rails. It alos supports other Ruby web frameworks like Sinatra and Merb. This gem has a wide variety of formatting options and can accept an active record object or even a simple array. It is quite configurable and can be easily extended to do AJAX pagination as well.

So, that’s all the terms that I can think of so far, I’ll try to add to this as and when I come across more terms. Nowadays, web development is more than a single framework, even one like Rails.  I have found the Rails community is more open and embracing of different ideas and frameworks – even to the extent of moving away from Rails and using lightweight frameworks like Sinatra. IMO that makes this glossary all the more relevant and fun.

As I have mentioned in the article I don’t profess to be an expert about or have even used all the tools and technologies listed above. I would love to get your comments and suggestions on what could be added.

And last but not the least – I wish you all an awesome 2012 – may all your efforts bear fruit this year.

Update (Jan 12 – 2012) : Since there has been some interest in this post, I have added a few more entries and provide more information where I can – stay tuned :-)

The loneliness of the software tester

Sometime back my favorite podcast Radiolab – released a short called “The Loneliness of the Goalkeeper” . The short was a re-broadcast of an english show on football (soccer) and the role of the goalkeeper in the game.

In soccer the goalkeeper plays a truly unique role – the opposite to the role of the remaining 10 players in the team and indeed one could argue the aim of a game of football. The goal keeper has to defend the net and prevent a goal being scored.

Of course, the other players in his team also help him in defending the net, but there are few differences that make the goalkeepers stand apart. The primary one, of course, is that he is allowed to use any part of his body to handle the ball during the play in a restricted area in front of the net – the penalty area. If the ball enters the net – even if it is through a mistaken strike, an ill-conceived pass or fumble from him or his own team – the goal counts against the team…

This means that a goalkeeper rarely ventures outside the penalty area – he is the last line of defense to prevent the ball from getting into the net. This is the loneliness of the goalkeeper – he is ultimately responsible for any goal and must protect the net from all comers – friend or foe…

Image via Wikipedia

The narrator of the piece goes on exploring the mental make-up of goalkeepers – the strange almost contrary nature that is needed to be a great goalkeeper. There is a subtle psychological difference in the mindset, brought on by the role of the goalkeeper. A goalkeeper does not have a single moment of triumph, there are only moments of disaster…  A goalkeeper’s mistakes are obvious and public while his success is not. After all – a goalkeeper who manages to prevent the opposition from scoring the entire game has simply done his job while a striker that scores that final goal takes his team to victory or saves them from defeat – is the hero!

As I listened to the podcast, I was struck by the similarities between the role of the goalkeeper and the role of a software tester. Unlike the rest of the team, the goalkeeper’s motive is to prevent a goal being scored – similarly the aim of a software tester is to find defects in code that is created by the team. In a way the software testing role requires an almost opposite mindset to that of a software developer, who wants to build a working product.

It is preached everywhere that a good programmer tests his code. Indeed nowadays it is a given that software should be written test first and the agile philosophy embraces this as a core tenet. I think the concept of TDD is brilliant and that it promotes good coding habits. But I think in all the hype there is one point that has for a long time not been emphasized and that is the writing code in the test driven style does not mean that your code will be defect free.

I have seen a lot of people claim that they write their code test first as if that somehow means that they don’t need to test it. This assumption then leads to an impression that all one needs to do for getting defect free code is to embrace the test driven development style. However what people don’t consider is that when a developer sits down to write code , he does so in a certain context. Code does not exist in a vacuum, it needs to interact with other code and in many cases the code itself might be a small section of a module of a feature in a platform.

The tests referred to in TDD are simply the ones that validate the requirements for the code that you are writing at that point of time. These are unit tests and do not test the software product – only the code being written is to be tested. Writing tests for even that narrow scope requires a mental shift. Programmers who are used to simply writing code (and I count myself as one this group) find this mental shift a pretty big hurdle. It is really hard to force yourself to sit down and think of the requirement and then do the coding and writing the tests needed to validate it (especially if you have a deadline or worse a manager breathing down your neck). And after you do all that, you find yourself having to mock the environment around your code and tests to get them to run consistently (or at all). Sometimes it is simply not feasible/cost-effective or even necessary, to write tests that cover every error that your code might meet. To tie this back to the football analogy, I would consider that software developers can make at the most –  defenders of the goal. They are restricted both by their role and by their mindset from being able to truly subject software to the test.

In contrast, a software tester approaches things from a totally different perspective. The role of the software tester is to find defects – it is his goal and his responsibility is to the quality of the software product as a whole. Testers approach software from a product perspective and not a code perspective. They look at the entire product, its various layers and the seams in the code, where the different features come together – and then look for defects. They write code (in fact I think good software testers need to write as much code as developers) to probe the product from various angles. Software testers need to test the various modules, individually and in combination and using different types of input.

Testing is a matter of diligence – the sheer focus needed to go through all the combinatorial options for all the inputs and apis available in a typical software product requires an intense mental focus and very tidy and organized mind. It is this mental makeup that make great software testers. It is not easy to find people with such a unique mindset – in fact I was reading the other day of a startup company that hires autistic people for software testing. Traits that make great software testers – intense focus, comfort with repetition, memory for detail – also happen to be characteristics of autism.

The role of a software tester is a lonely one – it is not something that will endear you with developers – after all you are pointing out defects in the code they wrote. If the software product fails the responsibility of failing to find the defect falls on the software tester. If the software tester does his job well and all the defects in a software product are fixed – he has merely done his job. Indeed if he does his job too well, and there are no defects to fix people will start wondering if there is a need for the role at all…

Like the goalkeeper – the software tester has a lonely job, but a vital one – he is the last line of defense – the responsibility of the overall quality of the software product lies with him. Software developers may have written the code for a killer feature but it is the software tester that ensures that it enhances the software product instead of breaking it….

Related articles

• Radio Lab Presents: The Loneliness of the Goalkeeper
• U.S. firm harnesses talents of young adults with autism (ctv.ca)
• Startup Company Succeeds at Hiring Autistic Adults to Test Software (seattlepi.com)

Needing an IDE should be considered a language smell

I’m a long time Visual Studio .Net (VS.Net) user – I have worked on the Microsoft .Net platform for a long time and have used the VS.Net tool during that time as my primary integrated development environment (IDE). It’s an awesome tool that has got better as the .Net platform progressed and matured with awesome features like code-completion (one of the first IDEs that had this), interactive debugging, refactoring and one of the most comprehensive documentation tools (MSDN) in the business.

What was frustrating, however, was that to write a simple program, I needed to fire up VS.Net (which is this huge beast of a program that used a lot of resources). On the other hand writing your code in Notepad and then compiling and linking from the command line while interesting was, in my opinion, not practical. I found myself looking for the code completion features and searching Google for C# keywords and .Net API syntax. Using Notepad seemed too difficult so I tried to compromise and I started looking at other “lighter” IDEs like Sharp Develop, Komodo Edit and Mono Develop to deal with this problem. But that meant having to learn my way around the new IDE with different keyboard shortcuts and user interfaces. The mental switching needed to work on another IDE was too high a barrier and despite several half-hearted tries I always found myself going back to VS.Net. The massive footprint of VS.Net was something I decided to ignore to avail myself of the many tools that made coding in C# easy.

Recently, I changed jobs and suddenly I was on Linux and programming in Ruby. I was apprehensive and I searched hard to find a replacement for VS.Net. I found Eclipse to be a bit too bulky and heavy, it was too much like VS.Net – resource hungry :-) Net Beans looked promising but then its community decided that Ruby and Rails support was not a priority which disqualified it for me. After some more looking I concluded that the Linux way seemed to be to use general purpose text editors like Vim and Emacs and customize them to ones specific needs.

Faced with coding without –  “gasp” – a proper enterprise IDE and having to learn Vim or Emacs – I decided to start of with a simpler text editor first. After looking around a bit more I settled on Red Car which is a JRuby open source editor that was copying TextMate (which I had heard was the editor that all the hip Ruby people used) and resigned myself to a drop in my productivity…

Surprisingly though this did not turn out to be the case. I was certainly writing less code but amazingly I was still more productive in Ruby and Rails than I was in C# and ASP.NET in terms of the amount of work done. Despite the lack of an IDE, I found the Ruby syntax to be concise and extremely expressive (I have a separate post on what I love about Ruby) which made it easy to code without having to resort to code completion or similar niceties. More importantly I found reading and understanding code in Ruby to be vastly easier than C# and I found myself not needing the features provided by VS.Net to get a mental model of the code base.

The more I worked with Ruby, the more I liked this way of working and I had a personal epiphany about programming language design. Some languages like C# or VB.Net, forced you to have a IDE just to be able to code in them, while others like Ruby let you simply code. This is because coding in a language like C# involves a lot of ceremony – you needed to write a lot of stuff to help the compiler understand the code – while Ruby tries to get out of your way and just let you code. The terseness and expressiveness of languages like Ruby allow programmers to easily keep larger amounts of functionality in their head.

This particular characteristic of languages should be recognized as a smell – just like a code smell… If you find yourself firing up an IDE as the first step, when you want to write or read a program in a particular language, that is a language smell. It indicates that the language is probably not at the right level of expressiveness for the problem it is trying to solve. To be clear, this particular postulate should be considered in terms of the language design and of course should not be the only way to choose a language. But, it certainly should be an important consideration, IMO,  especially since the language you choose will be something that you will be something you are going to spend most of your waking hours using and more importantly trying to understand.

I think if you come in at the start of your day and it takes you less time to load your code, read and understand what you have written and get into your coding zone, you are definitely going to be more productive, than if you have to fire up a program, wait for a while for it to load and then jump around several files to get your mental model back in your head to start coding. And if you have chosen the language you are using wisely, you will be more productive and in my case a lot happier as well.

Thoughts on Ruby – I think it’s great…

So it’s the middle of the year and like last year I am in the middle of a new language. This time however there are some differences from last year.

The biggest one is that this year I am actually working on the language – Ruby – while I am learning it and I think this makes a big difference. Though I know that one can learn a language using sample exercises and Code Katas – somehow I think the experience of working with a language everyday – gives one a real appreciation of what it is like to live with a language. When you are learning a language by choice – in your own time – you get to choose the problems and the Katas. When you have to work with the language and you use it daily on problems thrust upon you by seemingly unreasonable and sometime crazy requirements – you get a better understanding of the warts and wrinkles of the language along with the pretty bits.

I actually had a cursory look at Ruby sometime back, I had downloaded the language and looked at a tutorial on the web. I solved a couple of basic Project Euler questions with it. I remember thinking at the time – “Hmmmm, not really all that different from Python…” and deciding that learning it was not going to get me much. I was looking to exploring languages like F# and Erlang at the time, since they were based on different ways of solving programming problems.

So, when I started on this Ruby project this year, I came with certain preconceptions – but then something happened that changed how I approached Ruby completely. I read the online book – Why’s (Poignant) Guide to Ruby and I realized that the way I had coded Ruby thus far was not the way Ruby was intended to be coded. Before I go a little further I think I will digress a little give my impression of Why’s (Poignant) Guide to Ruby…

This is not the first time I had heard of Why and his book.

Image via Wikipedia

Indeed one would be hard pressed indeed to spend any time in the Ruby community and not have heard about Why the lucky stiff, his book and his sudden disappearance from the internet (another digression – I really am amazed at the fact this person was able to keep his virtual life so compartmentalized from his non-virtual one that he was able to disappear so completely!). My first impression after reading the first few pages was – Wow!! This guy must have a good collection of hallucinogens and Pink Floyd albums – to create those weird cartoons…

Image via Wikipedia

But, happily I persevered and was rewarded with a real appreciation of the beauty of Ruby. This book is not only a guide to the Ruby language – it is a guide to the Ruby philosophy – The Ruby Way. I would definitely recommend this book to all developers that are looking to learn Ruby, especially if you are coming from a background of working on the “big iron” languages like C# – this book will provide that mental shift in the way you code, that will let you really appreciate Ruby.

Well, now that the digression is out of the way, lets come back to the question coding style in Ruby. Rubyists tend to favor terse code that conveys its intent clearly and concisely. The Ruby culture uses idiomatic syntax and encourage succinctness in code. I really came to appreciate this when I saw the Ruby approach to a basic programming structure – the loop.

The canonical implementation of a for loop is in the form –

for <counter> <condition>
<begin block>
	<do something>
<end block>

In C it is –

for(i = 0; i < 10; i++)
{

}

In VB it is –

for i = 0 to 10

next

or in the case of iterating through an array

for each <item> in <group>
  <do something>
next

In Ruby the for loop can be expressed in a variety of ways –
The vanilla for loop is available –

for i in 0..10

end 

for  in 

end

But there is also –

5.times { <do something> }

1.upto(10) (<do something)

<group>.each {<do something>}

<group>.map
  <do something>
end .
.
.

Now it can be argued that functions like map are available in other mainstream programming languages as well – but the point I’m trying to make is – in the Ruby way of programming they become the usual way of doing things.

Ruby is a language designed for developers – but in order to fully appreciate and utilize this one must write (and read) Ruby the way Ruby is meant to be written. Once you embrace “The Ruby Way” you will see in Ruby a language that stays out of the way of your thinking. Instead of grappling with or worrying about syntax, you spend more of you time working on your solution. You can see the syntax is very easy to read. In fact the expressiveness of Ruby makes it easy to get to the intent of the code without getting hung up on the syntax. Reading code after you left it for awhile becomes a snap and you find yourself getting back to where you left of more easily.

I had written a post earlier regarding “the coding zone” – Ruby gets me into the coding zone more easily and lets me sustain it for longer than with any other language I know. It is liberating to be able to conceive of and code entire features in the time it took me to look up the idiosyncrasies of the syntax and battle the compiler of other languages. I find coding in Ruby to be a pleasant experience, not the least because of the incredible community around it. The Ruby community in a very real sense is one of the strengths of the language that put it ahead of Python in my book.

So (as you can probably tell from this long post) – I am really enjoying my latest foray into Ruby. That’s not to say Ruby is perfect – even if you ignore its real performance disadvantage – Ruby has some things that can definitely be improved – but with its great community I think it has a real chance of moving forward and fixing these things.

How to read code – a primer

I like programming – it’s what I do and I am blessed in that I get to spend most of my waking hours developing software. Like a lot of programmers I obsess over how good my code is and how I can get better at it.

Over the years there have been reading a lot of articles and books on software development. There has been a lot of ink spent (both physical and virtual) on ways to improve your “programming foo” and become a super ninja programmer ! There are some common pearls in all this ink and one of them is the advice on reading code. This advice, is usually a one liner couched in the midst of a bunch of other recommendations and usually along the lines of – find some great open source software or any piece of software that you admire, open up the source code (or print it out) and read it. While, this is on the whole, great advice there are some problems with actually putting it in practice.  In this post I endeavor to give some practical suggestions on reading code, but first let us enumerate the problems.

• The usual impression conveyed (in the posts that advise one to read code) is that the dispenser of the advice is a programming guru who can literally sit back in their chair with a page of code and read it like a novel. Well, I am sure there are some superb programmers out there who enjoy looking at pages of cryptic English-like statements over a cup of coffee and can hold entire class hierarchies and architectures in their heads. This post is not meant for them – this post is for poor slobs like me who find staring at reams of code a boring, frustrating and ultimately pointless exercise. Of course, it can be argued that one can learn simply by reading a single class or even a function of a the entire project code, but, IMO, except for the most simple problems, most software is interdependent. It is often impossible to appreciate the design decisions and the rationale behind a particular function or class layout without knowing the rest of the system…
• The next problem is getting code to read (actually before that you need to be able to identify code worth reading – check out this post for details on that). There is a lot of great software out there – both open source and freely available and licensed or proprietary. There are huge open source directories like Sourceforge and Google Code, and huge pieces of software like Open Office and Linux. If you are working in a software development company, you can probably get access to the proprietary code in your source control repository. A third common avenue are the programs distributed along with books on software development  or as part of resources for  education( Minix being the canonical example). Indeed we are actually spoiled for choice and from this universe of software identifying the ones that are good candidates for our purpose is a hard but essential task.
• Another problem is the language in which the program is written – reading someone else’s code is tough enough as it is, adding the burden of familiarizing yourself with the quirks and syntax of a new language while doing this, is, IMO, a recipe for disaster and immense frustration . You need to find code written in a language that you are familiar with. This particular problem is not relevant if you are going through the code distributed as part of a book or as an educational resource, since you would have the book or your mentor to explain things and set out the context. If, despite this forewarning, you are planning to read code written in a different language than you are used to (without the benefit of having a book or a mentor) I would advise, that you at least learn enough of the language to create your own programs  in it (“Hello World” does not count :-)) .
• The bit about context brings me to the next problem – figuring out what the code is doing is a lot harder if you are not familiar with the software itself. For example, it is far more difficult to go through the Linux code and figure out the concept of runlevels if you don’t use Linux daily and see the Linux boot sequence. Using the software gives one a context with which to read the code – this context includes the common terminology used, the functionality and features of the software, even the quirks and bugs that you experience.

I have realized that for me ‘reading code’ does not really describe the activities that I undertake – a better phrase for what I do is ‘code comprehension’. It is quite difficult for me to sit back with a laptop screen (or a printout) full of code and simply read through it. I need a lot more than simply a piece of code – I like to be able to look at documentation, play with the software, step through the code and even write tests for it before I really appreciate it. This is a significant investment of my time and effort, so I have to be very picky about the software I want to “read” (comprehend).

• The first filter I place on the code directory, when looking for code  is the language filter – for me this means – C# or VB.NET or Python or Javascript(while I am familiar with C++, Ruby and F# as well I do not consider myself at a level where I can understand other people’s code in them). Next is to look for software that I have used – this allows me the a bit of a leg up since I know what the code is meant to do, cannot do and (if I am familiar enough) its limitations. Good candidates are open source software that you use in your day job (for eg. I use Cruise Control.NET, NANT and NUnit which are open source tools written in C#)
• I happen to work in a software product company (a Microsoft shop), so one of the candidates for my reading list is the code in my companies source repository. If you happen to work in a software company, you can look at other projects, and even older versions of the software you are working on. In addition to providing insight on code, you get a pretty good idea of what was tried before and since. There are a few caveats though –
  • First, if you don’t have direct access to other projects, you need to ask permission – some companies are very touchy about their “intellectual property”.
  • Second, the quality of the software may not be as high as you think, since, in general, proprietary code does not get the kind of scrutiny open source code does. Warning signs to look out for are a lack of regular code reviews – if the software is not code reviewed the odds are that it would not be of good quality.
  • Third (this point is inspired from feedback provided by my friend Praseed), if the code in your company is business software (HR, Finance, ERP, etc) there is a lot of business context that needs to be understood first. Also, since most of this code tends to be factored by business functionality, it generally seems less modular than utility code or APIs.
• Look for well documented projects (this applies to open source as well as proprietary code).  By this I mean, that the documentation should highlight the overall design, and rationale for the way the code is. Simply having auto-generated Java Doc type documents cannot be considered documentation :-). One useful avenue to explore is software created as educational resources (like Minix ). Since, the target is to teach through the software, they are usually quite clearly documented and have plenty of material explaining the design rationale behind the code.

So, you have identified the software and downloaded the source code and documentation, so let’s get down and start spelunking ;-)

• Go through the design documentation and try to get a feel for the way the code has been built. Good software projects follow certain architectural patterns – these dictate the code organization. Once you get a handle on this, understanding the code becomes a whole lot easier. If you can create a class diagram of the code you can get a good idea of the layout.
• The next thing to do is to compile it and run it. This can be straightforward or tough depending on the process followed in the project and it’s documentation.
• Now it’s time to fire up your favorite IDE and go exploring. A good place to start your code exploration  would be to try to trace a functionality of the project that you are familiar with. This would let you go through the various layers and sub-systems and get a handle on how they inter-connect. For example when I was exploring NUnit – I started by writing a test and looking at the code classes I needed to do that.
• Try and identify the design patterns used in the code. If you do not know what design patterns are, then you need to stop reading this post right now and read this book. Familiarize yourself with design patterns – they form a great way to recognize and understand the design of well written code. This makes it easier to keep it in your head while reading code. It also helps you identify nuances and customizations made by the programmers more easily.
• Try to write tests for the code to fully understand it – this is really useful way to understand the dependencies between different parts of the code. When you try to write a test for the code you first need to satisfy (mock) all its dependencies. Next you need to understand the possible entry points as well as the exit values for the code. This improves your understanding of the code and get you to the next level.
• Finally, try to refactor the code. In this step you have moved from simply understanding the code to becoming familiar enough to be able to modify it. As the sophistication of your refactoring increases so too does your understanding. At this point you can if needed contribute your own code to the project :-)

“Code Reading” IMO is more than just reading – it is a distinct set of activities that together help one understand code. It might seem more intimidating than simply “reading code” but it is well worth then effort IMO.

Happy “code reading” :-)

Update: I came across this post by Joel Spolsky where he quotes Seth Gordon as saying code reading “Is just like reading the Talmud”… Yup, code reading is definitely not easy.

The Humble Programmer – Edsger W Dijkstra

I first heard of Edsger W Dijkstra in the context of agile programming.I was having a discussion regarding agile programming with some friends and explaining Test Driven Development and the concept of first creating tests that can prove show the correctness of the code before writing the code, when a friend mentioned that this sounded a lot like some of the arguments put forward by Prof. Dijkstra in his Turing award lecture in 1972. I found that hard to believe, after all, if this was known in 1972 then why is it only becoming popular now ?

So I started looking up Edsger W Dijkstra and realized that this man was one of the pioneering giants of software programming. He is the father of structured programming and one of the guiding heads responsible for much of the way we program computers today. There is a lot written about him all over the place – I shall focus on his Turing award lecture, that was titled “The Humble Programmer“. In this lecture, Prof. Dijkstra puts forth six arguments on the way software programming should be done. On reading these six arguments I cannot help but feel that this lecture was one of the main inspirations used by the authors of the agile programming movement and design patterns community.

The six arguments put forward in the lecture are as follows –

1. “A study of program structure had revealed that programs —even alternative programs for the same task and with the same mathematical content— can differ tremendously in their intellectual manageability. A number of rules have been discovered, violation of which will either seriously impair or totally destroy the intellectual manageability of the program.I now suggest that we confine ourselves to the design and implementation of intellectually manageable programs. The programmer only needs to consider intellectually manageable programs, the alternative alternatives he is choosing from are much, much easier to cope with.”
2. “As soon as we have decided to restrict ourselves to the subject of intellectually manageable programs, we have achieved once and for all a drastic reduction of the solution space to be considered. This argument is distinct from argument 1.”
3. “If one first asks oneself what the structure of a convincing proof would be and having found this, then construct a program satisfying this proofs requirements, the these correctness concerns turn out to be a very effective heuristic guidance. By definition this approach is only applicable if restrict ourselves to intellectually manageable programs.”
4. “The only mental tool by which a very finite piece of reasoning can cover a myriad of cases is called an “abstraction”. There are number of patterns of abstraction that play a vital role in the construction of programs. Knowledge of these patterns of abstraction are essential.”
5. “A programmer is fully aware of the limited size of his own skull; so he approaches the task of programming in full humility and avoids clever tricks.”
6. “The only solutionproblems we can solve in a satisfactory manner are those that finally admit a nicely factored solution.”

When you go through these arguments you see the seeds for the various movements in programming software today –

• Arguments 1 and 2 are the foundation of principles like KISS (Keep it Simple and Stupid) and YAGNI (You Ain’t Gonna Need It)
• Argument 3 forms the basis of efforts like  Test Driven Development and the Design By Contract.
• Argument 4 leads us to Design Patterns.
• Argument 5 and 6 preaches humility in programming, the use of abstraction to write readable well factored programs – beautiful code

I really enjoyed reading his lecture – I have the lecture here (The_Humble_Programmer)  if you want to read it. He must have been an extremely engaging speaker – a lot of his quotes are available here.

Till next time – Happy Programming !

Update: I have made a couple of corrections based of some of the comments here.  When I wrote that argument 3 is the basis for TDD I meant that in the lecture Dijkstra talks about first finding the structure of a proof and then constructing the program satisfying the proofs requirement. This is similar to the TDD approach of first writing a test and then writing code that satisfies the test.

Google Android and the CLI

Today morning I was reading about Brad Adams – Going Google and ruminating about what he wanted to do there when several earlier articles I had read suddenly came together and I had an inspiration that I thought I’d blog.

It would by really cool if Brad and Tim Bray (who recently left Sun and joined Google Android) get together and implement the CLI on the Android OS.

Brad is one of the architects of the CLI and has been one of the main driving forces behind the development of the .NET framework and it’s adoption. He is currently looking at what he wants to do when he starts at Google. He mentioned that he thinks the cloud plus devices is one of the dominant trends of the future and I agree.

I think the Android OS is an important part of this future and currently I am frustrated that there is only Java support for developing in it. Tim shares my frustrations and he is looking at getting other languages supported in the Android OS.  He is looking at Ruby right now – it’s open source and a dynamic language and it makes sense . But I feel the CLI (which is an open ECMA specification) is a great fit for the Android OS because it can be used to as a basis to quickly support a lot of languages.  Besides as a .NET developer myself, I think that having the capability to develop Android applications in C# or IronPython is a far more palatable proposition than doing it in Java :-)

I saw this article by Miguel de Icaza where he puts out an idea to incorporate the CLI into the browser engine so that we could use languages other than Javascript in our client-side scripting (my take on that is here). It occurs to me that it should be similarly possible to bring it into the Android OS as well.  There are currently efforts to port Mono that could be used as a starting point.

So what say guys – can we get the .NET CLI in Google Android ?  Become the opposite of the Apple iPhone and embrace developers instead of driving them away :-)

Ganesha – the original lateral thinker

There is an ancient tale from Hindu mythology that illustrates lateral thinking (also known as –  “out of the box” thinking) that I would  like to share –

One day Lord Siva and His consort Parvati were sitting atop their abode on Mt. Kailash with their sons Ganesha and Karthikeyan when the sage Narada dropped by for a visit. Narada had with him a special mango of knowledge, to offer to Siva. After accepting the mango from Narada,  Siva and Parvati decided to have a contest between their sons.

Ganesha

Karthikeyan

The first one who circumnavigates the world three times would get to the mango of knowledge. Without further ado Karthikeyan  jumped on his peacock and started off. Ganesha on the other hand was busy eating his favorite ladoos and decided to finish them first. Karthikeyan had completed two rounds by the time Ganesha finally got ready to compete :-)

Ganesha simply approached Siva and Parvati and deliberately walked around them –  He circled them once, twice and three times and then claimed the mango.

When, Siva and Parvati asked him how he could claim the mango when he had not circled the world even once – Ganesha replied – “You both are my world”. Delighted by the answer Siva and Parvati gave Ganesha the mango, which he immediately gobbled up with relish.

Two of the important traits of good software developers are “enlightened laziness” and “Out of the box” thinking.  This tale is an example of both enlightened laziness and out of the box thinking – confronted by the immense task of circumnavigating the world – Ganesha – by simply thinking a little and restating the problem comprehensively defeated his brother Karthikeyan.

So, my eager friends – the ones who are chomping at the bit after the initial presentation of a project – eager to rush into coding it, please spend some time contemplating your problem.  Another, homily you might want to consider is –  “Think twice, code once” – You, might just save yourself a LOT of time and effort !! :-)