On Distributed Development

 So, I have been doing distributed development at Bang the Table, for the past year. It’s a company with a very strong work from home culture. We even had a virtual birthday party for one of our directors over Skype video chat –

Birthday party at Bang the Table

Telecommuting / working remotely / working from home – these are various terms for an increasingly popular mode of working. In this model (of working) people use the amazing ability of the internet to communicate over vast distances instantly and modern collaborative software tools to effectively work together without actually being in the same physical room, office or even city.

There are some pretty powerful advantages to such an arrangement –

1. No need for any commuting, one can stay close to one’s family (a huge factor if you have children) and flexible working hours. It is a more relaxed way to work. I have been working in such an environment for about a year now, and with a new baby, the lack of commute and the freedom to pop downstairs and help out in case my wife needs an extra pair of hands has been awesome!
2. Another major advantage (at least for the company), is the reduced office infrastructure costs. In places where getting an office building thats a reasonably close to where your employees live is difficult, this can be a major boon. Also, not forcing your employees into having to commute is a major plus both for employee morale and the environment.
3. One has the opportunity to hire and work with people from a larger pool rather than being limited by a single geographical location, like a city or state or even a country. This can be very powerful advantage, sometimes finding all the right people you need in a single location can be a major problem.
4. An “advantage” that is often touted is that the geographical separation allows for better efficiency, since, theoretically, one could have employees coming online and starting their workday as others end theirs – making use of the entire 24 hours on the clock for work. I believe, at least for software development, this is a fallacy and attempts to leverage this so called “advantage” is a big reason for the failure of certain companies to successfully leverage this model of working – but more on this later…

In the software development field we have been working in such a distributed model for a long while now and I have seen this model evolve, influenced both by the advances in the technology and tools available for collaboration as well as the evolution of the software development methodology itself. I’d like to take a few lines to write about this evolution and distinguish between what I consider to be “distributed development” model and what I call “outsourced development”.

Image via Wikipedia

In the outsourced development model  – a software project is broken up into separate well defined phases, each phase having definite outcomes with concrete artifacts (usually in the form of documents). This means that (in theory) one can easily move around parts of a project between various teams who would then rely on the artifacts produced in the previous phase to execute the next phase of the project. Other than a short period between phases, where one team handed over the project to the other, there would be very little actual interaction between teams. This form of software development follows the waterfall methodology and while it is a good fit for outsourcing, it is, in practice, prone to problems, that, for the most part arise due to a lack of communication between the various stakeholders of the project and the members of the team(s).

What I call “distributed development” (there is a more generic definition on wikipedia that is not limited to software development) is a refinement on the outsourced development approach that attempts to leverage technological advances in internet connectivity and better communication tools to try and minimise the disconnect introduced by working remotely in software teams. Distributed development places emphasis on communication and interaction between all the members in the virtual team. The idea is to reduce the feeling that we are working in different physical spaces and that everyone is off working on some discrete task on their own. This kind of thinking is in-line with the agile philosophy which emphasizes individuals and interactions over processes and tools.

At Bang the Table, we get pretty close to what I consider to be an ideal distributed development model and I’ve come up with a few guidelines are a result of my experiences working here and from what I learned listening to a Hanselminutes podcast about working remotely –

1. One can use many tools and technologies to do your job. But regardless of what you use, patience and trust are the biggest success factors in this kind of working. I initially put this point lower down in the list but on consideration decided that this is the most important tip. Without trust the whole thing is an exercise in futility. One needs to learn to trust in the skills of the developer to do his or her bit and the patience to listen and accept that problems and delays might happen. The best way to build trust is by having open and rich channels of communication and meeting and interacting with everyone in person.
2. Reliable and fast internet is an absolute must and its not that easy to get – at least where I come from. Ask if the person you are working has a backup power supply and a good internet connection. In terms of power supply, the need for one depends a lot on the place the person is working from – if it’s a big city then usually a UPS that provides an hour of backup should be fine, unless there is chronic power problem. In smaller towns and such, they might need to have something more substantial like an inverter or a portable generator.  Modern laptops with their longer battery life and portability are a major help since they allow one to ignore smaller outages and coupled with the appropriate wireless or cellular data connection can even allow you to simply shift your base of operations to other places with power and internet.
3. This guideline, I think is going to be a bit controversial and is the reason I think that 24 hours work day idea I have mentioned is not really a good idea. If you have a team that spans countries and (in my case) continents, it is very important to have good overlap between your timezones – at least 4 hours of overlap is needed in my opinion, since this will allow you to actually collaborate rather than be limited to a short meeting at the beginning or end of your day. After all, developing software in a team is, in my experience, a collaborative activity, requiring lots of discussion (and argument)  and having your stakeholder at hand while you work through a problem is very useful. No amount of documentation and pretty pictures can substitute someone  being there, clarifying and providing context for you.
4. You need to have an open channel of communication during the time of overlap while you are working – this can be as simple as an open IRC (Internet Relay Chat) channel. We use Skype at work and I think the ability to call or video chat on demand make it far superior to IRC. Another option to explore is to use something like a Google+ Hangout while you are working – obviously this will mean the both you need to have reliable high bandwidth internet. Paid options for software tools are there as well like Microsoft Lync which may be a more integrated solution if you have a Microsoft based IT infrastructure with Microsoft Exchange (I got this from the podcast – haven’t used this myself since we don’t use MS Exchange). There are variety of communicators and tools out there to tryout – have a look at this link where Scott provides a deep dive into the communications tools available for remote workers.
5. Schedule regular meetings and get togethers in the real world. If this means that someone needs to travel somewhere from time to time, consider that the cost of doing distributed development. I would consider this an investment in the team’s productivity and efficiency. When everyone has met in person and worked together on issues, you build up mutual trust and respect. In my year of working at Bang the Table,  I think we have had get togethers at least once every quarter. Sometimes we just got together and spent a few days simply working together with no other agenda, sometimes it was a conference or a training.
6. Distributed development tends to work well with relatively flat and simple organizational structure. The idea is that everyone should feel equally invested and responsible for the software project. The challenge of course is that this particular setup does not scale very well and can be problematic for large organizations.
7. A good online project management tool is a must to ensure you are heading in the right direction. We are constantly trying out new tools for this part of our development. I have found JIRA to be good for support and task based work while, I am liking Trello for more open ended new development.
8. At work we use a Rackspace server as a development server and for testing. This is a machine on which we stage our commits from our local machines and have our testing done on. Having a machine to deploy your code on also forces you to test it in the right environment and also brings deployment considerations into your design and development schedule.

It’s been a fascinating experience for me – working from home and I think I have learnt a lot about the vagaries of distributed development. In fact, I have been experimenting with a more distributed approach to pair programming with a couple of friends of mine.

We used Vim in a GNU Screen session to set up a development environment that we could share between us and then while one person typed we used Skype to chat with each other. We followed the classic test driven mechanic – one person writes a test and the next person passes the test cycle. The experience was remarkably powerful, since I could see when the editor was being manipulated by the other person, the thought process going on as the code was being typed in. We built a small Python program – I came from that experience with a real appreciation of what can be done with some of the simple (and powerful) unix tools already out there.

I came across another tool Tmux which is similar to screen and is used a lot in the Mac when I was listening to this podcast on the Changelog. There is a very nice screencast showing how to use Tmux I found recently, you can checkout.  I also came across this awesome website called pair.io that spins up a server for you to use in your pair programming sessions.

Related articles

• Hanselman Blog – Remote Work
• Hanselminutes – The Plight of the Remote Worker with Pete Brown
• The Changelog – Tmux with Brian Hogan and Josh Clayton
• Pair Programming with Tmux – a screencast
• Bang the Table
• On working remotely
• Wemux – Tmux enhancement for remote pairing

Advertisement

A Ruby on Rails glossary.

So it’s been about a year since I started working on the Ruby language in the context of the Rails framework. It’s been awesome,  the combination of Ruby’s developer friendly language style and the incredible productivity of the Rails framework makes this a killer framework for web applications. Building the basic code is simply a matter of typing in some commands that generate all the scaffolding of code needed for a basic website. The rest is usually a matter of tweaking and adding to the generated scaffold.

Image by dullhunk via Flickr

Like every web technology the Ruby on Rails framework has its own set of idioms, that one needs to get familiar with and since I am new to Ruby, for me this includes some Ruby idioms as well. I saw a post today that lists a glossary of Rails terms and I thought I’d do my own. So here goes –

1. Application Servers – Thin, Mongrel, Passenger, Unicorn, Pow – These are application servers that run the Ruby web application and respond to the requests. They are generally integrated with a web-server like Apache or Nginx, though Mongrel is technically capable of running on its own as a web-server (though since it is not multi-threaded this configuration is not useful for anything except for very light loads or on a development machine).
2. Active Record – Active record is the ORM tool used in the Rails framework to abstract the database handling. It is an implementation of the active record architectural pattern described by Martin Fowler thus – “An object that wraps a row in a database table or view, encapsulates the database access, and adds domain logic on that data.” The Active Record code is available as a separate gem that is downloaded as a dependency to Rails.
3. Active Resource – A simple way to think of active resource is that it’s like active record but for REST based web-services. Active record provides a wrapper around REST actions providing a transparent proxy between a business entity and REST based web-services.
4. Active Support – This is a set of utility classes and standard library extensions that are useful in Rails application development. It includes classes for dealing with caching, dates, time zones, text, etc. Like active record and active resource, active support is available as a separate gem that is installed as a dependency of the Rails gem.
5. Bundler – Bundler manages the gem dependencies of a ruby application in a repeatable way across multiple machines. It automatically looks for and installs / updates all the gems needed by the application when it is deployed on a machine. It can do this is a repeatable way across multiple machines and deployments.
6. Capistrano – Capistrano is a tool for executing command on groups of servers over SSH. It is used mainly in deploying web applications and is capable recognizing different types of server environment (like staging and production, etc).
7. Capybara – This is an advanced version of the Webrat tool. It lets you do acceptance testing of applications using various drivers (Selenium and Rack are built-in) by simulating a real user hitting your web application.
8. CoffeeScript – Coffee-script is a little language that compiles into java-script. It has rapidly become very popular and has been included in Rails 3.1 as the default method of writing JavaScript code in a Rails application. I am personally a bit on the fence here but I like its terse syntax.
9. Cucumber – Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable DSL and serves as documentation, automated tests and development-aid – all rolled into one format. Cucumber along with RSpec are main tools used for the BDD methodology favored in many Rails projects.
10. DSL(Domain Specific Language) – A domain-specific language is a language dedicated to a particular domain or the representation of a particular problem or solution. DSL’s are more common than you think, some good examples in the Ruby being – Cucumber, Rake, Haml etc.
11. ERB (Embedded Ruby) – ERB is a simple and powerful templating system for Ruby. It is part of the Ruby core and it allows one to place ruby within other files (any text document).
12. FactoryGirl – This is a gem that can be used for creating object factories in an automated fashion. This is useful in testing or even for populating an application when deployed
13. Gem – At its most basic form, a Ruby gem is a package. It has the necessary files and information for being installed on the system. There is an online repository for Ruby gems at http://rubygems.org/ For me this is the biggest find and literally the “hidden gem” of my Ruby learning experience has been – Ruby Gems.  It’s like having an app-store for your development needs, where most of the best gems are freely available to you!  The number of awesome, useful gems actually available to ruby developers means that we have a huge advantage over other frameworks. A testament to the power of Ruby gems has been that Microsoft has started its own package management library – Nuget which, seems to be simply a straight lift of the Ruby Gems concept, except of course they have incorporated it into the Visual Studio as an extension. Nuget was released by the same team that created the Microsoft version of Rails – ASP.Net MVC so you can see the Rails influence…
14. Gherkin – Gherkin is a DSL  that emerged from the Cucumber project. It was designed to succinctly describe behavior in a form that can be acted upon by a tool like Cucumber. It is described as business readable DSL and can be used in any project that has need of such a tool. The lexer and parser for the language has been create and the library is available in several frameworks including Ruby, JVM, Javascript (Node.js) and as a .NET dll.
15. Git – Git is a free and open source, distributed version control system. It was written by Linus Torvalds to replace the old source control system for Linux and has become one of the tools of choice for a lot of Ruby projects. Git is typically used along with Github which is an online source control repository that leverages Git.
16. Haml – Haml is an HTML abstraction language. Haml functions as a replacement for inline page templating systems such as PHP, ASP, and ERB, the templating language used in most Ruby on Rails applications.
17. Heroku – Heroku is a cloud based platform as a service (PaaS) which was one of the first to have Ruby and Rails support. One could build a Rails application and directly deploy it on Heroku without having to set up a web-server. The platform has evolved to support multiple languages including Java, Scala, Node.js, Clojure and Python.
18. jQuery – jQuery has become the JavaScript framework of choice for web developers everywhere. It is a fast, framework that leverages CSS style selectors to allow developers to traverse, animate and handle events in the HTML DOM while abstracting browser specific quirks as far as possible. It has become so popular that it displaces Prototype as the default JavaScript framework used in Rails 3 applications.
19. Memcached – A distributed open source caching system, memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering. It is a generic store and can be used in any situation where caching is needed and is available in multiple languages and technologies. It is used in a lot of Rails applications even though today, other object stores like Redis are getting popular.
20. Merb – Like Ruby on Rails, Merb is an MVC framework. But, it a lighter and more barebones version. I haven’t used Merb – I’d love it if someone could give a more apt description of what Merb is and how it compares to Rails and Sinatra.
21. Open classes / Monkey Patching – Open classes is a feature of the Ruby language where you can simply open and add methods to existing classes. Active support library uses this feature to add a lot of its helper functions on the Ruby core classes like string. A popular name for this particular practice is “Monkey Patching”.
22. Paperclip – Its a complete (and easy) file attachment library for active record. It provides validation features based on the properties of the file like size, type, etc. The files themselves are saved in the filesystem. It has some basic image manipulation features as well, like creating thumbnails etc. There is a dependency on the ImageMagick library (an awesome image manipulation library) in order to use Paperclip.
23. Prototype – Prototype is a JavaScript Framework that aims to ease development of dynamic web applications. It was originally the default JavaScript framework bundled with Rails.
24. Rack –  Rack is a gem that provides a simple interface between a web-server and the Ruby web framework. Rack is used in various Ruby web frameworks, like Rails, Sinatra and Merb.
25. Rake – Rake is the ruby version of make but much more developer friendly. It provides capabilities that are similar to make but it uses the Ruby syntax instead of XML files or makefile syntax to define tasks.
26. REST(Representational State Transfer) – REST is a set of principles that define how Web standards, such as HTTP and URIs, are supposed to be used. Well, actually REST as a concept is more abstract than that – its architectural style that can be applied to distributed systems. REST in the context of Rails, mainly is used to refer to the way Rails exploits REST concepts in the way that Rails web applications are structured.
27. RJS(Ruby JavaScript) – It is a ruby helper used to generate JavaScript that are used to decorate HTML pages. It is used mainly in AJAX calls, and in situations where you wanted to conditionally emit JavaScript. The general opinion though was that it sucked and was avoided in favor of the unobtrusive style of jQuery.
28. Routing – Routing is the aspect of the Rails web application that deals with the translation of a URL request to a specific action in the application. In Rails, it is the job of the router to recognize URLs and dispatch them to a controller’s action. It can also generate paths and URLs, avoiding the need for hardcoded strings in your views.
29. RSpec – RSpec is a Behaviour-Driven Development tool for Ruby programmers. RSpec helps you do the TDD part of that equation, focusing on the documentation and design aspects of TDD. Like Cucumber RSpec uses a DSL to simplify design and implementation tests.
30. Ruby Debug – Just like it says on the tin – it is a debugger for ruby applications. Like the other tools it is gem that can be installed and can run your ruby application in debug mode.
31. RVM(Ruby Version Manager) – This is an awesome tool from Wayne Seguin that is used to install, manage and work with multiple versions of the Ruby programming language on your machine. It even allows you to segregate your gems so you could work with multiple versions of the same gem for different applications.
32. SASS & SCSS – These are CSS abstraction languages (I believe SCSS is a reference to the version that supports CSS3). They are compiled down to CSS. They allow you use variables, modules and include other files. I am not very familiar with them and would welcome more information from readers.
33. Sinatra – From the website – “Sinatra is a DSL for quickly creating web applications in Ruby with minimal effort.” Like Merb, Sinatra is a minimalist web framework. Again, like Merb, I haven’t used Sinatra at all and would welcome some comments to flesh out this entry.
34. Test::Unit – This is the default testing library available with Ruby. It is from the xUnit family of testing frameworks and is a part of the core Ruby libraries. It is mostly not used in favor of more powerful BDD style tools like RSpec.
35. UJS(Unobtrusive Javascript)– “Unobtrusive JavaScript” is a general approach to the use of JavaScript in web pages. The general principles of this approach includes –
    1. Separation of behavior from the markup – The HTML in a document is meant to describe the structure of the document and should not contain programmatic elements in it. This is similar to the principle of separation of the structure from the presentation of the document when talking about the separation of CSS and HTML.
    2. Should not pollute the global namespace – UJS code should add as little as possible to the global object or global namespace of the environment in which it runs.
    3. Degrade gracefully – If the browser rendering the document does not have the capability needed the code should be able to degrade gracefully without breaking the rest of the application.
36. Vim – IMO Vim, the text editor that just celebrated its 20th anniversary in 2011 has experienced a resurgence of sorts with the rise of Ruby and Rails. In fact there is a large number of Vim plugins that have been created to make developers even more productive and happy on this text editor. Indeed after Textmate (the other favored tool for Rails developers) I think Vim has the biggest following among Rails developers and the fact that its available even on Linux and Windows only makes it cooler…
37. Web Servers – Apache, nginx, Lighthttpd , Thin – In the Rails world web servers options include Apache, nginx and lighty(lighthttpd) of which Apache is by far the most popular. I have included Thin in this list as well since it includes both a web-server and an application server. With advent of PaaS providers like Heroku, this particular aspect of web application development is becoming abstracted out, just as the database is getting abstracted on the other end.
38. Webrat – This is a headless browser that is used in web application testing. It lets you run in browser test without the associated overhead of actually instantiating a browser. This does not execute any JavaScript but is great for doing tests of simple websites without much JS interactions.
39. Will_Paginate – This is the most popular pagination library out there for Ruby on Rails. It alos supports other Ruby web frameworks like Sinatra and Merb. This gem has a wide variety of formatting options and can accept an active record object or even a simple array. It is quite configurable and can be easily extended to do AJAX pagination as well.

So, that’s all the terms that I can think of so far, I’ll try to add to this as and when I come across more terms. Nowadays, web development is more than a single framework, even one like Rails.  I have found the Rails community is more open and embracing of different ideas and frameworks – even to the extent of moving away from Rails and using lightweight frameworks like Sinatra. IMO that makes this glossary all the more relevant and fun.

As I have mentioned in the article I don’t profess to be an expert about or have even used all the tools and technologies listed above. I would love to get your comments and suggestions on what could be added.

And last but not the least – I wish you all an awesome 2012 – may all your efforts bear fruit this year.

Update (Jan 12 – 2012) : Since there has been some interest in this post, I have added a few more entries and provide more information where I can – stay tuned :-)

The loneliness of the software tester

Sometime back my favorite podcast Radiolab – released a short called “The Loneliness of the Goalkeeper” . The short was a re-broadcast of an english show on football (soccer) and the role of the goalkeeper in the game.

In soccer the goalkeeper plays a truly unique role – the opposite to the role of the remaining 10 players in the team and indeed one could argue the aim of a game of football. The goal keeper has to defend the net and prevent a goal being scored.

Of course, the other players in his team also help him in defending the net, but there are few differences that make the goalkeepers stand apart. The primary one, of course, is that he is allowed to use any part of his body to handle the ball during the play in a restricted area in front of the net – the penalty area. If the ball enters the net – even if it is through a mistaken strike, an ill-conceived pass or fumble from him or his own team – the goal counts against the team…

This means that a goalkeeper rarely ventures outside the penalty area – he is the last line of defense to prevent the ball from getting into the net. This is the loneliness of the goalkeeper – he is ultimately responsible for any goal and must protect the net from all comers – friend or foe…

Image via Wikipedia

The narrator of the piece goes on exploring the mental make-up of goalkeepers – the strange almost contrary nature that is needed to be a great goalkeeper. There is a subtle psychological difference in the mindset, brought on by the role of the goalkeeper. A goalkeeper does not have a single moment of triumph, there are only moments of disaster…  A goalkeeper’s mistakes are obvious and public while his success is not. After all – a goalkeeper who manages to prevent the opposition from scoring the entire game has simply done his job while a striker that scores that final goal takes his team to victory or saves them from defeat – is the hero!

As I listened to the podcast, I was struck by the similarities between the role of the goalkeeper and the role of a software tester. Unlike the rest of the team, the goalkeeper’s motive is to prevent a goal being scored – similarly the aim of a software tester is to find defects in code that is created by the team. In a way the software testing role requires an almost opposite mindset to that of a software developer, who wants to build a working product.

It is preached everywhere that a good programmer tests his code. Indeed nowadays it is a given that software should be written test first and the agile philosophy embraces this as a core tenet. I think the concept of TDD is brilliant and that it promotes good coding habits. But I think in all the hype there is one point that has for a long time not been emphasized and that is the writing code in the test driven style does not mean that your code will be defect free.

I have seen a lot of people claim that they write their code test first as if that somehow means that they don’t need to test it. This assumption then leads to an impression that all one needs to do for getting defect free code is to embrace the test driven development style. However what people don’t consider is that when a developer sits down to write code , he does so in a certain context. Code does not exist in a vacuum, it needs to interact with other code and in many cases the code itself might be a small section of a module of a feature in a platform.

The tests referred to in TDD are simply the ones that validate the requirements for the code that you are writing at that point of time. These are unit tests and do not test the software product – only the code being written is to be tested. Writing tests for even that narrow scope requires a mental shift. Programmers who are used to simply writing code (and I count myself as one this group) find this mental shift a pretty big hurdle. It is really hard to force yourself to sit down and think of the requirement and then do the coding and writing the tests needed to validate it (especially if you have a deadline or worse a manager breathing down your neck). And after you do all that, you find yourself having to mock the environment around your code and tests to get them to run consistently (or at all). Sometimes it is simply not feasible/cost-effective or even necessary, to write tests that cover every error that your code might meet. To tie this back to the football analogy, I would consider that software developers can make at the most –  defenders of the goal. They are restricted both by their role and by their mindset from being able to truly subject software to the test.

In contrast, a software tester approaches things from a totally different perspective. The role of the software tester is to find defects – it is his goal and his responsibility is to the quality of the software product as a whole. Testers approach software from a product perspective and not a code perspective. They look at the entire product, its various layers and the seams in the code, where the different features come together – and then look for defects. They write code (in fact I think good software testers need to write as much code as developers) to probe the product from various angles. Software testers need to test the various modules, individually and in combination and using different types of input.

Testing is a matter of diligence – the sheer focus needed to go through all the combinatorial options for all the inputs and apis available in a typical software product requires an intense mental focus and very tidy and organized mind. It is this mental makeup that make great software testers. It is not easy to find people with such a unique mindset – in fact I was reading the other day of a startup company that hires autistic people for software testing. Traits that make great software testers – intense focus, comfort with repetition, memory for detail – also happen to be characteristics of autism.

The role of a software tester is a lonely one – it is not something that will endear you with developers – after all you are pointing out defects in the code they wrote. If the software product fails the responsibility of failing to find the defect falls on the software tester. If the software tester does his job well and all the defects in a software product are fixed – he has merely done his job. Indeed if he does his job too well, and there are no defects to fix people will start wondering if there is a need for the role at all…

Like the goalkeeper – the software tester has a lonely job, but a vital one – he is the last line of defense – the responsibility of the overall quality of the software product lies with him. Software developers may have written the code for a killer feature but it is the software tester that ensures that it enhances the software product instead of breaking it….

Related articles

• Radio Lab Presents: The Loneliness of the Goalkeeper
• U.S. firm harnesses talents of young adults with autism (ctv.ca)
• Startup Company Succeeds at Hiring Autistic Adults to Test Software (seattlepi.com)

Needing an IDE should be considered a language smell

I’m a long time Visual Studio .Net (VS.Net) user – I have worked on the Microsoft .Net platform for a long time and have used the VS.Net tool during that time as my primary integrated development environment (IDE). It’s an awesome tool that has got better as the .Net platform progressed and matured with awesome features like code-completion (one of the first IDEs that had this), interactive debugging, refactoring and one of the most comprehensive documentation tools (MSDN) in the business.

What was frustrating, however, was that to write a simple program, I needed to fire up VS.Net (which is this huge beast of a program that used a lot of resources). On the other hand writing your code in Notepad and then compiling and linking from the command line while interesting was, in my opinion, not practical. I found myself looking for the code completion features and searching Google for C# keywords and .Net API syntax. Using Notepad seemed too difficult so I tried to compromise and I started looking at other “lighter” IDEs like Sharp Develop, Komodo Edit and Mono Develop to deal with this problem. But that meant having to learn my way around the new IDE with different keyboard shortcuts and user interfaces. The mental switching needed to work on another IDE was too high a barrier and despite several half-hearted tries I always found myself going back to VS.Net. The massive footprint of VS.Net was something I decided to ignore to avail myself of the many tools that made coding in C# easy.

Recently, I changed jobs and suddenly I was on Linux and programming in Ruby. I was apprehensive and I searched hard to find a replacement for VS.Net. I found Eclipse to be a bit too bulky and heavy, it was too much like VS.Net – resource hungry :-) Net Beans looked promising but then its community decided that Ruby and Rails support was not a priority which disqualified it for me. After some more looking I concluded that the Linux way seemed to be to use general purpose text editors like Vim and Emacs and customize them to ones specific needs.

Faced with coding without –  “gasp” – a proper enterprise IDE and having to learn Vim or Emacs – I decided to start of with a simpler text editor first. After looking around a bit more I settled on Red Car which is a JRuby open source editor that was copying TextMate (which I had heard was the editor that all the hip Ruby people used) and resigned myself to a drop in my productivity…

Surprisingly though this did not turn out to be the case. I was certainly writing less code but amazingly I was still more productive in Ruby and Rails than I was in C# and ASP.NET in terms of the amount of work done. Despite the lack of an IDE, I found the Ruby syntax to be concise and extremely expressive (I have a separate post on what I love about Ruby) which made it easy to code without having to resort to code completion or similar niceties. More importantly I found reading and understanding code in Ruby to be vastly easier than C# and I found myself not needing the features provided by VS.Net to get a mental model of the code base.

The more I worked with Ruby, the more I liked this way of working and I had a personal epiphany about programming language design. Some languages like C# or VB.Net, forced you to have a IDE just to be able to code in them, while others like Ruby let you simply code. This is because coding in a language like C# involves a lot of ceremony – you needed to write a lot of stuff to help the compiler understand the code – while Ruby tries to get out of your way and just let you code. The terseness and expressiveness of languages like Ruby allow programmers to easily keep larger amounts of functionality in their head.

This particular characteristic of languages should be recognized as a smell – just like a code smell… If you find yourself firing up an IDE as the first step, when you want to write or read a program in a particular language, that is a language smell. It indicates that the language is probably not at the right level of expressiveness for the problem it is trying to solve. To be clear, this particular postulate should be considered in terms of the language design and of course should not be the only way to choose a language. But, it certainly should be an important consideration, IMO,  especially since the language you choose will be something that you will be something you are going to spend most of your waking hours using and more importantly trying to understand.

I think if you come in at the start of your day and it takes you less time to load your code, read and understand what you have written and get into your coding zone, you are definitely going to be more productive, than if you have to fire up a program, wait for a while for it to load and then jump around several files to get your mental model back in your head to start coding. And if you have chosen the language you are using wisely, you will be more productive and in my case a lot happier as well.

Thoughts on Ruby – I think it’s great…

So it’s the middle of the year and like last year I am in the middle of a new language. This time however there are some differences from last year.

The biggest one is that this year I am actually working on the language – Ruby – while I am learning it and I think this makes a big difference. Though I know that one can learn a language using sample exercises and Code Katas – somehow I think the experience of working with a language everyday – gives one a real appreciation of what it is like to live with a language. When you are learning a language by choice – in your own time – you get to choose the problems and the Katas. When you have to work with the language and you use it daily on problems thrust upon you by seemingly unreasonable and sometime crazy requirements – you get a better understanding of the warts and wrinkles of the language along with the pretty bits.

I actually had a cursory look at Ruby sometime back, I had downloaded the language and looked at a tutorial on the web. I solved a couple of basic Project Euler questions with it. I remember thinking at the time – “Hmmmm, not really all that different from Python…” and deciding that learning it was not going to get me much. I was looking to exploring languages like F# and Erlang at the time, since they were based on different ways of solving programming problems.

So, when I started on this Ruby project this year, I came with certain preconceptions – but then something happened that changed how I approached Ruby completely. I read the online book – Why’s (Poignant) Guide to Ruby and I realized that the way I had coded Ruby thus far was not the way Ruby was intended to be coded. Before I go a little further I think I will digress a little give my impression of Why’s (Poignant) Guide to Ruby…

This is not the first time I had heard of Why and his book.

Image via Wikipedia

Indeed one would be hard pressed indeed to spend any time in the Ruby community and not have heard about Why the lucky stiff, his book and his sudden disappearance from the internet (another digression – I really am amazed at the fact this person was able to keep his virtual life so compartmentalized from his non-virtual one that he was able to disappear so completely!). My first impression after reading the first few pages was – Wow!! This guy must have a good collection of hallucinogens and Pink Floyd albums – to create those weird cartoons…

Image via Wikipedia

But, happily I persevered and was rewarded with a real appreciation of the beauty of Ruby. This book is not only a guide to the Ruby language – it is a guide to the Ruby philosophy – The Ruby Way. I would definitely recommend this book to all developers that are looking to learn Ruby, especially if you are coming from a background of working on the “big iron” languages like C# – this book will provide that mental shift in the way you code, that will let you really appreciate Ruby.

Well, now that the digression is out of the way, lets come back to the question coding style in Ruby. Rubyists tend to favor terse code that conveys its intent clearly and concisely. The Ruby culture uses idiomatic syntax and encourage succinctness in code. I really came to appreciate this when I saw the Ruby approach to a basic programming structure – the loop.

The canonical implementation of a for loop is in the form –

for <counter> <condition>
<begin block>
	<do something>
<end block>

In C it is –

for(i = 0; i < 10; i++)
{

}

In VB it is –

for i = 0 to 10

next

or in the case of iterating through an array

for each <item> in <group>
  <do something>
next

In Ruby the for loop can be expressed in a variety of ways –
The vanilla for loop is available –

for i in 0..10

end 

for  in 

end

But there is also –

5.times { <do something> }

1.upto(10) (<do something)

<group>.each {<do something>}

<group>.map
  <do something>
end .
.
.

Now it can be argued that functions like map are available in other mainstream programming languages as well – but the point I’m trying to make is – in the Ruby way of programming they become the usual way of doing things.

Ruby is a language designed for developers – but in order to fully appreciate and utilize this one must write (and read) Ruby the way Ruby is meant to be written. Once you embrace “The Ruby Way” you will see in Ruby a language that stays out of the way of your thinking. Instead of grappling with or worrying about syntax, you spend more of you time working on your solution. You can see the syntax is very easy to read. In fact the expressiveness of Ruby makes it easy to get to the intent of the code without getting hung up on the syntax. Reading code after you left it for awhile becomes a snap and you find yourself getting back to where you left of more easily.

I had written a post earlier regarding “the coding zone” – Ruby gets me into the coding zone more easily and lets me sustain it for longer than with any other language I know. It is liberating to be able to conceive of and code entire features in the time it took me to look up the idiosyncrasies of the syntax and battle the compiler of other languages. I find coding in Ruby to be a pleasant experience, not the least because of the incredible community around it. The Ruby community in a very real sense is one of the strengths of the language that put it ahead of Python in my book.

So (as you can probably tell from this long post) – I am really enjoying my latest foray into Ruby. That’s not to say Ruby is perfect – even if you ignore its real performance disadvantage – Ruby has some things that can definitely be improved – but with its great community I think it has a real chance of moving forward and fixing these things.

How to read code – a primer

I like programming – it’s what I do and I am blessed in that I get to spend most of my waking hours developing software. Like a lot of programmers I obsess over how good my code is and how I can get better at it.

Over the years there have been reading a lot of articles and books on software development. There has been a lot of ink spent (both physical and virtual) on ways to improve your “programming foo” and become a super ninja programmer ! There are some common pearls in all this ink and one of them is the advice on reading code. This advice, is usually a one liner couched in the midst of a bunch of other recommendations and usually along the lines of – find some great open source software or any piece of software that you admire, open up the source code (or print it out) and read it. While, this is on the whole, great advice there are some problems with actually putting it in practice.  In this post I endeavor to give some practical suggestions on reading code, but first let us enumerate the problems.

• The usual impression conveyed (in the posts that advise one to read code) is that the dispenser of the advice is a programming guru who can literally sit back in their chair with a page of code and read it like a novel. Well, I am sure there are some superb programmers out there who enjoy looking at pages of cryptic English-like statements over a cup of coffee and can hold entire class hierarchies and architectures in their heads. This post is not meant for them – this post is for poor slobs like me who find staring at reams of code a boring, frustrating and ultimately pointless exercise. Of course, it can be argued that one can learn simply by reading a single class or even a function of a the entire project code, but, IMO, except for the most simple problems, most software is interdependent. It is often impossible to appreciate the design decisions and the rationale behind a particular function or class layout without knowing the rest of the system…
• The next problem is getting code to read (actually before that you need to be able to identify code worth reading – check out this post for details on that). There is a lot of great software out there – both open source and freely available and licensed or proprietary. There are huge open source directories like Sourceforge and Google Code, and huge pieces of software like Open Office and Linux. If you are working in a software development company, you can probably get access to the proprietary code in your source control repository. A third common avenue are the programs distributed along with books on software development  or as part of resources for  education( Minix being the canonical example). Indeed we are actually spoiled for choice and from this universe of software identifying the ones that are good candidates for our purpose is a hard but essential task.
• Another problem is the language in which the program is written – reading someone else’s code is tough enough as it is, adding the burden of familiarizing yourself with the quirks and syntax of a new language while doing this, is, IMO, a recipe for disaster and immense frustration . You need to find code written in a language that you are familiar with. This particular problem is not relevant if you are going through the code distributed as part of a book or as an educational resource, since you would have the book or your mentor to explain things and set out the context. If, despite this forewarning, you are planning to read code written in a different language than you are used to (without the benefit of having a book or a mentor) I would advise, that you at least learn enough of the language to create your own programs  in it (“Hello World” does not count :-)) .
• The bit about context brings me to the next problem – figuring out what the code is doing is a lot harder if you are not familiar with the software itself. For example, it is far more difficult to go through the Linux code and figure out the concept of runlevels if you don’t use Linux daily and see the Linux boot sequence. Using the software gives one a context with which to read the code – this context includes the common terminology used, the functionality and features of the software, even the quirks and bugs that you experience.

I have realized that for me ‘reading code’ does not really describe the activities that I undertake – a better phrase for what I do is ‘code comprehension’. It is quite difficult for me to sit back with a laptop screen (or a printout) full of code and simply read through it. I need a lot more than simply a piece of code – I like to be able to look at documentation, play with the software, step through the code and even write tests for it before I really appreciate it. This is a significant investment of my time and effort, so I have to be very picky about the software I want to “read” (comprehend).

• The first filter I place on the code directory, when looking for code  is the language filter – for me this means – C# or VB.NET or Python or Javascript(while I am familiar with C++, Ruby and F# as well I do not consider myself at a level where I can understand other people’s code in them). Next is to look for software that I have used – this allows me the a bit of a leg up since I know what the code is meant to do, cannot do and (if I am familiar enough) its limitations. Good candidates are open source software that you use in your day job (for eg. I use Cruise Control.NET, NANT and NUnit which are open source tools written in C#)
• I happen to work in a software product company (a Microsoft shop), so one of the candidates for my reading list is the code in my companies source repository. If you happen to work in a software company, you can look at other projects, and even older versions of the software you are working on. In addition to providing insight on code, you get a pretty good idea of what was tried before and since. There are a few caveats though –
  • First, if you don’t have direct access to other projects, you need to ask permission – some companies are very touchy about their “intellectual property”.
  • Second, the quality of the software may not be as high as you think, since, in general, proprietary code does not get the kind of scrutiny open source code does. Warning signs to look out for are a lack of regular code reviews – if the software is not code reviewed the odds are that it would not be of good quality.
  • Third (this point is inspired from feedback provided by my friend Praseed), if the code in your company is business software (HR, Finance, ERP, etc) there is a lot of business context that needs to be understood first. Also, since most of this code tends to be factored by business functionality, it generally seems less modular than utility code or APIs.
• Look for well documented projects (this applies to open source as well as proprietary code).  By this I mean, that the documentation should highlight the overall design, and rationale for the way the code is. Simply having auto-generated Java Doc type documents cannot be considered documentation :-). One useful avenue to explore is software created as educational resources (like Minix ). Since, the target is to teach through the software, they are usually quite clearly documented and have plenty of material explaining the design rationale behind the code.

So, you have identified the software and downloaded the source code and documentation, so let’s get down and start spelunking ;-)

• Go through the design documentation and try to get a feel for the way the code has been built. Good software projects follow certain architectural patterns – these dictate the code organization. Once you get a handle on this, understanding the code becomes a whole lot easier. If you can create a class diagram of the code you can get a good idea of the layout.
• The next thing to do is to compile it and run it. This can be straightforward or tough depending on the process followed in the project and it’s documentation.
• Now it’s time to fire up your favorite IDE and go exploring. A good place to start your code exploration  would be to try to trace a functionality of the project that you are familiar with. This would let you go through the various layers and sub-systems and get a handle on how they inter-connect. For example when I was exploring NUnit – I started by writing a test and looking at the code classes I needed to do that.
• Try and identify the design patterns used in the code. If you do not know what design patterns are, then you need to stop reading this post right now and read this book. Familiarize yourself with design patterns – they form a great way to recognize and understand the design of well written code. This makes it easier to keep it in your head while reading code. It also helps you identify nuances and customizations made by the programmers more easily.
• Try to write tests for the code to fully understand it – this is really useful way to understand the dependencies between different parts of the code. When you try to write a test for the code you first need to satisfy (mock) all its dependencies. Next you need to understand the possible entry points as well as the exit values for the code. This improves your understanding of the code and get you to the next level.
• Finally, try to refactor the code. In this step you have moved from simply understanding the code to becoming familiar enough to be able to modify it. As the sophistication of your refactoring increases so too does your understanding. At this point you can if needed contribute your own code to the project :-)

“Code Reading” IMO is more than just reading – it is a distinct set of activities that together help one understand code. It might seem more intimidating than simply “reading code” but it is well worth then effort IMO.

Happy “code reading” :-)

Update: I came across this post by Joel Spolsky where he quotes Seth Gordon as saying code reading “Is just like reading the Talmud”… Yup, code reading is definitely not easy.

The Humble Programmer – Edsger W Dijkstra

I first heard of Edsger W Dijkstra in the context of agile programming.I was having a discussion regarding agile programming with some friends and explaining Test Driven Development and the concept of first creating tests that can prove show the correctness of the code before writing the code, when a friend mentioned that this sounded a lot like some of the arguments put forward by Prof. Dijkstra in his Turing award lecture in 1972. I found that hard to believe, after all, if this was known in 1972 then why is it only becoming popular now ?

So I started looking up Edsger W Dijkstra and realized that this man was one of the pioneering giants of software programming. He is the father of structured programming and one of the guiding heads responsible for much of the way we program computers today. There is a lot written about him all over the place – I shall focus on his Turing award lecture, that was titled “The Humble Programmer“. In this lecture, Prof. Dijkstra puts forth six arguments on the way software programming should be done. On reading these six arguments I cannot help but feel that this lecture was one of the main inspirations used by the authors of the agile programming movement and design patterns community.

The six arguments put forward in the lecture are as follows –

1. “A study of program structure had revealed that programs —even alternative programs for the same task and with the same mathematical content— can differ tremendously in their intellectual manageability. A number of rules have been discovered, violation of which will either seriously impair or totally destroy the intellectual manageability of the program.I now suggest that we confine ourselves to the design and implementation of intellectually manageable programs. The programmer only needs to consider intellectually manageable programs, the alternative alternatives he is choosing from are much, much easier to cope with.”
2. “As soon as we have decided to restrict ourselves to the subject of intellectually manageable programs, we have achieved once and for all a drastic reduction of the solution space to be considered. This argument is distinct from argument 1.”
3. “If one first asks oneself what the structure of a convincing proof would be and having found this, then construct a program satisfying this proofs requirements, the these correctness concerns turn out to be a very effective heuristic guidance. By definition this approach is only applicable if restrict ourselves to intellectually manageable programs.”
4. “The only mental tool by which a very finite piece of reasoning can cover a myriad of cases is called an “abstraction”. There are number of patterns of abstraction that play a vital role in the construction of programs. Knowledge of these patterns of abstraction are essential.”
5. “A programmer is fully aware of the limited size of his own skull; so he approaches the task of programming in full humility and avoids clever tricks.”
6. “The only solutionproblems we can solve in a satisfactory manner are those that finally admit a nicely factored solution.”

When you go through these arguments you see the seeds for the various movements in programming software today –

• Arguments 1 and 2 are the foundation of principles like KISS (Keep it Simple and Stupid) and YAGNI (You Ain’t Gonna Need It)
• Argument 3 forms the basis of efforts like  Test Driven Development and the Design By Contract.
• Argument 4 leads us to Design Patterns.
• Argument 5 and 6 preaches humility in programming, the use of abstraction to write readable well factored programs – beautiful code

I really enjoyed reading his lecture – I have the lecture here (The_Humble_Programmer)  if you want to read it. He must have been an extremely engaging speaker – a lot of his quotes are available here.

Till next time – Happy Programming !

Update: I have made a couple of corrections based of some of the comments here.  When I wrote that argument 3 is the basis for TDD I meant that in the lecture Dijkstra talks about first finding the structure of a proof and then constructing the program satisfying the proofs requirement. This is similar to the TDD approach of first writing a test and then writing code that satisfies the test.

Ganesha – the original lateral thinker

There is an ancient tale from Hindu mythology that illustrates lateral thinking (also known as –  “out of the box” thinking) that I would  like to share –

One day Lord Siva and His consort Parvati were sitting atop their abode on Mt. Kailash with their sons Ganesha and Karthikeyan when the sage Narada dropped by for a visit. Narada had with him a special mango of knowledge, to offer to Siva. After accepting the mango from Narada,  Siva and Parvati decided to have a contest between their sons.

Ganesha

Karthikeyan

The first one who circumnavigates the world three times would get to the mango of knowledge. Without further ado Karthikeyan  jumped on his peacock and started off. Ganesha on the other hand was busy eating his favorite ladoos and decided to finish them first. Karthikeyan had completed two rounds by the time Ganesha finally got ready to compete :-)

Ganesha simply approached Siva and Parvati and deliberately walked around them –  He circled them once, twice and three times and then claimed the mango.

When, Siva and Parvati asked him how he could claim the mango when he had not circled the world even once – Ganesha replied – “You both are my world”. Delighted by the answer Siva and Parvati gave Ganesha the mango, which he immediately gobbled up with relish.

Two of the important traits of good software developers are “enlightened laziness” and “Out of the box” thinking.  This tale is an example of both enlightened laziness and out of the box thinking – confronted by the immense task of circumnavigating the world – Ganesha – by simply thinking a little and restating the problem comprehensively defeated his brother Karthikeyan.

So, my eager friends – the ones who are chomping at the bit after the initial presentation of a project – eager to rush into coding it, please spend some time contemplating your problem.  Another, homily you might want to consider is –  “Think twice, code once” – You, might just save yourself a LOT of time and effort !! :-)

Juggling code – the coding zone and burnouts..

Software programming is a very mentally intensive activity. In any non-trivial software system the coder has to juggle a large number of mental models. Like a juggler, a coder,  has to mentally juggle not only the actual code that he/she is writing, but,  details of the code that it is related to, the details of the data being manipulated, the possible errors to be handled, the reliability and performance of the code, it’s security characteristics, the requirement that is being implemented and it’s design and usability, etc (depending on the code there may be more to think of or if you lucky, less :-)). Unlike a juggler who generally juggles things of similar size,  a coder mentally juggles problems whose complexity vary by several orders of magnitude (1 – 10⁹).

Given all this, it takes time for coders to become truly productive when they sit down and start working on something.  Once you get into the what I call the – coding zone , you find the ideas flowing through you seamlessly – coders in the zone lose sense of time and place – the problems and solutions are clear and you find beautiful code coming from your keyboard. Coding when you are in the zone is an immensely satisfying task – it’s like the zone that sportspeople talk about – when they are breaking records, it seems like they are unstoppable and every movement is a beautiful ballet…

This is also why almost all good coders HATE BEING INTERRUPTED !! Whether it is a simple phone call or even a well meaning colleague coming over to tap you on your shoulder and ask a question – the effect is the same as though the coder was invited to a long meeting. It takes time and effort to get back to being productive  after the interruption.

There are several other things that contribute to this problem –

1. “Open Office” plans where you are compelled to hear your neighbors  conversations.
2. Having one phone for several people in your area so you cant disconnect it and have to attend it on the off chance it yours.
3. Conversations over information that can be sent by email or IM or SMS or any of the multitude of asynchronous forms of communication available today.

I have seen several ways to combat this  –

1. Some people wear head-phones to block the ambient noise and subtly indicate to people they are working on something and interruptions are not encouraged (YMMV – I have seen people ignore the subtle indication and come over anyway).
2. Some people deal with all their email and IM at scheduled intervals – this way everyone gets their reply and people learn to come with the questions at those times.
3. If you are lucky enough to have a cabin then disconnecting the phone and leaving a message on the door is often effective.
4. Some companies even plan their meetings to happen only on certain days so everyday disruption is minimized.
5. A common inclination is to work at times when no-one else is around to bother you. This is a reason why coders are night owls :-)

Another effect of software programming is burn-out… This is the opposite of being in the coding zone, but it seems to be consequence of being in one…  Like I mentioned in the beginning of this post – software programming is a very mentally intensive activity. Coders have frequently felt mentally burned-out after intensive coding sessions.  This happens more quickly on projects which you don’t find interesting or enjoyable. Sometimes you can continue only for a day, other times it’s a month but invariably – burn-out happens.The key is to recognize it for what it is and deal with it.

Indeed,  when I previously mentioned that coders lose all sense of time when in the zone, I did not mean that they should spend all their time coding.  I am not  applauding coders that brag about sitting for 36 hours at a computer churning out code. Those, that spend 16 hours a day at the terminal and spend their nights dreaming about code are, invariably, the ones whose work the rest of the team has to spend the rest of the month fixing. Like in all things,  there is a balance that needs to be maintained. Spending long amounts of time in intense concentration is tiring, and it is important to give things a rest. It is usually great to take some time off doing something else, like mountain biking, mixed martial arts, flying a plane or playing an instrument (these are pastimes of some of my friends :-)). Some people like physical activity others like mental activities like video games, or chess. The important thing is to have a balance. Sometimes, when you are grappling with a hard problem it is useful to stop thinking about it consciously and let your sub-conscious chew over it.

When you are no longer in the zone and are spinning your wheels, a break is the most productive thing you can do.

The Zen of Programming is being able to get into the zone and more importantly to recognize when you are no longer in it and take that break ! :-)

Update: I came across this article the other day that got me thinking about burn-out.  I mentioned before that burn-out happens more quickly when doing something you don’t find interesting or enjoyable – this advise goes in spades when you are doing something you feel is morally wrong or that goes against your conscience.  Guilt is a catalyst that will accelerate both the speed and the intensity of your burn-out.

IMHO if given a choice it is much more satisfying to do something you believe in at a lower salary than something you don’t at a higher one.

Learning Software Programming Takes Time

This comic created by the Abstruse Goose is one of my all time favorites.

It embodies my frustration at people who think that all  becoming a software programmer involves is learning a little computer syntax , reading a few books and typing out a lot of code.

The books that claim you can “Teach Yourself” X in Y “days/hours” etc are written by charlatans out to make a quick buck (I think sometimes it is the publishers that foist such titles on the authors in the hope to sell a few more copies). All they do is frustrate the people who are genuinely trying to learn software programming and provide PHB types with ammunition – after all if programming can be learned in X time then any schmuck who can read and knows a bit of typing should be able to become a programmer, at least by 2X the time.

Interestingly, no one writes a book on “Teaching Yourself” building a bridge or a skyscraper in X days – after all building construction is a popular metaphor in software engineering. The presence of these books are IMO a symptom of the ignorance that people have about what it takes to write solid software.

As this post points out – it can take up to ten years for an artist, researcher or a sportsman to be considered a master in his or her field. Software programming is no different.