You can scroll the shelf using ← and → keys
You can scroll the shelf using ← and → keys
This is an excerpt from our new book, “Rapid Rails with Hobo”
Hobo is a software framework that radically reduces the effort required to develop database-driven, interactive web-sites and web-based applications. Strictly speaking it’s more of a “half-framework” — Hobo builds on the amazingly successful Ruby on Rails and that’s where much of the functionality comes from. The original motivation for the Hobo project can be summed up pretty succinctly with a single sentiment: “Do I really have to code all this stuff up again?”.
In other words Hobo is about not re-inventing the wheel. In software-engineer-speak, we call that code reuse. If you mention that term in a room full of experienced programmers you’ll probably find yourself the recipient of various frowns and sighs; you might even get laughed at. It all sounds so simple – if you’ve done it before just go dig out that code and use it again. The trouble is, the thing you want to do this time is just a bit different, here and there, from what you did last time. That innocuous sounding “just a bit different” turns out to be a twelve headed beast that eats up 150% of your budget and stomps all over your deadline. Re-use, it turns out, is a very tough problem. Real programmers know this. Real programmers code it up from scratch.
Except they don’t. Ask any programmer to list the existing software technologies they drew upon to create their Amazing New Thing and you had better have a lot of time to spare. Modern programming languages ship with huge class libraries, we rely on databases that have unthinkable amounts of engineering time invested in them, and our web browsers have been growing more and more sophisticated for years. Nowadays we also draw upon very sophisticated online services, for example web based mapping and geo-location, and we add features to our products that would otherwise have been far beyond our reach.
So it turns out the quest for re-use has been a great success after all—we just have to change our perspective slightly, and look at the infrastructure our application is built on rather than the application code itself. This is probably because our attitude to infrastructure is different—you like it or lump it. If your mapping service doesn’t provide a certain feature, you just do without. You can’t dream of coding up your own mapping service, and some maps is better than no maps.
We’ve traded flexibility for reach, and boy is it a good trade.
Programmers get to stand on the shoulders of giants. Small teams with relatively tiny budgets can now successfully take on projects that would have been unthinkable a decade ago. How far can this trend continue? Can team sizes be reduced to one? Can timelines be measured in days or weeks instead of months and years? The answer is yes, if you are willing to trade flexibility for reach.
In part, this is what Hobo is about. If you’re prepared for your app to sit firmly inside the box of Hobo’s “standard database app”, you can be up and running with startlingly little effort. So little in fact that you can just about squeeze by without even knowing how to program. But that’s only one part of Hobo. The other part comes from the fact that nobody likes to be boxed in. What if I am a programmer, or I have access to programmers? What if I don’t mind spending more time on this project?
We would like this “flexibility for reach” tradeoff to be a bit more fluid. Can I buy back some flexibility by adding more programming skills and more time? In the past this has been a huge problem. Lots of products have made it incredibly easy to create a simple database app, but adding flexibility has been an all-or-nothing proposition. You could either stick with the out-of-the-box application, or jump off the “scripting extensions” cliff, at which point things get awfully similar to coding the app from scratch.
This, we believe, is where Hobo is a real step forward. Hobo is all about choosing the balance between flexibility and reach that works for your particular project. You can start with the out-of-the box solution and have something up and running in your first afternoon. You can then identify the things you’d like to tweak and decide if you want to invest programming effort in them. You can do this, bit by bit, on any aspect of your application, from tiny touches to the user-interface, all the way up to full-blown custom features.
In the long run, and we’re very much still on the journey, we hope you will never again have to say “Do I really have to code all this up again?”, because you’ll only ever be coding the things that are unique to this particular project. To be honest that’s probably a bit of a utopian dream, and some readers will probably be scoffing at this point—you’ve heard it all before. But if we can make some progress, any progress in that direction, that’s got to be good, right? Well we think we’ve made a ton of progress already, and there’s plenty more to come!
A brief look at the history leading up to Hobo might be helpful to put things in context. We’ll start back in ancient times — 2004. At that time the web development scene was hugely dominated by Java with its “enterprise” frameworks like EJB, Struts and Hibernate. It would be easy, at this point, to launch into a lengthy rant about over-engineered technology that was designed by committee and is painful to program with. But that has all been done before. Suffice it to say that many programmers felt that they were spending way to much time writing repetitive “boilerplate” code and the dreaded XML configuration files, instead of focusing on the really creative stuff that was unique to their project. Not fun and definitely not efficient.
One fellow managed to voice his concerns much more loudly than anyone else, by showing a better way. In 2004 David Heinemeier Hansson released a different kind of framework for building web apps, using a then little-known language called Ruby. A video was released in which Hansson created a working database-driven Weblog application from scratch in less than 15 minutes. That video was impressive enough to rapidly circulated the globe, and before anyone really even knew what it was, the Ruby on Rails framework was famous.
Like most technologies that grow rapidly on a wave of hype, Rails (as it is known for short) was often dismissed as a passing fad. Five years later the record shows otherwise. Rails is now supported by all of the major software companies and powers many household-name websites.
So what was, and is, so special about Ruby on Rails? There are a thousand tiny answers to that question, but they all pretty much come down to one overarching attitude. Rails is, to quote its creator, opinionated software. The basic idea is very simple: instead of starting with a blank slate and requiring the programmer to specify every little detail, Rails starts with a strong set of opinions about how things should work. Conventions which “just work” 95% of the time. “Convention over Configuration” is the mantra. If you find yourself in the 5% case where these conventions don’t fit, you can usually code your way out of trouble with a bit of extra effort. For the other 95% Rails just saved you a ton of boring, repetitive work.
In the previous section we talked about trading flexibility for reach. Convention over configuration is pretty much the same deal: don’t require the programmer to make every little choice; make some assumptions and move swiftly on. The thinking behind Hobo is very much inspired by Rails. We’re finding out just how far the idea of convention over configuration can be pushed. For my part, the experience of learning Rails was a real eye-opener, but I immediately wanted more.
I found that certain aspects of Rails development were a real joy. The “conventions”—the stuff that Rails did for you—were so strong that you were literally just saying what you wanted, and Rails would just make it happen. We call this “declarative programming”. Instead of spelling out the details of a process that would achieve the desired result, you just declare what you want, and the framework makes it happen. What not how.
The trouble was that Rails achieved these heights in some areas, but not all. In particular, when it came to building the user interface to your application, you found yourself having to spell things out the long way.
It turned out this was very much a conscious decision in the design of Ruby on Rails. David Heinemeier Hansson had seen too many projects bitten by what he saw as the “mirage” of high level components:
I worked in a J2EE shop for seven months that tried to pursue the component pipe dream for community tools with chats, user management, forums, calendars. The whole shebang. And I saw how poorly it adapted to different needs of the particular projects.
On the surface, the dream of components sounds great and cursory overviews of new projects also appear to be “a perfect fit”. But they never are. Reuse is hard. Parameterized reuse is even harder. And in the end, you’re left with all the complexity of a Swiss army knife that does everything for no one at great cost and pain.
I must say I find it easy to agree with this perspective, and many projects did seem, in hindsight, to have been chasing a mirage. But it’s also a hugely dissatisfying position. Surely we don’t have to resign ourselves to re-inventing the wheel forever? So while the incredibly talented team behind Rails have been making the foundations stronger, we’ve been trying to find out how high we can build on top of those foundations. Rather than a problem, we see a question — why do these ideas work so well in some parts of Rails but not others? What new ideas do we need to be able to take convention over configuration and declarative programming to higher and higher levels? Over the last couple of years we’ve come up with some pretty interesting answers to those questions.
In fact one answer seems to be standing out as the key. It’s been hinted at already, but it will become clearer in the next section when we compare Hobo to some other seemingly similar projects.
There are a number of projects out there that bear an external resemblance to Hobo. To name a few, in the Rails world we have Active Scaffold and Streamlined, and the Python language has Django, a web framework with some similar features.
There is some genuine overlap between these projects and Hobo. All of them (including Hobo) can be used to create so called “admin interfaces”. That is, they are very good at providing a straightforward user-interface for creating, editing and deleting records in our various database tables. The idea is that the site administrator, who has a good understanding of how everything works, does not need a custom crafted user-interface in order to perform all manner of behind-the-scenes maintenance tasks. A simple example might be editing the price of a product in a store. In other words, the admin interface is a known quantity: they are all largely the same.
Active Scaffold, Streamlined, Django and Hobo can all provide working admin sites like these with very little or even no programming effort. This is extremely useful, but Hobo goes much further. The big difference is that the benefits Hobo provides apply to the whole application, not just the admin interface, and this difference comes from Hobo’s approach to customization.
Broadly speaking, these “admin site builder” projects provide you a very complete and useful out-of-the-box solution. There will be a great number of options that can be tweaked and changed, but these will only refine rather than reinvent the end result. Once you’ve seen one of these admin-sites, you’ve pretty much seen them all. That’s exactly why these tools are used for admin sites – it generally just doesn’t matter if your admin site is very alike any other. The same is far from true for the user-facing pieces of your application—those need to be carefully crafted to suit the needs of your users.
Hobo has a very different approach. Instead of providing options, Hobo provides a powerful parameterization mechanism that lets you reach in and completely replace any piece of the generated user-interface, from the tiny to the large.
This difference leads to something very significant: it gets you out of making a difficult all-or-nothing decision. An admin site builder does one thing well, but stops there. For every piece of your site you need to decide: admin interface or custom code? With Hobo you can start off using the out-of-the-box UI as a rough prototype, and then gradually replace as much or as little as you need in order to get the exact user experience you are after.
Once again we find ourselves back at the original idea: making a tradeoff between flexibility and reach. The crucial difference with Hobo, is that you get to make this trade-off in a very fine-grained way. Instead of all-or-nothing decisions (admin-site-builder vs. custom-code), you make a stream of tiny decisions. Should I stick with Hobo’s automatically generated form? Sidebar? Button? How long would it take me to replace that with something better? Is it worth it?
There is a wide spectrum of possibilities, ranging from a complete out-of-the-box solution at one end to a fully tailored application at the other. Hobo lets you pick any point on this spectrum according to whatever makes sense right now. Not only that but you don’t have to pick a point for the app as a whole. You get to make this decision for each page, and even each small piece of each page.
The previous section posed the question: “how can the ideas of declarative programming be taken to higher and higher levels?”. We mentioned before that one particular answer to this question has stood out as crucial: it is the approach we have taken to customization. It’s not what your components can do, it’s how they can be changed that matters. This makes sense—software development is a creative activity. Developers need to take what you’re giving them and do something new with it.
It is this difficulty of customization that lies at the heart of concerns with high-level components: David Heinemeier Hansson again:
…high-level components are a mirage: By the time they become interesting, their fitting will require more work than creating something from scratch.
The typical story goes like this: you need to build something that “surely someone must have done before?”; you find a likely candidate – maybe an open-source plugin or an application that you think you can integrate; then as you start the work of adjusting it to your needs it slowly becomes apparent that it’s going to be far harder than you had anticipated. Eventually you end up wishing you had built the thing yourself in the first place.
To the optimistic however, a problem is just an opportunity waiting to be taken. We’re hitting a limit on the size of the components we can build—too big and it the effort to tailor them makes it counterproductive. Turn that around and you get this: if you can find a way to make customization easier, then you can build bigger components. If it’s the “fitting” that’s the problem, let’s make them easier to fit! That’s exactly what we’re doing.
At the time of writing we are just mopping up the last few bugs on the list before the release of Hobo version 1.0. It looks like we’re finished! In fact we’re just getting started.
Obviously the whole point in discovering the secrets of how to build high-level components, is that you want to build some high level components! In other words there are two distinct aspects to the Hobo project: getting the underlying technology right, and then building some cool stuff with it. Hobo 1.0 will ship with a decent library of useful “building blocks” to get your app up and running quickly, but there’s so much more we’d like to see. This is where the magic of open-source needs to come into play. The better Hobo gets, the more developers will want to jump on board, and the bigger the library will grow.
Although the underlying framework is the most technically challenging part of the project, in the long run there’s much more work to be done in the libraries. And writing the code is just part of the story. All these contributions will need to be documented and catalogued too.
We’ve started putting the infrastructure in place with “The Hobo Cookbook” website (http://cookbook.hobocentral.net) – a central home for both the “official” and user-contributed documentation.
It would be remiss not to mention that all these wonderful productivity gains do come at a cost – a Hobo application does have an extra performance overhead compared to a “normal” Rails application. Experience has shown it’s not really a big problem – many people are using Hobo to prototype, or to create a very niche application for a small audience. In these cases the performance overhead just doesn’t matter. If you do have a more serious application that may need to scale, there are well known techniques to apply, such as prudent use of caching.
The argument is pretty much the same as that told by early Rails coders to their Java based critics. It’s much better to save a ton of development time, even if it costs you some of your raw performance. The time saved can be used to work on performance improvements in the architecture of the app. You typically end up with an app that’s actually faster than something built in a lower-level, “faster” language.
Another way to look at it—it was about four our five years ago that Rails was getting a lot of push-back about performance. In those four or five years, Moore’s Law has made our servers somewhere between five and ten times faster. If Rails was fast enough in 2005 (it was), Hobo is certainly fast enough today.
Having said all that, it’s always nice to give people more performance out-of-the-box and postpone the day that they have to resort to app-specific efforts. Just as Rails has focused a lot on performance in the last couple of years, this is definitely an area that we will focus on in the future.
One of the most common criticisms leveled against Hobo is that it is “too magic”. This tends to come from very experienced developers who like to know exactly how everything is working. Because Hobo gives you so much out-of-the-box, it’s inevitable that you’ll be scratching your head a bit about where it all comes from in the early days. Fortunately this is mostly just a matter of the learning curve. Once you’ve oriented yourself, it’s pretty easy to understand where the various features come from, and hence where to look when you need to customize.
As Hobo has developed, we’ve definitely learnt how important it is to make things as clear and transparent as we can. The changes from Hobo 0.7 to 0.8 removed a great deal of hard to understand “magical” code. This is definitely a trend that will continue. We’re very confident that future versions will be able to do even more for you, while at the same time being easier to understand. It’s a challenge—we like challenges!
Even higher level
One of the really interesting things we’ve learnt through releasing Hobo as open source, has been that it has a very strong appeal to beginners. It is very common for a post to the “hobousers” discussion group to start “I am new to web programming” or “This is my first attempt to create a web app”. It seems that, with Hobo, people can see that a finished result is within their reach. That is a powerful motivator.
Now that we’ve seen that appeal, it’s really interesting to find out how far we can push it. We’ve already seen simple Hobo applications created by people that don’t really know computer programming at all. Right now these people are really rather limited, but perhaps they can go further.
Hobo has ended up serving two very different audiences: experienced programmers looking for higher productivity, and beginners looking to achieve things they otherwise couldn’t. Trying to serve both audiences might sound like a mistake, but in fact it captures what Hobo is all about. Our challenge is to allow the programmer to choose their own position on a continuous spectrum from “incredibly easy” to “perfectly customized”.
Hopefully this introduction has whetted you’re appetite and you’re keen to roll up your sleeves and find out how it all works. While this section has been a bit on the philosophical side, the rest of the book is eminently practical. From now on we’ll dispense with all the high-brow pontificating and teach you how to make stuff. Enjoy!
Excellent introduction Tom.