You don’t do Repository Driven Development yet? Where have you been?

I have been doing a lot of software development and related work by now and there are some things I just don’t get. One of them is “Why are people not using repositories more?” and practice what I would call Repository Driven Development (RDD sic!). Once you are using them, the benefits are just so obvious, that it is hard to imagine why you would not want to allow easy reuse of your artifacts, seemless distribution, ease of use for your end users and more. But let’s step back for a second and start at the beginning.

Working a lot with mobile development and in the enterprise development sector in terms of actual development as well as training and teaching I am observing a trend towards using repositories in various shapes and names for the artifacts you are working with more and more. The choice and actual usage of these repositories seems to strongly influence the ease and control during your software life cycle. If you are not using a dedicated tool – you are missing out.

To get the biggest benefits you have to, in my humble opinion, move towards using more and more of these repositories and dedicated tools and practice what I would like to call Repository Driven Development (RDD). So what do I mean by all this in detail?

Remember back in the day when version control systems were painful to use and needed a server to run? I am sure there were a few of you that decided not to bother and just “be careful” and take “regular copies”. Admittedly I was in the situation as well and rest assured – I can not imagine ever doing that again. Nowadays it is a no brainer to use tools like git and anybody doing any development will use a version control system of some sort. This is what I call the source code repository” stage of RDD and hopefully suitable tools like git, mercurial or others are used by everybody now.

In the next stages tool usage is a lot less common today, but the benefits to be gained are as dramatic. So what are the next stages?

Once your code is in a source code repository your might be manually checking code out and building the software. Ideally you would actually use a continuous integration server like Hudson to do that job for you on a regular basis. So when I say do the build – what do I actually mean. Just compile it? Or run the unit tests? Or do the whole thing by running e.g. mvn clean install? While all these are good steps, they only confirm that the build can produce an output artifact. And the tools available here range from shell scripts and make to powerful tools like Maven, Tesla or Gradle.

To be truly useful however this output artifact needs to be shared with e.g. other developers, other CI server builds or even other teams like the QA team. This is achieved by what I call a build output repository. Typically these are repository servers like Sonatype Nexus, but they can also be shared directories or local package distribution servers like a rpm or deb server. In all cases the main point will be that they are distributing binary artifacts for reuse, therefore avoid the need to build the component for downstream use.

The downstream use is often another build or assembly creation that produces an output suitable for deployment to the end users. They would for example pull together a bunch of jar artifacts into a war and then assemble some wars together into an ear for deployment and put those build or assembly outputs into another repository. I would call these software distribution repositories and in some instances they can be controlled by the same party as the build output repository and in fact use the same tools like Nexus or APT or RPM repository server.

Often however these tools are controlled by entities specializing in the distribution of the artifacts to their end users. They can be e.g. a large publically available repository catered to binary artifacts for development and distribution like the Central Repository or the NuGet Gallery but also more end user focussed repositories like a Linux distributions package repository or an Eclipse update site e.g. for the Android development tools. Even more specifically catering to end users places these repositories can take be tools like the Google Play Store.

Whatever repository you use to distribute your software, in all cases you will be gaining tremendous benefits from using a repository rather than reinventing the wheel in terms of package format, accessibility and discoverability for the artifacts, payment integration and so on.

The benefits from a build output repository as well as a software distribution repository are of much larger scale than the mere usage of a version control system, yet many companies still think they should work without any of these tools and roll their own development infrastructure to share binaries and then use things like install package creation tools and offer links on their website for manual (!!) download of that installer.

As a natural next step from providing your artifacts in a software distribtion repository follows, that the end user platform uses a repository for managing the installed components. Linux users have been wondering for years how Mac and Windows users can stand dealing with setup.exe, installers and manually installing software with dmg files. Now thanks to app stores on phones even mainstream users are waking up to the fact that manually installing and updating software is just totally last century. I would call this last repository the deployment repository.

It is often a combination of some sort of front end tool to access one or multiple software distribution repositories and a tool that holds inventory of installed applications and takes care of necessary updates. Front end tools can be as shiny as the Google Play store application on your phone, a plain GUI tool like Synaptic or a mere command line tools like apt or yum.

In many cases these tools are called package manager under the hood, but don’t actually have a user visible component. They hide in the background and manage applications, but do with minimal interaction relieving the user from unnecessary maintenance.

The tools to move applications from the software distribution repository to the deployment repository vary widely and include automation tools like Chef and Puppet that make deployment to lots of clients possible in an automated fashion.

It is possible to do all that without tools. But there are dedicated tools for all the migrations from one level to the next and repository tools for managing each level. Using the tools and repositories will greatly improve the transitions from one repo to the next as well as greatly reduce the work necessary at each repo level and still make it smoother in terms of getting your software from the source to the actual users. And in the end.. thats what we all really want. Ship it!

So what is repository driven development” then? Nothing more than the mindset to be aware of the fact there are tools for all these repository management tasks and the transitions in between them and that you can get tremendous benefits, if you choose to use them rather than reinventing the wheel and build e.g. your own deployment scripts or force the users to do manual installs because you just provide an installer and not an operating specific package.

Just think of the benefits you gained when you first started using version control systems and the first occassion that safed your rear end. And then think of the advantages your users get from easy discoverability for your software, painless installs and automatic updates. And now think of all the good stuff that awaits when you start doing some RDD. What are you waiting for?

manfred