iDoc: Hedtek’s open source community documentation system

iDoc is an open source community documentation system for the production and use of help documentation for web sites and applications. It’s more than just a vanilla documentation system:  We believe is important is for the users of an application to be able to contribute back and help others learn to use an application application — iDoc allows this by allowing users to sign up and comment on help pages. It is also possible to set iDoc to allow any user to create or edit help pages, indeed, this is currently the default behaviour!

Why another documentation system?

There were several motivations that inspired this project.

The foremost reason was that we wanted an easy-to-use documentation system for the Manchester PLE, where our desire was to host very short (from a few seconds to something less than a minute)  video’s to help newcomers use the system. This isn’t to imply that the PLE is hard to use, but rather that some potential users will have no idea, e.g., what a blog is, nor how to use blogs within the system. Equally, we don’t see ourselves producing all the documentation, rather we expect some to be produced and/or improved by our communities of users.

Most documentation systems we investigated involved the use of some specialised markup to produce the content from some other source. This isn’t desirable for a system geared towards general users of an application, as they may not be technically skilled, and, even if they were, we imagined that they most likely wouldn’t want to learn yet another syntax for creating content. It’s for this reason that iDoc provides WYSIWYG editors to create pages and add comments. You can even use the WYSIWYG editor to embed videos from YouTube and Vimeo in both help text and comments, and this will be expanded in the future to provide for videos from other sources.

The second reason was to test a new development approach for use within Hedtek. The approach we are now using is user story-driven approach, Behavior Driven Development (BDD). In short there are various advantages to BDD, but pretty much its a scenario-based design technique, which leverages the automated system development testing espoused by Test Driven Design (TDD) and enables accurate estimations of the development resource needed to implement the system under consideration. Also importantly, BDD uses iterations in the development, in a similar way that SCRUM uses sprints to deliver working systems at the end of each sprint. We wanted a smaller (but non-trivial and useful) project to get up to speed and make sure we were comfortable in the process before using it on larger projects where a new process could simply be one more unknown that might add problems to the development process. We are glad to say that BDD worked tremendously well for us and is now our default approach for all new projects we undertake.

Finally, there was the desire to have a non-trivial project developed in a carefully engineered way as an example in a new Software Engineering course we are developing for and delivering in the Advanced Master’s programme in the University of Manchester’s School of Computer Science.

The current iDoc status

After two of four iterations, iDoc has reached the stage where it provides useful documentation facilities. It provides a clean URL syntax for pages, WYSIWYG editors for content, and drag-and-drop editors for positioning pages in a documentation tree. Users can sign up, develop and publish page content, and comment on pages. There are some administration features that can, if so desired, limit who can create and edit help page body content, and there is the ability to select community moderators who can remove unsanitary comments from comment lists. There is also search functionality to more easily find relevant parts of the documentation, backed by Xapian search so the search is capable of case-insensitive searches, wildcards, boolean queries and all the other niceties of an advanced full-text search.

Technologies

iDoc has been developed with the latest stable Ruby on Rails stack (version 2.3.5). It requires backing by a database, and was developed using a PostgreSQL database. However, iDoc is currently relational database agnostic. The site uses Haml to provide HTML templates, and Compass to provide semantic CSS. Development made use of several Ruby and Rails development tools, including RSpec, Cucumber and Webrat. For dependency resolution, iDoc was recently migrated to make use of Bundler, a new Ruby system for specifying and installing libraries using the RubyGem package manager. It should be easy to install iDoc onto a system with Ruby and Xapian installed, and deployment can be automated with Capistrano. Current installations of iDoc are served up with Apache and Passenger, although again there shouldn’t be any problems with other Rails server technologies, such as Mongrel, Thin, Unicorn or even a plain old FastCGI server.

For Xapian integration, a new Ruby gem, Xapor, was developed to allow for the specification of Xapian indexes very easily through ActiveRecord. The interface is inspired very much by other popular search plugins, such as ThinkingSphinx. Xapor is currently at version 0.1.2, with the features required for iDoc currently implemented. It is intended to continue development of this gem to provide a much more sophisticated search integration with Xapian. Xapor currently also relies on the gem XapianFu, which provides an easier interface for Xapian than the plain Ruby bindings but isn’t particularly sophisticated for ActiveRecord integration. I open sourced Xapor on Github and the gem is available on Rubygems.

Future development

There are several more features that we want to integrate into future versions of iDoc. Some of these include:

• Notifications. It would be highly desirable to have notifications available to users. These would include things such as page additions, deletions, updates and new comments. Currently, it is planned that these would be served as an RSS feed, although other notification systems could also be added (e.g. twitter and/or email)
• Published APIs. iDoc isn’t intended as a standalone system, and although it is easy currently to just embed help links or navigate the documentation by hand, it would be much nicer if some degree of automation could be performed.
• An improved user login experience. Login currently relies on registration, but could be extended to include the use of other signup systems, such as Facebook Connect, Twitter OAuth and OpenID, to allow users to re-use their existing accounts on these servers rather than creating a new account.
• Automatic generation of content sections. This is an important improvement, and would improve the navigation of documentation pages. The general principle would be to automatically provide in-page tables of contents at the top of documentation pages, and provide the ability to edit subsections rather than the entire page as a block. [Stop press: Now partially implemented and soon to be rolled out]
• Interface customisation. While this is certainly possible currently, it requires a customiser to hand edit CSS in order to do so. It would be much nicer if there was a web-interface method to customise at least some of the interface elements for an installation, and possibly even on a per-user level at some point in the future.

As iDoc is open source, we hope that others will extend the system in useful ways.