How to fix the Unix configuration nightmare

As seen on freshmeat.net and slashdot. Some of the comments are excellent. There was quite a bit of discussion on sourceforge too. Update June 2003: Hiveconf 0.2 has been announced, a project which has similar goals to those I describe below.. A fair bit of design work has been done, and basic features implemented in python.

Summary: Unix configuration has plenty of room for improvement. Various attempts have been made to close the gap with the popular desktop operating systems, but the right recipe is yet to be found. I propose an approach for combining free software with easy configuration, making it easier for developers to get it right without losing diversity.

The Linux distributions and the BSD ports system are like this, taking an existing wealth of software and working with its quirks to streamline the installation and upgrading of applications and system software.

GNU have done a lot of good work in standardising the compilation of software on different operating systems (automake and configure), parsing command line options (getopts), and internationalisation (gettext).

One area that's still a mess, however, is configuring applications

There are useful general trends, like putting system-wide configuration in /etc, and personal preferences in files beginning with . in your home folder.

However, there is a complete zoo of configuration file formats. Some use the old Windows .INI format. Others use simple variable assignments. And some use natural language style syntax.

Now a certain amount of diversity is healthy. But in this case it makes it a bit of a black art to build user-friendly and powerful configuration tools for all the software on a machine.

There have been various noble attempts to make it all easier to get a handle on.

Some popular packages, like apache and samba, have had point-and-click front-ends built for them. But each front-end works differently. Other packages, notably the linux kernel and Debian debconf, have a range of different interfaces depending on the user's taste, from a raw text file and vi, to a text-based menu system, to graphical point-and-click.

Another approach is to custom build a consistent set of front ends for a lot of different software configurations, such as the GNOME and KDE control panels, the linuxconf project, and dotfile.

And then there's the Microsoft way. I don't think I need to go into much detail here as to what a mess the Windows registry is from a Unix geek's perspective. It does provide a certain amount of power for graphical applications, but the text-level access is shoddy and cryptic, the naming system opaque, and a binary database seems to be an inherently unstable and inflexible way of storing configuration data.

A new approach

But despite an apparently messy situation, I think that Unix may not be very far off from a workable solution.

Obviously it's possible - reports are that MacOS X has pulled off a very nice graphical front-end to Unix [*]. However, that front-end is the bit they're keeping closed, so the other Unix systems are going to need to build something free.

The solution I think lies in combining the best of the approaches I listed above. It's vital that it be something that can work with existing programs, but also is attractive to developers to incorporate into new versions and new projects. Without integration into the applications, it's always going to be a bit of a dodgy hack that makes it difficult to change the configuration files in a diversity of ways.

Like previous successful Unix evolutions, it needs to plug into current technology in a way that gives something back to the developer pretty quickly, and make it easy and inviting to be used in both existing and new projects. Getting the developer involved is crucial for making sure the configuration manager keeps up to date with changes in the application, which I believe is a big part of the success of standards like gettext and configure.

It needs to accomodate diversity (the opposite of the One Registry approach, which I think will never fly on Unix). And it needs to be tenacious: it should be useful even with minimal information, but take good advantage of as much information as is given.

It needs to help both the user and the developer. It helps the developer by automating and cleaning up the their configuration handling and documentation. That's a big load off their plate. It helps the user by providing both flexibility and consistency.

Here's an outline of how it might work:

• A core system would handle parsing, verification and storage of text-based configuration files in one or two basic formats.

• The master copy of the configuration is always left in the native text files (in /etc and ~/.*). This is where linuxconf used to have trouble [*]. In early versions linuxconf kept its own copy of the distribution-specific configuration, as I found out once when linuxconf took over my system and then later something stuffed up with the network configuration - it was very difficult to just switch to manual and edit the offending text file without losing linuxconf completely. Keeping the master config in text means you can always quickly and easily use a text editor to fix a broken config when nothing else works, for example using an emergency rescue boot disk. This is a good argument against using some sort of binary scheme (Windows Registry, SQL) for everything. This basic philosophy is I think a key part of the success and flexibility of the Unix init script system. You can always follow the entire boot process in text using /etc/inittab and /etc/rc* and /etc/init.d. That's the sort of Unix advantage we want to keep in the configuration system as well.

  Another option is to allow plugins to handle how the data is stored. The native format is text files as described above. Other options would be XML, LDAP, SQL, etc. The system admin should be able to easily switch a particular daemon or application (or all of a user's preferences maybe) from one to the other at will. All formats should be able to handle comments made by the system admin about their changes - visible whenever someone else goes to change that same option (whether in text or through an interface). Being able to mark sets of options to enable and disable as a group would be nice too. Versioning (probably complex enough that you'd want it handled in a plugin) would be luxurious.

• In order for some features to work, it might be necessary for application developers to switch to the use of the configuration manager for their internal routines. However, this should be possible wihout breaking backward compatibiliy with the original configuration file format for most features.

• A key element would be the configuration format description file. This would list the configuration options for a given piece of software, giving for each one the name, type (boolean, list, string, filename, internet address, etc.), options, category (for sub-sections within the config), and help text (short and long).

  It would be good to be able to write descriptions of how options relate to each other: e.g. one option might disable a bunch of other options.

  Ideally you would be able to add optional hints to help with layout of the options on the screen under various configuration editor systems. A specification for common configuration tasks, such as adding a new virtual web server, could be very useful. There might be various global options, such as how to reload the configuration while a program is running, and what general software category the application lies in (e.g. internet service, communication, hardware, etc.).

  Done well this could be a big attraction for developers, because it could eliminate a lot of duplicated effort involved in managing configuration systems. By creating a single place for storing configuration information, it could be used to automatically generate and update all related documenation: user guides, man pages, web page help, commented template config files, and programmatic help output.

• Because so much language is involved in configuration options, it needs to be internationalised from the ground up.

• A clear method would be provided for application options to be inherited (or even locked) down the path from application defaults, to a network, a given system, and finally to users.

• The core system would be kept small and efficient, so that it is easy to add into all sorts of projects without weighing them down. It would have interfaces in all of the common Unix languages: C, C++, perl, PHP, shell scripting, python, java, etc.

• Plugins would allow the addition of a lot of diversity, and would be of atleast two types:
  • File format plugins would allow the programming of new configuration file formats. This includes parsing, verification, editing, and writing. You could expect heaps of such plugins, preferably provided with an application, or by a third party. The idea is though that you could get somewhere without the involvement of the developer, although not as far as if they are involved. Some plugins might also allow the use of SQL tables (instead of text files) for storing preferences, however text would be generally enouraged unless there were specific reasons for a given app to use something else.

  • Editor plugins would give various methods for changing configuration files. One could be a simple syntax checking wrapper on your favourite text editor. Then there's text-based menus (curses), graphical (GNOME, KDE, etc.), distribution specific (e.g. Debian debconf), network client/server, and web forms (useful for web applications that have a lot of configuration options).

    Because applications can store a lot of meta data about configuration options, it should be possible to provide a fairly rich interface to the user. For example, options which specify filenames or folders could be verified by the confuration editor, and flagged if they do not exist.

    It should be possible to invoke an editor from within an application (using the application's File -> Preferences... menu to invoke the same GUI front end as the app), or directly from the command line, or from some sort of control panel that is aware of installed apps (this awareness might need plugins for different distributions - we can't rely on the developer for something so important).

    It should also be possible to programmatically edit configurations, either through a standard command line, or through the various language bindings such as perl. Network administrators would love such tools. Some things are simply better done from the command line or in a script, and that is what this system should support.

• It all needs to be language, desktop, distribution and operating system neutral, so as to avoid turning off any potential software developers who might find it useful.

With such a system Unix could go even further than Windows and Mac, providing the ease-of-use of graphical configuration, with the flexibility and security of remote text-based configuration, and the power of programmatic access to read and change configurations throughout the system.

If this seems like a good idea, then all we need is a team of programmers and documenters to write it. It seems like a lot of work, but with the right touch, and strategic initial energy, it would be something that attracted contributions from a lot of different angles. Personally I can go no further than making this description. So if this approach inspires you, please take it and run with it!

[*] Dig a little deeper into MacOS X and you find that only the very basic parts of the Unix daemons (such as apache and samba) are configurable through pretty point-and-click dialogs. For more advanced apache settings, you need to work with the usual text configuration files.

[*] My original version of this piece implied that linuxconf still keeps some sort of internal registry. In fact they got rid of that behaviour about 4 years ago. My apologies to the linuxconf developers for displaying my obsolete knowledge in a way that made their work look bad.

$Id: unix-config.html,v 1.9 2003/06/24 22:23:26 maffew Exp $