By Michel Dagenais, May 23 2003.
Typical systems often contain over 1000 installed packages, with about twice as many textual configuration files residing in /etc. System administration tools to manage these dissimilar files hardly achieve their goals. For instance, webmin or linuxconf suffer from the fact that they constantly need to adjust to small changes in configuration file formats and locations. Moreover, they cannot easily accomodate inherited configuration parameters, to simplify the configuration of multiple similar systems. Other tools have been developed for updating the configuration for groups of machines, including cfengine and hand tailored solutions, where configuration files are processed through the m4 preprocessor and redistributed with rdist.
This problem is widely recognized and regularly sparks heated discussions. Editorials on this very topic were posted recently to freshmeat ( http://freshmeat.net/articles/view/565/). The problem is to get a consensus on a better replacement, and even more on how to make the transition. In newer systems without legacy problems, the management of configuration information is being solved nicely. A few recent systems, in production today, are worth mentioning.
The GNOME Configuration System, gconf, consists in an API/library used by programs to get/set configuration information and receive notification, a daemon to access and cache the configuration entries and perform notification, and storage backends to read/write the configuration data from XML files or other sources. It is used for GNOME desktop applications configuration. Many configuration sources may be stacked, to allow for default, user, and mandatory settings. Other levels may be added, like a department default level, between the user and default level.
In its current implementation, gconf does not offer notification for changes in system defaults, only for changes in user preferences. Moreover, its prerequisites (it requires several libraries notably Orbit/CORBA and GLib) make it unappealing for system administration and for adoption by other desktop systems (e.g. KDE). This may be changing with the development of a low overhead messaging data bus named D-BUS which would remove the requirements for Orbit/CORBA and GLib, and offer system level notification.
The KDE desktop uses a simular system named kconfig.
Debconf is another similar system, but closer to system administration. It is mostly used for installation time configuration data and thus only manages a small subset of the configuration information. Every parameter is represented as a typed question for which the answer may be obtained interactively, or may be retrieved from a repository, if the question was asked previously on this system or on another system in the same group. Debconf could perhaps be used to manage more configuration parameters, but its expressive power remains limited. Noteworthy is the highly successfull distributed aspect of it. Unlike in Webmin or Linuxconf, each package maintainer provides and updates the necessary interface between the package and the configuration system.
Darwin/Mac OS X supports XML property lists to store configuration data. These are made available through a well defined API. The configd daemon is used for notification.
The GNOME Setup Tools is a related project but only adresses the configuration editing aspect. It provides converters between XML and text representations of configuration files. The text files reside in /etc and are used by applications while their XML representation is used by the graphical configuration editors.
Quite clearly, there is a general trend towards structured configuration data, in XML files or SQL/LDAP databases, available through a simple API, enabling multiple stacked sources, and coupled to a change notification service.
The structured configuration data format and associated API removes the need for all the custom configuration data parsers in system daemons and in configuration editors. It also allows configuration parameters inheritance which can simplify managing similar systems in a department, in a cluster, or even for diskless stations sharing a common NFS root; the topmost layer would be selected based on the IP or Ethernet address. Even for a single system, inheritance may be used to separate local changes from default values, and simplify reapplying these changes when upgrading a software package.
Notification services are convenient for some applications (e.g., changing the theme across all applications on a user desktop), but may not be desirable in others (e.g., the system administrator changes the default background color and all users see their background change suddenly without knowing why).
Any new configuration system must take into account the installed base of hundreds of applications using unstructured configuration files. Several adaptation mechanisms are possible involving conversion filters between the old unstructured format and the new structured format, and wrappers around these applications.
Most if not all the new configuration systems are based on an XML representation. This structured format is suitable for both tabular information (e.g., password files) as well as more complex nested declarations (e.g., dhcpd and Apache configurations). Furthermore, it comes with several well supported API (DOM, SAX) implemented in libraries available for most programming languages. The underlying storage may be a simple file, an XML database, a mapping to a simpler SQL or LDAP database, or a remote configuration server. A different dynamically loadable backend could be used for each different storage format. A configuration system API must support a few additional features for access control and notification.
Access control is an almost orthogonal issue. The configuration information may be split across several files, each with specific access restrictions, enforced directly by the operating system, or by a configuration server daemon. Similarly, XML and SQL database servers already include access control functions.
Notification services imply functions to register interest in changes for some portion of the configuration parameters space. Notification messages may be delivered in several ways. Gconf based applications specify callbacks to CORBA objects. Other applications may want to have a signal sent to a specific process (e.g. SIGHUP), or have a script executed (/etc/init.d/dhcpd restart). Of course there are security implications in requesting a notification daemon to execute a script at a later time; the script should not be run with more privileges that the application which registered for notification.
Computed entries is something else to consider. Often, several parameters depend on another parameter (e.g., netmask and dhcp dynamic range which depend on the host/network IP address). The dependencies must be tracked properly to insure correct updates and notification. This is similar to spreadsheet cells which may contain either a value or a formula. To specify a computed entry, a reference to another element, a script file name or a script function could be entered. The actual computation must be performed in the context of the application using the parameter, thus being implemented in the API library or in the application itself.
While conceptually simple, and extremely useful, the stacking of configuration parameters brings a few problems. An analogous problem is the stacking of several file trees, which is refered to in the literature as Translucent File Systems, or Union Mounts. When looking for a named file, each layer is searched from top to bottom and the search stops as soon as one is found. Negative file entries may be used to hide/erase in an upper layer a file present in a lower layer.
Listing directories is more problematic. A directory in an upper layer may be opaque and entirely mask the content of corresponding directories in lower layers. Alternatively, a directory may be translucent and its content merged with the content of lower layers. The order in which the entries should be merged is another problem (before, after, merged in alphabetical order). This ordering may, for instance, represent the order in which applets appear on the panel.
The multiple derivatives of a software package are perhaps an even closer analogy to configuration parameters stacking. For example, hundreds of patches, in diff format, exist for adding or correcting features for the Linux kernel. While some patches may be applied together, others conflict. Each patch is a changeset over a specific version of the kernel. Some patches are built over an already patched kernel, thus forming a tree of changesets. When a patch is applied out of context, it may or not be rejected. Differences among XML files may be represented using the XUpdate format.
The following example is an excerpt from httpd.conf. It represents the
default layer.
<Directory /var/www>
Options MultiViews Indexes SymLinksIfOwnerMatch ExecCGI
</Directory>
<Directory /home/www>
Options MultiViews SymLinksIfOwnerMatch
</Directory>
For a specific host Foo a layer above could
specify that the ExecCGI
option should be removed, which may be expressed as:
/Directory/1/Options/ExecCGI="FALSE"
If the order of the Directory
entries is changed in the default layer,
the modification in the above layer will become useless. Replacing the path
component based on the location (1) by something with more semantic value
(/var/www) would give:
/Directory/"/var/www"/Options/ExecCGI="FALSE"
However, if the name of the directory is changed from /var/www to /var/html this would still miss the intended purpose. In that specific case, assigning a unique identifier to each branch may, while inconvenient, provide the intended semantic.
In the first model of stacked trees, as exemplified above, the end result may differ from the intended purpose and create unwanted surprises. In the second model, in which differences are stored (in diff or XUpdate format) in the upper layers, conflicts may occur when the base layers are changed too much or the stacking is modified. The first is more convenient while the second may be safer. In either case, inheritance remains the best tool to simplify system configuration management.
Every application would retrieve configuration information from several configuration repositories (e.g. localhost:/home/john/.xml-conf, localhost:/etc/xml-conf/default, server.mygroup.mycompany.com:/etc/xml-conf/mygroup). To simplify the architecture, insure serialization and allow notification, a single process should be responsible for serving each configuration repository, receiving changes and sending change notification. Typically, one user daemon would run for each active user, serving its home directory configuration repository, and one system daemon would serve the system default repository.
Each application could directly contact the daemon associated with each layer in its configuration stack. However, since all applications with the same user id use the same layers, it should be more efficient to have all their requests go to the same user daemon. That daemon would in turn connect to other daemons as needed. This reduces the number of connections to one per application plus a few for daemon to daemon exchanges. Moreover, it allows for caching frequently requested configuration parameters.
The notification mechanism, while desirable, is a major complexity item in a configuration system. Low resources or seldom changing systems (e.g. embedded systems, bootstrap disks, thin clients or cluster nodes) should be able to do without it. It must then be implemented as a separate optional facility. The D-Bus systems looks promising in that area. This low overhead messaging system may be used to retrieve configuration data and to obtain notification services. Yet, the configuration API would also have a simpler backend which accesses local files directly without connecting to the D-Bus.
A configuration editor would use the configuration API and allow viewing and editing the configuration parameters hierarchy. Different editors may exist (GUI, Web, Text), possibly sharing some code. A challenge is to clearly show which layer/configuration repository provides which data element and where modifications will be written. It may be sufficient to simply view the stacking result and assume that the topmost layer is written to; individual layers would be accessed by changing the stack, removing topmost layers as needed.
Most existing configuration tools (e.g., Linuxconf, Webmin) have specific UI controls/druids designed for the various tasks. Is it possible to have only a generic interface to do the same job as efficiently? The computed values and the notification mechanisms may alleviate the need for most of the specific per subsystem configuration work often performed by these. Some simple extensions to the configuration parameters descriptions may allow more advanced generic configuration editing (icons, specialized types, Glade forms/druids description files to manipulate some of the configuration parameters...).
Since several similar configuration API exist (e.g., gconf, debconf, kconfig) it could be possible to provide gateways such that an editor may be used with several of these systems.
Eventually, most major applications may use a structured configuration API. If a few similar API are in common use, wrappers may be used as gateways to achieve interoperability. Until then, adapters to simplify the transition must be devised. These adapters all involve conversion filters between the legacy ad hoc textual formats and structured formats such as XML.
When an application changes to use a structured format, it is expected that all the tools manipulating its configuration parameters are updated at the same time and no adapter is required. Alternatively, if a configuration editor expects a structured format but the application still uses a textual format, conversion filters may be used at the input and output of the configuration editor. This is exactly how the Gnome Setup Tools operate.
More problematic is when the configuration parameters are stored in a layered structured format while the application still expects a textual format and cannot easily be modified. Three methods may be used in that case:
/etc/xml-conf/layers/Default /etc/xml-conf/layers/GroupA /etc/xml-conf/layers/GroupB /etc/xml-conf/layers/Host1 /etc/xml-conf/layers/Host2 /etc/xml-conf/layers/Host3For each host, the corresponding textual files would be derived and stored in /etc. For systems with a shared NFS root, the textual files would instead be stored in:
/etc/xml-conf/derived/Host1 /etc/xml-conf/derived/Host2 /etc/xml-conf/derived/Host3In the shared /etc, configuration files would be replaced by links to corresponding entries in /etc/xml-conf/self. Then, at boot time, the appropriate /etc/xml-conf/derived/ subdirectory, (Host1, Host2 or Host3), would be mounted on /etc/xml-conf/self. Many equivalent organizations are possible for the mount points and the symbolic links in order to optimize the disk space and reduce the number of links.
A simple, flexible and low overhead system is needed but, above all, the
challenge is the migration of these thousands of applications with text based
configuration files. Fortunately, most systems are customized by modifying
only a small number of configuration files, typically 10 on a simple client,
20 on a simple server and 30 on a more elaborate server. Frequently modified
configuration files and associated applications include the following.
/etc/network/interfaces (ifupdown)
/etc/hostname (/etc/init.d/hostname.sh in sysvinit)
/etc/firewall (/etc/init.d/firewall)
/etc/lilo.conf (lilo)
/etc/group (nsswitch)
/etc/cups/client.conf (cups)
/etc/fstab (mount, umount, swapon/getmnt in libc)
/etc/passwd (nsswitch)
/etc/shadow (nsswitch)
/etc/sane.d/plustek.conf (sane)
/etc/X11/XF86Config-4 (Xserver)
/etc/modules (/etc/init.d/modutils)
/etc/inittab (init in sysvinit)
/etc/dhcpd.conf (dhcpd)
/etc/apt/sources.list (apt)
/etc/hosts (nsswitch)
/etc/gdm/gdm.conf (gdm)
/etc/apache/httpd.conf (apache)
/etc/postfix/* (postfix)
It is thus possible to migrate the applications one by one, until enough are converted to insure that most stations may be customized simply through the inheritable configuration sources. In the files mentioned, several are already accessed through an API (group, passwd, shadow, hosts) which could be used as wrapper to the new system. A few are only used from easy to modify scripts (hostname, firewall, modules), at least one is already managed by debconf (XF86Config-4), leaving a few frequently used configuration files requiring more work (interfaces, lilo, cups, fstab, inittab, dhcpd.conf, sources.list, gdm.conf, httpd.conf, postfix, sane).
The D-Bus messaging system is likely to be used as a base for newer versions of gconf and may also be considered for adoption by KDE. Indeed, Havoc Pennington is a major contributor to both gconf and D-Bus and has hinted in the past that a new version of gconf, without the CORBA prerequisite, may be used for system configuration. Both KDE and GNOME could have their own flavor of API on top of a common configuration library and change notification system.
The GNOME Setup Tools may be another important piece of the puzzle. It provides a specialized configuration editor for several of the most common system configuration parameters. Furthermore, it contains scripts to convert between XML and textual formats for the most common system configuration files.
Although very many people recognize the serious problems associated with the current ad hoc storage of configuration parameters in thousands of /etc text files, all the related projects so far had little success. There is therefore much to be learned about what not to do!
It has been said that to popularize a project one should release early, release often. When proposing a different organization, the motto may be provide tangible benefits early on. The inherited configuration parameters possibly offer the highest potential benefits. Configuring a new machine may be as simple as selecting and stacking configuration parameters packages: departemental configuration (network parameters and servers), specific hardware configuration (network/graphics/sound card), public Web site configuration, strong firewall configuration...; it becomes as simple as selecting software packages to install.