Chapter 9. The Automounter

The automounter is a tool that automatically mounts NFS filesystems when they are referenced and unmounts them when they are no longer needed. It applies NIS management to NFS configuration files so that you can edit a single NIS map and have it affect client mount information throughout the network. Using the automounter, you don't have to keep /etc/vfstab files up-to-date by hand.[13] Mount information, including the server's name, filesystem pathname on the server, local mount point and mount options, is contained in automounter maps, which are usually maintained in NIS maps.

[13]The automounter is included in Solaris, Compaq's Tru64 Unix, SGI's IRIX, IBM's AIX, and other commercial Unix operating systems. A public domain version called amd is available on http://www.cs.columbia.edu/~ezk/am-utils/ and amd runs on almost any Unix system. Because it is kernel- and server-independent, the amd automounter is easily migrated to other NFS client platforms.

Why would you want to bother with another administrative tool? What's wrong with putting all of the remote filesystem information in each hosts' /etc/vfstab file? There are many motivations for using the automounter:

/etc/vfstab files on every host become much less complex as the automounter handles the common entries in this file.
The automounter maps may be maintained using NIS, streamlining the administration of mount tables for all hosts in the network the same way NIS streamlines user account information.
Your exposure to hanging a process when an NFS server crashes is greatly reduced. The automounter unmounts all filesystems that are not in use, removing dependencies on fileservers that are not currently referenced by the client.
The automounter extends the basic NFS mount protocol to find the "nearest server" for replicated, read-only filesystems. The NFS server that is closest to the client -- going through the fewest number of bridges and routers -- will handle the mount request. Distributing client load in this manner reduces the load on the more heavily used network hardware.

In a large and dynamic NFS environment, it is difficult to keep the vfstab file on each machine up-to-date. Doing so requires creating mount points and usually hand-editing configuration files; automatic distribution of vfstab files is made difficult by the large number of host-specific entries in each. As you add new software packages or filesystems on the network, you usually have to edit every vfstab file. Using the automounter, you change one NIS map and allow the automounter to provide the new mount point information on all NIS clients.

Adding NFS servers is usually accompanied by a juggling of directories. It is likely that every client will be required to mount filesystems from the new server. As new NFS servers add filesystems to the network, the clients develop new dependencies on these servers, and their vfstab files grow in complexity.

Users cannot simply mount filesystems at their whim without root privileges. The automounter handles this problem by performing the mount as the filesystems are referenced, which is usually the point at which users decide they need to perform the mount themselves. Some users request that their machines mount only those filesystems of interest to them to eliminate the possibility that their machines will hang if a server containing "uninteresting" files hangs. The automounter eliminates dependencies on these unrelated NFS servers by imposing a working-set notion on the set of mounted filesystems. When a filesystem is first referenced, the automounter mounts it at the appropriate place in the local filesystem. After several minutes (ten by default), the automounter attempts to unmount all filesystems that it previously mounted. If the filesystem is quiescent, and therefore probably uninteresting to the client, then the automounter's umount( ) system call succeeds, and the client is relieved of the server dependency. If the filesystem is busy, the automounter ignores the error returned by umount( ).

Using the automounter also adds another level of transparency to the network. Once a client's /etc/vfstab file is created, the client has a static idea of where each remote filesystem is located. It becomes difficult for the system administrator to move tools, users, or any other directory without going to each host and changing the /etc/vfstab files to reflect the change. The automounter makes the location of NFS filesystems even more transparent to NFS clients by removing hardcoded server names and pathnames from the clients' /etc/vfstab files.

Placing NFS filesystems in automounter maps greatly simplifies the administrative overhead of adding or reconfiguring NFS servers. Because the maps may be maintained using NIS, a single file is propagated to all NFS clients. Editing the individual /etc/vfstab files is not required. The automounter is also conducive to simpler mounting schemes. For example, mounting 50 directories of tools and utilities under /tools produces an unwieldy vfstab file. In addition, the tools mount point becomes a bottleneck, since any directory stat( ) or getwd( ) call that touches it also touches all NFS servers with filesystems mounted in /tools. More frequently, tools and utilities are mounted haphazardly, creating administrative problems. Simply remembering where things are is difficult, as users become confused by irregular naming schemes.

Managing /tools with the automounter offers several advantages. All of the individual mount points are replaced by a single map that creates the appropriate mount points as needed. The automounter mount point contains only the handful of entries corresponding to the working set of tools that the user employs at any one time. It's also much simpler to add a new tool: instead of having to create the mount point and edit /etc/vfstab on every host in the network, you simply update the NIS-managed automounter map.

Finally, the automounter looks for a filesystem on one of several servers. Manual pages, read-only libraries, and other replicated filesystems will be mounted from the first server in a set to respond to the mount request. In addition to providing a simple load-balancing scheme similar to that of NIS, the automounter removes single-host dependencies that would make a diskless or dataless workstation unusable in the event of a server crash.

9.1. Automounter maps

The behavior of the automounter is governed by its maps. An indirect map is useful when you are mounting several filesystems with common pathname prefixes (as seen on the clients, not necessarily on the servers). A good example is the /tools directory described previously, although home directories also fit the indirect map model well. A direct map is used for irregularly named filesystems, where each mount point does not have a common prefix with other mount points. Some good examples of mounts requiring direct maps are /usr/local and /usr/man.

Direct and indirect maps vary in how the automounter emulates the underlying mount point. For a direct map, the automounter looks like a symbolic link at each mount point in the map. With an indirect map, the automounter emulates a directory of symbolic links, where the directory is the common pathname prefix shared by all of the automounter-managed mount points. This is confusing and is best explained by the examples that follow.

The master map is a meta-map (a map describing other maps). It contains a list of indirect maps and direct mount points and tells the automounter where to look for all of its map information. We'll look at a typical master map after seeing how the indirect and direct maps are used to mount NFS filesystems.

9.1.1. Indirect maps

Indirect maps are the simplest and most useful automounter convention. They correspond directly to regularly named filesystems, such as home directories, desktop tools, and system utility software. While tools directories may not be consistently named across fileservers, for example, you can use NFS mounts to make them appear consistent on a client machine. The automounter replaces all of the /etc/vfstab entries that would be required to effect this naming scheme on the clients.

Each indirect map has a directory associated with it that is specified on the command line or in the master map (see Section 9.2.1, "The master map" later in this chapter). The map itself contains a key, which is the name of the mount point in the directory, optional NFS mount options, and the server:pathname pair identifying the source of the filesystem. Automounter maps are usually named auto_contents, where contents describes the map. The map name does not have to correspond to its mount point -- it can be anything that indicates the map's function. Maps are placed in /etc or maintained via NIS.

The best way to understand how an indirect map works is to look at an example. We'll look at an automounter map and equivalent vfstab file for a directory structure like this:

/tools/deskset 
/tools/sting 
/tools/news 
/tools/bugview

Here is an indirect automounter map for the /tools directory, called auto_tools:

deskset         -ro      mahimahi:/tools2/deskset 
sting                    mahimahi:/tools2/sting 
news                     thud:/tools3/news 
bugview                  jetstar:/usr/bugview

The first field is called the map key and is the final component of the mount point. The map name suffix and the mount point do not have to share the same name, but adopting this convention makes it easy to associate map names and mount points. This four-entry map is functionally equivalent to the /etc/vfstab excerpt:

mahimahi:/tools2/desket - /tools/deskset  nfs - - ro 
mahimahi:/tools2/string - /tools/sting    nfs - -  
thud:/tools3/news       - /tools/news     nfs - -  
jetstar:/usr/bugview    - /tools/bugview  nfs - -

Notice that the server-side mount points have no common pathname prefixes, but that the client's vfstab and automounter map establish a regularly named view of filesystems.

There are basically two kinds of automounters: older ones that use symbolic links and newer ones that don't. Using the auto_tools map, older implementations of the automounter emulate tools in a directory of symbolic links. When any process on the client makes a reference to something in /tools, the automounter completes the appropriate NFS mount and makes a symbolic link in /tools pointing to the actual mount point for the filesystem. Suppose you go to execute /tools/news/bin/rn. Using the automounter effectively breaks this pathname up into three components:

The prefix /tools picks an automounter map. (We will see in the section Section 9.2.1, "The master map" just how /tools refers to the auto_tools map.) In this case, the map for the /tools directory is the auto_tools map.
The next pathname component is the key within this map. news selects the server filesystem thud:/tools3/news ; the automounter mounts this filesystem and makes a link to it in /tools on the client.
The remainder of the path, bin/rn, is passed to the NFS server thud since it is relative to the directory from which the news toolset is mounted.

Keep in mind that this list applies to older automounters that use symbolic link map entries to NFS mount points. There are problems with using symbolic links, and newer automounters solve them. The newer automounters don't use symbolic links and effectively put the NFS mounts "in place." The next section will explain in more detail.

Note that the automounter map doesn't contain any information about the /tools directory itself, only about the subdirectories in it that are used for mount points. This makes it extremely easy to relocate a set of mount points -- you simply change the master map that associates the directory /tools with the map auto_tools. We'll come back to the master map later on.

9.1.2. Inside the automounter

At this point, it's useful to take a look under the hood of the automounter. This background makes the operation of indirect maps a little clearer and will make direct maps much easier to understand.

As mentioned before, automounter implementations come in two designs. The first one is a purely user-level approach that relies heavily on symbolic links. The second is a hybrid user-level and kernel-level approach called the autofs automounter, which eschews symbolic links.

9.1.2.1. User-level automounters

The original automounters were strictly user-level daemons that required no support in the kernel. SunOS 4.x automounters were all user-level, as were the automounters in Solaris 2.0 through Solaris 2.3. As many automounters were derived from SunOS 4.x or Solaris code, you'll find that several non-Solaris implementations are still user-level.

Before walking through the sequence of automounter operations in detail, some knowledge of mount information is necessary. The mount( ) system call takes the filesystem type (ufs, nfs, hsfs, etc.) and mount point from the /etc/vfstab table, and a packet of parameters that are type-specific. For NFS mounts, the argument vector passed to mount( ) includes the server's hostname and a socket address (IP host address and port number pair) to be used for sending requests to that server. For normal NFS mounts, the remote server's hostname and IP address are used, and the IP port number is the well-known NFS port number 2049. The kernel uses this information to put together an RPC client handle for calling the remote NFS server.

User-level automounters capitalize on this architecture by creating a set of mount arguments that points to itself, a process on the local host, with a different port number than 2049. In effect, a system running the automounter has mounted a daemon on each mount point, instead of a remote filesystem. NFS requests for these mount points are intercepted by the automounter, since it appears to be a regular, remote NFS server to the kernel. No kernel modifications are necessary to run the automounter, and the automounter's functions are transparent to user processes.

We'll take a look at how the user-level automounter works using the indirect auto_tools map discussed earlier. The NFS client host is named wahoo. From boot time, the complete sequence of events is:

The user-level automounter advertises the /tools mount point in /etc/mnttab, making it look like any other NFS-mounted filesystem except for the more verbose information about the server's IP address and port:
```
/etc/mnttab excerpt 
thud:/export/home/thud /tmp_mnt/home/thud nfs rw,dev=218980f 929944999
wahoo:(pid161)     /tools      nfs     ro,ignore,map=/etc/auto_
tools,indirect,dev=2180009 920935886
```
The first mnttab entry is for a normal NFS mount point listed in the vfstab file. The second is for an indirect map and was added when the automounter was started. Instead of a server:directory pair, the automounter entry contains its process ID and the local host's name. The device numbers for NFS-mounted filesystems are simply unique values assigned by the kernel on each mount operation. This entry is added to mnttab when the automounter starts up and reads its maps.
A user goes to execute /tools/news/bin/rn. The kernel performs a lookup of the executable's pathname and finds that the tools component is a mount point. An NFS lookup request for the next component, news, is sent to the listed process -- the automounter -- via a loopback RPC mechanism.
The user-level automounter emulates a directory of symbolic links under the indirect map mount point. The lookup request on the news component is received by the automounter daemon, and it returns information identical to that received when performing a lookup on a symbolic link on a remote NFS server. The automounter looks up the appropriate filesystem in /etc/auto_tools and mounts it in its staging area, /tmp_mnt. This operation uses the mount( ) system call, which places a new entry in the mnttab file.
Now that the automounted filesystem has been referenced, the user-level automounter adds a symbolic link to its emulated directory. The new link in /tools points to the newly mounted filesystem. The equivalent command-line operations are:
```
# mount thud:/tools3/news /tmp_mnt/tools/news 
# ln -s /tmp_mnt/tools/news /tools/news
```
The client-side process receives the reply from its lookup request and goes to read the link. This time, the automounter returns the contents of the symbolic link, which points to the automounter staging area. Note that the automounter fabricates a response to the client's readlink request; it looks like there's a symbolic link on the disk but it's really an artifact of the automounter. The client process follows the link's target pathname to the appropriate subdirectory of /tmp_mnt.
The client process can now trace every pathname in /tools/news to a subdirectory of /tmp_mnt/tools/news, through the new entry in /etc/mnttab and the symbolic link emulation provided by the automounter. A client process pathname lookup finds /tools in the mount table and sends its query to the automounter. The automounter's link points to /tmp_mnt/news, which is also listed in the mount table. To the client, the automounter looks exactly like a directory and a symbolic link.

If this seems to be a convoluted mechanism for mounting a single filesystem, it is. However, this approach is taken to minimize the number of NFS mounts performed and to thereby improve performance by keeping /etc/mnttab as small as possible. When you mount several subdirectories of the same remote filesystem, only one NFS mount is required. The various subdirectories of this common mount point are referenced by symbolic links, not by individual mounts. In the sample indirect map earlier, mahimahi:/tools2 contains several utilities. /tools2 will be mounted on the NFS client when the first utility in it is referenced, and references to other subdirectories of /tools2 simply contain links back to the existing mount in /tmp_mnt.

The staging area /tmp_mnt is a key to the indirect map mechanism. If the staging area concept is eliminated, then the indirect map mount point becomes another directory filled with direct mounts. The primary advantage of indirect maps is that they allow the mount points in a directory to be managed independently -- the mounts occur when a process references the mount point, and not the parent directory itself. We'll look at some problems with direct mounts shortly.

As a result of linking /tools to the actual NFS mount point, a user would encounter the following:

% cd /tools/bin
% /usr/bin/pwd
/tmp_mnt/tools/bin

In other words, instead of pwd displaying /tools/bin, it gets /tmp_mnt/tools/bin. This behavior breaks lots of software. For example, a program might record the current working directory, and cache it in a file. A subsequent invocation of the program might read the cache, and attempt to access /tmp_mnt/tools/bin, and find that it isn't there. This is because a user-level automounter responds to attempts to access /tools, not /tmp_mnt/tools. For the remainder of this chapter, we will refer to this issue as the "pwd problem."

There are other side-effects of the user-level automounter that may catch the user off-guard. The automounter creates and controls the indirect map mount point. It emulates the entire directory, so that no user, even the superuser, can create entries in it. This has an important implication for creating indirect maps: they cannot be mounted over an existing directory, because the automounter hides the underlying files. If a directory must contain a mixture of automounter mount points and "normal" directory entries, a direct map must be used.

This is an important but subtle point: when you poke at a user-level automounter mount point with ls, it appears that there is a directory filled with symbolic links. In reality, this directory and the links in it do not exist on any disk. If this hurts to think about, it's really no different than the way NFS itself works: there may be no filesystem called /tools/news on your local disk, but NFS makes it look like it's there. The user-level automounter speaks to the NFS protocol, allowing it to fabricate replies to NFS RPC calls that are indistinguishable from the real thing.

Because the user-level automounter controls the contents of a readdir NFS RPC reply, ls behaves strangely. The user-level automounter displays only currently mounted links in the directory it emulates. If no reference is made to a subdirectory of the indirect map directory, it appears empty:

% cd /tools 
% ls 
% ls /tools/news 
bin lib spool 
% cd /tools 
% ls -l 
total 1 
lrwxrwxrwx  1 root           19 Aug 31 12:59 news -> /tmp_mnt/tools/news

Why not display potential mounts as well? Doing so could result in a great deal of unintended mounting activity -- a mount storm -- when ls -l is executed in this directory. A newer automounter described in the next section allows you to browse potential mounts, as well as fix the pwd problem described earlier. Another approach is to use hierarchical mounts, as described later in this chapter.

9.1.2.2. The autofs automounter

The pwd problem described in the previous section was solved in Solaris 2.4 with the introduction of a hybrid user-level and kernel-level automounter, which retained a user-level automounter daemon, but introduced a new filesystem known as autofs. The autofs filesystem is a pseudo-filesystem that allows you to mount automounter points like /tools/news as directory objects directly underneath /tools, instead of as symbolic links. The automounter daemon is no longer an NFS server, but instead responds to requests from the in-kernel autofs filesystem to mount NFS filesystems on the mount points that autofs creates.

Let's take a look at how adding autofs changes how the automounter works using the example of the indirect auto_tools map. The NFS client host is still named wahoo. From boot time, the complete sequence of events is:

The autofs automounter advertises the /tools mount point in /etc/mnttab, making it look like any other NFS-mounted filesystem except for the more verbose information about the server's IP address and port:
```
/etc/mnttab excerpt
thud:/export/home/thud /home/thud nfs nosuid,dev=218980f 929944999
auto_tools /tools     autofs
ignore,indirect,nosuid,dev=2b40002 922482272
```
The first mnttab entry is for a normal NFS mount point listed in the vfstab file. Note that the mount point is /home/thud and not /tmp_mnt/home/thud. The second is for an indirect map and was added when the automounter was started. Instead of a process ID and the local host's name, the entry simply has the map name and a filesystem type of autofs. The device numbers for autofs-mounted filesystems are assigned by the kernel on each mount operation. This entry is added to mnttab when the automounter starts up and reads its maps.
A user goes to execute /tools/news/bin/rn. The kernel performs a lookup of the executable's pathname, and finds that the tools component is a mount point. The kernel invokes the lookup entry point of the autofs filesystem request for tools. The kernel then proceeds to the next component in the pathname, news, and again invokes the lookup entry point of autofs. The autofs filesystem sends a request to the automounter daemon (automountd ) -- via a loopback RPC mechanism -- to mount news.
The automounter daemon receives the request from autofs. The request includes the name of the map (auto_tools), the entry in the map autofs is interested in (news), and the mount point the client wants to mount news onto (/tools/news). The automounter daemon examines the tools indirect map looking for the entry:
```
news      thud:/tools3/news
```
The automounter daemon checks if /tools/news exists, and if not, creates the news directory under /tools. Because /tools is an autofs filesystem, the result of the mkdir( ) system call from the daemon is a call to the mkdir entry point in the autofs filesystem.
The automounter daemon then determines that the news map entry is to be satisfied by NFS, and so does the equivalent of:
```
# mount -F nfs thud:/tools3 /tools/news
```
The results of the mount are returned -- again, via a loopback RPC mechanism -- to autofs.
The autofs filesystem receives the reply from the automounter daemon, and now the kernel can proceed with the next components in the pathname, bin and rn. Because the automounter daemon mounted an NFS filesystem onto /tools/news, the automounter is not involved in the processing of bin and rn.

As with the user-level automounter, the original autofs automounter didn't display potential mounts of indirect maps like auto_tools. The next section describes an enhanced autofs automounter that supports the ability to browse the potential mounts under a mount point such as /tools.

9.1.2.3. The enhanced autofs automounter: Browsing indirect maps

In Solaris 2.6, the autofs automounter was modified so that displaying directories of the mount points of indirect maps shows every entry; in other words, it allows a user to browse the map:

% cd /tools 
% ls -l 
total 4 
dr-xr-xr-x  1 root  root   19 Aug 31 12:59 bugview
dr-xr-xr-x  1 root  root   19 Aug 31 12:59 deskset
dr-xr-xr-x  1 root  root   19 Aug 31 12:59 news
dr-xr-xr-x  1 root  root   19 Aug 31 12:59 sting

When the readdir entry point in the autofs filesystem is called on /tools for the first time, there are no autofs directories underneath it, and so, autofs makes an RPC call to the automounter daemon to read the auto_tools map to return the list of map entries. The map entries are used to construct a directory listing for the ls command. Note that the attributes of the directories are faked. This is because we want to avoid mount storms, as described in Section 9.1.2.1, "User-level automounters".

Now see what happens we start to populate /tools with real entries:

% ls /tools/news 
bin lib spool 
% cd /tools 
% ls -l 
total 1 
dr-xr-xr-x  1 root  root   19 Aug 31 12:59 bugview
dr-xr-xr-x  1 root  root   19 Aug 31 12:59 deskset
drwxrwxr-x 5  root  other 512 Jun 10 17:03 news
dr-xr-xr-x  1 root  root   19 Aug 31 12:59 sting

Invoking the ls command on /tools/news causes /tools/news to be NFS-mounted from thud:/tools3/news. When the readdir entry point in the autofs filesystem is called on /tools for the second time, there is now an NFS directory, news, underneath it. Thus, autofs combines the list of map entries with the list of NFS-mounted directories.

By default, indirect maps can be browsed, but browsing can be turned off with the -nobrowse option to an indirect map.

9.1.3. Direct maps

Direct maps define point-specific, nonuniform mount points. The best example of the need for a direct map entry is /usr/man. The /usr directory contains numerous other entries, so it cannot be an indirect mount point. Building an indirect map for /usr/man that uses /usr as a mount point will "cover up" /usr/bin and /usr/etc. A direct map allows the automounter to complete mounts on a single directory entry.

The key in a direct map is a full pathname, instead of the last component found in the indirect map. Direct maps also follow the /etc/auto_contents naming scheme. Here is a sample /etc/auto_direct:

/usr/man        wahoo:/usr/share/man 
/usr/local/bin  mahimahi:/usr/local/bin.sun4

The automounter registers the entire direct mount point pathname in the mnttab file, instead of the parent directory of all of the mount points:

auto_direct /usr/local/bin autofs ignore,direct,intr,ro,dev=2cc000a  933723158

The mnttab entry's map type is listed as direct. Operation of the automounter on a direct mount point is similar to the handling of an indirect mount. The autofs automounter is passed the entire direct mount point pathname in the RPC from autofs, since the mount point is the key in the map. See Table 9-1 for automounter map entry formats.

A major difference in behavior is that the real direct mount points are always visible to ls and other tools that read directory structures. The automounter treats direct mounts as individual directory entries, not as a complete directory, so the automounter gets queried whenever the directory containing the mount point is read. Client performance is affected in a marked fashion if direct mount points are used in several well-traveled directories. When a user reads a directory containing a number of direct mounts, the automounter initiates a flurry of mounting activity in response to the directory read requests. Section 9.5.3, "Conversion of direct maps" describes a trick that lets you use indirect maps instead of direct maps. By using this trick, you can avoid mount storms caused by multiple direct mount points.

Table 9-1. Automounter map entry formats

Key	Mount options	Server:directory pair
indirect map: `deskset`		`mahimahi:/tools2/deskset`
direct map: `/usr/man`	`-ro`	`thud:/usr/man`

Key

Mount options

Server:directory pair

indirect map: deskset

mahimahi:/tools2/deskset

direct map: /usr/man

-ro

thud:/usr/man


8.9. Client/server ratios		9.2. Invocation and the master map