 |  |
Chapter 2. Host Configurations
The goal of this chapter is to review system administration from the perspective of the individual hosts on a network. This chapter presumes that you have a basic understanding of system administration. Consequently, many of the more basic issues are presented in a very cursory manner. The intent is more to jog your memory, or to fill an occasional gap, than to teach the fundamentals of system administration. If you are new to system administration, a number of the books listed in Appendix B, "Resources and References" provide excellent introductions. If, on the other hand, you are a knowledgeable system administrator, you will probably want to skim or even skip this chapter.Chapter 1, "Network Management and Troubleshooting" lists several reasons why you might not know the details of your network and the computers on it. This chapter assumes that you are faced with a networked computer and need to determine or reconstruct its configuration. It should be obvious that if you don't understand how a system is configured, you will not be able to change its configuration or correct misconfigurations. The tools described in this chapter can be used to discover or change a host's configuration.
As discussed in Chapter 1, "Network Management and Troubleshooting", if you have documentation for the system, begin with it. The assumption here is that such documentation does not exist or that it is incomplete. The primary focus is network configuration, but many of the techniques can easily be generalized.
If you have inherited a multiuser system that has been in service for several years with many undocumented customizations, reconstructing its configuration can be an extremely involved and extended process. If your system has been compromised, the intruder has taken steps to hide her activity, and you aren't running an integrity checker like tripwire, it may be virtually impossible to discover all her customizations. (tripwire is discussed briefly in Chapter 11, "Miscellaneous Tools".) While it may not be feasible, you should at least consider reinstalling the system from scratch. While this may seem draconian, it may ultimately be much less work than fighting the same battles over and over, as often happens with compromised systems. The best way to do this is to set up a replacement system in parallel and then move everyone over. This, of course, requires a second system.
If rebuilding the system is not feasible, or if your situation isn't as extreme as that just described, then you can use the techniques described in this chapter to reconstruct the system's configuration.
Whatever your original motivation, you should examine your system's configuration on a regular basis. If for no other reason, this will help you remember how your system is configured. But there are other reasons as well. As you learn more, you will undoubtedly want to revisit your configuration to correct problems, improve security, and optimize performance. Reviewing configurations is a necessary step to ensure that your system hasn't been compromised. And, if you share management of a system, you may be forced to examine the configuration whenever communications falter.
Keep a set of notes for each system, giving both the configuration and directions for changing the configuration. Usually the best place to start is by constructing a list of what can be found where in the vendor documentation you have. This may seem pointless since this information is in the documentation. But the information you need will be spread throughout this documentation. You won't want to plow through everything every time you need to check or change something. You must create your own list. I frequently write key page numbers inside the front covers of manuals and specifics in the margins throughout the manual. For example, I'll add device names to the manpages for the mount command, something I always seem to need but often can't remember. (Be warned that this has the disadvantage of tying manuals to specific hardware, which could create other problems.)
When reconstructing a host's configuration, there are two basic approaches. One is to examine the system's configuration files. This can be a very protracted approach. It works well when you know what you are looking for and when you are looking for a specific detail. But it can be difficult to impossible to find all the details of the system, particularly if someone has taken steps to hide them. And some parameters are set dynamically and simply can't be discovered just from configuration files.
The alternative is to use utilities designed to give snapshots of the current state of the system. Typically, these focus on one aspect of the system, for example, listing all open files. Collectively, these utilities can give you a fairly complete picture. They tend to be easy to use and give answers quickly. But, because they may focus on only one aspect of the system, they may not provide all the information you need if used in isolation.
Clearly, by itself, neither approach is totally adequate. Where you start will depend in part on how quickly you must be up to speed and what specific problems you are facing. Each approach will be described in turn.
2.1. Utilities
Reviewing system configuration files is a necessary step that you will have to address before you can claim mastery of a system. But this can be a very time-consuming step. It is very easy to overlook one or more key files. If you are under time pressure to resolve a problem, configuration files are not the best place to start.Even if you plan to jump into the configuration files, you will probably want a quick overview of the current state of the system before you begin. For this reason, we will examine status and configuration utilities first. This approach has the advantage of being pretty much the same from one version of Unix to the next. With configuration files, the differences among the various flavors of Unix can be staggering. Even when the files have the same functionality and syntax, they can go by different names or be in different directories. Certainly, using these utilities is much simpler than looking at kernel configuration files.
TIP: The output provided by these utilities may vary considerably from system to system and will depend heavily on which options are used. In practice, this should present no real problem. Don't be alarmed if the output on your system is formatted differently.
2.1.1. ps
The first thing any system administrator should do on a new system is run the ps command. You are probably already familiar with ps so I won't spend much time on it. The ps command lists which processes are running on the system. Here is an example:
In this example, the first and last columns are the most interesting since they give the owners and the processes, along with their arguments. In this example, the lines, and consequently the arguments, have been truncated, but this is easily avoided. Running processes of interest include portmap, inetd, sendmail, telnetd, and chargen.bsd4# ps -aux USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 6590 22.0 2.1 924 616 ?? R 11:14AM 0:09.80 inetd: chargen [2 root 1 0.0 0.6 496 168 ?? Ss Fri09AM 0:00.03 /sbin/init -- root 2 0.0 0.0 0 0 ?? DL Fri09AM 0:00.52 (pagedaemon) root 3 0.0 0.0 0 0 ?? DL Fri09AM 0:00.00 (vmdaemon) root 4 0.0 0.0 0 0 ?? DL Fri09AM 0:44.05 (syncer) root 100 0.0 1.7 820 484 ?? Ss Fri09AM 0:02.14 syslogd daemon 109 0.0 1.5 828 436 ?? Is Fri09AM 0:00.02 /usr/sbin/portmap root 141 0.0 2.1 924 616 ?? Ss Fri09AM 0:00.51 inetd root 144 0.0 1.7 980 500 ?? Is Fri09AM 0:03.14 cron root 150 0.0 2.8 1304 804 ?? Is Fri09AM 0:02.59 sendmail: accepti root 173 0.0 1.3 788 368 ?? Is Fri09AM 0:01.84 moused -p /dev/ps root 213 0.0 1.8 824 508 v1 Is+ Fri09AM 0:00.02 /usr/libexec/gett root 214 0.0 1.8 824 508 v2 Is+ Fri09AM 0:00.02 /usr/libexec/gett root 457 0.0 1.8 824 516 v0 Is+ Fri10AM 0:00.02 /usr/libexec/gett root 6167 0.0 2.4 1108 712 ?? Ss 4:10AM 0:00.48 telnetd jsloan 6168 0.0 0.9 504 252 p0 Is 4:10AM 0:00.09 -sh (sh) root 6171 0.0 1.1 464 320 p0 S 4:10AM 0:00.14 -su (csh) root 0 0.0 0.0 0 0 ?? DLs Fri09AM 0:00.17 (swapper) root 6597 0.0 0.8 388 232 p0 R+ 11:15AM 0:00.00 ps -aux
There are a number of options available to ps, although they vary from implementation to implementation. In this example, run under FreeBSD, the parameters used were -aux. This combination shows all users' processes (-a), including those without controlling terminals (-x), in considerable detail (-u). The options -ax will provide fewer details but show more of the command-line arguments. Alternately, you can use the -w option to extend the displayed information to 132 columns. With AT&T-derived systems, the options -ef do pretty much the same thing. Interestingly, Linux supports both sets of options. You will need to precede AT&T-style options with a hyphen. This isn't required for BSD options. You can do it either way with Solaris. /usr/bin/ps follows the AT&T conventions, while /usr/ucb/ps supports the BSD options.
While ps quickly reveals individual processes, it gives a somewhat incomplete picture if interpreted naively. For example, the inetd daemon is one source of confusion. inetd is used to automatically start services on a system as they are needed. Rather than start a separate process for each service that might eventually be run, the inetd daemon runs on their behalf. When a connection request arrives, inetd will start the requested service. Since some network services like ftp, telnet, and finger are usually started this way, ps will show processes for them only when they are currently running. If ps doesn't list them, it doesn't mean they aren't available; they just aren't currently running.
For example, in the previous listing, chargen was started by inetd. We can see chargen in this instance because it was a running process when ps was run. But, this particular test system was configured to run a number of additional services via inetd (as determined by the /etc/inetd.conf configuration file). None of these other services show up under ps because, technically, they aren't currently running. Yet, these other services will be started automatically by inetd, so they are available services.
In addition to showing what is running, ps is a useful diagnostic tool. It quickly reveals defunct processes or multiple instances of the same process, thereby pointing out configuration problems and similar issues. %MEM and %CPU can tell you a lot about resource usage and can provide crucial information if you have resource starvation. Or you can use ps to identify rogue processes that are spawning other processes by looking at processes that share a common PPID. Once you are comfortable with the usual uses, it is certainly worth revisiting ps periodically to learn more about its other capabilities, as this brief discussion just scratches the surface of ps.
2.1.2. top
Although less ubiquitous, the top command, a useful alternative to ps, is available on many systems. It was written by William LeFebvre. When running, top gives a periodically updated listing of processes ranked in order of CPU usage. Typically, only the top 10 processes are given, but this is implementation dependent, and your implementation may let you select other values. Here is a single instance from our test system:
15 processes: 2 running, 13 sleeping
CPU states: 0.8% user, 0.0% nice, 7.4% system, 7.8% interrupt, 84.0% idle
Mem: 6676K Active, 12M Inact, 7120K Wired, 2568K Cache, 3395K Buf, 1228K Free
Swap: 100M Total, 100M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
6590 root 35 0 924K 616K RUN 0:15 21.20% 20.75% inetd
144 root 10 0 980K 500K nanslp 0:03 0.00% 0.00% cron
150 root 2 0 1304K 804K select 0:03 0.00% 0.00% sendmail
100 root 2 0 820K 484K select 0:02 0.00% 0.00% syslogd
173 root 2 0 788K 368K select 0:02 0.00% 0.00% moused
141 root 2 0 924K 616K select 0:01 0.00% 0.00% inetd
6167 root 2 0 1108K 712K select 0:00 0.00% 0.00% telnetd
6171 root 18 0 464K 320K pause 0:00 0.00% 0.00% csh
6168 jsloan 10 0 504K 252K wait 0:00 0.00% 0.00% sh
6598 root 28 0 1556K 844K RUN 0:00 0.00% 0.00% top
1 root 10 0 496K 168K wait 0:00 0.00% 0.00% init
457 root 3 0 824K 516K ttyin 0:00 0.00% 0.00% getty
214 root 3 0 824K 508K ttyin 0:00 0.00% 0.00% getty
213 root 3 0 824K 508K ttyin 0:00 0.00% 0.00% getty
109 daemon 2 0 828K 436K select 0:00 0.00% 0.00% portmapIn a sense, ps is a more general top since it gives you all running processes. The advantage to top is that it focuses your attention on resource hogs, and it provides a repetitive update. top has a large number of options and can provide a wide range of information. For more information, consult its Unix manpage.[5]
[5]Solaris users may want to look at process management utilities included in /usr/proc/bin.
2.1.3. netstat
One of the most useful and diverse utilities is netstat. This program reports the contents of kernel data structures related to networking. Because of the diversity in networking data structures, many of netstat 's uses may seem somewhat unrelated, so we will be revisiting netstat at several points in this book.One use of netstat is to display the connections and services available on a host. For example, this is the output for the system we just looked at:
The first column gives the protocol. The next two columns give the sizes of the send and receive queues. These should be 0 or near 0. Otherwise, you may have a problem with that particular service. The next two columns give the socket or IP address and port number for each end of a connection. This socket pair uniquely identifies one connection. The socket is presented in the form hostname.service. Finally, the state of the connection is given in the last column for TCP services. This is blank for UDP since it is connectionless. The most common states are ESTABLISHED for current connections, LISTEN for services awaiting a connection, and TIME_WAIT for recently terminated connections. Any of the TCP states could show up, but you should rarely see the others. An excessive number of SYN_RECEIVED, for example, is an indication of a problem (possibly a denial-of-service attack). You can safely ignore the last few lines of this listing.bsd4# netstat -a Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp 0 0 bsd4.telnet 205.153.60.247.3473 TIME_WAIT tcp 0 17458 bsd4.chargen sloan.1244 ESTABLISHED tcp 0 0 *.chargen *.* LISTEN tcp 0 0 *.discard *.* LISTEN tcp 0 0 *.echo *.* LISTEN tcp 0 0 *.time *.* LISTEN tcp 0 0 *.daytime *.* LISTEN tcp 0 0 *.finger *.* LISTEN tcp 0 2 bsd4.telnet sloan.1082 ESTABLISHED tcp 0 0 *.smtp *.* LISTEN tcp 0 0 *.login *.* LISTEN tcp 0 0 *.shell *.* LISTEN tcp 0 0 *.telnet *.* LISTEN tcp 0 0 *.ftp *.* LISTEN tcp 0 0 *.sunrpc *.* LISTEN udp 0 0 *.1075 *.* udp 0 0 *.1074 *.* udp 0 0 *.1073 *.* udp 0 0 *.1072 *.* udp 0 0 *.1071 *.* udp 0 0 *.1070 *.* udp 0 0 *.chargen *.* udp 0 0 *.discard *.* udp 0 0 *.echo *.* udp 0 0 *.time *.* udp 0 0 *.daytime *.* udp 0 0 *.sunrpc *.* udp 0 0 *.syslog *.* Active UNIX domain sockets Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr c3378e80 dgram 0 0 0 c336efc0 0 c3378f80 c3378f80 dgram 0 0 0 c336efc0 0 c3378fc0 c3378fc0 dgram 0 0 0 c336efc0 0 0 c336efc0 dgram 0 0 c336db00 0 c3378e80 0 /var/run/log
A couple of examples should clarify this output. The following line shows a Telnet connection between bsd4 and sloan using port 1082 on sloan:
The next line shows that there was a second connection to sloan that was recently terminated:tcp 0 2 bsd4.telnet sloan.1082 ESTABLISHED
Terminated connections remain in this state for a couple of minutes, during which time the socket pair cannot be reused.tcp 0 0 bsd4.telnet 205.153.60.247.3473 TIME_WAIT
Name resolution can be suppressed with the -n option if you would rather see numeric entries. There are a couple of reasons you might want to do this. Typically, netstat will run much faster without name resolution. This is particularly true if you are having name resolution problems and have to wait for requests to time out. This option can help you avoid confusion if your /etc/services or /etc/hosts files are inaccurate.
The remaining TCP entries in the LISTEN state are services waiting for a connection request. Since a request could come over any available interface, its IP address is not known in advance. The * in the entry *.echo acts as a placeholder for the unknown IP address. (Since multiple addresses may be associated with a host, the local address is unknown until a connection is actually made.) The *.* entries indicate that both the remote address and port are unknown. As you can see, this shows a number of additional services that ps was not designed to display. In particular, all the services that are under the control of inetd are shown.
Another use of netstat is to list the routing table. This may be essential information in resolving routing problems, e.g., when you discover that a host or a network is unreachable. Although it may be too long or volatile on many systems to be very helpful, the routing table is sometimes useful in getting a quick idea of what networks are communicating with yours. Displaying the routing table requires the -r option.
There are four main ways entries can be added to the routing table -- by the ifconfig command when an interface is configured, by the route command, by an ICMP redirect, or through an update from a dynamic protocol like RIP or OSPF. If dynamic protocols are used, the routing table is an example of a dynamic structure that can't be discovered by looking at configuration files.
Here is an example of a routing table from a FreeBSD system:
At first glance, output from other systems may be organized differently, but usually the same basic information is present. In this example, the -n option was used to suppress name resolution.bsd1# netstat -rn Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 205.153.60.2 UGSc 0 0 xl0 127.0.0.1 127.0.0.1 UH 0 0 lo0 172.16.1/24 172.16.2.1 UGSc 0 7 xl1 172.16.2/24 link#2 UC 0 0 xl1 172.16.2.1 0:10:7b:66:f7:62 UHLW 2 0 xl1 913 172.16.2.255 ff:ff:ff:ff:ff:ff UHLWb 0 18 xl1 172.16.3/24 172.16.2.1 UGSc 0 2 xl1 205.153.60 link#1 UC 0 0 xl0 205.153.60.1 0:0:a2:c6:e:42 UHLW 4 0 xl0 906 205.153.60.2 link#1 UHLW 1 0 xl0 205.153.60.5 0:90:27:9c:2d:c6 UHLW 0 34 xl0 987 205.153.60.255 ff:ff:ff:ff:ff:ff UHLWb 1 18 xl0 205.153.61 205.153.60.1 UGSc 0 0 xl0 205.153.62 205.153.60.1 UGSc 0 0 xl0 205.153.63 205.153.60.1 UGSc 2 0 xl0
The first column gives the destination, while the second gives the interface or next hop to that destination. The third column gives the flags. These are often helpful in interpreting the first two columns. A U indicates the path is up or available, an H indicates the destination is a host rather than a network, and a G indicates a gateway or router. These are the most useful. Others shown in this table include b, indicating a broadcast address; S, indicating a static or manual addition; and W and c, indicating a route that was generated as a result of cloning. (These and other possibilities are described in detail in the Unix manpage for some versions of netstat.) The fourth column gives a reference count, i.e., the number of active uses for each of the routes. This is incremented each time a connection is built over the route (e.g., a Telnet connection is made using the route) and decremented when the connection is torn down. The fifth column gives the number of packets sent using this entry. The last entry is the interface that will be used.
If you are familiar with the basics of routing, you have seen these tables before. If not, an explanation of the first few lines of the table should help. The first entry indicates the default route. This was added statically at startup. The second entry is the loopback address for the machine. The third entry is for a remotely attached network. The destination network is a subnet from a Class B address space. The /24 is the subnet mask. Traffic to this network must go through 172.16.2.1, a gateway that is defined with the next two entries. The fourth entry indicates that the network gateway, 172.16.2.1, is on a network that has a direct attachment through the second interface xl1. The entry that follows gives the specifics, including the Ethernet address of the gateway's interface.
In general, it helps to have an idea of the interfaces and how they are configured before you get too deeply involved in routing tables. There are two quick ways to get this information -- use the -i option with netstat or use the ifconfig command. Here is the output for the interfaces that netstat generates. This corresponds to the routing table just examined.
For our purposes, we are interested in only the first four entries. (The other interfaces include the loop-back, lo0, and unused interfaces like ppp0*, the PPP interface.) The first two entries give the Ethernet address and IP address for the xl0 interface. The next two are for xl1. Notice that this also gives the number of input and output packets and errors as well. You can expect to see very large numbers for these. The very low numbers indicate that the system was recently restarted.bsd1# netstat -i Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll xl0 1500 <Link> 00.10.5a.e3.37.0c 2123 0 612 0 0 xl0 1500 205.153.60 205.153.60.247 2123 0 612 0 0 xl1 1500 <Link> 00.60.97.92.4a.7b 478 0 36 0 0 xl1 1500 172.16.2/24 172.16.2.13 478 0 36 0 0 lp0* 1500 <Link> 0 0 0 0 0 tun0* 1500 <Link> 0 0 0 0 0 sl0* 552 <Link> 0 0 0 0 0 ppp0* 1500 <Link> 0 0 0 0 0 lo0 16384 <Link> 6 0 6 0 0 lo0 16384 127 localhost 6 0 6 0 0
The format of the output may vary from system to system, but all will provide the same basic information. There is a lot more to netstat than this introduction shows. For example, netstat can be run periodically like top. We will return to netstat in future chapters.
2.1.4. lsof
lsof is a remarkable tool that is often overlooked. Written by Victor Abel, lsof lists open files on a Unix system. This might not seem a particularly remarkable service until you start thinking about the implications. An application that uses the filesystem, networked or otherwise, will have open files at some point. lsof offers a way to track that activity.The program is available for a staggering variety of Unix systems, often in both source and binary formats. Although I will limit this discussion to networking related tasks, lsof is more properly an operating system tool than a networking tool. You may want to learn more about lsof than described here.
In its simplest form, lsof produces a list of all open files. You'll probably be quite surprised at the number of files that are open on a quiescent system. For example, on a FreeBSD system with no one else logged on, lsof listed 564 open files.
Here is an example of the first few lines of output from lsof:
The most useful fields are the obvious ones, including the first three -- the name of the command, the process ID, and its owner. The other fields and codes used in the fields are explained in the manpage for lsof, which runs about 30 pages.bsd2# lsof COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME swapper 0 root cwd VDIR 116,131072 512 2 / swapper 0 root rtd VDIR 116,131072 512 2 / init 1 root cwd VDIR 116,131072 512 2 / init 1 root rtd VDIR 116,131072 512 2 / init 1 root txt VREG 116,131072 255940 157 /sbin/init ...
It might seem that lsof returns too much information to be useful. Fortunately, it provides a number of options that will allow you to tailor the output to your needs. You can use lsof with the -p option to specify a specific process number or with the -c option to specify the name of a process. For example, the command lsof -csendmail will list all the files opened by sendmail. You only need to give enough of the name to uniquely identify the process. The -N option can be used to list files opened for the local computer on an NFS server. That is, when run on an NFS client, lsof shows files opened by the client. When run on a server, lsof will not show the files the server is providing to clients.
The -i option limits output to Internet and X.25 network files. If no address is given, all such files will be listed, effectively showing all open socket files on your network:
As you can see, this is not unlike the -a option with netstat. Apart from the obvious differences in the details reported, the big difference is that lsof will not report connections that do not have files open. For example, if a connection is being torn down, all files may already be closed. netstat will still report this connection while lsof won't. The preferred behavior will depend on what information you need.bsd2# lsof -i COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME syslogd 105 root 4u IPv4 0xc3dd8f00 0t0 UDP *:syslog portmap 108 daemon 3u IPv4 0xc3dd8e40 0t0 UDP *:sunrpc portmap 108 daemon 4u IPv4 0xc3e09d80 0t0 TCP *:sunrpc (LISTEN) inetd 126 root 4u IPv4 0xc3e0ad80 0t0 TCP *:ftp (LISTEN) inetd 126 root 5u IPv4 0xc3e0ab60 0t0 TCP *:telnet (LISTEN) inetd 126 root 6u IPv4 0xc3e0a940 0t0 TCP *:shell (LISTEN) inetd 126 root 7u IPv4 0xc3e0a720 0t0 TCP *:login (LISTEN) inetd 126 root 8u IPv4 0xc3e0a500 0t0 TCP *:finger (LISTEN) inetd 126 root 9u IPv4 0xc3dd8d80 0t0 UDP *:biff inetd 126 root 10u IPv4 0xc3dd8cc0 0t0 UDP *:ntalk inetd 126 root 11u IPv6 0xc3e0a2e0 0t0 TCP *:ftp inetd 126 root 12u IPv6 0xc3e0bd80 0t0 TCP *:telnet inetd 126 root 13u IPv6 0xc3e0bb60 0t0 TCP *:shell inetd 126 root 14u IPv6 0xc3e0b940 0t0 TCP *:login inetd 126 root 15u IPv6 0xc3e0b720 0t0 TCP *:finger lpd 131 root 6u IPv4 0xc3e0b500 0t0 TCP *:printer (LISTEN) sendmail 137 root 4u IPv4 0xc3e0b2e0 0t0 TCP *:smtp (LISTEN) httpd 185 root 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 198 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 199 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 200 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 201 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 202 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 10408 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 10409 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 10410 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 25233 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 25236 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) telnetd 58326 root 0u IPv4 0xc3e0eb60 0t0 TCP bsd2.lander.edu:telne t->sloan.lander.edu:1184 (ESTABLISHED) telnetd 58326 root 1u IPv4 0xc3e0eb60 0t0 TCP bsd2.lander.edu:telne t->sloan.lander.edu:1184 (ESTABLISHED) telnetd 58326 root 2u IPv4 0xc3e0eb60 0t0 TCP bsd2.lander.edu:telne t->sloan.lander.edu:1184 (ESTABLISHED) perl 68936 root 4u IPv4 0xc3dd8c00 0t0 UDP *:eicon-x25 ping 81206 nobody 3u IPv4 0xc3e98f00 0t0 ICMP *:*
If you specify an address, then only those files related to the address will be listed:
One minor problem with this output is the identification of the telnet user as root -- a consequence of root owning telnetd, the server's daemon. On some systems, you can use the PID with the -p option to track down the device entry and then use lsof on the device to discover the owner. Unfortunately, this won't work on many systems.bsd2# lsof -i@sloan.lander.edu COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME telnetd 73825 root 0u IPv4 0xc3e0eb60 0t0 TCP bsd2.lander.edu:telnet-> sloan.lander.edu:1177 (ESTABLISHED) telnetd 73825 root 1u IPv4 0xc3e0eb60 0t0 TCP bsd2.lander.edu:telnet-> sloan.lander.edu:1177 (ESTABLISHED) telnetd 73825 root 2u IPv4 0xc3e0eb60 0t0 TCP bsd2.lander.edu:telnet-> sloan.lander.edu:1177 (ESTABLISHED)
You can also use lsof to track an FTP transfer. You might want to do this to see if a transfer is making progress. You would use the -p option to see which files are open to the process. You can then use -ad to specify the device file descriptor along with -r to specify repeat mode. lsof will be run repeatedly, and you can see if the size of the file is changing.
Other uses of lsof are described in the manpage, the FAQ, and a quick-start guide supplied with the distribution. The latter is probably the best place to begin.
2.1.5. ifconfig
ifconfig is usually thought of as the command used to alter the configuration of the network interfaces. But, since you may need to know the current configuration of the interfaces before you make changes, ifconfig provides a mechanism to retrieve interface configurations. It will report the configuration of all the interfaces when called with the -a option or of a single interface when used with the interface's name.Here are the results for the system we just looked at:
bsd1# ifconfig -a
xl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 205.153.60.247 netmask 0xffffff00 broadcast 205.153.60.255
ether 00:10:5a:e3:37:0c
media: 10baseT/UTP <half-duplex>
supported media: autoselect 100baseTX <full-duplex> 100baseTX <half-dupl
ex> 100baseTX 10baseT/UTP <full-duplex> 10baseT/UTP <half-duplex> 10baseT/UTP
xl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 172.16.2.13 netmask 0xffffff00 broadcast 172.16.2.255
ether 00:60:97:92:4a:7b
media: 10baseT/UTP <half-duplex>
supported media: autoselect 100baseTX <full-duplex> 100baseTX <half-dupl
ex> 100baseTX 10baseT/UTP <full-duplex> 10baseT/UTP 10baseT/UTP <half-duplex>
lp0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> mtu 1500
tun0: flags=8010<POINTOPOINT,MULTICAST> mtu 1500
sl0: flags=c010<POINTOPOINT,LINK2,MULTICAST> mtu 552
ppp0: flags=8010<POINTOPOINT,MULTICAST> mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
inet 127.0.0.1 netmask 0xff000000You can ignore the entries for lp0, tun0, sl0, and ppp0. In fact, if you don't want to see these, you can use the combination -au to list just the interfaces that are up. Similarly, -d is used to list just the interfaces that are down.
While netstat allows you to get basic information on the interfaces, if your goal is configuration information, ifconfig is a better choice. First, as you can see, ifconfig supplies more of that sort of information. Second, on some systems, netstat may skip interfaces that haven't been configured. Finally, ifconfig also allows you to change parameters such as the IP addresses and masks. In particular, ifconfig is frequently used to shut down an interface. This is roughly equivalent to disconnecting the interface from the network. To shut down an interface, you use the down option. For example, ifconfig xl1 down will shut down the interface xl1, and ifconfig xl1 up will bring it back up. Of course, you must have root privileges to use ifconfig to change configurations.
Since ifconfig is used to configure interfaces, it is typically run automatically by one of the startup scripts when the system is booted. This is something to look for when you examine startup scripts. The use of ifconfig is discussed in detail in Craig Hunt's TCP/IP Network Administration.
2.1.6. arp
The ARP table on a system maps network addresses into MAC addresses. Of course, the ARP table applies only to directly connected devices, i.e., devices on the local network. Remote devices, i.e., devices that can be reached only by sending traffic through one or more routers, will not be added to the ARP table since you can't communicate with them directly. (However, the appropriate router interface will be added.)Typically, addresses are added or removed automatically. If your system needs to communicate with another system on the local network whose MAC address is unknown, your system sends an ARP request, a broadcast packet with the destination's IP address. If the system is accessible, it will respond with an ARP reply that includes its MAC address. Your system adds this to its ARP table and then uses this information to send packets directly to the destination. (A simple way to add an entry for a directly connected device to the ARP table is to ping the device you want added. ping is discussed in detail in Chapter 3, "Connectivity Testing".) Most systems are configured to drop entries from the ARP table if they aren't being used, although the length of the timeout varies from system to system.
At times, you may want to examine or even change entries in the ARP table. The arp command allows you to do this. When arp is invoked with the -a option, it reports the current contents of the ARP table. Here is an example from a Solaris system:
The format or details may vary from system to system, but the same basic information should be provided.sol1# arp -a Net to Media Table Device IP Address Mask Flags Phys Addr ------ -------------------- --------------- ----- --------------- elxl0 205.153.60.1 255.255.255.255 00:00:a2:c6:0e:42 elxl0 205.153.60.53 255.255.255.255 00:e0:29:21:3c:0b elxl0 205.153.60.55 255.255.255.255 00:90:27:43:72:70 elxl0 mail.lander.edu 255.255.255.255 00:90:27:9c:2d:c6 elxl0 sol1 255.255.255.255 SP 00:60:97:58:71:b7 elxl0 pm3.lander.edu 255.255.255.255 00:c0:05:04:2d:78 elxl0 BASE-ADDRESS.MCAST.NET 240.0.0.0 SM 01:00:5e:00:00:00
For Solaris, the first column gives the interface for the connection. The next two are the IP address and its mask. (You can get just IP numbers by using the -n option.) There are four possible flags that may appear in the flags column. An S indicates a static entry, one that has been manually set rather than discovered. A P indicates an address that will be published. That is, this machine will provide this address should it receive an ARP request. In this case, the P flag is for the local machine, so it is natural that the machine would respond with this information. The flags U and M are used for unresolved and multicast addresses, respectively. The final column is the actual Ethernet address.
This information can be useful in several ways. It can be used to determine the Ethernet hardware in this computer, as well as the hardware in directly connected devices. The IEEE assigns to the manufacturers of Ethernet adapters unique identifiers to be used as the first three bytes of their Ethernet addresses. These addresses, known as Organizationally Unique Identifiers (OUI), can be found at the IEEE web page at http://standards.ieee.org/regauth/oui/index.html. In other words, the first three bytes of an Ethernet address identify the manufacturer. In this case, by entering on this web page 00 60 97, i.e., the first three bytes of the address 00 60 97 58 71 b7, we find that the host sol1 has a 3COM Ethernet adapter. In the same manner we can discover that the host 205.153.60.1 is Bay Networks equipment.
WARNING: OUI designations are not foolproof. The MAC address of a device may have been changed and may not have the manufacturer's OUI. And even if you can identify the manufacturer, in today's world of merger mania and takeovers, you may see an OUI of an acquired company that you don't recognize.If some machines on your network are reachable but others aren't, or connectivity comes and goes, ARP problems may be the cause. (For an example of an ARP problem, see Chapter 12, "Troubleshooting Strategies".) If you think you might have a problem with IP-to-Ethernet address resolution on your local network, arp is the logical tool to use to diagnose the problem. First, look to see if there is an entry for the destination and if it is correct. If it is missing, you can attempt to add it using the -s option. (You must be root.) If the entry is incorrect, you must first delete it with the -d option. Entries added with the -s option will not time out but will be lost on reboot. If you want to permanently add an entry, you can create a startup script to do this. In particular, in a script, arp can use the -f option to read entries from a file.
The usual reason for an incorrect entry in an arp table is a duplicated IP address somewhere on your network. Sometimes this is a typing mistake. Sometimes when setting up their computers, people will copy the configuration from other computers, including the supposedly unique IP number. A rogue DHCP server is another possibility. If you suspect one of your hosts is experiencing problems caused by a duplicate IP number on the network, you can shut down the interface on that computer or unplug it from the network. (This is less drastic than shutting down the computer, but that will also work.) Then you can ping the IP address in question from a second computer. If you get an answer, some other computer is using your IP address. Your arp table should give you the Ethernet address of the offending machine. Using its OUI will tell you the type of hardware. This usually won't completely locate the problem machine, but it is a start, particularly for unusual hardware.[6]
[6]You can also use arp to deliberately publish a bad address. This will shut up a connection request that won't otherwise stop.
2.1.7. Scanning Tools
We've already discussed one reason why ps may not give a complete picture of your system. There is another much worse possibility. If you are having security problems, your copy of ps may be compromised. Crackers sometimes will replace ps with their own version that has been patched to hide their activities. In this event, you may have an additional process running on your system that provides a backdoor that won't show up under ps.One way of detecting this is to use a port scanner to see which ports are active on your system. You could choose to do this from the compromised system, but you are probably better off doing this from a remote system known to be secure. This assumes, however, that the attacker hasn't installed a trapdoor on the compromised host that is masquerading as a legitimate service on a legitimate port.
There are a large number of freely available port scanners. These include programs like gtkportscan, nessus, portscan, and strobe, to name just a few. They generally work by generating a connection request for each port number in the range being tested. If they receive a reply from the port, they add it to their list of open ports. Here is an example using portscan:
The arguments are the destination address and beginning and ending port numbers. The result is a list of port numbers and service names for ports that answered.bsd1# portscan 205.153.63.239 1 10000 -vv This is a portscanner - Rafael Barrero, Jr. Email me at rbarrero@polymail.calpoly.edu For further information. Enjoy! Port: 7 --> echo Port: 9 --> discard Port: 13 --> daytime Port: 19 --> chargen Port: 21 --> ftp Port: 23 --> telnet Port: 25 --> smtp Port: 37 --> time Port: 79 --> finger Port: 111 --> sunrpc Port: 513 --> login Port: 514 --> shell
Figure 2-1 shows another example of a port scanner running under Windows NT. This particular scanner is from Mentor Technologies, Inc., and can be freely downloaded from http://www.mentortech.com/learn/tools/tools.shtml. It is written in Java, so it can be run on both Windows and Unix machines but will require a Java runtime environment. It can also be run in command-line mode. Beware, this scanner is very slow when used with Windows.
Figure 2-1. Chesapeake Port Scanner
Most administrators look on such utilities as tools for crackers, but they can have legitimate uses as shown here. Keep in mind that the use of these tools has political implications. You should be safe scanning your own system, but you are on very shaky ground if you scan other systems. These two tools make no real effort to hide what they are doing, so they are not difficult to detect. Stealth port scanners, however, send the packets out of order over extended periods of time and are, consequently, more difficult to detect. Some administrators consider port scans adequate justification for cutting connections or blocking all traffic from a site. Do not use these tools on a system without authorization. Depending on the circumstances, you may want to notify certain colleagues before you do a port scan even if you are authorized. In Chapter 12, "Troubleshooting Strategies", we will return to port scanners and examine other uses, such as testing firewalls.One last word about these tools. Don't get caught up in using tools and overlook simpler tests. For example, you can check to see if sendmail is running by trying to connect to the SMTP port using telnet. In this example, the test not only tells me that sendmail is running, but it also tells me what version of sendmail is running:
In the same spirit:lnx1# telnet 205.153.63.239 25 Trying 205.153.63.239... Connected to 205.153.63.239. Escape character is '^]'. 220 bsd4.lander.edu ESMTP Sendmail 8.9.3/8.9.3; Wed, 8 Mar 2000 09:38:02 -0500 (EST) quit 221 bsd4.lander.edu closing connection Connection closed by foreign host.
clearly shows ipfw is not running on this system. All I did was try to use it. This type of application-specific testing is discussed in greater detail in Chapter 10, "Application-Level Tools".bsd1# ipfw list ipfw: getsockopt(IP_FW_GET): Protocol not available
|  | |
| 1.3. Troubleshooting and Management |  | 2.2. System Configuration Files |
