emlog -- the EMbedded-system LOG-device Jeremy Elson - 6 June 2000 jelson@circlemud.org Emlog web page: http://www.circlemud.org/~jelson/software/emlog -------------------------------------------------------------------------- What is emlog? ============== emlog is a Linux kernel module that makes it easy to access the most recent (and *only* the most recent) output from a process. It works just like "tail -f" on a log file, except that the storage required never grows. This can be useful in embedded systems where there isn't enough memory or disk space for keeping complete log files, but the most recent debugging messages are sometimes needed (e.g., after an error is observed). The emlog kernel module implements simple character device driver. The driver acts like a named pipe that has a finite, circular buffer. The size of the buffer is easily configurable. As more data is written into the buffer, the oldest data is discarded. A process that reads from an emlog device will first read the existing buffer, then see new text as it's written, similar to monitoring a log file using "tail -f". How is emlog used? ================== 1: Configure, compile, and install emlog First, decide which major number you would like to use for emlog. This is configured in emlog.h using the constant EMLOG_MAJOR_NUMBER. The default is 241, which is in the "local/experimental use" range according to the kernel documentation (similar to the 10/8 or 192.168/16 IP networks). Setting the major number to 0 will cause the kernel to dynamically assign a major number to emlog. Next, compile using the Makefile provided. Typing 'make' should generate a single object file, 'emlog.o'. Insert the module into the kernel using the 'insmod' command; e.g. 'insmod emlog.o'. 2: Create device files for emlog Next, you must use 'mknod' to create device files that your processes can write to. The major number of the device files should be whatever number you selected in Step 1 (e.g., 241). The minor number is used to indicate the *size* of the ring buffer for that device file, specified as the the number of kilobytes (e.g., 1024 bytes). For example, to create an 8K buffer called 'testlog': % mknod /tmp/testlog c 241 8 You can create as many devices as you like. Internally, emlog uses the file's inode number to identify which buffer it refers to. 3: Write to and read from your new device file Once the device file has been created, simply write to your device file as you would any normal named pipe, e.g. % echo hello > /tmp/testlog Writes will never block because the buffer never runs out of space; old data is simply overwritten by new data. You can read from the log in the normal way, e.g. using cat. Note that reads block, just like "tail -f", waiting for new log data. For example: % cat /tmp/testlog hello [we immediately see the hello that we wrote in the previous step] _ [... and here's the cursor. the 'cat' process is now blocked, waiting for new input. New data will be displayed as it is written to the device by other processes.] ^C [use control-c, for example, to stop reading.] 4: Remove emlog when you're done Type 'rmmod emlog' will remove the emlog kernel module and free all associated buffers. This won't work until all emlog device files are closed. Other Usage Notes ================= emlog will allocate a fixed-size buffer on behalf of a device file if one of the following two conditions is true: 1- A process has the file open for reading or writing 2- A process has written text to the pipe that has not been read In other words, buffers are persistent, even after a process closes the pipe. If another process later reads the pipe, the text will still be there. Note that it is possible (naturally) to fill virtual memory by creating many such pipes, writing to all of them, and never reading the data out of them. All buffers will be freed when the emlog kernel module is removed. Troubleshooting =============== Q: When I try insert the module using 'insmod', I get 'I/O error". A: That probably means the major device number being registered by emlog is already in use by another device driver. Try changing the major device number in emlog.h (or, change it to 0 in order to get a dynamically assigned major number). Q: I'm seeing "I/O error" at a time *other* then when the module is inserted. A: Oops - you've found a bug in emlog. Please report it. Q: When I try to access an emlog device file for reading or writing, I get the error "no such device". A: This probably means either that the emlog kernel module is not loaded; or, that the major number of the device file does not match the major number that emlog registered. To see which major number is being used by emlog, type 'cat /proc/devices | grep emlog'. Q: When I try to access an emlog device file for reading or writing, I get the error "invalid argument". A: The *minor* number of the emlog device file must be a number between 1 and 128, representing the number of kilobytes (1,024 bytes) that should be used for emlog's ring buffer. Make sure you're specifying a valid minor number in your 'mknod' statement. Q: I see "no memory" errors when I try opening new emlog files. A: Looks like you're out of virtual memory, sport. Q: When I try to remove the emlog driver ("rmmod emlog"), I get the error "Device or resource busy". A: That means a process is currently using an emlog device. You have to wait until all processes close all emlog device files until the driver can be removed. Try using "lsof" to see which files are in use by which processes. Q: You've made my computer crash. A: Sorry. If you can reproduce the problem I'll try to fix it. Known Bugs ========== emlog identifies buffers based solely on the inode number of the device file being accessed. If two device files on two different filesystems happen to have the same inode number, they will share the same buffer, as if they were the same device file. Currently, the poll() function for emlog files is unimplemented. Therefore, using the select() function with an emlog file will not work. (However, non-blocking reads DO work, e.g. by setting O_NONBLOCK using ioctl()). Bug reports, patches, complaints, praise, and submissions of Central Services Form 27B/6, are welcomed by the author (Jeremy Elson, . Who wrote emlog, and why? ========================= Emlog was written by Jeremy Elson at the University of Southern California's Information Sciences Institute as part of the SCADDS project . SCADDS is an embedded systems research project. We use small PC/104-bus-based single-board-PCs using Linux. We wanted to save the debugging output from certain processes, but since these things have 16MB of disk space and 32MB of RAM, keeping complete log files was not an option. These tiny nodes do have serial ports running PPP, though, so it's possible to walk over to a node with a laptop, plug in a serial cable, and then telnet into the box. Using emlog, we can always keep the most recent debug messages from our processes; in case of an error, we can plug in a debug console and see what went wrong. This work was supported by DARPA under grant No. DABT63-99-1-0011 as part of the SCADDS project, and was also made possible in part due to support from Cisco Systems.