This document covers best practices for kernel development in Chromium OS, including debugging tips, platform bringup info, committing changes, sending code upstream, and using upstream repos for testing & development.
Note for Googlers: There are additional Google-specific notes and work-in-progress notes at go/chromeos-kernel-tips-and-tricks.
First, make sure you're running a dev or test image. Then ensure that verity is disabled on the target before running the update_kernel.sh script, or it will complain and abort. Verity can be disabled using the command /usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification --partition <partition number> on the target followed by a reboot.
Do an incremental build of the kernel:
(chroot) $ FEATURES="noclean" cros_workon_make --board=${BOARD} --install chromeos-kernel-[x_y]
To enable debug options like lockdep and KASAN, add USE="debug" to the command line above. This is highly recommended because the default build is optimized for performance rather than debugging purpose. Note that the debug build bloats the size of kernel image, and the image may not be able to fit into its partition on some older devices. The debug build also takes much longer to boot.
You can also enable serial port at the same time by USE="debug pcserial".
Update the kernel on the target:
(chroot) $ ~/trunk/src/scripts/update_kernel.sh --remote <ip of target>
update_kernel.sh takes additional arguments that allow you to install to the secondary kernel partition and set the “bootonce” flag, which can make it easier to recover from a bad kernel install. E.g.
(chroot) ./update_kernel.sh --remote=$IP --rootfs=/dev/sda5 --partition=/dev/sda4 --bootonce
Be aware, however, that after you reboot again, you'll go back to your old kernel on partition 2 (which can be very confusing... “hey where did my new feature go??”).
Note: cros deploy does not currently support deploying kernel packages.
Also note that using cros_workon_make leaves build artifacts in your source directory under the build directory. When you do a regular emerge of the kernel (and are cros_workon-ed) this will slow things down because the entire source directory gets copied. So delete the build directory when you're done.
However, using emerge is an alternate method, for example:
(chroot) emerge-${BOARD} --nodeps chromeos-kernel-[x_y] && ./update_kernel.sh --remote=$IP --remote_bootargs
cros_workon_make is faster than emerge if you just want to do a build test.--install though if you want to deploy the resulting kernel (and in that case emerge is equally fast).One time I really screwed up my system by recovering (after bad kernel installation) with ‘dd if=/dev/sdb of=/dev/sda’. I forgot the ‘2’ after each drive specification. This overwrote my internal partition table with an exact copy of the USB stick's partition table, including the GUIDs. When I subsequently tried to boot USB, the system always seemed to boot off the internal disk. ‘rootdev -s’ reported (internal partition) /dev/sda3. After an hour or so, consultation with Bill showed that I really was booting the kernel from /dev/sda2, but the kernel found the matching GUID on sda before even looking at sdb. This was recovered with:
$ a=$(uuidgen) $ cgpt add -i 3 -u $a /dev/sda
which generates and installs a new GUID for sda3.
One issue is often to figure out how to recover if you flash a bad kernel. Booting from USB and running chromeos-install is one solution, but that's slow. There are a couple approaches that can be useful to recover quickly.
Always have a good USB stick connected to the device.
Make sure you use a serial-enabled coreboot firmware.
If the kernel on internal storage does not boot anymore:
Boot from USB (press Ctrl-U during FW bootup, you may have to do this repeatedly if on a serial console)
Copy kernel and modules back to internal storage (instructions below assume eMMC)
VERSION=$(uname -r) dd if=/dev/sda2 of=/dev/mmcblk0p2 mkdir /tmp/mnt mount /dev/mmcblk0p3 /tmp/mnt rm -rf /tmp/mnt/lib/modules/${VERSION} cp -a /lib/modules/${VERSION}/tmp/mnt/lib/modules/ dd if=/dev/sda2 of=/dev/mmcblk0p4 umount /tmp/mnt mount /dev/mmcblk0p5 /tmp/mnt rm -rf /tmp/mnt/lib/modules/${VERSION} cp -a /lib/modules/${VERSION}/tmp/mnt/lib/modules/ umount /tmp/mnt # Optional, only if USB stick has rootfs verification on # /usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification -i /dev/mmcblk0 sync reboot
System should boot from internal storage again
Alternatively flash a known good working image to the device and then use update_kernel.sh to target the other kernel partition (typically KERN-B) instead of the live kernel partition. Boot the device into the A slot kernel (KERN-A) and then run update_kernel.sh like this:
(chroot) ./update_kernel.sh --remote=$IP --rootfs=/dev/sda5 --partition=/dev/sda4 --bootonce
The bootloader will attempt to boot the kernel on the sda4 partition (KERN-B) and kernel modules will be updated to the sda5 rootfs partition (ROOT-B). If the kernel crashes early on then a reboot will fallback to the A slot kernel and rootfs that is known to be good and working.
See disk format for more info on partition layouts, as you may need to use a different partition number depending on how you installed your kernel or which one you want to replace.
Kernel configuration in Chromium OS has an extra level of indirection from the normal .config file. So do the instructions - see this page for more information.
See also the cros-kernel eclass documentation.
The built kernel config is available at /build/$BOARD/boot/config in the chroot.
On a running system, the kernel config is not loaded by default (to save memory), so you'll need to use modprobe to load it first:
(DUT)# modprobe configs; zcat /proc/config.gz
Kconfig changes (changes that affect chromeos/config) should be normalized by running chromeos/scripts/kernelconfig olddefconfig
The built kernel command line is avalable at ~/trunk/src//build/images/$BOARD/latest/config.txt in the chroot. The update_kernel.sh script will use this file for the command line when updating the device, unless --remote-bootargs is used, in which case it will try to use the target device args.
It's possible to modify the command line of an installation on a target device as well.
For example, to enable the console on a recovery image on USB stick /dev/sdb:
```bash sudo make_dev_ssd.sh -i /dev/sdb --partitions 2 --save_config ./foo vi ./foo # add the updated command line, for example: earlycon=uart,mmio32,0xfedc6000,115200,48000000 save & exit vi sudo make_dev_ssd.sh -i /dev/sdb --partitions 2 --set_config ./foo sudo make_dev_ssd.sh -i /dev/sdb --recovery_key ```
This extracts the command line from the kernel partition using vbutil, allowing you to edit it and write it back.
If you're booting with depthcharge, you can modify the command line thusly:
(chroot) cros_workon-${board} depthcharge vi /src/platform/depthcharge/src/board/${board}/board.c
Call the commandline_append() function containing your command line addition:
#include "boot/commandline.h" static int board_setup(void) { commandline_append("earlycon=uart,mmio32,0xfedc6000,115200,48000000"); }
Rebuild depthcharge, and build it into the image.
FIXME: this seems redundant with the device update method above
FIXME: Does this still apply to current boards? If yes, to which ones?
Place kernel blob into a file (<original_kernel>), either using dd on the target or by dismantling chromiumos_image.bin generated by build_image. Store the desired kernel command line in a file <new_cmd_line> and then use the following to change the kernel command line:
vbutil_kernel --repack <modified_kernel> --config <new_cmd_line> \ --signprivate <path_to>/vboot_reference/``tests/devkeys/<key> \ --oldblob <original_kernel>
where <key> is kernel_data_key.vbprivk for the main kernel or recovery_kernel_data_key.vbprivk for the flash drive based recovery kernel The keys can be found in the vboot_reference repository. Then dd the <modified_kernel> file back to where <original_kernel> came from.
The command line to boot a kernel with verified rootfs disabled can be obtain by editing the regular command line as follows:
vbutil_kernel --verify <original_kernel> --verbose | tail -1 | sed ' s/dm_verity[^ ]\+//g s|verity /dev/sd%D%P /dev/sd%D%P || s| root=/dev/dm-0 | root=/dev/sd%D%P | s/dm="[^"]\+" //' > new_cmd_line
FIXME: seems redundant with the vbutil info above
FIXME: Does this still apply to current boards? If yes, to which ones?
Instead of booting the kernel from USB as described above, it can be installed directly on the SSD of the target device. With modern H2C Bios, this requires signing the blob with the development key and booting with the target machine's development mode switch set appropriately. Also, since there are two kernel/root partition pairs in our partition scheme, we need to select which one we want to use. Usually we stay with the current pair.
To sign with the devkey as per the Disk Format doc https://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format#TOC-Quick-development:
vbutil_kernel --pack new_kern.bin --keyblock /usr/share/vboot/devkeys/kernel.keyblock --signprivate <keys_path>/kernel_data_key.vbprivk --version 1 --config config.txt --bootloader /lib64/bootstub/bootstub.efi --vmlinuz vmlinuz
Transfer new_kern.bin to the target system. I prefer scp, but it can be placed on USB stick as well.
Identify the preferred kernel partition. This will be either sda2 or sda4. rootdev -s will identify the root partition, and that can be used to identify the currently booted kernel partition.
| Kernel | Root | |
|---|---|---|
| pair A | /dev/sda2 | /dev/sda3 |
| pair B | /dev/sda4 | /dev/sda5 |
Copy the image to the partition.
dd if=new_kern.bin of=/dev/sda2
dev_debug_vboot can be used to verify the kernel partition has a properly signed image. Indeed, it will actually tell you in what modes (ie, development, recovery, neither) your kernel will boot.
localhost ~ # dev_debug_vboot : TEST: verify HD kernel B with firmware A key Key block: Size: 0x4b8 Flags: 7 !DEV DEV !REC :
cgpt can be used as an alternative to rootdev above to find the currently preferred kernel partition.
localhost ~ # cgpt show /dev/sda start size part contents : 4096 32768 2 Label: "KERN-A" Type: ChromeOS kernel UUID: B87DAA9E-E82E-B449-B93A-5EB0BD81BCEC Attr: priority=3 tries=0 successful=1 : 36864 32768 4 Label: "KERN-B" Type: ChromeOS kernel UUID: 4581FC5C-58D1-8148-9FC4-E4B983C90782 Attr: priority=0 tries=0 successful=0 :
FIXME: Does this still apply to current boards? If yes, to which ones?
Copy the new bzImage file into the /efi/boot/ directory on your USB key‘s partition 12. The /efi/boot/grub.cfg file will look for the kernel called vmlinuz, but you can edit that config file to add a line to look for your test kernel too. For example, here’s my USB key's partition 12:
blackadder$ mount | grep vfat /dev/sdc12 on /media/disk type vfat (rw,nosuid,nodev,uhelper=hal,shortname=mixed,uid=100135,utf8,umask=077,flush) blackadder$ ls -l /media/disk/efi/boot/ total 6600 -rwx------ 1 wfrichar root 262656 Apr 21 10:21 bootx64.efi* -rwx------ 1 wfrichar root 2851056 Apr 21 10:12 bzImage* -rwx------ 1 wfrichar root 1040 Apr 21 08:51 grub.cfg* -rwx------ 1 wfrichar root 2821296 Apr 19 11:19 vmlinuz* blackadder$ cat /media/disk/efi/boot/grub.cfg set timeout=10 set default=0 menuentry "bzImage normal" { linux /efi/boot/bzImage quiet console=tty2 init=/sbin/init boot=local rootwait root=/dev/sda3 ro noresume noswap i915.modeset=1 loglevel=1 } menuentry "bzImage serial normal" { linux /efi/boot/bzImage earlyprintk=serial,ttyS0,115200 console=ttyS0,115200 i nit=/sbin/init boot=local rootwait root=/dev/sda3 ro noresume noswap i915.modese t=1 loglevel=7 } menuentry "bzImage serial add_efi_memmap" { linux /efi/boot/bzImage add_efi_memmap earlyprintk=serial,ttyS0,115200 console =ttyS0,115200 init=/sbin/init boot=local rootwait root=/dev/sda3 ro noresume nos wap i915.modeset=1 loglevel=7 } menuentry "vmlinuz normal" { linux /efi/boot/vmlinuz quiet console=tty2 init=/sbin/init boot=local rootwait root=/dev/sda3 ro noresume noswap i915.modeset=1 loglevel=1 } menuentry "vmlinuz serial debug" { linux /efi/boot/vmlinuz earlyprintk=serial,ttyS0,115200 console=ttyS0,115200 i nit=/sbin/init boot=local rootwait root=/dev/sda3 ro noresume noswap i915.modese t=1 loglevel=7 }
When the USB key boots, I'll see a menu that lets me select which boot path to use.
FIXME: Does this still apply to current boards? If yes, to which ones?
Copy the new bzImage file into the /boot directory on your USB key‘s partition 3. The /boot/extlinux.conf file will look for the kernel called vmlinuz, but you can edit that config file to add a line to look for your test kernel too. For example, here’s my USB key's partition 3:
blackadder$ mount | grep sdc3 /dev/sdc3 on /media/C-KEYFOB type ext3 (rw,nosuid,nodev,uhelper=hal) blackadder$ ls -l /media/C-KEYFOB/boot total 6940 lrwxrwxrwx 1 root root 19 Apr 23 01:55 System.map -> System.map-2.6.32.9 -rw-r--r-- 1 root root 1313402 Apr 23 00:12 System.map-2.6.32.9 -rw-r----- 1 root root 2851056 Apr 26 10:30 bzImage lrwxrwxrwx 1 root root 15 Apr 23 01:55 config -> config-2.6.32.9 -rw-r--r-- 1 root root 74534 Apr 23 00:12 config-2.6.32.9 -rw-r--r-- 1 root root 409 Apr 23 01:53 extlinux.conf -r--r--r-- 1 root root 14336 Apr 23 01:53 extlinux.sys lrwxrwxrwx 1 root root 16 Apr 23 01:55 vmlinuz -> vmlinuz-2.6.32.9 -rw-r--r-- 1 root root 2821296 Apr 23 00:12 vmlinuz-2.6.32.9 blackadder$ cat /media/C-KEYFOB/boot/extlinux.conf DEFAULT chromeos-usb PROMPT 1 TIMEOUT 20 label chromeos-usb menu label chromeos-usb kernel vmlinuz append quiet console=tty2 init=/sbin/init boot=local rootwait root=/dev/sdb3 ro noresume noswap i915.modeset=1 loglevel=1 label chromeos-test menu label chromeos-test kernel bzImage append console=tty1 init=/sbin/init boot=local rootwait root=/dev/sdb3 ro noresume noswap i915.modeset=1 loglevel=7
When the USB key boots, I can hit TAB to see the list of boot choices, and can pick the one I want by entering the label.
If you need to load kernel modules from a location other than the root filesystem, module locking must be disabled. Either a kernel command line option can be used:
lsm.module_locking=0
Or, on images with dm-verity disabled (--noenable_rootfs_verification), the restriction can be disabled via the exposed sysctl:
echo 0 >/proc/sys/kernel/chromiumos/module_locking
If you need to block kernel modules for specific overlays, modify the overlay-/chromeos-base/chromeos-bsp-/chromeos-bsp--.ebuild file.
Add the following two lines to the end of the src_install() function:
insinto "/etc/modprobe.d" doins "${FILESDIR}/<blocklist>"
The ${FILESDIR} variable points to the files/ directory within the chromeos-bsp-/ directory. Within this directory, add your (ex cros-blocklist.conf).
For each kernel module you wish to block, add the following line to :
blacklist <module name>
You can also use # comments within these files to explain why the kernel module needs to be blocked.
git supports multiple branches coexisting in the same directory tree, and kernel make system supports placing the kernel build output in a separate directory (using the O=<path> make command line parameter).
To create separate builds get per kernel git branch, while in the cloned kernel source tree root create a build directory for your current branch, for instance:
mkdir ../build/<branch_name>
and then just add O=../../build/<branch_name> to make invocations described above. Or use the following bash script to take care of all make command line parameters other than make targets:
kmake () { b=$(git branch 2>/dev/null | grep '^\*' | awk '{print $2}') if [ "${b}" == "" ]; then echo "not in a git tree" return fi build_dir="../build/${b}" if [ ! -d "${build_dir}" ]; then echo "build directory ${build_dir} does not exist" return fi make_jobs=$(expr 2 \* $(cat /proc/cpuinfo | grep -c '^processor')) make ARCH=i386 O=${build_dir} -j "${make_jobs}" $* }
FIXME: The instructions the link leads to seem to apply only to ancient boards running U-Boot.
Please take a look at doc on network based development. While setting up your environment might appear to be harder and more time consuming, in many cases it will allow to test kernel modifications much faster and easier than the ways described below.
When adding new files to the kernel, please add a regular Google copyright header to them. In particular this is true for any code that will eventually find its way upstream (which should include practically everything we do). The main reason for this is that there‘s no concept of “The Chromium OS Authors” outside of our project, since it refers to the AUTHORS file that isn’t bundled with the kernel.
Each file type has its own SPDX comment format, discussed here:
C header files:
/* SPDX-License-Identifier: GPL-2.0 */ /* * <short description> * * Copyright 2019 Google LLC. */
C source files:
// SPDX-License-Identifier: GPL-2.0 /* * <short description> * * Copyright 2019 Google LLC. */
For reference, old drivers already existing in upstream might still have the full text format, which would look like below.
/* * Copyright 2018 Google LLC. * * This software is licensed under the terms of the GNU General Public * License version 2, as published by the Free Software Foundation, and * may be copied, distributed, and modified under those terms. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. */
Make sure that your patch builds fine with allmodconfig:
mkdir -p ../build/x86-64../build/arm64 # Native build (x86-64) make O=../build/x86-64 allmodconfig make O=../build/x86-64all -j50 2>&1|tee ../v3.18-build/x86-64.log # arm64 build CROSS_COMPILE=aarch64-cros-linux-gnu- ARCH=arm64 O=../build/arm64 make allmodconfig CROSS_COMPILE=aarch64-cros-linux-gnu- ARCH=arm64 O=../build/arm64 make -j64 >/dev/null
Test build with Chrome OS config:
cd src/third_party/kernel/v4.19 git checkout linux-next/master # Checkout config options only git checkout m/master -- chromeos # Normal emerge (chroot) emerge-${BOARD} -av chromeos-kernel-4_19
See the Kernel Design page for some more details.
For changes which cannot be submitted upstream to the official Linux Kernel repository, the commit message is important. We use the following conventions:
An example subject line is: CHROMIUM: ARM: tegra: Add initial support for aebl
If not sure, use git log to check commit messages of earlier commits for the same file or other files in the same directory.
Do not include configuration changes (i.e. changes to files within chromeos/config) with other code changes. See the next section for these.
Files may not be suitable for submission upstream because they have Chromium OS-specific information, or may be based on other changes which are local to the Chromium OS project. Such changes may not be upstreamed, but the Chromium OS project team will continue to maintain the changes.
When a commit involves configuration changes, make sure that any code changes are separated out into a different commit. The configuration commit should contain only changes to files within the chromeos/config directory tree.
The commit message should start with CHROMIUM: config:
An example message is: CHROMIUM: config: enable aebl config
See the Contributing Guide for details on how to upload your changes, get them tested & reviewed, and ultimately get them into the tree.
So the first step is to figure out what are the problems:
Look at kernel messages (dmesg), on boot, or at any time, and find errors or warnings that should not be there. e.g. what does dmesg -w give.
Look at /var/log/messages (contains kernel logs from dmesg, as well as logs from most system services).
Look at top output, to check if certain processes are hogging CPU or memory.
Build the kernel with USE=kasan. KASan is a great tool to find memory issues in the kernel (use it with the other tests below).
Other debugging kernel options:
USE=ubsanUSE=lockdebugUSE=kmemleakUSE=failslab. Then configure in /sys/kernel/debug/failslab (setting probability to 10 and times to 1000 is a good start).FAIL_MMC_REQUESTCONFIG_FAIL_MMC_REQUEST, CONFIG_FAULT_INJECTION and CONFIG_FAULT_INJECTION_DEBUG_FS, and then configure in /sys/kernel/debug/mmc{n}/fail_mmc_request/.Stress tests:
Single iteration suspend test: powerd_dbus_suspend
Multi-iteration suspend test: suspend_stress_test
Reboot loops, keeping ramoops at each reboot to analyse failures (setup SSH keys first):
#!/bin/bash IP=$1 i=0 while true; do while ! scp root@$IP:/sys/fs/pstore/console-ramoops-0 ramoops-$i; do sleep 1 done ssh root@$IP reboot sleep 20 i=$((i+1)) done
Then run this to extract out the bad ramoops
mkdir bad; ls ramoops-* | xargs -I{} sh -c \ 'tail -n 1 {} | grep -v "reboot: Restarting system" && cp {} bad/{}'
restart ui in a loop.
Run tests (autotests, CTS, etc.)
Stress test cpufreq by changing frequency constantly
Balloons (from crbug.com/468342, or src/platform/microbenchmarks/mmm_donut.py
Unbind/rebind drivers (may be nice with kasan/kmemleak, too):
cd /usr/local find /sys/bus/\*/drivers/\*/\* -type l -maxdepth 0 | grep -v "module$" > list sync cat list | xargs -I{} sh -c 'echo {}; cd \`dirname {}\`; echo \`basename {}\` > unbind; echo \`basename {}\` > bind; sleep 5' # see what crashes, edit list to remove bad drivers, continue
Run the following commands on the target. This needs to be done just once after an install.
touch /var/lib/crash_sender_paused touch /home/chronos/"Consent To Send Stats" chown chronos:chronos /var/lib/crash_sender_paused chown chronos:chronos /home/chronos/"Consent To Send Stats" sync; sync; sync
The crashes will then appear in /var/spool/crash.
One advanced debugging technique is to use printk and other syslog output functions to tell you what your code is doing.
dev_[info/warn/err] or pr_[info/warn/err]), reboot, see what happens.dev_dbg/pr_dbg in the kernel code can be enabled by setting #define DEBUG at the top of the source file (before all includes).dump_stack calls in places may also be very useful.Sometimes adding too many printks changes behaviour (Heisenbug), or makes the system unusable, so be careful where you put them (probably a bad idea to put them on every timer tick for example, or on each incoming network packet in a wifi driver). If such granularity is needed however, there are some options:
ftrace, see below.ratelimit to minimize the number of prints. See example CL.If you need to see kernel log messages (e.g., over UART) before the full console driver is running, earlyprintk or earlycon may help you. Find more info in the kernel parameters guide.
Note that unlike with earlyprintk, you often don't need any hardware-specific arguments to use earlycon -- you only need to add earlycon to the kernel command line. The kernel can pick up the appropriate console parameters from either the Device Tree (via /chosen/stdout-path) or ACPI (via the SPCR table).
Pitfall: Chrome OS kernel command lines typically include an empty console= parameter by default, which prevents directing kernel logs to the default console (earlycon or otherwise). Remove this if you want to direct kernel logs to your console.
Caveats apply: architecture and driver support varies. For example, ACPI/SPCR earlycon support is not fully integrated in Chrome OS as of this writing.
Dynamic debugging allows one to enable/disable debugging messages in kernel code at runtime (e.g., calls to dev_dbg or pr_debug).
Using dynamic debugging requires the CONFIG_DYNAMIC_DEBUG config option to be enabled. By default dynamic debug is disabled on Chrome OS.
If using menuconfig, the following enables it:
Kernel hacking
---> printk and dmesg options
---> [*] Enable dynamic printk() support
Once the kernel is compiled with CONFIG_DYNAMIC_DEBUG, you can use the following commands to control the output.
echo "+p" > /sys/kernel/debug/dynamic_debug/control
echo "-p" > /sys/kernel/debug/dynamic_debug/control
echo "module cros_ec_spi +p" > /sys/kernel/debug/dynamic_debug/control echo "module cros_ec_proto +p" > /sys/kernel/debug/dynamic_debug/control
cat /sys/kernel/debug/dynamic_debug/control
See Dynamic Debug for complete details and syntax.
ftrace allows you to trace events in the kernel (e.g. function calls, graphs), without introducing too much overhead. This is especially useful to debug timing/performance issues, or for cases when adding printk changes the behaviour.
It is possible to add custom messages by using trace_printk.
Example, to trace functions starting with rt5667 and mtk_spi:
cd /sys/kernel/debug/tracing echo "rt5677*" > set_ftrace_filter echo "mtk_spi_*" >> set_ftrace_filter echo function > current_tracer echo 1 > tracing_on # Look at the trace cat trace
trace-cmd, available on test images, provides a nice frontend to the tracing infrstructure. With trace-cmd, the above becomes:
# 'record' configures ftrace and writes to trace-cmd.dat (default file). trace-cmd record -p function -l 'rt5677*' -l 'mtk_spi_*' # Hit Ctrl^C to stop recording # 'report' formats trace-cmd.dat, dumps to stdout. trace-cmd report
See the trace-cmd man pages or the LWN trace-cmd HOWTO for more info.
Another example:
cd /sys/kernel/debug/tracing # Sample output: blk function_graph function nop. These are valid values you can echo into current_tracer cat available_tracers # By default this should output 'nop' cat current_tracer # function_graph is useful too echo "function" > current_tracer # This should output "all functions enabled" by default cat set_ftrace_filter # You can also append with "echo *nl80211* >> set_ftrace_filter" echo *nl80211* > set_ftrace_filter # Should be the number of functions enabled. wc -l set_ftrace_filter # Clear out the tracing pipe of the previous junk. You will need to Ctrl-C kill this after a while cat trace_pipe > /dev/null # You should see nothing, now start performing actions that will lead to your module/code being called. cat trace_pipe
Another ftrace article: https://lwn.net/Articles/370423/
Other tricks:
It is also possible to start tracing on boot by adding kernel parameters (useful to debug early hangs).
It is possible to ask the kernel to dump the ftrace buffer to uart on oops, this is useful to debug hangs/crashes:
echo 1 > /proc/sys/kernel/ftrace_dump_on_oops.
Dumping the whole buffer may take an enormous amount of time at serial rate, but sometimes it's worth it.
BUG/WARN and friends provide nice backtraces. These can be very useful for figuring out what code path is triggering a hard to reproduce issue.
~/trunk/src/platform/dev/contrib/kernel_decode_stack -b kukui
Sometimes gdb is more useful (aarch64, update as needed):
aarch64-cros-linux-gnu-gdb /build/kukui/usr/lib/debug/boot/vmlinux disas /m function
TODO: This is anecdotal, and may not be an optimal or fully correct solution. Please verify and remove the TODO.
You have a few options:
1. Googler-only: Check out go/xstability. Clicking on sample crashes here go/crash with the filter set for that particular crash. Click on a sample report. Below the “Report Time” and “Client ID” you should “Files” with a link to “upload_file_kcrash”. This has the stack trace towards the end.
TODO: Add more details on this
2. If you are debugging a local crash on your device, look for the crash in /var/log/messages (unlikely that it would be saved there) or /sys/fs/pstore/console-ramoops. You may see some symbols preceded by question marks in the stack trace, something like the below.
<5>[ 25.801932] Call Trace: <5>[ 25.801947] [<ffffffffc008c064>] ieee80211_amsdu_to_8023s+0xec/0x2df [cfg80211] <5>[ 25.801968] [<ffffffffc02af0f2>] __iwl7000_ieee80211_sta_ps_transition+0x154a/0x21dc [iwl7000_mac80211] <5>[ 25.801987] [<ffffffffc03154e4>] ? iwl_mvm_send_lq_cmd+0x8e/0x9c [iwlmvm] <5>[ 25.802003] [<ffffffffc0324409>] ? iwl_mvm_rs_tx_status+0xf9c/0x1f5cd /4 [iwlmvm] <5>[ 25.802023] [<ffffffffc02b06f2>] __iwl7000_ieee80211_mark_rx_ba_filtered_frames+0x96e/0xcb0 [iwl7000_mac80211] <5>[ 25.802041] [<ffffffff9e4ee0f0>] ? kmem_cache_free+0x8a/0xc5 <5>[ 25.802059] [<ffffffffc02b08a1>] __iwl7000_ieee80211_mark_rx_ba_filtered_frames+0xb1d/0xcb0 [iwl7000_mac80211] <5>[ 25.802080] [<ffffffffc02b0dc6>] __iwl7000_ieee80211_rx_napi+0x392/0x46a [iwl7000_mac80211] <5>[ 25.802098] [<ffffffffc0316578>] iwl_mvm_rx_rx_mpdu+0x749/0x78b [iwlmvm] <5>[ 25.802113] [<ffffffffc0310f16>] iwl_mvm_enter_d0i3+0x359/0xe7f [iwlmvm] <5>[ 25.802128] [<ffffffffc023d504>] iwl_pci_unregister_driver+0xfdb/0x1439 [iwlwifi] <5>[ 25.802143] [<ffffffffc023e883>] iwl_pcie_irq_handler+0x57d/0x7d1 [iwlwifi] <5>[ 25.802157] [<ffffffff9e48c255>] ? free_irq+0x8a/0x8a <5>[ 25.802168] [<ffffffff9e48c272>] irq_thread_fn+0x1d/0x3c <5>[ 25.802179] [<ffffffff9e48be1a>] irq_thread+0x117/0x21a <5>[ 25.802191] [<ffffffff9e921dda>] ? __schedule+0x589/0x5d3 <5>[ 25.802202] [<ffffffff9e48b863>] ? kzalloc.constprop.37+0x1c/0x1c <5>[ 25.802214] [<ffffffff9e48bd03>] ? irq_thread_check_affinity+0x8f/0x8f <5>[ 25.802227] [<ffffffff9e45183b>] kthread+0xc0/0xc8 <5>[ 25.802238] [<ffffffff9e45177b>] ? __kthread_parkme+0x6b/0x6b <5>[ 25.802249] [<ffffffff9e92389c>] ret_from_fork+0x7c/0xb0 <5>[ 25.802259] [<ffffffff9e45177b>] ? __kthread_parkme+0x6b/0x6b
There are a few ways you can resolve the “? some_symbol + 0xoffset” format into a line of source code. For example, you can enter the cros_sdk chroot and load up the vmlinux file in gdb.
Be careful to use the gdb binary from the cross-toolchain of the $BOARD you are debugging on. TODO(crbug.com/995661): Chromium OS runs 32-bit ARM userspace on ARM64 boards and there is no good programmatic way of getting the right gdb tuple in such case, so just use aarch64-cros-linux-gnu-gdb with them for the time being.
(cr) user@machine /build/samus $ gdb="$(portageq-$BOARD envvar CHOST)-gdb" (cr) user@machine /build/samus $ file /build/$BOARD/usr/lib/debug/boot/vmlinux | grep -q aarch64 && gdb="aarch64-cros-linux-gnu-gdb" (cr) user@machine /build/samus $ ${gdb} /build/$BOARD/usr/lib/debug/boot/vmlinux
Next, use the list command to print the code at given address
Reading symbols from /build/samus/usr/lib/debug/boot/vmlinux...done. (gdb) list *( iwl_mvm_send_lq_cmd+0x8e) 0x12b5 is in iwl_mvm_send_lq_cmd (/mnt/host/source/src/third_party/kernel/v3.14/drivers/net/wireless-3.8/iwl7000/iwlwifi/mvm/utils.c:752). 747 }; 748 749 if (WARN_ON(lq->sta_id == IWL_MVM_STATION_COUNT)) 750 return -EINVAL; 751 752 return iwl_mvm_send_cmd(mvm, &cmd); 753 } 754 755 /** 756 * iwl_mvm_update_smps - Get a request to change the SMPS mode (gdb)
3. A slightly more tedious way of getting symbols is to symbolize the whole kernel using objdump -
cd /build/samus/var/cache/portage/sys-kernel/chromeos-kernel-3_14 # Pick a proper output location - the resulting file is > 2GB in size! objdump -e vmlinux > /tmp/objdump-output.txt grep your_kernel_symbol /tmp/objdump-output.txt
More information here and here.
KGDB is an in-kernel debugger implementation, which allows developers to attach a local GDB instance on their development machine to debug the kernel on a remote test machine, using a serial connection. You can find some information here:
To use KGDB with Chromium OS requires two steps for the test machine:
Step 1 can be done by building with USE="kgdb":
USE="kgdb vtconsole" emerge-${BOARD} chromeos-kernel-${VER}
Step 2 can be done by adding kgdboc=$TTY to the kernel config.txt, where $TTY depends on the board -- for many systems, this should be ttyS0, but some ARM SoCs use ttyS2.
Once you configure the target device, you can break into debug mode with the Alt-SysRq-G shortcut (see Linux SysRq docs); e.g.:
Alt-VolUp-G on the DUT keyboard (note: for this to work, you need to have console=ttyS0,115200n8 parameter, but you can also set loglevel=0 if you want to keep the console quiet and avoid associated slowdowns)<enter> ` Z G (brk-g) if using servoecho g > /proc/sysrq-triggerthen attach to the target console with your cross-targeted GDB:
${CROSS_ARCH}-gdb \ /build/${BOARD}/usr/lib/debug/boot/vmlinux \ -ex "set remotebaud 115200" \ -ex "target remote $(dut-control cpu_uart_pty | cut -d: -f2)"
Once attached, you can use standard GDB commands, though report has it that not everything works well (e.g., stepping and breakpoints) -- YMMV.
Besides basic GDB commands, you can make use of Linux-specific KDB commands via the monitor command. For more info, run this while attached:
(gdb) monitor help Command Usage Description ---------------------------------------------------------- [...]
You can get a list of modules and addresses in kgdb with monitor lsmod. Then you can add symbol files using the base addresses found there:
add-symbol-file /build/${BOARD}/usr/lib/debug/lib/modules/3.8.11/kernel/drivers/net/wireless/mwifiex/mwifiex.ko.debug 0xbf077000 add-symbol-file /build/${BOARD}/usr/lib/debug/lib/modules/3.8.11/kernel/drivers/net/wireless/mwifiex/mwifiex_sdio.ko.debug 0xbf0a0000
If you're in kgdb and want to get back to kdb:
maintenance packet 3 Ctrl-Z kill -9 %
Use dut-console script with -k parameter:
dut-console -p 9999 -c cpu -k
dut-console will also print instructions on how to attach gdb from inside the chroot.
If you want to use both KGDB and a standard serial console over the same serial port, you need to run a program like kdmx or agent-proxy to multiplex your connection. Both can be found at:
https://kernel.googlesource.com/pub/scm/utils/kernel/kgdb/agent-proxy/
kdmx is probably easier to deal with. If your serial port is at /dev/pts/80, you can start it with:
agent-proxy/kdmx/kdmx -n -b 115200 -p /dev/pts/80 -s /tmp/kdmx_tty_
You can find the ttys to use for console in /tmp/kdmx_tty_trm and for gdb in /tmp/kdmx_tty_gdb. Thus connect to the terminal with:
cu --nostop -l $(cat /tmp/kdmx_tty_trm)
and attach gdb with:
${CROSS_ARCH}-gdb \ /build/${BOARD}/usr/lib/debug/boot/vmlinux \ -ex "target remote $(cat /tmp/kdmx_tty_gdb)"
If telnet is more your style, use agent-proxy with:
agent-proxy 127.0.0.1:5510^127.0.0.1:5511 0 /dev/pts/80,115200
Then connect to the terminal with:
telnet localhost 5510
and attach gdb with:
${CROSS_ARCH}-gdb \ /build/${BOARD}/usr/lib/debug/boot/vmlinux \ -ex "target remote localhost:5511"
monitor ftdump calls sleeping allocation functions (as of 2016-11-17)To bisect along the upstream stable branch, first identify and test the merge commit and the chromeos branch.
Merge:
commit f5edda0c2aefe22f338c3a00c0aa52161976d4b1
Merge: ce070a331d16 399849e4654e
Author: Guenter Roeck <groeck@chromium.org>
Date: Wed Jul 1 08:17:19 2020 -0700
CHROMIUM: Merge 'v4.19.131' into chromeos-4.19
Merge of v4.19.131 into chromeos-4.19
Changelog:
----------------------------------------------------------------
Aaron Plattner (1):
ALSA: hda: Add NVIDIA codec IDs 9a & 9d through a0 to patch table
Aditya Pakki (1):
rocker: fix incorrect error handling in dma_rings_init
ChromeOS:
commit ce070a331d1697048ebfdb9011be299bc77940dc
Author: Benjamin Gordon <bmgordon@chromium.org>
Date: Thu Mar 26 13:23:28 2020 -0600
CHROMIUM: LSM: Convert symlink checks to MNT_NOSYMFOLLOW
Start a regular git bisect, identifying the merge as bad and chromeos branch as good:
git bisect git checkout f5edda0c2aef # Build and test git bisect bad git checkout ce070a331d16 # Build and test git bisect good
Git is smart enough to bisect along the upstream branch, rooted at the common branch point (the previous upstream merge).
At each bisection point, you need to merge in the chromeos branch. The device may not boot and function correctly if you do not do this:
git merge --no-commit ce070a331d16 # Resolve merge conflicts # Build/deploy/test kernel # Reset git state and continue bisection git reset --hard git bisect [good|bad]
Use perf! One easy perf method is to use perf top on the target device to look for hotspots.
On specific platforms, other tools may be available. For example on Intel, both socwatch and VTune can be used provided the proper kernel drivers are loaded.
FIXME: add some stuff on importing perf output into flamegraph tool, etc.
Let‘s suppose you’ve spotted a juicy new commit in Linus's upstream linux kernel tree that you just must have. Instead of creating a new branch and manually applying the changes, use git cherry-pick to do it for you. In addition, the repository maintainers appreciate it if the cherry-picked commit still contains the original author and git hash of the original upstream commit.
For “simple” UPSTREAM cherry-picks, one should first try using fromupstream.py script to prepare CLs “automagically”. Doug Anderson (author) provided examples for use.
Otherwise, the follow steps use git cherry-pick -x to do most of the work:
NAME git-cherry-pick - Apply the changes introduced by some existing commits SYNOPSIS git cherry-pick [--edit] [-n] [-m parent-number] [-s] [-x] [--ff] <commit>... DESCRIPTION Given one or more existing commits, apply the change each one introduces, recording a new commit for each. This requires your working tree to be clean (no modifications from the HEAD commit). OPTIONS ... -x When recording the commit, append to the original commit message a note that indicates which commit this change was cherry-picked from. Append the note only for cherry picks without conflicts. Do not use this option if you are cherry-picking from your private branch because the information is useless to the recipient. If on the other hand you are cherry-picking between two publicly visible branches (e.g. backporting a fix to a maintenance branch for an older release from a development branch), adding this information can be useful.
First, add Linus's tree as a remote to the chromium-os kernel tree (assuming the chromium-os root is ~/chromiumos):
cd ~/chromiumos/src/third_party/kernel git remote add linus git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git git remote update
This will take a little while as git fetches all upstream commits. Luckily, git is smart and won't refetch commits already in the chromium-os tree.
Once the tree is updated, take a brief look at whats been happening upstream recently to a particular path (--oneline shows short-form upstream hashes and the brief commit message):
git log --oneline linus/master /path/of/interest
We can view that juicy commit using its upstream hash:
git show <upstream_commit_hash>
To backport the commit to the chromium-os tree, first start a new branch from the current Tip of Tree (ToT). Then cherry-pick with -x to preserve the original author and hash, and -s to sign-off-by the commit:
repo sync . repo start my_upstream_commit . git cherry-pick -x -s <upstream_commit_hash>
Add TEST= and BUG= lines at the bottom of the patch description. Also, remember to keep the patch subject intact with only an addition of UPSTREAM: or BACKPORT: as a new prefix. Use UPSTREAM: if you are applying an upstream patch as-is, or BACKPORT: if you had to change the code to make it run with an older kernel version.
NOTE: Do not make functional changes to backported patches! Downstream changes in backports should be strictly limited to resolving conflicts. If you need to make a functional change to a backport (ie: changing a delay, tweaking a default value, etc), backport the change from upstream as-is and follow up with a separate patch with CHROMIUM prefix.
Now, the upstream commit is on its own branch, let's upload it to gerrit, like usual:
repo upload .
This will generate a gerrit change for review.
After review, submit the patch in gerrit like usual.
When backporting patches from Linus‘s kernel tree, you should tag your patch with UPSTREAM (or BACKPORT, if modifications were needed). But what about patches that are “on their way” upstream, but haven’t been merged for an official release yet?
FROMLIST: use this tag when a patch has been sent to a public mailing list for review, but hasn't yet been merged anywhere. Before submitting a patch like this, try to address any review comments made in the public forum. Please also include a link to the list the patch was obtained from. For example:
FROMLIST: bibble: a patch to fix everything ... ... original description verbatim, including any tags, ... e.g. Signed-off-by, Reviewed-by, etc. ... (am from https://lore.kernel.org/patchwork/patch/1060242/) ... ... any additional downstream information goes here, e.g. ... - (also found at A-LINK-BASED-ON-MESSAGE-ID), ... - BUG=, ... - TEST=, ... - Change-Id, ... - list of conflicts (e.g. generated by git), ... - cherry-picker's Signed-off-by, if not present in original description, ... - etc. ...
UPSTREAM: this tag should be used exclusively for patches that have actually landed in Linus' tree, not for cherry-picks from maintainer trees.
BACKPORT: follow the same rules as UPSTREAM, except that if you have to make changes to the patch, you should label it with BACKPORT and document what you had to change.
FROMGIT: use this tag for cherry-picks of patches from maintainer trees, which have been applied in preparation for an upcoming release.
Although it is a good reference for “what's going into the next release” never backport a patch straight from linux-next. Always source either a maintainer tree or a mailing list post.
When including patches from maintainer trees, be specific about your source tree and branch. For example, for a patch from the for-next branch in the chrome-platform tree:
FROMGIT: spi: mediatek: Only do dma for 4-byte aligned buffers ... (cherry picked from commit 1ce24864bff40e11500a699789412115fdf244bf git://git.kernel.org/pub/scm/linux/kernel/git/chrome-platform/linux.git for-next)
BACKPORT: FROMLIST: or BACKPORT: FROMGIT: follow the same rules as FROMLIST or FROMGIT, except that if you have to make changes to the patch, you should also label it with BACKPORT (in addition to FROMLIST/FROMGIT) and document what you had to change.
Previous discussions defining this practice:
../../../platform/dev/contrib/fromupstream.py -b b:123489157 \ -t "Deploy kukui kernel with USE=kmemleak, no kmemleak warning in __arm_v7s_alloc_table" \ 'git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git#next/032ebd8548c9d05e8d2bdc7a7ec2fe29454b0ad0'
Add project url in ~/.pwclientrc
[options] default=kernel [kernel] url=https://patchwork.kernel.org/ [lore] url=https://lore.kernel.org/patchwork/xmlrpc/
Then run:
../../../platform/dev/contrib/fromupstream.py -b b:132314838 -t "no crash with CONFIG_FAILSLAB" 'pw://10957015/' # or ../../../platform/dev/contrib/fromupstream.py -b b:132314838 -t "no crash with CONFIG_FAILSLAB" 'pw://kernel/10957015/'
In CrOS chroot (gerrit deps prints dependencies from top to bottom, so its better to use tac so that the bottom-most CL is set to ready first):
# If the CL of interest is HEAD, else substitute the gerrit CL number. cl=$(git log -1 --format='%(trailers:key=Change-Id,valueonly)') deps=( $(gerrit --raw deps "${cl}" | tac) ) gerrit label-v "${deps[@]}" 1 gerrit label-cq "${deps[@]}" 2
So you have an email/patch on patchwork, but you didn‘t subscribe to the mailing list, so you can’t reply to/review the change.
To fetch the email into your IMAP/gmail account:
python2.7 ./imap_upload.py patch.mbox --gmailmbox downloaded from patchwork doesn't include replies to the patch (e.g. reviewer comments). To obtain mbox containing replies, download mbox.gz files from https://lore.kernel.org/lkml/ instead.
There are various ways of building mainline Linux, but it can be useful to use existing Chrome OS tooling to build a non-Chrome-OS-flavored kernel. See the cros-kernel eclass documentation for tips on how to use the “fallback” configuration system to build any (e.g., mainline) kernel tree within the existing Portage-based flow.
Note: Chrome OS kernels often support hardware that is not yet supported in an upstream kernel release. Ensuring hardware support for your system is not covered here.
If given target device is not building kernel-next, you can switch by unmerging the standard kernel and then building kernel-next normally:
cros_workon --board=${BOARD} stop sys-kernel/chromeos-kernel emerge-${BOARD} --unmerge sys-kernel/chromeos-kernel cros_workon --board=${BOARD} start sys-kernel/chromeos-kernel-next cros_workon_make --board=${BOARD} sys-kernel/chromeos-kernel-next --install ~/trunk/src/scripts/update_kernel.sh --board=${BOARD} --remote=hostname...
Changes to parts of the kernel which are not purely Chrome OS- specific should be upstreamed where possible. This includes just about any part of the kernel: ARM- and x86-specific changes, driver patches and changes within the main kernel and mm source. You can start with a code review if you like. Take a look on the kernel mailing list to get a feel for how people submit and review patches.
To upstream, create a remote to track upstream.
For example the main kernel:
git remote add upstream git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git git fetch upstream git checkout -b send-upstream upstream/master
You can then create a commit within this branch. This can be done either by cherry-picking the commit from another branch and perhaps changing the commit message:
git cherry-pick my-change git commit --amend # edit the message and save
or using git am to turn a patch into a commit:
git am my-change.patch
or manually applying a patch, and then committing:
patch -p1 < my-change.patch git add ... git commit # create a suitable message
There are two aspects of having correct patches to send upstream: not having Chromium OS-specific details, and meeting all the Linux kernel requirements.
Verifying these details is as simple as loading the patch file in your favorite editor. Edit the file manually to become compliant; this will, of course, have no affect on the source or commit message stored by git.
Once all of the above is true, you can move on to checking for compliance with the Linux Kernel guidelines.
First off, make sure the Kernel builds with patch applied.
For style, the patman tool (see below) will automatically run checkpatch.pl on your change. If you‘d like to run the checkpatch.pl tool manually, here’a an example workflow:
git format-patch HEAD~ scripts/checkpatch.pl 0001-my-change.patch # make improvements git add ... git commit --amend # rinse and repeat
It is possible to send out patches using git send-email manually, but for most usecases using the patman CLI is sufficient and can save a lot of time.
(See the next section for first-time credential setup for using patman or git send-email.)
Patman automates patch creation, checking, change list creation, cover letter, sending to the mailing list, etc. You can find patman in the U-Boot tree (src/third_party/u-boot/files/tools/patman). It usually should be run outside of the chroot, so you could create an alias, or a symlink to somewhere in your path:
alias patman='~/chromiumos/src/third_party/u-boot/files/tools/patman/patman' # or ln -s ~/chromiumos/src/third_party/u-boot/files/tools/patman/patman ~/bin
To use patman, amend your top commit to have the line:
Series-to: LKML <linux-kernel@vger.kernel.org> Series-cc: (anyone you want to Cc all patches in the series to)
Then type:
patman -n
to generate patches, check that they will go to the right place, and send them. Or:
patman
to generate patches and send them.
Various options are available. Particularly useful ones are:
Full documentation is available in the README (patman -h) or here. Take a look at the automated change list creation and the alias support also.
If you have never sent email from the command-line, or from git send-email, then there is some setup required.
NOTE: Googlers who can access pre-released Google-corp binaries should use instructions from this internal site instead: http://go/sendgmail.
The following instructions are for open-source contributors or Googlers who are directly using gLaptops for sending out patches.
Install git send-email
git send-email --help shows an error, you'll need to install itapt-get install git-emailDecide on the email address, password, and mail server to use
You must configure git's send-email command with the details of how to send email from your identity. The rest of this section will explain how to set up a google-mail based account (e.g. an @gmail.com address, @chromium.org, etc). If you have a different mail server, please contact the system administrator (or check some help docs related to your email service) for the correct settings.
NOTE: For Googlers, note that DMARC restrictions prevent usage of your @google.com email address. Use http://go/chromium-account to obtain an @chromium.org address.
For google-mail-based addresses, it's recommended to use an “App Password” for convenience when storing your real password on disk is undesireable (which should be most cases). Follow these instructions to obtain an App Password, and use it as the smtppass value in the next section.
Edit your ~/.gitconfig
Open up your ~/.gitconfig file to include the following stanza:
[sendemail] smtpserver = smtp.gmail.com smtpserverport = 587 smtpencryption = tls smtpuser = YOUR_EMAIL_ADDRESS smtppass = PASSWORD confirm = always
Remember to swap in the YOUR_EMAIL_ADDRESS with your full email address, and PASSWORD with your password (or App Password).
To use the following script, you will need to have created a patch file using git format-patch. Also note that you will have to recreate the patch file, and re-check your patch file each time you check in code to your source tree.
This script might be useful also, as it checks a series of patches, checks for Chrome OS-specific commit tags and prints a summary at the end. Put it in your path and run it from anywhere.
#! /bin/sh # Pass a list of patchfiles to check for compliance KERNEL=./scripts/ OUT=$(tempfile) while (( "$#" )); do ERRCP= ERR= "${KERNEL}/checkpatch.pl" $1 || ERRCP=1 grep BUG= $1 && ERR="$ERR BUG" grep TEST= $1 && ERR="$ERR TEST" grep "Change-Id" $1 && ERR="$ERR Change-Id" grep "Review URL" $1 && ERR="$ERR Review URL" if [ -n "${ERR}" ]; then echo "Bad $1 ($ERR)" >>$OUT else echo "OK $1" >>$OUT fi shift done cat $OUT