I have 2 500G hard drives partitioned with a few 'normal' partitions for /boot, swap, etc but mostly in a raid 1 array, which is divided into several LVM partitions. For many Fedora releases, I have done fresh installs, preserving the /home LVM partition, and rotating the root partition among several 30-40G LVM partitions for /. This has always worked splendidly until Fedora 21/22. Recently I decided that it was time to move up to Fedora 22, and with failing that, to Fedora 21. I have tried several Fedora 22 and Fedora 21. live install disks (desktop, server) and the server isos. With the live disks, the installer doesn't see my existing LVM partitions, so I can't try to install into them. With the 21 and 22 server isos, the installer sees the LVM partitions and appears to proceed normally, but the final reboot boots into a grub rescue shell, and the only partition visible to this shell is my /boot partition. I also tried the Fedora 22 netinstall iso, which also ultimately resulted in reboot into the grub rescue shell.
I have googled several variations of 'grub doesn't recognize LVM partitions' without turning up anything helpful.
Have there been changes to the LVM software that might result in an incompatibility between grub and older LVM versions?
Best,
-jmw-
On Sat, Jul 11, 2015 at 4:25 PM, John Wright jwright2@san.rr.com wrote:
I have 2 500G hard drives partitioned with a few 'normal' partitions for /boot, swap, etc but mostly in a raid 1 array, which is divided into several LVM partitions. For many Fedora releases, I have done fresh installs, preserving the /home LVM partition, and rotating the root partition among several 30-40G LVM partitions for /. This has always worked splendidly until Fedora 21/22. Recently I decided that it was time to move up to Fedora 22, and with failing that, to Fedora 21. I have tried several Fedora 22 and Fedora 21. live install disks (desktop, server) and the server isos. With the live disks, the installer doesn't see my existing LVM partitions, so I can't try to install into them. With the 21 and 22 server isos, the installer sees the LVM partitions and appears to proceed normally, but the final reboot boots into a grub rescue shell, and the only partition visible to this shell is my /boot partition. I also tried the Fedora 22 netinstall iso, which also ultimately resulted in reboot into the grub rescue shell.
I have googled several variations of 'grub doesn't recognize LVM partitions' without turning up anything helpful.
Have there been changes to the LVM software that might result in an incompatibility between grub and older LVM versions?
I'm pretty sure GRUB doesn't understand LVM thin volumes, and maybe not LVM RAID. But otherwise there are no changes I'm aware of that would prevent this from working. It sounds more likely there's some sort of udev or multipath confusion at boot time from lives that's helping to cause misidentification.
All I can suggest is to file a bug report against anaconda. Boot the live media, launch the installer, go to custom partitioning, and then before changing anything, drop to tty2, go into /tmp and collect the logs. What I usually do is: tar -acf f22analogs.tar.gz *log scp f22analogs.tar.gz chris@blah.local:~/Desktop/
And there untar it, and file a bug, attaching each log file as a separate attachment. Then post the bug URL here.
If I get a moment to reproduce it in Rawhide, then I'll update the bug and hopefully it gets fixed for Fedora 23. Assuming it's a bug. It's also possible I'll find a work around, in which case I'll put that in the bug report also.
On Sat, 11 Jul 2015, John Wright wrote:
I have 2 500G hard drives partitioned with a few 'normal' partitions for /boot, swap, etc but mostly in a raid 1 array, which is divided into several LVM partitions. For many Fedora releases, I have done fresh installs, preserving the /home LVM partition, and rotating the root partition among several 30-40G LVM partitions for /. This has always worked splendidly until Fedora 21/22. Recently I decided that it was time to move up to Fedora 22, and with failing that, to Fedora 21. I have tried several Fedora 22 and Fedora 21. live install disks (desktop, server) and the server isos. With the live disks, the installer doesn't see my existing LVM partitions, so I can't try to install into them. With the 21 and 22 server isos, the installer sees the LVM partitions and appears to proceed normally, but the final reboot boots into a grub rescue shell, and the only partition visible to this shell is my /boot partition. I also tried the Fedora 22 netinstall iso, which also ultimately resulted in reboot into the grub rescue shell.
I have googled several variations of 'grub doesn't recognize LVM partitions' without turning up anything helpful.
Have there been changes to the LVM software that might result in an incompatibility between grub and older LVM versions?
Best,
-jmw-
Bug report filed, bugzilla Bug 1242193
-jmw-
There are messages in the storage.log about md126 and md127 being degraded. That would be a problem for discovering anything on those, even if they come up OK degraded, Anaconda won't install to degraded arrays. But I also don't know what messages you get about this, it should give an error message rather than silently fail.
Two things to update the bug with. What live media installer does see this correctly? Was it Fedora 20 or Fedora 21? Or both? And then also, if you boot the Fedora 22 Workstation (live) media, launch the installer, go to custom partitioning, drop to shell - capture:
cat /proc/mdstat > mdstat.txt mdadm -Dv /dev/md126 > md126.txt mdadm -Dv /dev/md127 > md127.txt journalctl -b -l -o short-monotonic > journal.log
scp those off machine or cp elsewhere and attach those to the bug too. For the mdadm commands, make sure you use the md device designation returned by the first command. I'm pretty sure it'll be md126 and md127 again, but I suppose that could change on subsequent boots.
Chris Murphy
I have done several netinstalls on other computers with no problem. However, the computer I am having a problem with has raid1. For the set root entry I have tried: copied set root from other menu entries set root='hd0,msdos1' When the error message no /dev/root drops to dracut I have done 'ls -l /dev/by-label*' and set root = to the uuid
All with no luck in getting past the /dev/root error
Searching for a solution to this problem has found nothing.
What am I missing?
Thanks,
David
On Sun, Jul 12, 2015 at 10:33 AM, dwoody5654 dwoody5654@gmail.com wrote:
I have done several netinstalls on other computers with no problem. However, the computer I am having a problem with has raid1. For the set root entry I have tried: copied set root from other menu entries set root='hd0,msdos1' When the error message no /dev/root drops to dracut I have done 'ls -l /dev/by-label*' and set root = to the uuid
All with no luck in getting past the /dev/root error
Searching for a solution to this problem has found nothing.
What am I missing?
I don't understand if you're getting this post-install, or when booting the netinstall media on a system with pre-existing mdadm raid1? You shouldn't have to make any modifications to the grub menu entry in either case, so I don't know why you think you need to do that.
The fact you're getting to a dracut prompt suggests this is not a problem with the bootloader not finding its root (set root in GRUB speak is referring to its root, i.e. the device and partition that /boot/grub2 is located on); it's sounding more like a problem with mounting sysroot, which suggests either a problem with the grub root=UUID= value (which should be for the fs volume UUID as reported by blkid, not a partition UUID), or a problem with fstab, or a problem with assembling the raid.
Most useful would be putting the rdsosreport.txt that dracut usually prepares in such cases, it will tell you at the dracut prompt where this is, it should be in /run/, but if one isn't generated maybe most of the useful information can be found with:
journalctl -b -l -o short-monotonic > journal.log blkid > blkid.txt
scp those somewhere, or copy to usb stick, and then post them somewhere and then a URL in this thread where to find them.
On 07/12/2015 01:37 PM, Chris Murphy wrote:
On Sun, Jul 12, 2015 at 10:33 AM, dwoody5654 dwoody5654@gmail.com wrote:
I have done several netinstalls on other computers with no problem. However, the computer I am having a problem with has raid1. For the set root entry I have tried: copied set root from other menu entries set root='hd0,msdos1' When the error message no /dev/root drops to dracut I have done 'ls -l /dev/by-label*' and set root = to the uuid
All with no luck in getting past the /dev/root error
Searching for a solution to this problem has found nothing.
What am I missing?
I don't understand if you're getting this post-install, or when booting the netinstall media on a system with pre-existing mdadm raid1? You shouldn't have to make any modifications to the grub menu entry in either case, so I don't know why you think you need to do that.
Thanks for your reply. Sorry for not including enough info in first post.
I created a menu entry in the current F20 grub.cfg with menu_entry name of 'Remote Install'. So I boot to the netinstall from F20. This has been working up until the current issue. The 2 pertinent lines are: linux /boot/vmlinuz-remote repo=hd:md126:/david/Fedora-Server-netinst-i386-21.iso noselinux ks.device=00:14:6c:55:f1:85 ks=hd:md126:/david/ks.cfg --noip6 ramdisk_size=8192 panic=30 initrd /boot/initrd-remote.img
The set root line is what I think I do not have set correctly. The /dev/root error comes up during the initial netinstall boot up. The file rdsosreport.txt is below.
The fact you're getting to a dracut prompt suggests this is not a problem with the bootloader not finding its root (set root in GRUB speak is referring to its root, i.e. the device and partition that /boot/grub2 is located on); it's sounding more like a problem with mounting sysroot, which suggests either a problem with the grub root=UUID= value (which should be for the fs volume UUID as reported by blkid, not a partition UUID), or a problem with fstab, or a problem with assembling the raid.
Most useful would be putting the rdsosreport.txt that dracut usually prepares in such cases, it will tell you at the dracut prompt where this is, it should be in /run/, but if one isn't generated maybe most of the useful information can be found with:
journalctl -b -l -o short-monotonic > journal.log blkid > blkid.txt
scp those somewhere, or copy to usb stick, and then post them somewhere and then a URL in this thread where to find them.
Do not have a place to post a file and reference it by url.
The sdc1 is the flash drive I used to save a copy of the rdsosreport.txt to.
The file contents for rdsosreport.txt is:
+ cat /lib/dracut/dracut-038-30.git20140903.fc21 dracut-038-30.git20140903.fc21 + cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-remote repo=hd:md126:/david/Fedora-Server-netinst-i386-21.iso noselinux ks.device=00:14:6c:55:f1:85 ks=hd:md126:/david/ks.cfg --noip6 ramdisk_size=8192 panic=30 + '[' -f /etc/cmdline ']' + for _i in '/etc/cmdline.d/*.conf' + '[' -f /etc/cmdline.d/99-anaconda-disable-disk-activation.conf ']' + echo /etc/cmdline.d/99-anaconda-disable-disk-activation.conf /etc/cmdline.d/99-anaconda-disable-disk-activation.conf + cat /etc/cmdline.d/99-anaconda-disable-disk-activation.conf rd.dm=0 rd.md=0 rd.lvm=0 rd.luks=0 + cat /proc/self/mountinfo 1 1 0:2 / / rw shared:1 - rootfs rootfs rw,size=1659784k,nr_inodes=204355 15 1 0:15 / /sys rw,nosuid,nodev,noexec,relatime shared:2 - sysfs sysfs rw 16 1 0:4 / /proc rw,nosuid,nodev,noexec,relatime shared:7 - proc proc rw 17 1 0:5 / /dev rw,nosuid shared:8 - devtmpfs devtmpfs rw,size=1659796k,nr_inodes=204361,mode=755 18 15 0:16 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:3 - securityfs securityfs rw 19 17 0:17 / /dev/shm rw,nosuid,nodev shared:9 - tmpfs tmpfs rw 20 17 0:11 / /dev/pts rw,nosuid,noexec,relatime shared:10 - devpts devpts rw,gid=5,mode=620,ptmxmode=000 21 1 0:18 / /run rw,nosuid,nodev shared:11 - tmpfs tmpfs rw,mode=755 22 15 0:19 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:4 - tmpfs tmpfs ro,mode=755 23 22 0:20 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:5 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 24 15 0:21 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:6 - pstore pstore rw 25 22 0:22 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:12 - cgroup cgroup rw,cpuset 26 22 0:23 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:13 - cgroup cgroup rw,cpu,cpuacct 27 22 0:24 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:14 - cgroup cgroup rw,memory 28 22 0:25 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,devices 29 22 0:26 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,freezer 30 22 0:27 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,net_cls,net_prio 31 22 0:28 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,blkio 32 22 0:29 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,perf_event 33 1 0:30 / /var/lib/nfs/rpc_pipefs rw,relatime shared:20 - rpc_pipefs rpc_pipefs rw 55 15 0:32 / /sys/kernel/config rw,relatime shared:21 - configfs configfs rw + cat /proc/mounts rootfs / rootfs rw,size=1659784k,nr_inodes=204355 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,nosuid,size=1659796k,nr_inodes=204361,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 + blkid /dev/sda1: UUID="2b912a3c-ecac-4551-ea43-d2f031ce1e57" UUID_SUB="1d86f129-3945-c0f3-a875-1d0c37c7a5fd" LABEL="star11.home.com:root" TYPE="linux_raid_member" PARTUUID="00072692-01" /dev/sda2: UUID="a92ca2fd-998f-d542-feb8-59fa8bd7ab33" UUID_SUB="74923772-ac33-195b-aaf3-28c64a08b7a3" LABEL="star11.home.com:home" TYPE="linux_raid_member" PARTUUID="00072692-02" /dev/sdb1: UUID="2b912a3c-ecac-4551-ea43-d2f031ce1e57" UUID_SUB="203b9602-aa3d-5ef1-d457-ad02b99eac03" LABEL="star11.home.com:root" TYPE="linux_raid_member" PARTUUID="0009d086-01" /dev/sdb2: UUID="a92ca2fd-998f-d542-feb8-59fa8bd7ab33" UUID_SUB="d0b6524a-2250-776b-95c3-b0502ba2d0d4" LABEL="star11.home.com:home" TYPE="linux_raid_member" PARTUUID="0009d086-02" /dev/sdc1: UUID="a9682244-3d0f-4469-ae80-5d6ef5fd148f" TYPE="ext2" PARTUUID="c3072e18-01" + blkid -o udev ID_FS_UUID=2b912a3c-ecac-4551-ea43-d2f031ce1e57 ID_FS_UUID_ENC=2b912a3c-ecac-4551-ea43-d2f031ce1e57 ID_FS_UUID_SUB=1d86f129-3945-c0f3-a875-1d0c37c7a5fd ID_FS_UUID_SUB_ENC=1d86f129-3945-c0f3-a875-1d0c37c7a5fd ID_FS_LABEL=star11.home.com:root ID_FS_LABEL_ENC=star11.home.com:root ID_FS_TYPE=linux_raid_member ID_FS_PARTUUID=00072692-01
ID_FS_UUID=a92ca2fd-998f-d542-feb8-59fa8bd7ab33 ID_FS_UUID_ENC=a92ca2fd-998f-d542-feb8-59fa8bd7ab33 ID_FS_UUID_SUB=74923772-ac33-195b-aaf3-28c64a08b7a3 ID_FS_UUID_SUB_ENC=74923772-ac33-195b-aaf3-28c64a08b7a3 ID_FS_LABEL=star11.home.com:home ID_FS_LABEL_ENC=star11.home.com:home ID_FS_TYPE=linux_raid_member ID_FS_PARTUUID=00072692-02
ID_FS_UUID=2b912a3c-ecac-4551-ea43-d2f031ce1e57 ID_FS_UUID_ENC=2b912a3c-ecac-4551-ea43-d2f031ce1e57 ID_FS_UUID_SUB=203b9602-aa3d-5ef1-d457-ad02b99eac03 ID_FS_UUID_SUB_ENC=203b9602-aa3d-5ef1-d457-ad02b99eac03 ID_FS_LABEL=star11.home.com:root ID_FS_LABEL_ENC=star11.home.com:root ID_FS_TYPE=linux_raid_member ID_FS_PARTUUID=0009d086-01
ID_FS_UUID=a92ca2fd-998f-d542-feb8-59fa8bd7ab33 ID_FS_UUID_ENC=a92ca2fd-998f-d542-feb8-59fa8bd7ab33 ID_FS_UUID_SUB=d0b6524a-2250-776b-95c3-b0502ba2d0d4 ID_FS_UUID_SUB_ENC=d0b6524a-2250-776b-95c3-b0502ba2d0d4 ID_FS_LABEL=star11.home.com:home ID_FS_LABEL_ENC=star11.home.com:home ID_FS_TYPE=linux_raid_member ID_FS_PARTUUID=0009d086-02
ID_FS_UUID=a9682244-3d0f-4469-ae80-5d6ef5fd148f ID_FS_UUID_ENC=a9682244-3d0f-4469-ae80-5d6ef5fd148f ID_FS_TYPE=ext2 ID_FS_PARTUUID=c3072e18-01 + ls -l /dev/disk/by-id /dev/disk/by-path /dev/disk/by-uuid /dev/disk/by-id: total 0 lrwxrwxrwx 1 root 0 9 Jul 12 18:45 ata-HL-DT-STDVD-RAM_GSA-H55L -> ../../sr0 lrwxrwxrwx 1 root 0 9 Jul 12 18:45 ata-ST1000DM003-1CH162_S1DH17X6 -> ../../sdb lrwxrwxrwx 1 root 0 10 Jul 12 18:45 ata-ST1000DM003-1CH162_S1DH17X6-part1 -> ../../sdb1 lrwxrwxrwx 1 root 0 10 Jul 12 18:45 ata-ST1000DM003-1CH162_S1DH17X6-part2 -> ../../sdb2 lrwxrwxrwx 1 root 0 9 Jul 12 18:45 ata-ST1000DM003-1CH162_Z1D81JHH -> ../../sda lrwxrwxrwx 1 root 0 10 Jul 12 18:45 ata-ST1000DM003-1CH162_Z1D81JHH-part1 -> ../../sda1 lrwxrwxrwx 1 root 0 10 Jul 12 18:45 ata-ST1000DM003-1CH162_Z1D81JHH-part2 -> ../../sda2 lrwxrwxrwx 1 root 0 9 Jul 12 18:45 usb-_Patriot_Memory_070826F26C1B6391-0:0 -> ../../sdc lrwxrwxrwx 1 root 0 10 Jul 12 18:45 usb-_Patriot_Memory_070826F26C1B6391-0:0-part1 -> ../../sdc1 lrwxrwxrwx 1 root 0 9 Jul 12 18:45 wwn-0x5000c50065676574 -> ../../sda lrwxrwxrwx 1 root 0 10 Jul 12 18:45 wwn-0x5000c50065676574-part1 -> ../../sda1 lrwxrwxrwx 1 root 0 10 Jul 12 18:45 wwn-0x5000c50065676574-part2 -> ../../sda2 lrwxrwxrwx 1 root 0 9 Jul 12 18:45 wwn-0x5000c5006d65acfc -> ../../sdb lrwxrwxrwx 1 root 0 10 Jul 12 18:45 wwn-0x5000c5006d65acfc-part1 -> ../../sdb1 lrwxrwxrwx 1 root 0 10 Jul 12 18:45 wwn-0x5000c5006d65acfc-part2 -> ../../sdb2
/dev/disk/by-path: total 0 lrwxrwxrwx 1 root 0 9 Jul 12 18:45 pci-0000:00:13.2-usb-0:4:1.0-scsi-0:0:0:0 -> ../../sdc lrwxrwxrwx 1 root 0 10 Jul 12 18:45 pci-0000:00:13.2-usb-0:4:1.0-scsi-0:0:0:0-part1 -> ../../sdc1
/dev/disk/by-uuid: total 0 lrwxrwxrwx 1 root 0 10 Jul 12 18:45 a9682244-3d0f-4469-ae80-5d6ef5fd148f -> ../../sdc1 + for _i in '/etc/conf.d/*.conf' + '[' -f /etc/conf.d/systemd.conf ']' + echo /etc/conf.d/systemd.conf /etc/conf.d/systemd.conf + cat /etc/conf.d/systemd.conf systemdutildir="/usr/lib/systemd" systemdsystemunitdir="/usr/lib/systemd/system" systemdsystemconfdir="/etc/systemd/system" + command -v lvm + lvm pvdisplay
Thanks a lot for your help,
David
On Sun, Jul 12, 2015 at 1:33 PM, dwoody5654 dwoody5654@gmail.com wrote:
I created a menu entry in the current F20 grub.cfg with menu_entry name of 'Remote Install'. So I boot to the netinstall from F20.
OK now I'm lost on the advantage of modifying the current bootloader to do this, rather than boot from either Fedora 21 netinstall media, or boot.fedoraproject.org media.
This has been working up until the current issue. The 2 pertinent lines are: linux /boot/vmlinuz-remote repo=hd:md126:/david/Fedora-Server-netinst-i386-21.iso noselinux ks.device=00:14:6c:55:f1:85 ks=hd:md126:/david/ks.cfg --noip6 ramdisk_size=8192 panic=30 initrd /boot/initrd-remote.img
I've never seen this kind of notation before, and I don't see a root=UUID= line so it seems like systemd is going to have no idea what volume to mount as root fs.
What are vmlinuz-remote and initrd-remote.img? set root='hd0,msdos1' is telling GRUB that those files are local, on the primary drive's first partition. So where did they come from?
A particular Fedora installer version expects a particular environment setup by a precreated initramfs which is on the netinstall.
I thought noselinux was deprecated a long time ago in favor of selinux=0 or enforcing=0, with the latter being preferred since it still labels files correctly.
The set root line is what I think I do not have set correctly. The /dev/root error comes up during the initial netinstall boot up.
The set root line in the grub.cfg tells GRUB where the kernel and initramfs are located. That's it. Obviously it found /boot/vmlinuz-remote and /boot/initrd-remote.img or the system wouldn't have gotten to dracut. So it did exactly what you asked it to do. I just don't know why you're booting the system this way instead of using install media written to a USB stick or pxeboot or something.
The file rdsosreport.txt is below.
It looks really short, I'm expecting a lot more data including journal entries and there are none.
Chris Murphy
-
Chris Murphy -
dcw -
John Wright