cwRsync, Windows 7 and UNIX Targets

Lately whenever I've had to rsync anything to my E O N-based NAS from my Windows 7 machine, I've had permissions issues on the NAS.  Specifically, any sub-directories I sync over are created with ridiculous permissions e.g. 0500 or something odd.  No files are able to be transferred until I manually login to the NAS and run a something similar to:

find -type d -exec chmod 0755 {} \;

That got annoying very quickly.

I came across the rsync --no-perms flag which alleviated the problem to some extent in that the directories were at least writable during the initial transfer but the permissions still had to be resolved before transferring anything else over.

I'd gotten used to the whole issue but then came across this post and self-researched answer by karikas that talked about exactly my situation.  Turns out it's a known issue of cygwin (which is what the cwRsync application is using) and Windows 7.  The solution is to set the following environment variable in Windows 7:

set CYGWIN=nontsec

Building a Diskless MythTV Frontend with Mythbuntu 10.04 - Lucid Lynx - Part 1

I just upgraded to Mythbuntu 10.04 which was released just a few days back on April 29th.  Since I was already running Mythbuntu 9.10 on my MythTV backend server I was able to upgrade it to Lucid using the well documented and fairly simple steps on the ubuntu.com site:

$ sudo apt-get install update-manager-core

$ sudo do-release-upgrade

After upgrading the backend server I decided to finally figure out why my diskless frontend server wasn't working.  After a day of investigating it turned out to be a faulty stick of memory.  With that replaced, I needed to configure a diskless frontend on the MythTV backend server so I could use PXE to boot into a frontend without having to worry about anything else.  Since Mythbuntu hasn't had a graphical control panel to create a diskless frontend since Karmic, this blog post documents everything I had to do to get my diskless frontend up and running.  Some of the setup already existed but I'm documenting it here for future use.

Requirements for PXE Booting

PXE is a way for computers to boot up using resources found on the network.  On some computers you have to press F12 or some other key to get them to PXE boot but on all computers that have network cards that support PXE you should be able to set the first boot option in the BIOS to something like "Network" to allow your computer to PXE boot.

During a PXE boot a network card will perform the following tasks without any intervention from any OS stored anywhere on the computer:

1. automatically obtain an IP address,
2. automatically obtain a boot kernel and initial ramdisk, and
3. boot up the computer using that kernel and initial ramdisk 

After booting up the kernel using the initial ramdisk in step 3, the OS can either use a built-in hard drive or the computer's RAM as a root disk or it can NFS mount a volume from a remote server and use it as a root disk.  The NFS scenario is most common for computers that PXE boot and is the one I use.

For a computer to be able to automatically obtain an IP address you need a DHCP server in your network.  However, this DHCP server has to be configured so it can inform the PXE booting computer where the kernel and initial ramdisk files are located.  The kernel and initial ramdisk files are hosted on a TFTP server.  Finally the NFS mounted root disk needs an NFS server which will export the directory of files that will become the root directory of a diskless MythTV frontend.

In my home network I have a separate DHCP server hosted on a Linux machine that used to be my main Linux server.  Now it just serves DHCP until I can move that functionality into my Cisco router.  The other two pieces of the puzzle i.e. the TFTP server and the NFS server are handled by the Mythbuntu backend server and are closely tied to each other.  However, there is no reason why they can't be split up onto their own servers.

The next post will focus on the steps needed to build a diskless image using the Linux Terminal Server Project's utilities.  The post after that will focus on configuring the individual servers needed to put it all together.  These three posts should serve as a more or less complete guide to setting up a diskless Mythbuntu frontend.

Spaces in /etc/fstab

I needed to mount a Samba share from my windows gaming / media center PC to my MythTV backend so I could navigate to all my videos from a single location instead of having to worry about which server they were on.  To that end I shared out the Videos folder from my account on the Windows PC.  Since my user on the Windows PC is "Shahbaz Javeed" that presented a problem when trying to auto-mount it using fstab on my MythTV host - spaces aren't allowed in any field of /etc/fstab.  Any spaces are considered field delimiters.  The solution is to escape all spaces with \040.  My /etc/fstab entry now looks like so:

//frey/Users/Shahbaz\040Javeed/Videos   /var/lib/mythtv/videos/frey cifs     guest,ro       0       0

This works swimmingly!

Accessing the KDE Wallet from the Cmdline

I needed to write a script that would contact my Exchange server at work via IMAP and list all the messages in the Calendar folder. The idea was to see if it was possible to perform a one-way sync from the Exchange server to a specific calendar in Kontact. I was going to embed my IMAP password in the script - security hole, I know - but we have a password policy that requires the password to change every 30 days. Since I didn't want to edit the script every month I decided to see if it was possible to use the password for kmail that's already stored in my KDE wallet.

I didn't have any luck finding a perl interface to the KDE wallet. However, thanks to the good folks at #kde on freenode, I found that the KDE wallet - and lots of other applications as well - expose their interfaces over D-Bus. This was the first time I'd dealt with D-Bus so it took some getting used to but I figured out how to read my kmail password from my KDE wallet. KDE comes with the handy qdbus program that allows command-line testing of the D-Bus interface.

What follows are step-by-step instructions on how to use qdbus to open your KDE wallet and read your kmail password. I'll incorporate this into my one-way sync experiment using the Net::DBus perl module but I wanted to put this out there in case someone else was looking at that.

Introduction to qdbus
A quick intro to qdbus before we get started so you can explore other options instead of just kwalletd:

The following command shows all applications exposing a DBus interface:

$ qdbus

:1.50
org.gtk.vfs.Daemon
:1.51
:1.52
:1.54
org.kde.kwalletd
:1.56
org.kde.printer-applet-3206
:1.57
net.update-notifier-kde-3203
:1.58

The numbers and strings refer to applications, however since most applications expose a recognizable string it's common to use just the strings and ignore the numbers.

The following command lists all the DBus paths exposed by the kwalletd application:

$ qdbus org.kde.kwalletd
/
/MainApplication
/modules
/modules/kwalletd

The /MainApplication path is mainly used when you want to interact with the application itself and you'll find many applications that expose a /MainApplication path. I haven't explored this much but it looks like it should be interesting.

With that introduction to qdbus you should have enough to explore further on your own.

Getting a Password from a KDE Wallet
The following steps will open your default KDE wallet and get your kmail password. Each step will have an explanation, the command issued and the output of that command.

We will use the /modules/kwalletd path in the DBus interface for org.kde.kwalletd for all our password-getting needs. You can get a list of all the methods and signals exposed in the /modules/kwalletd path by using the following command:

$ qdbus org.kde.kwalletd /modules/kwalletd

method bool org.kde.KWallet.isOpen(QString wallet)
method bool org.kde.KWallet.isOpen(int handle)
method bool org.kde.KWallet.keyDoesNotExist(QString wallet, QString folder, QString key)
method QString org.kde.KWallet.localWallet()
method QString org.kde.KWallet.networkWallet()
method int org.kde.KWallet.open(QString wallet, qlonglong wId, QString appid)
method int org.kde.KWallet.openAsync(QString wallet, qlonglong wId, QString appid, bool handleSession)
method int org.kde.KWallet.openPath(QString path, qlonglong wId, QString appid)

Formatting's a bit messed up going forward. Not sure why :-(

However, let's get to work obtaining the password. First, we will open the default KDE wallet - called kdewallet. We will call the org.kde.KWallet.open method which expects a wallet name string, what appears to be a wallet id (similar to a file handle it seems) and finally an application id string. We will use "kdewallet" as the wallet name since that's the name of the default wallet in KDE. We don't know the value of the wallet id so we'll just specify 0. The application id is interesting because KDE wallet prompts the currently logged in user with the application id of any applications that call the org.kde.KWallet.open method which we're abbreviating to just open since that uniquely identifies it in the method list for the /modules/kwalletd path. Specifying a meaningful id here goes a long way to helping the user click on "Allow", "Allow Once" or "Allow Never". With all that in mind, let's use the following command:

$ qdbus org.kde.kwalletd /modules/kwalletd org.kde.KWallet.open kdewallet 0 "KOrganizer-Exchange 1-way Sync"

470467109

This results in popping up a dialog box like so:


For the purpose of these experiments, I chose "Allow Once". Once I've allowed it, the qdbus call returns a wallet id - similar to a file handle - that we'll use in all our other method calls. I did see that if I waited too long the dialog box remains visible but qdbus times out. However, the next time you make the org.kde.KWallet.open call it returns a valid wallet id without prompting which means the permission grant is persistent. I'll have to deal with the timeouts in my perl code somehow. The next step is to see what's stored in my wallet. This isn't strictly necessary if you already know what you want but serves to walk through my own discovery process. Notice I'm passing in the newly given wallet id as well as the full application id as I sent earlier.

$ qdbus org.kde.kwalletd /modules/kwalletd folderList 470467109 "KOrganizer-Exchange 1-way Sync"

AdobeAIR
Amarok
Form Data
Network Management Passwords
bilbo
kblogger
kmail
mailtransports

Let's list the contents of the kmail folder:

$ qdbus org.kde.kwalletd /modules/kwalletd entryList 470467109 kmail "KOrganizer-Exchange 1-way Sync"

account-242017858
account-990222852

I know the account-242017858 account is the one I need the password from because the other account is older. So let's see how to retrieve that password:

$ qdbus org.kde.kwalletd /modules/kwalletd readPasswordList 470467109 kmail account-242017858 "KOrganizer-Exchange 1-way Sync"

account-242017858: [the password here]

There you go folks! That's all it takes. Please let me know if you found this helpful.

Determining 64-bitness of your CPU

It looks like /proc/cpuinfo isn't the only way to find out whether your CPU is 64bit capable. In an effort to determine the most reliable way to find out this information I came across this page. A quick summary:

If you see any output from the following command, you're running a 64-bit capable CPU.

grep ^flags /proc/cpuinfo | grep ' lm '

The following command, if it exists on your system, will tell you the width of your physical and logical CPUs:

lshw -C cpu | grep width

The lshw command is available natively on my Ubuntu 9.04 system and is available from rpmforge.org for RHEL5 and CentOS5 systems.

Partially extracting a tarball

I've never had to extract just a small part of a tarball before so it took a little bit of digging to determine how to do that. It's simple with other archivers e.g. zip, rar et al but tar requires just a couple of extra options to make it happen.

Let's say you have a gzipped tarball that contains all your logs from /var/log with fully qualified paths. One thing to remember is that tar will always remove any leading / from all paths it processes. This means if you ran the tar command as follows:

tar zcf /tmp/logs.tar.gz /var/log

you will have a gzipped tarball containing files such as:

var/log/httpd/access.log
var/log/httpd/error.log
var/log/messages
var/log/dmesg

With this in mind, if you want to extract just your web server log files located in var/log/httpd, you can use the following commandline:

tar zxf /tmp/logs.tar.gz --wildcards 'var/log/httpd/*'

The quotes are necessary so the wildcard doesn't get extrapolated. That command will result in the var/log/httpd directory structure being created in your current dir and you will find that directory populated with the access.log and error.log files.

Implementing a Web Downloader in bash

While looking for a pure bash implementation of wget or curl I came across this post at LinuxQuestions.org. I haven't tried it yet but I'm trying to figure out if there's a more system-agnostic way of doing the same thing. Keen readers will notice the script uses the /dev/tcp/hostname/port "file" to make a connection to a remote host. Based on what I know that's a Solaris-specific way of connecting to a remote host. I'm thinking of something using telnet and the like that can download binary files as well. I have a couple of ideas and I'll update once I've either found a solution or given up and switched to perl.

The Magic SysRq Key - Commandline Edition

So earlier today I was stuck trying to bounce a remote host that had a Software RAID volume. It turns out for some reason (I still suspect a faulty volume in the RAID1 volume) nothing was able to write to the mounted RAID volume. All processes kept getting hung and drove the load up to 25. Responsiveness was still good since this load was actually due to I/O waiting processes that kept getting some CPU time and not CPU-intensive processes trying to share a couple of processors.

In any case, after manually failing the partition that was causing problems with the RAID volume I still wasn't able to bounce the box since the RAID volume won't stop or sync and actually halted the shutdown process. I thought about the magic SysRq but realized I was ssh'ed in and that won't work. For some reason I didn't think of searching for a way too invoke the magic SysRq processes from the commandline. I looked for a way to force a shutdown and found this blog. It worked like a charm!

Intriguingly enough, when the host came back up the RAID volume remounted and both partitions that formed the RAID volume were fine as reported by mdadm. What's more all seemed well with the mounted volume and I was able to read/write to it just fine. I'll be keeping an eye on that for a bit.

KMail Integration with Microsoft Exchange

I found this interesting post on the kde-pim mailing list which is very to-the-point and informative on how to connect KMail to Exchange. You do need IMAP enabled on the Exchange server but once you get over that hurdle - and believe me, in a pure Microsoft shop it's a big hurdle - you're good to go.

I haven't gotten Calendar integration working just yet but I haven't put a lot of time into it yet. This page on the KDE Wiki explains how to use webdav to perform one-way syncs against an Exchange server using a plugin for KOrganizer.

Edit: Even brought it to my attention that the KDE Wiki link is no longer working.  It seems the KDE wiki has changed structure since I was last on it.  I found Jason Kasper's blog, however, where he's talking about a ruby-based solution to the 1-way sync between an Exchange OWA server and KOrganizer.