Too Busy For Words - The PaulWay Weblog
09 10 2008

Thu, 09 Oct 2008

Look Out Eddie Van Halen
Back in the days when Icehouse was in, Crowded House was big and I was getting deeply hooked into Yello, I had a Roland Juno 6. It was my Dad's, but I played it a fair bit. One of the leaders in the analog synthesis days between full-on knob-for-everything setups like the Moog and the start of MIDI and digital control, it is still legendary for producing huge bass lines and stunning synth leads. Then our house burned down and took the Juno with it, and though I often thought of getting another synthesizer I never did.

Until now. It started with playing the piano at friends and relatives houses; then Kate suggested I could accompany her violin playing. As I got more into LMMS I started realising that having a keyboard to record lines and work out notes and melodies on was going to be very useful. So I did some research and found the Roland Juno G, which sat between the full-on knob tweaking of Nords and Moogs (all digital, now, of course, but still faithfully emulating the analogue sound synthesis process), the 'play the demo song' integrated-speaker cheap synthesizer market, and the 'it has 4096 patches, all pianos' professional keyboard. This may sound like a no man's land, but the market segment is for people who want a range of instruments, the ability to fiddle with how they sound, and don't need heavy 'piano-action' keys. Unfortunately, they don't make the Juno G anymore.

Fortunately, it's successor is the Juno Stage, which is basically version 2 - all the features of the G but without the confusion between it and the Juno D. You get knobs to control attack and release, low and high frequency rolloff, and cutoff and resonance of the filter - which you can twiddle on the fly. It comes with 1024 different patches, a variety of modes including split keyboard (SuperSaw on the left and piano on the right is a favourite) and lots of nice features that I haven't truly discovered yet. So I bought it, brought it home, and started practicing again.

Gradually my fingers are warming up again, playing scales and old tunes I used to know. But what has amazed me is the amount of pure inspiration I'm getting from the sounds. A new patch will make me start writing new melodies out of thin air, and when I find that some presets consist of an arpeggio and drum rhythm on the left hand, new mystical tunes will flow out of my right hand and almost amaze me in the process. That and the joy of working out the chord progressions (the title of this post is a nod to the classic synth line of 'Jump' by Van Halen - I hit the first two chords (C, F in my playing) and then had to figure out the next (B) later by experimentation - I don't know what the actual song used but it's easiest to play on G, C, and F) for songs I remember. Playing the Doctor Who theme or the theme to "Axel F" or "Fletch" (yay Harold Faltermeyer) is always a blast, and it all came right back to me.

So I'm now doing regular practice of my own devising, before I seek out someone to teach me how to play more. I'll report how I go plugging it into the computer (yay USB MIDI interface) in another post.

posted at: 13:18 | path: /personal | permanent link to this entry

Perl threaded database query processing
In my work I've recently had to implement several pieces of code which follow this basic pattern:

  1. Retrieve data from the database
  2. Process data
  3. Store data somewhere.
Because of Perl DBI's habit (on the systems I've used) of grabbing all the data from the database into memory before actually giving it to the caller, and because that data can often get large enough to get my process swapping or killed, what this usually turns into is:

  1. Get a list of 'grouping' items (e.g. days, months, IP addresses, etc.)
  2. For each item in that group:
    1. Retrieve data from the database for that item.
    2. Process data
    3. Store data somewhere.
This runs into an unfortunate problem when the database server you're talking to takes a noticeable time to process your query - the whole thing slows down hugely. A typical slowdown I've seen is in the order of 500% - and both the database and the client processors are mostly idle during that time, as each query has to be individually fetched, processed, dumped back to the client, and then processed. It suffers the same problem if the time to process each group of data is significant - by the time you've got back to fetching the next group, the database has gone off and done other things and needs to get its disk heads back in the right place for your data.

These days we have processors capable of doing multiple things at the same time, and so it would be nice if the client could be processing rows at the same time as it's also requesting more data from the database. This is where Perl's threads and Thread::Queue libraries come in. It seems to me to be a generalisable task, so I'm sharing my first attempt at doing this in a generalisable way here. My main subroutine is:

######################################
sub Thread_Process {
######################################
    # We take one query which returns a list of items, a query which
    # returns other rows based on each of those items, and a function
    # which processes those rows.  We then run the processor function
    # in parallel to the fetching process to utilise the connection
    # to the database and keep the local processor active.
    # Requirements:
    # The item query must be executed and ready to return rows.  It
    #   can return any number of fields.
    # The rows query must be ready to be executed, and will be
    #   executed with the row_args_ref and then the items from each
    #   row in the item query in turn (as arrays).
    # The function takes as its last argument the Thread::Queue object
    #   that data will be passed through.  It must know exactly how
    #   many items it will take from each row, and that should match
    #   the number of items returned in the query.  For reasons as yet
    #   unclear, we can't pass references of any kind on the queue,
    #   so we pass the fields in each row as single scalars.  Any
    #   arguments that it needs should be given in fn_args_ref.  It
    #   should exit on receiving an undef.
    my ($items_qry, $rows_qry, $row_args_ref, $fn_ref, $fn_args_aref) = @_;
    my ($items_aref) = $items_qry->fetchall_arrayref;
    unless (ref $items_aref eq 'ARRAY') {
        carp "Warning: got no rows from item query\n";
        return 0;
    }
    
    my $queue = Thread::Queue->new();
    my $thread = threads->create($fn_ref, @$fn_args_aref, $queue);
    foreach my $item_aref (@$items_aref) {
        $rows_qry->execute(@$row_args_ref, @$item_aref);
        while (my $aref = $rows_qry->fetchrow_arrayref) {
            $queue->enqueue(@$aref);
        }
    }
    $queue->enqueue(undef);
    $thread->join();
    return scalar @$items_aref;
}
A sample caller function would be:

sub Send_mail_to_everyone {
    my ($mail_handler, $template, $start_date, $end_date);
    my $servers_qry = $dbh->prepare(
        'select distinct mail_server from addresses'
       .' where birth_date between ? and ? and current = true',
    );
    my $args_ref = [$start_date, $end_date];
    $servers->execute(@$args_ref);
    my $email_qry = $dbh->prepare(
        'select user_name, server, full_name, birth_date'
       .' from addresses'
       .' where birth_date between ? and ? and server = ?'
       .' and current = true'
    );
    my $mailer_sub = sub {
        my ($queue) = @_;
        while (defined my $user_name = $queue->dequeue) {
            my $server = $queue->dequeue;
            my $full_name = $queue->dequeue;
            my $birth_date = $queue->dequeue;
            my $email_body = sprintf $template
                , $username, $server, $full_name, $birth_date;
            $mail_handler->send("$user_name@$server", body => $email_body);
        }
    };
    # Here most of the work gets done.
    Thread_Process($servers_qry, $email_qry, $args_ref, $mailer_sub, []);
}
Of course, this is somewhat of a contrived example, and gives you little in the way of feedback or error handling. But it's an example of how to use the Thread_Process subroutine. The mailer subroutine gets its $mail_hander and $template from being within the Send_mail_to_everyone routine.

There are two problems I've discovered so far. The first is that trying to do any kind of database operation within the subroutine doesn't work, because the database handle needs to be cloned. On the systems I've tested this on, unfortunately, $dbh->clone seems to be a no-op and the DBI engine complains that a different thread is using the database handle. I've tried passing $dbh->clone to the handler function, and doing the clone inside the handler function, but they change nothing.

More annoying is the fact that the memory used by the process continues to rise even if the number of outstanding rows is constant or dropping. I haven't traced this down, and haven't really the time now, but it seems to be related to the Thread::Queue object - I've tested variations of my handler routine that reuse existing memory rather than doing undef @array and push @array, $data in the handler, and this changes little.

What I don't know yet is whether either to package this up as a Perl module and start being a maintainer or whether it's too trivial or not generalised enough to be useful for anyone but me.

posted at: 13:11 | path: /tech/perl | permanent link to this entry

Anonymous Earpieces
I bought a "Stereo Headset Kit" at Landmark Computers in Braddon the other day. This consists of a pair of wireless headphones, a bluetooth dongle that functions as a stereo encoder, a CD, a USB cable, a charger and some basic instructions. It's almost impossible to find out exactly who made this - there's no actual brand on the box or devices and the only identifiable branding is of the USB connection software. The software does have a Linux version available but (almost inevitably) it isn't supplied - the disk is really Windows-only.

The dongle has a switch that allows it to function in either USB or Audio mode. In USB mode it is a fully-featured USB Bluetooth dongle - plugging it into my Fedora 9 install allowed me to see all nearby Bluetooth phones, computers and the headset. I haven't tried to see if I can get it to function as a Bluetooth audio device, but PulseAudio does apparently provide this. In Audio mode, it encodes input from the headphone jack on the dongle and sends it to the paired headset. This allows you to use other devices such as computers, phones and music players that don't have Bluetooth capability.

The headset supports the A2DP profile, which basically supports (reasonable quality) stereo audio over Bluetooth. The quality and the stereo separation are quite good and, although it might not be up to full studio monitoring quality, it is easily capable of delivering good quality audio for everyday use. It also supports the headset profile for phones that don't have A2DP capability, but the headphones don't have a microphone so you can't use it as a full headset.

The headphones are comfortable even after a couple of hours of use. They sit around the back of the head, and even for people who have a large skull (such as myself) they don't press in uncomfortably. They can also fold to be flat so they can easily fit in a pocket when not in use. The right speaker has a volume up/down control, a skip forward/back control (for phones that support such control) and a main button that can be used to turn the unit off and on and put it in pairing mode.

These are not a cheap device, at around $110. However, they are cheaper than many of the brand-name devices and are more comfortable than the BlueAnt X5s that I tried a while back. The larger speaker gives them a better bass response than smaller earphones and the lack of cord prevents all sorts of tangles and trip-ups. As someone who seems prone to turning away from the computer and pulling the earphones out of my ears by accident (and force), that's a good thing.

The major downside so far has been that, while they give perfectly good stereo audio between the encoder dongle and the headphones, the other two devices I've tried both have bizarre and annoying behaviour that makes them nearly unusuable. My phone, a Nokia 5310 XpressMusic easily capable of A2DP, will drop them down from A2DP to Headset protocol, detectable as a flat mono signal, and then eventually (in three tests) drop them altogether, often getting wedged on the song being played back (a different song each time which has played normally otherwise, so that wasn't the problem). This wedges the headphones too, requiring a little press of the hidden reset button with a handy bent paperclip. Don't have one to hand? Too bad. (Note to young players - the little hole beneath the reset button is actually to seal the rubber protector in place, and should not be poked into unless you want to put a damaging hole in the speaker diaphragm. And you don't.)

The computer was even worse. I got the laptop bluetooth working with the instructions at http://fedoraforum.org/forum/showthread.php?t=190468 and it all came through nicely. For two minutes. Then it went to Insanely Loud Mode. Once I'd dialled back all the settings in PulseAudio (which, for reasons as yet unclear, muted the audio completely at 60% main volume or 40% RhythmBox volume) it was listenable, although you could tell there was hard clipping going on somewhere before the volume reduction stage (audible as a 'crackliness' to the sound). Then, two minutes later, it dropped back to completely inaudible, and only by turning all the volumes up again did anything come out. Two minutes later it cycled back to insanely loud and kept doing this as long as I was prepared to put up with it. Adjusting the volume on the headset seemed to do little, although when I later connected it to my phone for a second test the volume on the headset was turned up very loud, and changing the volume on the headset altered the volume on the phone. So I assume that something in PulseAudio was 'helpfully' adjusting the volume, for reasons as yet unclear.

I haven't tested it with anything else that outputs A2DP via Bluetooth, so I haven't any other benchmarks to work against. But so far this is a device that works perfectly with its own adapter and appallingly with everything else; not a trait that endears it to me.

posted at: 13:11 | path: /tech | permanent link to this entry


All posts licensed under the CC-BY-NC license. Author Paul Wayper.