So I took backups, and copied files, and wrote new code, and converted old Django 1.2 code which worked in Django 1.4 up to the new standards of Django 1.6. Much of the site has been 404'ing for the last couple of days as I fix problems here and there. It's still work in progress, especially fixing the issues with URL compatibility - trying to make sure URLs that worked in the old site, in one Perl-based CGI system, work in the new site implemented in Django with a changed database structure.
Still, so far so good. My thanks once again to Daniel and Neill at Ace Hosting for their help and support.
Peter may be known to my readers, so I won't be otiose in describing him as a programmer with great experience who's worked in the Open Source community for decades. For the last couple of years he's been battling Leukaemia, a fight which has taken its toll - not only on him physically and on his work but also on his coding output. It's a telling point for all good coders to consider that he wrote tests on his good days - so that when he was feeling barely up to it but still wanted to do some coding he could write something that could be verified as correct.
I arrived while he was getting a blood transfusion at a local hospital, and we had spent a pleasurable hour talking about good coding practices, why people don't care about how things work any more, how fascinating things that work are (ever seen inside a triple lay-shaft synchronous mesh gearbox?), how to deal with frustration and bad times, how inventions often build on one another and analogies to the open source movement, and many other topics. Once done, we went back to his place where I cooked him some toasted sandwiches and we talked about fiction, the elements of a good mystery, what we do to plan for the future, how to fix the health care system (even though it's nowhere near as broken as, say, the USA), dealing with road accidents and fear, why you can never have too much bacon, what makes a good Linux Conference, and many other things.
Finally, we got around to talking about code. I wanted to ask him about a project I've talked about before - a new library for working with files that allows the application to insert, overwrite, and delete any amount of data anywhere in the file without having to read the entire file into memory, massage it, and write it back out again. Happily for me this turned out to be something that Peter had also given thought to, apropos of talking with Andrew Cowie about text editors (which was one of my many applications for such a system). He'd also independently worked out that such a system would also allow a fairly neat and comprehensive undo and versioning system, which was something I thought would be possible - although we differed on the implementation details, I felt like I was on the right track.
We discussed how such a system would minimise on-disk reads and writes, how it could offer transparent, randomly seekable, per-block compression, how to recover from partial file corruption, and what kind of API it should offer. Then Peter's son arrived and we talked a bit about his recently completed psychology degree, why psychologists are treated the same way that scientists and programmers are at parties (i.e. like a form of social death), and how useful it is to consider human beings as individual when trying to help them. Then it was time for my train back to Sydney and on to Canberra and home.
Computing is famous, or denigrated, as an industry full of introverts, who would rather hack on code than interact with humans. Yet many of us are extroverts who don't really enjoy this mould we are forced into. We want to talk with other people - especially about code! For an extrovert like myself, having a chance to spend time with someone knowledgeable, funny, human, and sympathetic is to see sun again after long days of rain. I'm fired up to continue work on something that I thought was only an idle, personal fantasy unwanted by others.
I can only hope it means as much to Peter as it does to me.
It's a token gesture, and I'd prefer something that actually causes a real change in the state of affairs. But hopefully the few people that visit my site will ask why its blacked out, and I'll tell them. Or they'll find out why for themselves. Or they'll know already.
Ultimately, the thing that worries me in all of this is that all the data collection, all the wire tapping and interception, all the bad cryptography and bastardised standards, all the spying and all the secrecy doesn't really improve our actual security. It hasn't found anything that normal detective work and normal policing and existing laws couldn't already deal with. It hasn't prevented any crimes, either against real people or against 'the state' or anything.
The 'baddies' are already adapting their methods and covering their tracks. There are far too many false positives, and much too much confirmation bias, to make the resulting 'intelligence' anything but a joke. The FBI already spends more money on covering up its mistakes - like its total waste of resources watching Brandon Mayfield - than it would if it had just asked him for an interview. Mean time they're missing the Boston Marathon bombers despite lots of evidence pointing to them. Then follows a lot of chest puffing and excuses and "we can't tell you the details, they're classified".
(Meanwhile, we have banks that are laundering money to supply to exactly the same terrorist organisations that get a slap-on-the-wrist fine and no jail time for anyone because they're "too big to fail". So not only did the NSA and all the security TLAs not find a massive source of funding for these organisations - something that's causing far more damage to USAdian society than 'terrorism' - but the entire rest of the government quietly brushed it under the carpet and pretended it didn't happen. Yeah, good one.)
Ultimately, all it's really about is perpetuating the existince of the security complex - mainly in the USA, but everywhere really. Its first imperative is to preserve itself, and it has all the means to do so. It has the secret courts and the secret laws to prevent legal challenge, and the arms and the blackmail material to prevent other attacks. And its paranoid level of secrecy and security makes it automatically treat any rein, any check on it, as a threat to its own existence - because, well, it would be.
So what REALLY scares me is that nothing we do will actually stop them at all. At this stage, it's basically impossible to even rein in the NSA's powers - and that'd be like taking a rabid tiger and smacking it on the nose to tell it to go away. To put in the high-level open oversight that lets the public see whether these agencies are actually doing anything useful with the vast quantities of money they control is a task that's beyond the realistic abilities of any government (to say nothing of the blackmail and subversive influence that any security agency can bring against anyone that wants to downsize them). Tackling the companies who run the prisons and supply the equipment and make a profit from all the unrest - that's just bordering on insane.
We've made the tiger, and we've fed the tiger because it said it would protect us, and we're on its backs because it's better than being in its jaws, and we've fed it more because we're afraid it might eat us, and it's only grown larger and hungrier. To be honest, I think the security complex will kill the world before climate change does.
This turned out to have unexpected complications: while liblzo supports the wide variety of compression methods all grouped together as "LZO", it does not actually created '.lzo' files. This is because '.lzo' files also have a special header, added checksums, and file contents lists a bit like a tar file. All of this is added within the 'lzop' program - there is no external library for reading or writing lzo files in the same way that zlib handles gz files.
Now, I see three options here:
LZO is a special case: it does a reasonable job of compression - not quite as much as standard gzip - but its memory requirements for compression can be miniscule and its decompression speed is very fast. It might work well for compression inside the file system, and is commonly used in consoles and embedded computers when reading compressed data. But for most common situations, even on mobile phones, I imagine gzip is still reasonably quick and produces smaller compressed output.
Now to put all the LZO work in a separate git branch and leave it as a warning to others.
Pretty much all food is expensive. They're just in the process of setting up their own small abbatoir, which will allow them to serve local meat at Australian food safety standards. Fish is caught locally (outside the protected areas, of course), but is variable - some times they have to serve frozen fish caught days or weeks ago. There are a few people gathering chicken eggs, available at the local co-op store.
Everything else is brought in via ship. It costs about $540 per cubic metre - much, much more if you need it to be refrigerated or frozen in transit. Ice creams typically cost $6 to $8, a half round of White Castello cheese costs $14.60, and we had the smallest roast chicken you'd ever seen for $20. Even self catering is reasonably expensive here. Likewise, all fuel, all cars, pretty much all building materials - all are shipped here from the mainland. Mail day is pretty spectacular.
I don't know what electricity costs on the island but it's main supply is a series of diesel generators. Wind, wave and solar power are being investigated but the impact on the views and possibly wildlife is considered a downside (although I'd argue that they need to think differently; bird deaths due to wind turbines are much lower than you'd think and I for one would love to see a couple of wind turbines on Transit Hill or in some of the valleys, providing good clean energy). All power lines are underground, which I think is great, and street lighting is kept to a minimum (partly to save power, partly to not interfere with the many bird species here). The other complication with renewable energy is that there's simply not enough base load and not enough distribution to mean that the variable power supply can easily be used. Wind is OK when you've got hundreds of turbines spread across a state, but not so good if they're all concentrated in a square kilometre area.
But the real reason you should pity Lord Howe Islanders is their internet connection.
There is no undersea fibre-optic cable running here. One was connected to Norfolk Island, 700km east, but they didn't connect Lord Howe Island (for some unknown reason). So all internet connections are via satellite. One of the two satellite companies servicing the island decided to stop service, and only took some of their existing customers back at higher cost and reduced rate of data. The other is not taking any new customers. The NBN satellites are already oversubscribed - so "satellite internet" for regions may already be bad - which means that Lord Howe Island has no option for new internet connections. There are only a few satellite uplinks to serve the entire population, so link congestion is high.
What does this mean? It means studying, getting email, and even getting basic information takes a lot longer. It's costly and unreliable. You could do great business on LHI - selling Kentia Palm seedlings (which used to be the main business on the island), for instance - except you can't do it using the internet and compete with other sites on the mainland. Keeping in touch with children - most go to boarding school on the mainland - is slow and some things like video calls are impossible. So many things we take for granted on the mainland, things that are possible with 3G connections and "just work" on ADSL, just do not work at all on the island. Bufferbloat is crippling here.
The islanders are already conversing with Malcolm Turnbull about capacity of the NBN satellites and getting better speed. But I can see how easily it's overlooked - the problems experienced by 300 people and their 330 or so guests can look small beside an electorate of 100,000 or so. The pity to me is that the internet is a great opportunity giver. People can run businesses, find help, and get opportunities to better themselves (almost) regardless of where they are. My trip to Lord Howe Island has really shown how much we can take for granted the availability of information that the internet brings.
Well, as luck would have it I recently bought several LiIon batteries at a good price, and thought I might as well have the working drill with a nice, working battery pack too. And I'd bought a nice Lithium Ion battery balancer/charger, so I can make sure the battery lasts a lot longer than the old one. So I made the new battery fit in the old pack:
First, I opened up the battery pack by undoing the screws in the base of the pack:
There were ten cells inside - NiMH and NiCd are 1.2V per cell, so that makes 12V. The pack contacts were attached to the top cell, which was sitting on its own plinth above the others. The cells were all connected by spot-welded tabs. I really don't care about the cells so I cut the tabs, but I kept the pack contacts as undamaged as possible. The white wires connect to a small temperature sensor, which is presumably used by the battery charger to work out when the battery is charged; the drill doesn't have a central contact there. You could remove it, since we're not going to use it, but there's no need to.
The new battery is going to sit 'forward' out of the case, I cut a hole for my replacement battery by marking the outline of the new pack against the side of the old case. I then used a small fretsaw to cut out the sides of the square, cutting through one of the old screw channels in the process.
I use "Tamiya" connectors, which are designed for relatively high DC current and provide good separation between both pins on both connectors. Jaycar sells them as 2-pin miniature Molex connectors; I support buying local. I started with the Tamiya charge cable for my battery charger and plugged the other connector shell into it. Then I could align the positive (red) and negative (black) cables and check the polarity against the charger. I then crimped and soldered the wires for the battery into the connector, so I had the battery connected to the charger. (My battery came with a Deanes connector, and the charger didn't have a Deanes connector cable, which is why I was putting a new connector on.)
Aside: if you have to change a battery's connector over, cut only one side first. Once that is safely sealed in its connector you can then do the other. Having two bare wires on a 14V 3AH battery capable of 25C (i.e. 75A) is a recipe for either welding something, killing the battery, or both. Be absolutely careful around these things - there is no off switch on them and accidents are expensive.
Then I repeated the same process for the pack contacts, starting by attaching a red wire to the positive contact, since the negative contact already had a black wire attached. The aim here is to make sure that the drill gets the right polarity from the battery, which itself has the right polarity and gender for the charger cable. I then cut two small slots in the top of the pack case to let the connector sit outside the case, with the retaining catch at the top. My first attempt put this underneath, and it was very difficult to undo the battery for recharging once it was plugged in.
The battery then plugs into the pack case, and the wires are just the right length to hold the battery in place.
Then the pack plugs into the drill as normal.
The one thing that had me worried with this conversion was the difference in voltages. Lithium ion cells can range from 3.2V to 4.2V and normally sit around 3.7V. The drill is designed for 12V; with four Lithium Ion cells in the battery, it ranges from 14.8V to 16.8V when fully charged. Would it damage the drill?
I tested it by connecting the battery to a separate set of thin wires, which I could then touch to the connector on the pack. I touched the battery to the pack, and no smoke escaped. I gingerly started the drill - it has a variable trigger for speed control - and it ran slowly with no smoke or other signs of obvious electric distress. I plugged the battery in and ran the drill - again, no problem. Finally, I put my largest bit in the drill, put a piece of hardwood in the vice, and went for it - the new battery handled it with ease. A cautious approach, perhaps, but it's always better to be safe than sorry.
So the result is that I now have a slightly ugly but much more powerful battery pack for the drill. It's also 3AH versus the 2AH of the original pack, so I get more life out of the pack. And I can swap the batteries over quite easily, and my charger can charge up to four batteries simultaneously, so I have something that will last a long time now.
I'm also writing this article for the ACT Woodcraft Guild, and I know that many of them will not want to buy a sophisticated remote control battery charger. Fortunately, there are many cheap four-cell all-in-one chargers at HobbyKing, such as their own 4S balance charger, or an iMAX 35W balance charger for under $10 that do the job well without lots of complicated options. These also run off the same 12V wall wart that runs the old pack charger.
Bringing new life to old devices is quite satisfying.
The other big generalisation is that this works purely on a cost amortisation calculation. I.e. that musicians are trying to cover costs, and therefore the cost of producing physical units and distributing them is the governing factor. Some musicians also look to make a living from their work, and that means setting a time period over which they hope to gain money from selling the product, an amount which they expect to live on, a number of units to sell, and so forth - all of which can vary widely and are complicated to fix. (Aside: this is why established artists push for extension to copyright - because theoretically they're extending the amount of time they gain from selling that product. This is a myth and a fairy story record labels tell them when they want them to support copyright extension.) It used to be that the cost of producing the units and distributing them were the major costs - see Courtney Love's calculation, for example - and therefore the label proposes to take that risk for the band (another fairy story); nowadays the distribution is free, and producing a new unit is cheap (in the case of digital distribution, it's totally free), so the ongoing costs of keeping the musicians alive and producing new music is the major cost for professionally produced music.
But I still think the big points made in the article is true: that the real cost of producing music - even music of reasonable quality - is coming down, that more music than ever is being produced and hence there's much more competition for listener's money, and that "hobby" artists who do it in their spare time and don't expect to make money out of their music (I'm one) drives the cost of actually getting music down too. So for professional musicians, who have sort of expected to make money out of music because their heroes of the previous generations did (due, as David points out, to a quirk in history that made the twentieth century great for this kind of oligopoly), it's a rude awakening to find out that people don't care about your twenty years in the industry or your great study of the art form, they care about listening to a catchy tune that's easy to get.
I also like the point that musicians are also inveterate software copiers. It's one reason I use LMMS and free plugins - because free, quality software does exist. I find it intensely hypocritical that professional musicians can criticise people for copying their music, when they may well have not paid a cent for all the proprietary software they use to produce it.
But to me this is really just about getting in touch with your audience. Companies like Magnatune exist to help quality artists find an audience by putting them in touch with an existing large subscriber base who wants new music. Deathmøle's insane success on Kickstarter shows that someone with an established audience can make it really big without having to sell their soul to big record labels. And Jeph himself is a great example of the way things work in the modern world, since Deathmøle is his side project - his main one is Questionable Content, which he also went into without having existing funding or requiring a big backer to grant him some money and take his rights in exchange. As Tim O'Reilly says, obscurity is a far greater threat to authors and creative artists than piracy; it doesn't matter if you're signed to the best record label there is, if they haven't actually publicised your work you might as well have not signed up at all. And, fortunately, these days we have this wonderful thing called the internet which allows artists to be directly in touch with their fans rather than having to hope that the record label will do the right thing by you and not, say, ignore you while promoting another band.
I wish David had made his point without the broad generalisations - I think it stands well without them.
The ever-thoughtful Charlie Stross has written an article about the problems facing the NSA. There's not going to be just one Edward Snowden or Bradley Manning, there's going to be heaps of them - because the Three Letter Acronym security departments are busy getting rid of all of the permanent employees who felt loyal to them and replacing them with contractors who have no more loyalty to them than the security department's loyalty to the contractor.
Now, I personally believe in being loyal to my employer. I (of course) honour the various clauses in my contract that say they get to own all my work for them, and that I won't sell or leak their secrets, and that I won't work for someone else without telling them. I believe in being loyal to the customers I work for and the people I work with. I believe that I am more valuable to an employer the longer I work there because I know the intricacies of the job better and am better at solving problems by recognising them and their underlying causes. These are things that a new employee will always struggle with.
But I believe that the big problem with employers these days is this pernicious idea that their workforce is interchangeable, not to be trusted, and best used by screwing them for as much work as you can get out of them and then throwing them away. It's an "Atlas Shrugged" mindset that believes that somehow the people at the top are being held back by the people at the bottom, and that therefore workers don't deserve any of the benefits of being at the top. It's also contributed to by the idea that companies poach people - especially "rock star" workers and people high up the ladder; the idea that those people (and their loyalty) can be bought just for their experience and to (somehow) change their environment just by being in it.
The "glory days" of jobs for life that Charlie talks about in his essay are really the times before the MBA school of management came into being; when people managed companies because they'd worked their way to the top. Those people knew the business intimately, they'd sweated over it for decades, they knew the people - and the employees knew them. There was much more of a feeling of trust in those organisations, because it was about personal relationships more than work relationships or "rightsizing" or "mission statements". Walt Disney was famous for remembering every person in the 700-strong Disney workforce. These days, one gets the impression that the management of some companies consider it a burden to even associate with the people more than a step down the org chart.
At the moment all we're really seeing, IMO, is the 'tit for tat' nature of the Prisoners Dilemma being played out in corporate workforces. If you want to find the point at which employers started cheating on their workforces, then you have to keep on going back - past the 1980s anti-union laws and workplace deregulation, past the 1880s and the weavers and miners unions, past the 1780s and the clearances... in fact, just keep going: its feudal lords demanding tithes, and high priests demanding donations, and kings demanding tributes. The Greeks famously invented Democracy, but even then slaves, women, and other "not our sort" people couldn't actually vote. The process of cheating on the people beneath you for your own gain has a long history - far longer, I would argue, than the history of the workers rebelling and demanding their own rights.
So now the workforce is no longer loyal to their employer, and we see the mistrust and second-guessing that usually accompanies standard Prisoners Dilemma situations. I think the two are evenly matched - the employer might seem to hold the power (because they write the contract the employeee must sign without change) but the employees are many, and their methods of working around the employer's restrictions and exploiting the employer's weaknesses are many and subtle. The employee has much more mobility than the employer, and while there are usually non-competition restrictions in the contract the number of times I've heard people subtly, and not so subtly, ignoring these (for example, sales people poaching client lists) makes it difficult for the employer to fight all those battles.
Overall, it's a pity, because I think a situation where employer and employee trust eachother and work together is much better than one where each is subtly trying to screw the other. Once you see it as a contest, though, it's all downhill from there. Many organisations try to rebuild trust, but the "team building exercise" is such a cliche for uncaring management that it's boring to repeat it. If you're trying to rebuild trust, but not fundamentally changing the management style and not addressing the needs and issues of the workers, then it's really just an exercise in paying some management consultant to take your money and laugh at you.
I currently work at a company which does have, at least in our Canberra offices, a lot of respect for its workers. It's easy to imagine being paid well for being a "subject matter expert" rather than having to go into management to keep climbing the pay ladder. We have regular functions every fortnight or so where you can speak to just about anyone - not 'town hall' meetings which in my experience are still basically management telling the workers the way it's going to be. And I think that there are many examples of companies that are doing the right things by their workers and seeing a lot of benefits - it's easy to cynically see what Google get out of paying for their sysadmins to have good internet connections, but the sysadmins get a decent deal out of it too, and the trust and understanding that goes with it is not easily bought.
So I do think there's hope. But I think we have to see a profound shift in the employers and their attitudes to staff before that changes. For a start, weed out the psychopaths and bullies in management before you complain about theft of office supplies. Promote people from within rather than always hiring top management from outside. Stop trying to win my trust with company slogans and mission statements, and start actually listening to me when I tell you about the opportunity that I can see right in front of you. Stop treating companies like feudal families, with their fiefdoms and strict hierarchy, and start treating us all like citizens.
In your article for the Sydney Morning Herald on the 31st of July 2013, you say fair use is "theft" in all but name.
And on your blog you have mentioned Bruce Sterling's piece "The Ecuadorian Library". In fact, you've quoted directly from it.
So, is that fair use? Or are you going to hand yourself in for copyright theft now?
Now, you theoretically don't make any money from quoting Bruce, so maybe you think that because it's not a commercial use that therefore you're not "stealing". But I don't think you can have it both ways.
If we follow your argument - that any use that has some kind of commercial gain is, in fact, theft - then it simply becomes a question of what "commercial gain" is. And that's where lawyers come in.
Because you've obviously gained from referencing a quotation from Shakespeare in your article title. You're probably gained by mentioning songs or stories in your books - also copyrighted. And where does that end? Should you be paying the people who wrote the thesaurus every time you look up a synonym? Should you be paying the authors whose work you cribbed on the Russo-Japanese war? Should you be paying Bruce Sterling a proportion of your royalties, as he's clearly influenced your thinking?
You're also presenting a slippery slope that cannot help anyone. An academic quotes your book? Clearly they must pay! Someone satirises it? Clearly they must pay! A student quotes from it? Well, clearly they must pay in proportion to how much they quoted - after all, some people might read your book and not use a thing from it, and others quote entire sections! Someone mentions it on a radio show? They should pay for the privilege! Someone sells your book second-hand? Well, obviously you should get a cut too!
You're also a successful author, having published eight books and translated more. So it's kind of convenient for you to say, now, that you should be paid more for all that work. It doesn't help the new author, struggling to make a living and trying to read and learn from everything they can.
And, let's face it, the spectre of some dread international conglomerate ripping off your work and not giving you any money for it is kind of the wrong way around, isn't it? After all, you've basically been published by them - big printing companies who control distribution, decide who is going to be released where and when, and decide the royalties they will offer you and how they'll pay. They don't need to steal other people's work, they've got authors begging to be published sending them manuscripts all the time. Pretending that you're threatened by hungry companies desperate to rip your work off, and ignoring the one that's already only paying you trivial amounts compared to their own salaries and bonuses, is not a very good distraction.
I have nothing against you personally. I only think that your logic in defending a system that offers a pittance to the people who actually write the words we read, and in turn demand that no-one use your work without paying for it, while at the same time using other people's work without paying for it, seems to be mixed up.
All posts licensed under the CC-BY-NC license. Author Paul Wayper.