operator

Tomato on a WRT54G limited to 25 or so Mbps with QoS - use a faster router for higher speeds

2011-07-24T23:50:00.000-07:00

This is a very specific post, if you couldn't already tell by the title - just a tidbit for anyone else curious and trying to make all the junk surrounding them in the world work for them instead of frustrate them.

If you have an internet connection of 25Mbps or less down, and you are using the wonderful tomato firmware on the works-pretty-good linksys/cisco WRT54G, you've been having a great time, things work wonderfully and they will continue to work wonderfully.

However, if you like me use Comcast high-speed internet and your area gets the infrastructure upgrade from DOCSIS 2 to DOCSIS 3 you'll find that you have 50Mbps of bandwidth available, and you might opt for that plan.

At that point you'll realize that tomato running QoS on a WRT54G can't keep up with more than 25Mbps so you won't get all the speed you want.

Never fear though, the works-pretty-well and slightly-newer WRT310N can handle it, and the almost-as-pretty but just-as-functional DD-WRT firmware does the job nicely if you follow the instructions well.

And just for another random tidbit from my life of trying to make things work today, if for any reason you attempt to use a Thule Rapid rail / aero cross bar with their Prologue bike rack, you'll need four more grade 8 1/4" washers per bike rack to get the bolts secure on the aero bar, but the instructions aren't aware of that and they don't give you the extra washers. Great rack, bad instructions, bad testing, Thule. Shame!

Cheers

Open links from external applications in a new window in Google Chrome on Mac OS X

2011-06-26T18:21:00.001-07:00

I love tabbed browsing. I switched to Firefox way back in the day partially because Firefox allowed tabbed browsing. I recently switched to Google Chrome, which also has tabbed browsing.

I also love virtual desktops. I switched to Linux on the desktop way back in the day partially because it supported virtual desktops. I currently use Mac OS X which has "Spaces", so I continue to have virtual desktops.

However, when you combine the two, you potentially have a problem. If you have a web browser open, but it is not on your current virtual desktop, and you click on a link in a different application (a newsreader, perhaps), do you want that external link to open in a new tab in an existing browser window on a different space, or do you want it to open a new browser window on your current space, and open the link in a new tab on that new window?

Turns out lots of people prefer the second option - they want a new window on their current space with the link opened in it. Now, adults may disagree, and Firefox as well as Safari have a preference setting which allows you to change the behavior to either of those options. Everyone's happy.

Except Google Chrome users. Chrome always opens external links in the most recently used browser window, where ever it is.

This causes some people to be upset, example: http://www.google.com/support/forum/p/Chrome/thread?tid=1eb2eb870ba29e97&hl=en

I was also upset. I am a programmer though, and I can make the computer do what I want normally. So I made it do what I want.

1) Download this AppleScript bundle, and unpack it in ~/Library/Scripts/
2) Double-click it (to register it with Launch Services)
3) Open Safari and go to the General tab in the Preferences, and set your app as the default browser.
4) Share and enjoy

A fair bit of credit for making this work goes to Pepjin de Vos - http://pepijndevos.nl/2010/05/open-external-links-in-running-browser/ - I repurposed his idea/script to implement my workaround here.

GTD - automating addition of new email info to an existing Toodledo

2011-05-15T08:10:00.000-07:00

This will probably be the last post about migrating from Things to toodledo, as I've got all my most frequently-used workflows automated at this point

The last thing I needed to do after migrating all my data and automating task intake from Mail was automating the addition of new information from Mail to an existing Toodledo task.

The specific use case is that I've already got a task tracked in Toodledo and there is email traffic concerning it - some with important information I'd like to easily reference - but the task isn't done yet. How do you make that information easily referenced? The most useful way in my opinion is to save a link to that message along with some metadata about it, in the note of the Toodledo task.

To do that quickly, I wrote a little Applescript that fishes out the date, the sender, and the message id, and puts that information on the clipboard with the message id embedded in a clickable link. Now you just have to go to the task in toodledo and hit the paste key, and you're all set.

As with my new task creation script, I use Mail Act-On to bind execution of this script to a hot-key and simultaneously chuck the message into an archiving folder - keeping me from mousing as much and keeping my inbox clean.

Here's the script - Copy Message Link.scpt - hopefully you find it useful

Cheers!
-Mike

GTD - intake automation with Toodledo and Mail.app

2011-05-13T14:50:00.000-07:00

I'm still in the process of converting my task workflows from Things to Toodledo, for reasons mentioned previously

One of the most important things for me is to have an automated workflow for task intake from email people send me to my task management application.

For Mail.app (my email application) I use Mail Act-On so that I can have a keyboard shortcut run an Applescript. For Toodledo I use Applescript to create a new email message to my Toodledo email address for new task creation, and I prepulate the email with the original subject, and an email body that contains information about who the email was from, when it turned up, and then a URL that will open the email in Mail.app if clicked.

I leave the message open so I can edit the Subject line (adding toodledo email syntax about when it is due and what folder to put it in etc) then I send it away and move the message out of my Mail.app inbox into a "to-process" folder where things are archived

The end result is a new toodledo task with a note that contains a link that will open the message. Fabulous.

Here's the script

GTD - migrating from Things to Toodledo - exporting Things data

2011-05-06T11:35:00.001-07:00

I'm migrating from Things to Toodledo for task-tracking, in order to satisfy a couple of new (to me) use cases that I can't handle with Things, but that it appears Toodledo will allow me to handle.

I have more than 3500 completed tasks in Things though, and I treasure my ability to search through the completed tasks to turn up info used for new tasks. It isn't acceptable to me to lose that data in the transition, but I don't want to have multiple task apps, so I want to migrate all of my data from Things to Toodledo.

Things supports certain things Toodledo doesn't, so the first thing you have to do is alter your Things data so it's in a style Toodledo can accept, and that may be imported well.

First thing is that Toodledo doesn't really support Projects. It supports "Goals" but to tell you the truth, migrating that data isn't easy, and I didn't attempt it. I simply made my Projects into Areas on Things, and moved the tasks in Things to those Contexts.

Toodledo also doesn't support delegation the same way Things does. In order to migrate your delegates, the best way I could think of to map it was to have a "Delegated" Context, and then take the Things delegate's name and tag the thing with it. That way you can search for specific delegates on Toodledo post-migration, and you'll know something is delegated, at least.

On Toodledo, you'll need to create Folders for each Area from Things, along with the "Logged" and "Delegated" contexts.

The second thing is that Toodledo doesn't allow you to import completed tasks. This is troublesome for me because I want that history. I settled on putting all my completed tasks in a "Logged" context so at least I could get them in Toodledo, then mark them all completed later.

Finally, Things URL format in notes needs massaging in order to generate clickable links on the toodledo site, and I implemented some logic to do so.

That's about it - if you run this Applescript from within Things, once for the "Next", "Someday" and "Logbook" lists, uploading to the toodledo site each time, you'll have all your data in there.

Then you just need to mark the things in the "Logged" Context as completed, and you've finished your migration. I used TaskSurfer to do that since the interface was so fast for completion-marking.

Cheers

GTD - migrating from Things to Toodledo - why?

2011-05-06T11:06:00.001-07:00

As mentioned previously, I'm not a strict GTD follower, but I'm not far from it and I am an obsessive task-list-maker. It's the only way I reliably get things done.

I've been using Things from Cultured Code for a long time - long enough to have created and completed 3617 tasks, actually - and it has worked fantastically for me. I love the interface, and it works well.

However, it does not have cloud sync working yet (though I understand it is in beta), and even with cloud sync it doesn't handle delegation or 3rd-party information sharing well. That was never a problem for me before, but I recently has a new use case turn up as a requirement for me and my task list: I needed to provide visibility into my tasks to 3rd parties.

To me, that means that 3rd parties should be able to look at my task list at any time, with no extra effort on my part. Requiring effort on either my part or the 3rd parties part would be too inefficient for something as constantly used and updated as my task list. Further, it should be possible to swap tasks back and forth with people if I want, with a minimum of effort. Things simply can't handle that.

Some features that Things has which I must maintain are full-featured offline functionality, a feature that eliminates most issue-tracking software like JIRA. I must be able to automate task intake from Mail.app to the task list as I had before. Finally, I must be able to access and modify my task list on my phone, in sync with the list on my desktop.

For all these reasons, I've decided to migrate to Toodledo - it appears to satisfy all these requirements, with the possible exception of good Mail.app -> Toodledo integration, but I'm aware of a couple ways to do it already just not quite as good as I like yet. At present it appears I'll use Toodledo's web site for 3rd party access, their iPhone app for phone access, and TaskSurfer for a desktop client. I'll continue to use Mail Act-On for Mail.app integration and I will either use Toodledo's email interface for task queueing from Mail.app, or I'll figure out how to drive TaskSurfer via Applescript.

The next couple posts will detail the scripts I've written in order to implement the migration from Things to Toodledo so that I don't lose my existing data from completed Things tasks as well as Things in progress.

GTD - intake automation with Things and Mail.app

2011-05-06T10:54:00.001-07:00

I'm not a strict Getting Things Done / GTD follower, but I do try to keep my inbox from becoming my todo list. It's easy to slide into that habit, but the inbox can't tell you anything about deadlines or priorities, so I really like to have a task management program outside my mail program, where I can line up tasks based on something besides when someone sent me mail.

I use Things from Cultured Code in order to manage my tasks, and I use Mail.app as an email client. I get mail rapidly enough that the simple act of creating a task in Things in response to a mail takes a significant amount of time each day - a great target for automation.

My first goal was simply to have a keyboard shortcut that would pop up a quick entry task creation dialog for Things in response to an email, while in Mail.app. That's easily done, but while we're automating why not make things significantly better? My definition of "better" here is to create the Things task with the email subject used as the task name, then the task notes prepopulated with a hyperlink to the email, made in a way that is stable no matter where the email is saved. Finally, I want the email automatically moved to an archive folder in Mail.app so my inbox is decluttered. The final bit of glue is the Mail.app plugin "Mail Act-On" which lets you create rules triggered by keyboard shortcuts.

I have a Mail Act-On rule set up in response to Ctrl-T that runs the Applescript I use to automate things, then moves the message to a folder.

All pretty easily done, and something I use all the time. So much in fact that I'm sharing it here - use and enjoy: NewThingsToDo2.scpt

Cheers

Monitoring TimeMachine backups with Nagios

2010-09-11T11:39:00.000-07:00

An automated backup strategy is the key to avoiding tears when your hard drive inevitably fails.

With Apple machines, this is typically handled at the desktop level with an external drive and TimeMachine. (Incidentally I'm partial to the OtherWorldComputing on-the-go's myself since they don't need a power adapter...)

The problem with TimeMachine and local backups, in an enterprise context, is that it's hard to tell whether people are actually backing things up or not. What if the TimeMachine drive has failed for some reason? You will keep right on living, without realizing your user (or perhaps you) aren't actually successfully backing things up.

This is exactly the sort of problem that monitoring systems are supposed to solve, and my personal choice for monitoring is still nagios (though I hear good things about OpenNMS). You don't have to use a monitoring system perhaps - you could just have the script send you mail similar to my previous MacPorts check, but I like nagios because it maintains a history of previous events and has a flexible communication infrastructure.

But how to monitor the local TimeMachine backup status of a laptop with a centralized monitoring server like nagios?

The strategy I chose is to have a script on the local machine run daily (via an /etc/periodic/ entry) that scans for successful backups, fishes out the most recent success timestamp, and uses ssh to make an entry on my nagios server in a file. Then the I wrote a quick nagios plugin that inspect a timestamp in a file and measure whether it is acceptably recent or not.

If this is interesting to you - grab both of those little scripts from the link above and enjoy.

If you have a totally different strategy for monitoring local backup status, I'd be curious to hear it - please add a comment.

Cheers

Script to automatically check for outdated ports on Mac OS X

2010-08-22T20:15:00.001-07:00

I use the fabulous MacPorts system to get some of the software I need for my Mac laptop.

I am also interested in keeping this software up to date, for a variety of reasons, but just to know that I'm getting the latest security patches alone has me interested in regular updates.

I hate checking for outdated ports manually over and over again though, aren't computers supposed to do that?

So I wrote a script that will periodically check if I have outdated ports, and if there are any it will send me an email with a list of the outdated ones.

You can fetch the script port-checkupdate from my website - there are some comments in the script, and it's really tiny so it should be easy to work with.

You will likely also want the mailtest self-contained mailer my friend Marek Gilbert knocked together as well - the script expects to use it

Cheers

Mac OS X laptop and TimeMachine whole disk encryption with successful restore

2010-07-18T10:27:00.000-07:00

At some point, if you work with sensitive information, you realize it's time to get serious about protecting the data from theft.

I'll assume you already have firewalls in place, surf the web safely and keep your software up to date. That should help reduce the risk the data is accessed while you are using your computer, but since MacBookPro laptops are portable and valuable, and TimeMachine backup drives are also portable and valuable, there is the real risk that they are physically stolen, putting your sensitive data at risk.

The solution to this is multi-part - first you need to make sure that your machine asks you for a password when it resumes from sleep or wakes up from a screensaver, and that it goes to sleep or to the screensaver quickly when unattended. This makes sure that someone can't just open the lid or move the mouse and get your data.

Now they are going to try to reboot, the second part is to make sure that you don't have auto-login enabled, so they'll have to enter a password to login after rebooting.

You are mostly safe at this point, but did you know that Mac OS X install DVDs have a feature which let you reset the root password on a laptop, if you boot from the DVD? They do. It is quite useful really - I've needed to use it - but it means that the password protection isn't worth much at this level, a thief that wants your data can easily access it.

The last part of the solution handles this problem, by using encryption to make sure your data is inaccessible even if someone has physical access to your machine. The goal is to have your whole disk encrypted, with a pre-boot password required to decrypt it and get the OS running. I use PGP Whole Disk Encryption ("PGP WDE") for this purpose - it is commercial software but doesn't cost too much compared to exposing the sensitive data, and it is easy to use.

Assuming you have installed PGP WDE and have your laptop hard drive encrypted at this point, you are mostly secure, but what about your TimeMachine backups? That little external drive next to your computer still holds all your sensitive data, but it is not encrypted - you have moved the problem around but not solved it completely yet.

The next step then is to use PGP WDE on the TimeMachine drive, so it is encrypted as well. Now your data is fully protected, but you have a new problem - if your laptop dies for some reason and you need to restore from your TimeMachine drive, you can't - the Mac OS X install DVD you would use for the restore can't access the TimeMachine drive to restore from it, because it is encrypted!

Luckily, this is a solved problem. In a nutshell, what you want to do is to re-partition your TimeMachine drive and put an OS X install on it that you can use as a rescue, and you want a copy of the Mac OS X install DVD on that rescue install. Use TimeMachine to backup to a second partition on the TimeMachine drive, and now following these excellent instructions you have a tested and working self-contained encrypted backup of your laptop with all the tools you need to restore your computer if you have problems.

Cheers

Graphing JMeter results from the command line

2009-09-29T15:42:00.000-07:00

I'm currently knee-deep in some performance testing, and I wanted to have fully automated JMeter runs, with attendant fully-automated graphing of the results.

There's lots of results to graph from the systems, but the JMeter log itself is a great source of data.

Only problem is, the best-looking script didn't seem to want to produce the two most important graphs I wanted to see! (aggregate throughput and aggregate users)

The script is here:

http://wiki.apache.org/jakarta-jmeter/LogAnalysis?action=AttachFile&do=get&target=jmetergraph.pl

From:
http://wiki.apache.org/jakarta-jmeter/LogAnalysis

All it needed was a small patch though, et voila, I've got the aggregate charts.

Maybe this will save others (http://markmail.org/message/yntlaygfepnpine7) the time?

[mhardy@tkdevvm(192.168.146.130) util]$ diff jmetergraph.pl ~/mike/Desktop/jmetergraph.pl
313,314d312
< $glabels{'entire'} = \%entire;
<

Accounting error

2009-06-28T17:50:00.000-07:00

I'm continually astounded by the general lackadaisical attitude towards accounting error in general society. Some examples:

- governments don't use accrual accounting, yet generate massive future liabilities. Fail!
- historic-cost accounting (the GAAP standard) makes no allowance for appreciation or inflation, a double-fail

There are definitely gray areas in implementing those things, but the benefits should outweigh the costs, and as a manager I definitely want to be aware of accrual effects, appreciation and inflation.

Boring perhaps but it's better than driving businesses (or countries) into the ground...

Berkshire Hathaway goals

2009-03-01T12:52:00.000-08:00

Reading through the Berkshire Hathaway annual report (something that is highly recommended - it's not nearly as dry as it sounds) you quickly run in to the operating goals. Honestly, thinking about them out of context, there's not much else you need to do either on a small scale (a 4 person project, a small business, a career, your family, etc) or on the scale they operate in order to be successful:

(1) maintaining Berkshire's Gibraltar-like financial position, which features huge amounts of excess liquidity, near-term obligations that are modest, and dozens of sources of earnings and cash;
(2) widening the "moats" around our operating businesses that give them durable competitive advantages;
(3) acquiring and developing new and varied streams of earnings;
(4) expanding and nurturing the cadre of outstanding operating managers who, over the years, have delivered Berkshire exceptional results.

Substitute credibility or trust for finances for some scenarios, other valuable qualities for others, or obviously keep it finance-related for economic things, but I don't think there's a lot more to it from what I've seen so far.

The devil's in the details implementing this game plan, surely, but I think it's important to realize/remember that the game plan itself is not complicated.

As I read through more of it, some other gems:

- Beware the investment activity that produces applause; the great moves are usually greeted by yawns.
- Beware geeks bearing formulas (as a geek, I agree)
- Regarding municipal finance: "The gap between
assets and a realistic actuarial valuation of present liabilities is simply staggering."

Mike 1, Banking Fees 0

2009-01-22T15:38:00.000-08:00

Along the same lines as credit card finance charges, I simply refuse to banking fees.

It isn't that I don't think they are justified, it's that I think they are a tax on ignorance and it's less costly to avoid fee-inducing banking situations.

To that end, I just got a "collected funds fee" at the bank, and investigated it. Here's the skinny from my banker:

"Yes, there's a difference between collected and available balance and your analysis is correct. Depending on where the funds are drawn from and if they've been collected is the determining factor. Funds might be available but not collected from the maker bank which means a portion of what is available is technically a loan to the customer and the reason for the fee (I was able to reverse it). Some customers find a balance between savings and checking or attach a line of credit to the operating account as an option to avoid the repeated fee."

So if you deposit checks regularly, you're going to see three balances. "Balance", "Available Balance", and "Collected Balance".

If you want to make sure you get no fees, find the lessor of those three numbers, and treat that as your "fee-free balance".

It's a game of inches...

Cheers

Credit Card Finance Charge Tidbit

2008-11-23T11:56:00.000-08:00

Okay, as some following this blog (if anyone is) may know, I'm not just doing internal server operations at Tacit Knowledge any more, I'm also the CFO, so naturally I do a lot of finance.

In order to want to take on a role like that, and to pursue excellence in it, you've got to have more than a little bit of interest in even the most mundane parts of finance. You have to want to truly understand the full workings of the little things or you've got no chance when the big things confront you.

With that as motivation, I'll mention that for my entire adult life I have engaged in a campaign against credit card finance charges. This may seem pointless, and I'll admit it is - on the "dollar value" side, it is not really worth a lot to achieve a state of "no finance charges ever" because even getting close to that state means there's not enough dollar value left in charges to justify getting to zero. But where's the fun in that? Let's get to zero.

The first thing I learned in this campaign was that the credit card companies use several strategies to make it nearly impossible. The first of these strategies is the non-fixed statement date. In other words, they do not align their statement dates with any fixed calendar day so you can never assume that if you make a full payment by the 1st of the month every month, you won't get a finance charge.

They move the date around every month, so every once in a while, a full statement period will be between and not inclusive of the two 1st dates, and they'll get you. How sneaky! That honestly took me a couple finance charges to figure out, but is easily defeated with a "semi-monthly payment plan" though - just send them two payments a month, on the 1st and 15th. Please note that if you don't have some automated way to make this extremely easy you're wasting your time and it is not worth it. I use Quicken with bill pay, personally.

Okay, so having done that, I thought I'd essentially won the game and needed to move on to other venues to exorcise the daemons of my financial obsessive compulsiveness.

Most of their other strategies revolve around keeping you in debt once you've gotten there, and since that's a problem solved with fiscal discipline (a much larger subject than tactical credit card payment plans) I'm not going to go into it here. You need Suze Orman or a Rich Dad, Poor Dad book, Quicken, and some soul-searching if you're in that situation (highly recommended if you are, though).

Then I just got hit with two finance charges in two consecutive months, and the game was back on. I simply couldn't understand what happened since I sent them more then the previous balance was.

I'm sure this is no surprise to some of you but for those that did not know it, I'll mention another subtle trick they play on you. If you *ever* take advantage of the ability to defer payment, as I did two months ago, you will obviously incur a finance charge. I'm not against that, I will pay to rent cash sometimes, and that month I did so.

But what you've done is you have established a new class of balance in your account - a "revolving balance". It's important to realize that this class of balance is treated very differently from "new charges" because now when you send payments in - even two a month! even more than all new charges! - all of the payments will be applied to "new charges" *first*, and only the excess will apply to the revolving balance.

Sneaky again! In this way, you may find yourself making multiple payments a month, for *more* than the sum of your new charges every month, and still owe a finance charge every month as they charge you to float whatever is left of the revolving bit from the past. Seems unfair to me, but hey, they're in the cartel of payment processing networks and I'm not, so I don't get a vote. I do have a new compensating strategy though, and will restate my "never get a finance charge" rules like so:

- make two payments a month, on the 1st and 15th, for whatever the balance is on those days
- if you ever establish a "revolving balance", make sure to establish a credit balance with the account with the next payment - compensating for any charges you may incur between your sending the semi-monthly payment that would otherwise make you run a balance

Now I think I've "got" them. We'll see.

Some will argue I should quit fussing and use the "AutoPay" option, but then if you actually wanted to rent cash for a month, you'd have to pick up the phone to do it or they'd pull the full balance from your checking account anyway. So that option gives you a low score on the "ability to control" things scale. My phone calls to Citibank to get the finance charges waived while I learn these lessons :-) average around 15 minutes, so as a solution design with AutoPay I'd be losing money on the opportunity cost of my hourly billable time.

This concludes your obsessively compulsive finance lesson for the day.

Cheers!

New Macbook Pros underwhelming

2008-10-22T05:36:00.000-07:00

Only new features I was able to discern:

- larger hard drive
- faster RAM (but limited to 4GB according to them, and there are no 1x4GB 1067MHz parts that I'm aware of)
- stiffer chassis

All the other stuff looked not-that-interesting

I'm especially bummed that they upped the speed of the RAM since that means our new 6GB strategy won't work as there are no parts, and they didn't introduce a 17" one that I can see.

But they're heavier at least!

3rd Gen and 4th Gen 17" MBPs are still the best bet for us I think, with 6GB in them.

Any one else have thoughts?

This is possibly material as we have enough people now that hardware refresh is always going on, so we're always potentially in the market for a laptop or two.

Bose QuietComfort2 headphones are quite nice

2008-09-25T17:14:00.000-07:00

This post is a bit of shilling.

First let me say that I know the QuietComfort2 headphones are nice because I've had a pair for around three years. I work with them on frequently, I travel with them all the time, I use the heck out of them basically and they're much loved.

But they broke on a recent trip, and that reminded me of another feature I like about them: support. Bose replaced them for me (3 year old headphones!!) with a new pair for $50 ($50!!). If you haven't bought them because they cost a lot (and they *do* cost a lot) you should know that you're going to get a set of headphones that will last and last, and if they don't, you'll be able to get a new pair without taking out a loan

Apple Bluetooth Not Available fix

2008-09-25T17:11:00.000-07:00

A lot of people have experienced the dreaded "Bluetooth Not Available" problem on their Apple machines, and I just recently had it happen to me.

I've seen it before, and just rebooted to resolve it but this time I researched more.

Some people report that resetting the SMC will fix it. Some report it's a temperature thing. Odds are it can be many things, so this fix may not work for you.

One forum post reported that they hadn't noticed it, but they were using VMWare, and one of their VMs had taken control of the bluetooth device - simply unhooking it from the VM allowed it to be used in OS X again.

Imagine my chagrin when I noticed that was what had happened to me...so if you get this message and use VMWare, check your VMs to see if one of them grabbed the device. Disconnecting it will get you going again.

MacBook Pro rev3+ works with 6GB ram (1x4GB SODIMM, 1x2GB SODIMM)

2008-09-25T17:08:00.000-07:00

1x4GB SODIMM RAM sticks just came out, and they're actually not that expensive ($179 @ NewEgg)

Read on a zdnet review site that a Rev3 or higher MacBook Pro would boot with 2x4GB sticks, but not work well

An enterprising forum member on some Australian tech site defied someone else's unsubstantiated data to show that in a 1x4GB+1x2GB SODIMM configuration it would work, so I ponied up to try it for myself along with a colleague.

Works just fine for me, second day running. Going to kit out the rest of the office with them.

VMs running with 2 CPUs and 3GB of RAM while your host machine is still humming along? Definitely worth $170...

Build systems, and work flow thoughts

2008-08-30T10:34:00.000-07:00

Long time, no post. No worries though, there are new tasks and new thoughts associated with them. Here's a couple.

First is a general problem associated with build systems. For some reason, it appears that dependency resolution and build/package/deploy scripting always get mixed up in build systems. I'm creating one now for a large Java system and the fact is that all the available tools and styles leave me unsatisfied.

A system that is basic might include dependencies persisted into the source tree (libraries etc committed) with ant scripting. This is functional but for systems with multiple modules, like the one I'm working with, it results in either a lack of fine-grained library control, or library clashes, or library redundancies - all of which are maintenance inefficiences. At the ant level you would either end up with a large complicated ant file that is hard to maintain or you would have many sub-build files which may have redundancy

A system that is more "modern" would use maven for dependency resolution and build, but the problem as I see it is that while you eliminate the operational inefficiency associated with library placement you add a large maintenance inefficiency in that you were already using a source control system and now you have a fair bit of work to do to maintain your maven repository as well, in order to back up your dependency declarations. Further, scripting the actual compilation and packaging of your artifacts can be done in maven but is not elegant, to my eyes.

I have heard that ivy+ant can help, but as ivy is built on maven repositories to meet declared dependencies there is still the maintenance cost of maintaining a repository, and scripting the build in ant reduces the problem at minimum to the cost incurred in the basic build system.

Which all leaves me honestly feeling as though from a pure dependency resolution and artifact production perspective that the basic system with libraries in the source tree and ant scripting for artifact production is still the global optimum for build systems. That just can't be the case, can it? I'd love to hear otherwise.

The only saving grace I'm aware of with maven is that there are enough value-adding plugins (e.g. IDE configuration generators, static analysis tools) that maven proponents assert the value of a maven system is large enough globally to overcome the local added costs of maintaining the repository. While I will grant that there are a large number of value-adding plugins, I'm not convinced that the same value couldn't be captured with what I would wager is a simpler system substituting like ant tasks for those maven plugins.

The internal debate goes on, though I am committed to using maven for the system. At the least I will come out of this with a thorough understanding of exactly how maven will work on a large project because we are certainly destined to find out.

One other thing I have an ongoing interest in is the general problem of how to efficiently complete tasks where more than one person could do parts of the task, and in general more than one type of specialized skill must be used to complete the task.

I've been watching articles on lean engineering, agile, scrum and XP flow by for a while, and I think something between a pull-based lean system and agile/scrum models how a highly skilled team works, while introspection on the real things that drive such a system would perhaps formalize (and make more efficient) how this realistic team works. That makes a little formal thought around the ideas useful, and I just recently read a great article that does so here:

http://leansoftwareengineering.com/ksse/feature-brigade/

The idea I see there is a generalization of a "bucket brigade" work team style to any production need, and then a refocus of that style to a specific application of software feature development, mapping in the lean and agile processes where necessary to communicate the idea. It appears it might work.

One thing I'm curious about though is that it appears the work style counts on the links in a chain being fixed - e.g. this one type of skill (or skill overlap) is always needed for this stream of tasks. In reality a stream of tasks is typically much less uniform and contains individual tasks that need a variety of skills (or skill overlaps) - never quite the same set twice in a row. I'm not sure how you would handle that, or even if it is handleable in a general work flow design like that.

Perhaps for each unique set of skills you'd have a different feature brigade, and just hope that in reality there was a finite number of linkages required and that exercising them didn't result in over-utilization of a resource shared between multiple brigades? Unfortunately, this seems to match reality frequently as well. It shouldn't be too hard to reduce the multiple chains to a single slightly branching chain though and still get the inventory/capacity alarms that a kanban board will give you while maintaining flexibility around the skills required for a given task.

Either way, if there is potential to have smooth high-velocity feature development it should be examined, and this is article definitely has some ideas I'd like to incorporate.

Cheers-
-Mike

If you upgrade JDKs, don't forget the security portions

2008-05-21T13:20:00.000-07:00

Ran into this today at work - the goal was to upgrade the JDK / JVM on our servers to a newer version with bugfixes for some issues the servers have.

We merrily got the new JDK and slowly promoted it out from dev to qa to staging and then to two servers in prod.

Naturally, there was a problem that wasn't discovered until we got to prod, but on testing other environments was present everywhere. What was the problem? there was a security exception preventing the server from accessing an https URL now over SSL.

Huh? Well, Java is very very strict about SSL / HTTPS connections, so if you try to use a Java program to connect to a server over SSL using an HTTPS URL, and that server doesn't have an SSL certificate that is signed by a CA in your JDK cacerts file, you'll have problems. The normal solution for this is to use keytool to add the CA to the JDK's cacerts file, or to make a keystore using keytool with the certificate in it.

Additionally, it is possible that the java.policy file itself has been changed to grant (or revoke) certain permissions from the JDK.

Both of these files live inside the JDK, so if you upgrade the JDK but forget to copy those files over (like we did) you're going to have some problems.

Finally, for very very old JDKs (1.3.1 and older, pity me) note that you will have other issues if you don't download the JCE and JSSE since without those, the old JDK can't do strong crypto and you'll have trouble too.

In order to discover whether you need to care about this or not, you should like at JAVA_HOME/jre/lib/ext (the JSSE and JCE files go in there) and JAVA_HOME/jre/lib/security/ to see if any files have been updated. Or you can use something like a "find . -mtime " to see what's not stock.

Good luck...

JDK GC tuning for app servers

2008-05-15T14:50:00.000-07:00

A quickie post here.

Lets say you get called in to review a server that's getting OutOfMemoryErrors

This is a bad thing, obviously! How do you fix it?

Well, it depends on the root cause, first of all you have to figure out what the problem is.

On old JVMs you turn on -verbose:gc and that's it, on newer ones you can turn on more details, some timestamps etc - all things that help you, but that's not the point of the post.

The point of this post is to send you over to get GCViewer from tagtraum at http://www.tagtraum.com/gcviewer-download.html

It's really the best option for visualizing what's going on.

It's not perfect though, and a trick I have to use nearly every time I'm visualizing the GC behavior in an app server via GCViewer, I have to filter the log file so it has nothing but GC lines:

cat $log | grep '\[' | grep GC > clean-gc.log

Then you fire GCViewer up and see what you've got

usually you get a line going "up and to the right" which is great for finance, but bad for GC signatures. That means you have a memory leak, or you have misconfigured your server to hold on to too many objects in caches, or to accept too many sessions that themselves have too many objects.

Sometimes you realize that you never run out of heap, but if you still get OOM errors, you are either running out of PermGen space or in rare cases are running out of operating system file handles by spawning too many threads (that error gets mapped to OOM, believe it or not)

Without GCViewer, it's difficult to clearly articulate the problem though, and also hard to see trends. Combined with how fast it is to use, I end up reaching for this particular utility every time.

Cheers

Installing windows on a laptop that had linux

2008-04-28T15:00:00.000-07:00

At my office, most of the people use macs, and of the people that don't, most use windows but some use linux.

I don't really care what people use, but when someone wants a windows laptop and the only available hardware in our inventory used to have linux on it, I need to pop in the old windows installation cd and get windows installed, right?

Problem is, the windows installer CD will not install by default onto a hard disk that has linux on it.

Why not? I have no idea. I'm fine with it obliterating all the data - I want it to actually. But it will say "Setup is investigating your something or other..." then get stuck on a black screen. The installer doesn't really start.

The solution? Boot the machine with some linux install cd (or something similar that you have handy) and zero out the first part of the hard disk. The first 800MB did it for me (it did that before I could even hit ctrl-c, probably just a few MB is enough). It is not enough to just re-partition the drive without zeroing out some data. (specifically I popped a fedora core install CD into the drive, typed "linux rescue" at the boot prompt, told it to skip networking and skip finding linux installs, then at the root prompt said 'dd if=/dev/zero of=/dev/hda bs=1M', let it run for a second, then hit ctrl-c, then when the prompt came back, said 'exit' and rebooted)

After that, you can use the windows installation cd with no issues.

$PATH issues on windows

2008-04-16T19:06:00.000-07:00

If you're busy perl-scripting on a windows machine, and you're getting unexpected behavior from the command-line utilities your shelling out to (say, 'find', or something), it may be because instead of getting the utility you think you're getting (/usr/bin/find, for example, from cygwin) you're getting something totally different (dos find, for example)

This can lead to a lot of confusion when you try to figure out why you aren't getting consistent results between systems you're working on.

Solution? Use full paths. If you want the cygwin find, say "/usr/bin/find" or you just can't be sure.

Beware DOS line endings with Perl open command in cygwin

2008-03-26T09:05:00.000-07:00

Just a quick note, if you are opening files in cygwin perl on windows, you need to beware the difference between line endings on unix and windows (CR vs CRLF)

You can either twiddle with what constitutes a line-break in perl or dos2unix the file, but if you do neither you'll get unexpected results

Patterns of deployment

2008-02-15T15:52:00.000-08:00

This link just flew across the transom, and since I'm think in automated deployment on most projects I'm on I thought it was too good not to pass on.

It's called "Patterns of deployment" and the best part I think is the info about host-specific / environment stuff. That's always a tough nut to crack

Windows locks files, I unlock them

2008-02-15T15:45:00.000-08:00

Still dealing with a recalcitrant windows environment where I'm trying to automate the deployment of a ColdFusion application.

Problem is, even after you stop the ColdFusion servers (and you ensure they're down via tasklist.exe inspection), some files are still locked.

It appears the only way to unlock them is to reboot the machine. (yes, I know this is abhorrent, believe me, I'd avoid it)

The way to do this easiest (I think) is to use the 'sc' Windows builtin to set the services to start manually (instead of automatically) (e.g. 'sc config "" start= demand')

Then reboot the machine, and check remotely until the machine's uptime is recent (via something like 'net statistics srvr|grep since 2>&1') via ssh, then parsing (m/\D+(\d+)\D(\d+)\D(\d+)\s+(\d+):(\d+)\s+(AM|PM)/) out the uptime (using Time::Local, and being careful if you don't get what you expect since that machines the machine is rebooting so the connection failed).

Once it's rebooted, you should be good to go, then just sc to set the start back to "auto" and you're done.

Easy (ha)

cygwin messes up windows permissions, cacl fixes them

2008-02-15T15:29:00.000-08:00

I recently had the misfortune to work in a windows environment via cygwin+openssh

This is a seemingly civilized configuration - to my unix-biased way of thinking - as cygwin gives me a shell and ssh is a very familiar way to access machines for logins and automation.

Unfortunately, Windows has a very complex permissions scheme (ACLs) and cygwin+openssh+scp just won't honor Windows permissions. No matter how I tried, the files would land on the server without properly inheriting the permissions of the directories they landed in.

Luckily, Windows provides a relatively easy way to fix this problem in the utility cacls.

What I ended up doing was using cacls with the '/S' switch to get the string representation of the parent directory of the location I copied files to, then used the '/S:' switch again, this time with the string I got from the first invocation, but now targeted at the actual directory I was working in.

You can use the nifty "AccessEnum" utility (google it) to verify your permissions a lot more quickly than a ton of right-click/sharing-tab examinations.

Poof, perfectly consistent permissions.

I'll continue choosing unix for my servers though, thank you...