It’s Time to Move On

Reading Time: 2 minutes

Seed OfficeAfter considering the matter for some time now I’ve decided to leave The University. I know this will come as a surprise to many and it was to me a little. The reasons are rather complex, but it boils down to the fact that I’ve achieved what I set out to achieve and that many threads are coming to a natural end.

I’ve been at Seed Software for more than 6 years, it’s been a remarkable journey and we’ve taken the business way ahead of anyone’s expectation. I’m hugely proud to have been a key player in taking Seed from a start-up to one of the biggest suppliers of software to the UK Fire Service and the most successful project of its type in the UK.

It is now though time to draw a line under this chapter and begin a new one. So I’m looking for a new role beginning in October this year. I’m keeping an open mind as to what that might be.

Update Septmber 22nd 2015

I’ve now agreed terms with another organisation. I’ll be leaving Seed Software in mid October.

–end of update–

My career is punctuated by large switches of environment. Embedded to VMS to *nix to Windows, flow orientation to OO, native to managed, waterfall to agile and of course keyboard to chainsaw when I worked in habitat management for a while.

Find out more about some of the key projects I’ve worked on or more about me in general on the “About” page.

Read more

Because… PostScript!

Reading Time: 2 minutes

Because it’s become somewhat of a tradition for me to do something silly on a Friday lunchtime, I thought I’d take on one of The Department‘s basic coding challenges.

The brief was simple;

I want a program that will print out the numbers 1,2,3,4,5,6,7,8,9 in a shuffled order. The order must be different each time the program runs. Note that the same number must be different each time. It should be possible to extend this to work with 52 numbers, in which case I can make a shuffled deck of cards.
You can use the Random number generator in C#, but you must make sure that the same number never appears twice, as a deck of cards which contains more than 4 aces has been known to raise suspicion.

As it was Friday lunchtime however I decided to make the solution anything but simple, firstly I replaced the numbers with the actual card names and secondly I thought I’d write it in PostScript because it demonstrates a totally different form of notation from the way we write the normal imperative languages like C# or Java.

/Suits [(Clubs)(Diamonds)(Hearts)(Spades)] def
/Cards [(Ace)(Two)(Three)(Four)(Five)(Six)(Seven)(Eight)(Nine)(Ten)(Jack)(Queen)(King)] def
/YCursorMax 720 def % 10 inches from bottom
/YCursor YCursorMax def 
/XCursorMin 72 def % 1 inch from the left
/XCursor XCursorMin def
/XColWidth 113 def % 1/4 of the printable page
/Helvetica findfont
12 scalefont 
setfont
/Deck [ 0 1 51 {} for ] def
0 1 50 {
    /SwapLeft exch def
    52 SwapLeft sub realtime rand mul exch mod
    SwapLeft add /SwapRight exch def
    Deck SwapLeft get
    Deck SwapRight get
    Deck exch SwapLeft exch put
    Deck exch SwapRight exch put
} for
0 1 3 {
    dup /Col exch def
    0 1 12 {
        Col 13 mul add Deck exch get
        dup 13 mod
        XCursor YCursor moveto
        Cards exch get show
        5 0 rmoveto
        (of) show
        5 0 rmoveto
        13 div cvi
        Suits exch get show
        /YCursor YCursor 20 sub def
    } for
    /XCursor exch 1 add XColWidth mul XCursorMin add def
    /YCursor YCursorMax def
} for
showpage

C#’s Parallel.Invoke is [Insert Hyperbole Here]

Reading Time: 2 minutes

Freeside VTIt’s simple – we can’t make processors go any faster. But we can add more cores.
The OS can schedule different processes to different cores but to take full advantage of the potential we need to be writing applications that run on multiple cores.
Unsurprisingly there’s a lot of investment in making this easier for programmers – to the point that it’s now easier than falling off a log…

Parallel.Invoke(
    () =>
    {
        if (!GetSomethingFromTheDb(out something))
            something = MakeUpSomethingFromSomewhere();
    },
    () =>
    {
        complexResult = SomeComplexOperation();
    },
    () =>
    {
        someObjectGraph = ReadAndParseSomeFile(fileName);
    });
SomeReallyComplexThing(something, complexResult, someObjectGraph);

The individual actions will be run in parallel, or to be more precise may be run in parallel. The task scheduler takes care of whether or not to actually run them in parallel and the degree of parallelism.

When we arrive at SomeReallyComplexThing all the previous tasks will have been done.

That’s ace. It’s a lot easier than messing about with threads.

Even before the Parallel library was around it wasn’t actually difficult, but you needed some lateral thinking…

Task[] tasks = new Task[] {
    Task.Factory.StartNew(()=>
    {
        if (!GetSomethingFromTheDb(out something))
            something = MakeUpSomethingFromSomewhere();
        }),
    Task.Factory.StartNew(()=>
    {
        complexResult = SomeComplexOperation();
    }),
    Task.Factory.StartNew(()=>
    {
        someObjectGraph = ReadAndParseSomeFile(fileName);
    })
};
Task.WaitAll(tasks);
SomeReallyComplexThing(something, complexResult, someObjectGraph);

Ok, I admit, I’m simplifying. You still need to understand the basic problems of concurrent programming but the mechanics of writing code that can take advantage of parallelism are now trivial.

Ipswich Town Centre Blues

Reading Time: 5 minutes

Ipswich isn’t the wealthiest large town in the UK but it’s certainly not in line for any EU deprivation grants. It does pretty well, yet there seem to be rather lot of empty shops in the town centre. It’s something I’ve particularly noticed since moving back to Suffolk (from Hull) 3 years ago.

So on a recent visit I thought I’d snap every empty shop I saw. I wasn’t prepared for just how many there were though!

Sure, I know that the effects that The Internet, out of town superstores and the remaining fallout from the global banking crisis have had on all town centres but some are faring better than others. Ipswich seems particularly bad.

Read more

Shell Attacks!

Reading Time: < 1 minute

Toad!
…because toads are lovely really
I’ve been having some networking issues recently so I was watching the router logs on my main gateway. I was genuinely amazed by the number of attempts to hack my ssh server, every few seconds I saw another line telling me that the firewall had rejected another attempted hack.

So I started pondering what I could do to try to stop this. Perhaps I could write a little script that works out who the appropriate admin is and emails them? Yes, but I think that might cause me rather a lot of trouble. I can’t exactly send the logs to my ISP or to the Police – simply attempting to connect to a ssh server isn’t exactly strong evidence of nefarious intent.

So instead I thought I’d just publish them – each day’s log is uploaded to the downloads section as a text file.
If I get bored and fancy writing some php then I’ll start stuffing them into a database so we can run some basic analysis on them – produce graphs and run some analysis.

The “Within” Pattern in C#

Reading Time: 2 minutes

Ziggy StardustHere’s a little pattern that I use a lot. It happens quite often that I need to perform a set of tasks in C# that are different, but all need the same set up and tear down.

Now I could use two methods, e.g.

object someResource=PerformSetUp(someParameters);
try
{
    someTask(someResource);
    if(someCondition)
        reportSomeError();
    anotherTask();
    someResource.Conclude("It was aliens");
}
finally
{
    PerformTearDown(someResource);
}

That’s a totally reasonable way of doing it. There are a few disadvantages though, it splits up the allocation and release of resources which can lead to tracking problems. It’s also a pain if you’ve got several things that need to be set up and torn down. You either end up with a lot of local variables flying around all the time or you make an object to hold the resources you need.

So I tend to use a method to do both the set up and tear down and pass into it an Action.

So the above example would become;


WithinSetUpAndTearDown((someResource)=>
{
    someTask(someResource);
    if(someCondition)
        reportSomeError();
    anotherTask();
    someResource.Conclude("It was aliens");
});

WithinSetUpAndTearDown is defined thus;

void WithinSetUpAndTearDown(Action<object width="300" height="150"> action)
{
    Handle someHandle = new someHandle.Open();
    try
    {
        using(object someResource = someDisposableAllocation(someHandle))
        {
            action(semeResource);
        }
    }
    finally
    {
        someHandle.Close();
    }
}

It’s sort of a user-defined version of the using keyword.Of course it has other uses too, there’s one particular applications that all share a particularly complex set of error handling conditions. I’ve used this pattern there too.

There is one application where I use this pattern rather a lot. C#’s “lock” keyword is really handy if you’re operating in a multithreaded environment, its big drawback is that there’s no way of specifying a timeout. This is important for deadlock resolution – a deadlock always means there’s a bug somewhere but if your concurrent operations just sit there deadlocked it may cause some pretty serious problems that you may not be aware of for some time. So you want to know. Rather than use the “lock” keyword then you can use this…

public static void WithObjectLocked(object lockMe, int waitMsecs, Action action)
{
    if(null == lockMe)
        throw new ArgumentNullException("lockMe");
    if(0 >= waitMsecs)
        throw new ArgumentException("waitMsecs must be greater that zero");

    bool gotLock = Monitor.TryEnter(lockMe, waitMsecs);
    if (!gotLock)
        throw new TimeoutException("Timeout of [" + waitMsecs.ToString() + "] milliseconds exceeded");
    try
    {
        action();
    }
    finally
    {
        Monitor.Exit(lockMe);
    }
}

Its functionality is identical to that of “lock”, but it will throw a “TimeoutException” if the lock cannot be obtained within the specified time. This at least gives you a fighting chance of finding operations that are taking far too long or are deadlocked.

I Never Could Get the Hang of Thursdays

Reading Time: 2 minutes
Made on  a Friday
Made on a Friday

I’ve just reached a natural point where I should release some software to the customer, but I can’t. I can’t because it’s a Friday and I don’t release software on a Friday.

This might at first seem like an odd little foible or possibly the symptom of some form of obsessiveness disorder but actually there’s method behind my madness.

There’s something psychologically important about Friday and about the concept of a week. If there’s no reason to set a deadline otherwise then it tends to be “by the end of the week of…” and that means Friday afternoon.
It’s clean, it’s logical. The product will arrive on the Friday night so the customer’s team can come in fresh from the weekend and start working with it on the Monday morning.

The problem is that this means finishing the product on a Friday afternoon. Of course we are all professional, but with all the professionalism in the world concentration levels aren’t at their highest on Friday afternoon. What’s more it’s the time a lot of people try to knock off early and it’s certainly the day when fewest people are prepared to stay late.

To exacerbate things software development itself is famously difficult to predict but rarely do people overestimate time. There’s usually a lot of pressure as the deadline approaches and anything that reasonably can be dropped, de-scoped or downscaled is.

Releasing a piece of software – and doing so properly – is not a trivial task. There are a lot of hoops to jump through to make sure that everything is watertight. Mistakes can be costly too, even if they’re just in the admin. Record the version number of a component incorrectly and you might spend days trying to reproduce a bug in the wrong place.
Although it’s mostly admin, there’s a lot of stuff that can go wrong with releasing software and it all takes time to fix.

The net result of the above is that delay tends to knock on into delay and by Friday morning there’s usually so much to do it’s only just theoretically possible if everything goes according to plan.

It is, of course, inevitable that it won’t.

So 2 minutes before the end of Friday you kick out the door what you have on the bench. It’s sort of the right shape. You then go to the pub and celebrate prematurely, because on Monday morning the issues will start getting logged because you went and spoiled a great piece of work by rushing it at the last minute.

So I try to release software on a Tuesday. Logically it makes no difference, Monday is one business day after Friday just as Wednesday is one business day after Tuesday but a lot more people are willing to stay late on a Tuesday and there’s something about not having the weekend there that makes customers much more willing to accept delivery on Wednesday lunch time than Monday lunch time.

I’ve been doing this for about 10 years now, it works really well.

[Written on a Friday, published on a Monday]

The Tube That Rocks the Remote Working World

Reading Time: 2 minutes

Slash And Myles Rock!Moose and JRock of UK national (and international) radio station Team Rock Radio had a crazy idea. They were going to broadcast the breakfast show whilst on the Tube.
To be precise they were going to travel from their bunker in a secret location under an an anonymous office building in Central London to Westfield Shopping Centre in Stratford using the London Underground system.
They planned to travel during the songs and do the links live from the stations using the WiFi.

Today they’re doing it. In fact they’ve just left Liverpool Street Station. This might seem like just another silly radio stunt but actually there’a very serious point behind it for them, but moreover for a good proportion of our working population.

Until recently it’s been really difficult and expensive to arrange communications of the kind of quality that it would require to broadcast a live radio show.

So Moose and JRock wanted to prove – or perhaps really just test – the fact that this is no longer the case.

The programme isn’t over yet, but thus far it’s worked really rather well. There have been a few audio glitches, but nothing serious.

This proves to the Team Rock organisation that they can be much more responsive in covering events and festivals and indeed broadcast from anywhere in the world as long as they have a half reasonable Internet connection – which today isn’t a difficult thing to obtain.

The wider implications though are significant. I’ve been a remote worker for a little over 3 years so I know that technology has improved such that remote working (and indeed home working) is entirely possible for Software Developers like me. I also happen to work for the Computer Science Department at The University of Hull so they understand it too.

Not everyone happens to be a technologist or indeed work for a technical employer. What Moose and JRock have done is to demonstrate very publicly and very effectively to a much wider audience that the technology for remote working is now there.

I really hope this has the impact that it should.

PS: They’ve now added the show to their On-Demand section.

Using Rank In Case in SQL Server 2005, 2008, 2012, 2014 etc.

Reading Time: < 1 minute

Here’s a quick tip for a Friday.

I have a table that records the history of certain items. I wanted to get a list of the updates between two dates – that’s easy enough. The problem was that, for each item, I also wanted the last change before the start of the time period.

A standard way to get the latest update for each item in a history table is use a nested pair of queries, the inner one using the RANK function to provide an order. The outer query then selects all the records with a RANK of 1 as these are the latest (or the earliest, or maximum, or minimum, depending on the order specified).

e.g.

select * from (
    select ml.*,rank() over(partition by itemID order by [timeColumn] desc) rk
    from someHistoricTable ml ) T1
    where rk=1

I started wondering if I could make the inner query return the ranking for only part of its result set.

What I wanted was a resultset that gave me all the records after a given date and a ranked list of those prior to it. I wondered if I could us the RANK function within a CASE statement.
Yes, apparently you can…

select * from (
        select ml.*,
        case
            when [time] < @startTime
                then rank() over(partition by [itemID] order by [timeColumn] desc)
            else 1
            end rk
        from someHistoricTable ml
        where [timeColumn] < @stopTime ) T1
    where rk=1 order by [time] desc

The case / rank bit splits the time, it ranks records that were earlier than the @startTime whereas if they’re later it just returns ‘1’. The outer select then just takes all rows that have an ‘rk’ of 1, these being the records that are either during the time period or the highest ranked row before.

Coding Exercise Silliness

Reading Time: 4 minutesIt happens quite often that we assign a simple programming task for one reason or another. One thing we find is that our victims often look for the “right” answer when in reality there isn’t one. Sure some answers are better than others but even that can be subjective.

So whilst a group were busy scribbling away I thought I’d demonstrate the point by doing the same problem… in as many different ways as I could think of.

The Task

The task we set them was;

Given the string “We Wish You A Merry Xmas”, output a list of the unique letters.

So I put the string into a string variable called “leString” and got to work. My favourite implementation is this;

Console.WriteLine(String.Concat(leString.ToLower()
    .Where(x => char.IsLetter(x)).Distinct()));

If you understand a little about LINQ, particularly the extension method syntax, then this is about as clear and concise as you can get. It’s possibly not the fastest way of doing it, but speed of execution is not the only concern.

I then tried to guess how I thought a junior programmer would approach it. The result is this.

HashSet<char> alreadyThere = new HashSet<char>();
foreach (char c in leString)
{
    char lc = char.ToLower(c);
    if (char.IsLetter(c) && !alreadyThere.Contains(lc))
    {
        alreadyThere.Add(lc);
        Console.Write(lc);
    }
}
Console.WriteLine();

This is perfectly valid and indeed relatively quick. I don’t think it’s quite as obvious what it’s doing as using .Distinct(), but if you’re not a LINQ Ninja then it’s OK.

I was right here in that it’s this kind of approach that most of them went for, although we saw arrays, lists and dictionaries used to store the list of chars and nobody realised that you could output the chars as you went so the real results looked more like this;

List<char> alreadyThere = new List<char>();
foreach (char c in leString)
{
    char lc = char.ToLower(c);
    if (char.IsLetter(c) && !alreadyThere.Contains(lc))
        alreadyThere.Add(lc);
}
foreach (char c in alreadyThere)
    Console.Write(c);
Console.WriteLine();

There’s a fundamentally different approach that we can take here. Essentially we want to remove duplicates. If we sort the string then all the duplicates will be next to each other. So all we have to do is iterate through the sorted string and avoid outputting the same char twice.
This is pretty easy to implement.

char lastchar = (char)0;
foreach (char c in leString.ToLower()
    .Where(x => char.IsLetter(x)).OrderBy(n => n))
{
    if (lastchar != c)
        Console.Write(c);
    lastchar = c;
}
Console.WriteLine();

It’s a bit of a nasty mix of LINQ and more traditional iteration but I still think it’s pretty clear. We take the string, eliminate anything that isn’t a letter and then sort the result.
We then iterate over the result. If this iteration’s char is different from the last iteration’s char then we output it. If it’s the same, we don’t.

You can also implement this using a for loop, but you need to make the sorted result into an array or list so that you can get a count. This is all rather convoluted so the foreach is preferable in my opinion.

You can implement it in LINQ too. The problem is how to assign “lastChar” after it’s tested. I’ve used a nasty little hack here – in C# an assignment actually has a return type that is the value of the assignment. So lastchar = p actually returns the value of p as well as assigning it to lastchar – hence the final .Select.

char lastchar = (char)0;
Console.WriteLine(String.Concat(leString.ToLower()
    .OrderBy(n => n)
    .Where(x => char.IsLetter(x) && lastchar != x)
    .Select(p => lastchar = p)));

This isn’t good code, it’s not really clear what it’s doing.

Here’s another slightly leftfield LINQ based approach. In C# the ++ actually returns a value, if it’s used postfix (i.e. i++) the value returned is what i was before it was incremented. If it’s used prefix (i.e. ++i) then it’s the value after it’s incremented.
We can use this little gem to index into a counter array…

int[] hits = new int[26];
Console.WriteLine(String.Concat(leString.ToLower()
    .Where(x => char.IsLetter(x) && 0 == hits[(byte)x - (byte)'a']++)));

This assumes that the string is 7 bit ASCII. It goes horribly wrong if it’s unicode. This should be a pretty quick implementation though.

Here’s another 1 liner (if you ignore the declaration of a HashSet). This works in a similar way to the previous example but is unicode safe. A HashSet is a collection of unique values, duplicates are not allowed. The HashSet.Add() method returns false if the entity you’re trying to add is already in the collection. So we can use that in the same way we used the array and the ++ operator above…

HashSet<char> alreadyThere = new HashSet<char>();
Console.WriteLine(String.Concat(leString.ToLower()
    .Where(x => char.IsLetter(x) && alreadyThere.Add(x))));

If you’re a fully paid up, card carrying lunatic you can of course do it parallel…

ConcurrentDictionary<char, int> alreadyThereDict = new ConcurrentDictionary<char, int>();
Console.WriteLine(String.Concat(leString.AsParallel()
    .Select(c=>char.ToLower(c))
    .Where(x => char.IsLetter(x) && alreadyThereDict.TryAdd(x,1))));

Rather irritatingly there’s no such thing as a ConcurrentHashSet so we have to use ConcurrentDictionary but the approach is identical.

Another different approach is this.

Console.WriteLine(String.Concat(leString.ToLower()
    .Intersect("abcdefghijklmnopqrstuvwxyz")));

The result of intersecting a string with a set that contains all possible characters must be a set containing the characters that are in the string. Clearly this isn’t unicode safe either, although it’s better than the array indexing example because this won’t blow up, it just won’t work with unicode.
The advantage of this approach however is that it very clearly defines what the dictionary of permissible character is. So if you wanted to define a slightly different dictionary you could – say consonants for instance.

Finally we return to the original method, but we do it slightly differently. Most of the functionality we think of as LINQ is provided through extension methods defined in the “Enumerable” class.
You don’t need to call an extension method as an extension method at all, you can in fact call it as a static method of its defining class. Thus we can perform the entire operation in a functional manner.

Console.WriteLine(
  String.Concat(
    Enumerable.Distinct(
      Enumerable.Where(
        Enumerable.Select(leString, char.ToLower), 
      char.IsLetter))));

If you’re not familiar with this style, basically you read it from the middle out. So we take “leString” and call Enumerable.Select on it, passing char.ToLower as the predicate for selection. The result of this operation is then used as the first parameter to the call to Enumerable.Where. The other parameter to the Where call is the predicate char.IsLetter. The result of this operation is then used as the first parameter to a call to Enumerable.Distinct and the result of that goes to String.Concat.
As with many things programming it’s horses for courses and this is not the right horse. It’s useful however to know that it can be done like this.

Over to you…

Naturally this isn’t an exhaustive list – it’s just the ones I happened to think of, if you can think of another way let me know!