Technology

Reading Time: < 1 minute

I’ve been having some networking issues recently so I was watching the router logs on my main gateway. I was genuinely amazed by the number of attempts to hack my ssh server, every few seconds I saw another line telling me that the firewall had rejected another attempted hack.

So I started pondering what I could do to try to stop this. Perhaps I could write a little script that works out who the appropriate admin is and emails them? Yes, but I think that might cause me rather a lot of trouble. I can’t exactly send the logs to my ISP or to the Police – simply attempting to connect to a ssh server isn’t exactly strong evidence of nefarious intent.

So instead I thought I’d just publish them – each day’s log is uploaded to the downloads section as a text file.
If I get bored and fancy writing some php then I’ll start stuffing them into a database so we can run some basic analysis on them – produce graphs and run some analysis.

Reading Time: 2 minutes

Here’s a little pattern that I use a lot. It happens quite often that I need to perform a set of tasks in C# that are different, but all need the same set up and tear down.

Now I could use two methods, e.g.

object someResource=PerformSetUp(someParameters);
try
{
    someTask(someResource);
    if(someCondition)
        reportSomeError();
    anotherTask();
    someResource.Conclude("It was aliens");
}
finally
{
    PerformTearDown(someResource);
}

That’s a totally reasonable way of doing it. There are a few disadvantages though, it splits up the allocation and release of resources which can lead to tracking problems. It’s also a pain if you’ve got several things that need to be set up and torn down. You either end up with a lot of local variables flying around all the time or you make an object to hold the resources you need.

So I tend to use a method to do both the set up and tear down and pass into it an Action.

So the above example would become;

WithinSetUpAndTearDown((someResource)=>
{
    someTask(someResource);
    if(someCondition)
        reportSomeError();
    anotherTask();
    someResource.Conclude("It was aliens");
});

WithinSetUpAndTearDown is defined thus;

void WithinSetUpAndTearDown(Action<object width="300" height="150"> action)
{
    Handle someHandle = new someHandle.Open();
    try
    {
        using(object someResource = someDisposableAllocation(someHandle))
        {
            action(semeResource);
        }
    }
    finally
    {
        someHandle.Close();
    }
}

It’s sort of a user-defined version of the using keyword.Of course it has other uses too, there’s one particular applications that all share a particularly complex set of error handling conditions. I’ve used this pattern there too.

There is one application where I use this pattern rather a lot. C#’s “lock” keyword is really handy if you’re operating in a multithreaded environment, its big drawback is that there’s no way of specifying a timeout. This is important for deadlock resolution – a deadlock always means there’s a bug somewhere but if your concurrent operations just sit there deadlocked it may cause some pretty serious problems that you may not be aware of for some time. So you want to know. Rather than use the “lock” keyword then you can use this…

public static void WithObjectLocked(object lockMe, int waitMsecs, Action action)
{
    if(null == lockMe)
        throw new ArgumentNullException("lockMe");
    if(0 >= waitMsecs)
        throw new ArgumentException("waitMsecs must be greater that zero");

    bool gotLock = Monitor.TryEnter(lockMe, waitMsecs);
    if (!gotLock)
        throw new TimeoutException("Timeout of [" + waitMsecs.ToString() + "] milliseconds exceeded");
    try
    {
        action();
    }
    finally
    {
        Monitor.Exit(lockMe);
    }
}

Its functionality is identical to that of “lock”, but it will throw a “TimeoutException” if the lock cannot be obtained within the specified time. This at least gives you a fighting chance of finding operations that are taking far too long or are deadlocked.

Reading Time: 2 minutes

Moose and JRock of UK national (and international) radio station Team Rock Radio had a crazy idea. They were going to broadcast the breakfast show whilst on the Tube.
To be precise they were going to travel from their bunker in a secret location under an an anonymous office building in Central London to Westfield Shopping Centre in Stratford using the London Underground system.
They planned to travel during the songs and do the links live from the stations using the WiFi.

Today they’re doing it. In fact they’ve just left Liverpool Street Station. This might seem like just another silly radio stunt but actually there’a very serious point behind it for them, but moreover for a good proportion of our working population.

Until recently it’s been really difficult and expensive to arrange communications of the kind of quality that it would require to broadcast a live radio show.

So Moose and JRock wanted to prove – or perhaps really just test – the fact that this is no longer the case.

The programme isn’t over yet, but thus far it’s worked really rather well. There have been a few audio glitches, but nothing serious.

This proves to the Team Rock organisation that they can be much more responsive in covering events and festivals and indeed broadcast from anywhere in the world as long as they have a half reasonable Internet connection – which today isn’t a difficult thing to obtain.

The wider implications though are significant. I’ve been a remote worker for a little over 3 years so I know that technology has improved such that remote working (and indeed home working) is entirely possible for Software Developers like me. I also happen to work for the Computer Science Department at The University of Hull so they understand it too.

Not everyone happens to be a technologist or indeed work for a technical employer. What Moose and JRock have done is to demonstrate very publicly and very effectively to a much wider audience that the technology for remote working is now there.

I really hope this has the impact that it should.

PS: They’ve now added the show to their On-Demand section.

Reading Time: < 1 minute

Here’s a quick tip for a Friday.

I have a table that records the history of certain items. I wanted to get a list of the updates between two dates – that’s easy enough. The problem was that, for each item, I also wanted the last change before the start of the time period.

A standard way to get the latest update for each item in a history table is use a nested pair of queries, the inner one using the RANK function to provide an order. The outer query then selects all the records with a RANK of 1 as these are the latest (or the earliest, or maximum, or minimum, depending on the order specified).

e.g.

select * from (
    select ml.*,rank() over(partition by itemID order by [timeColumn] desc) rk
    from someHistoricTable ml ) T1
    where rk=1

I started wondering if I could make the inner query return the ranking for only part of its result set.

What I wanted was a resultset that gave me all the records after a given date and a ranked list of those prior to it. I wondered if I could us the RANK function within a CASE statement.
Yes, apparently you can…

select * from (
        select ml.*,
        case
            when [time] < @startTime
                then rank() over(partition by [itemID] order by [timeColumn] desc)
            else 1
            end rk
        from someHistoricTable ml
        where [timeColumn] < @stopTime ) T1
    where rk=1 order by [time] desc

The case / rank bit splits the time, it ranks records that were earlier than the @startTime whereas if they’re later it just returns ‘1’. The outer select then just takes all rows that have an ‘rk’ of 1, these being the records that are either during the time period or the highest ranked row before.

Reading Time: 4 minutesIt happens quite often that we assign a simple programming task for one reason or another. One thing we find is that our victims often look for the “right” answer when in reality there isn’t one. Sure some answers are better than others but even that can be subjective.

So whilst a group were busy scribbling away I thought I’d demonstrate the point by doing the same problem… in as many different ways as I could think of.

The Task

The task we set them was;

Given the string “We Wish You A Merry Xmas”, output a list of the unique letters.

So I put the string into a string variable called “leString” and got to work. My favourite implementation is this;

Console.WriteLine(String.Concat(leString.ToLower()
    .Where(x => char.IsLetter(x)).Distinct()));

If you understand a little about LINQ, particularly the extension method syntax, then this is about as clear and concise as you can get. It’s possibly not the fastest way of doing it, but speed of execution is not the only concern.

I then tried to guess how I thought a junior programmer would approach it. The result is this.

HashSet<char> alreadyThere = new HashSet<char>();
foreach (char c in leString)
{
    char lc = char.ToLower(c);
    if (char.IsLetter(c) && !alreadyThere.Contains(lc))
    {
        alreadyThere.Add(lc);
        Console.Write(lc);
    }
}
Console.WriteLine();

This is perfectly valid and indeed relatively quick. I don’t think it’s quite as obvious what it’s doing as using .Distinct(), but if you’re not a LINQ Ninja then it’s OK.

I was right here in that it’s this kind of approach that most of them went for, although we saw arrays, lists and dictionaries used to store the list of chars and nobody realised that you could output the chars as you went so the real results looked more like this;

List<char> alreadyThere = new List<char>();
foreach (char c in leString)
{
    char lc = char.ToLower(c);
    if (char.IsLetter(c) && !alreadyThere.Contains(lc))
        alreadyThere.Add(lc);
}
foreach (char c in alreadyThere)
    Console.Write(c);
Console.WriteLine();

There’s a fundamentally different approach that we can take here. Essentially we want to remove duplicates. If we sort the string then all the duplicates will be next to each other. So all we have to do is iterate through the sorted string and avoid outputting the same char twice.
This is pretty easy to implement.

char lastchar = (char)0;
foreach (char c in leString.ToLower()
    .Where(x => char.IsLetter(x)).OrderBy(n => n))
{
    if (lastchar != c)
        Console.Write(c);
    lastchar = c;
}
Console.WriteLine();

It’s a bit of a nasty mix of LINQ and more traditional iteration but I still think it’s pretty clear. We take the string, eliminate anything that isn’t a letter and then sort the result.
We then iterate over the result. If this iteration’s char is different from the last iteration’s char then we output it. If it’s the same, we don’t.

You can also implement this using a for loop, but you need to make the sorted result into an array or list so that you can get a count. This is all rather convoluted so the foreach is preferable in my opinion.

You can implement it in LINQ too. The problem is how to assign “lastChar” after it’s tested. I’ve used a nasty little hack here – in C# an assignment actually has a return type that is the value of the assignment. So lastchar = p actually returns the value of p as well as assigning it to lastchar – hence the final .Select.

char lastchar = (char)0;
Console.WriteLine(String.Concat(leString.ToLower()
    .OrderBy(n => n)
    .Where(x => char.IsLetter(x) && lastchar != x)
    .Select(p => lastchar = p)));

This isn’t good code, it’s not really clear what it’s doing.

Here’s another slightly leftfield LINQ based approach. In C# the ++ actually returns a value, if it’s used postfix (i.e. i++) the value returned is what i was before it was incremented. If it’s used prefix (i.e. ++i) then it’s the value after it’s incremented.
We can use this little gem to index into a counter array…

int[] hits = new int[26];
Console.WriteLine(String.Concat(leString.ToLower()
    .Where(x => char.IsLetter(x) && 0 == hits[(byte)x - (byte)'a']++)));

This assumes that the string is 7 bit ASCII. It goes horribly wrong if it’s unicode. This should be a pretty quick implementation though.

Here’s another 1 liner (if you ignore the declaration of a HashSet). This works in a similar way to the previous example but is unicode safe. A HashSet is a collection of unique values, duplicates are not allowed. The HashSet.Add() method returns false if the entity you’re trying to add is already in the collection. So we can use that in the same way we used the array and the ++ operator above…

HashSet<char> alreadyThere = new HashSet<char>();
Console.WriteLine(String.Concat(leString.ToLower()
    .Where(x => char.IsLetter(x) && alreadyThere.Add(x))));

If you’re a fully paid up, card carrying lunatic you can of course do it parallel…

ConcurrentDictionary<char, int> alreadyThereDict = new ConcurrentDictionary<char, int>();
Console.WriteLine(String.Concat(leString.AsParallel()
    .Select(c=>char.ToLower(c))
    .Where(x => char.IsLetter(x) && alreadyThereDict.TryAdd(x,1))));

Rather irritatingly there’s no such thing as a ConcurrentHashSet so we have to use ConcurrentDictionary but the approach is identical.

Another different approach is this.

Console.WriteLine(String.Concat(leString.ToLower()
    .Intersect("abcdefghijklmnopqrstuvwxyz")));

The result of intersecting a string with a set that contains all possible characters must be a set containing the characters that are in the string. Clearly this isn’t unicode safe either, although it’s better than the array indexing example because this won’t blow up, it just won’t work with unicode.
The advantage of this approach however is that it very clearly defines what the dictionary of permissible character is. So if you wanted to define a slightly different dictionary you could – say consonants for instance.

Finally we return to the original method, but we do it slightly differently. Most of the functionality we think of as LINQ is provided through extension methods defined in the “Enumerable” class.
You don’t need to call an extension method as an extension method at all, you can in fact call it as a static method of its defining class. Thus we can perform the entire operation in a functional manner.

Console.WriteLine(
  String.Concat(
    Enumerable.Distinct(
      Enumerable.Where(
        Enumerable.Select(leString, char.ToLower), 
      char.IsLetter))));

If you’re not familiar with this style, basically you read it from the middle out. So we take “leString” and call Enumerable.Select on it, passing char.ToLower as the predicate for selection. The result of this operation is then used as the first parameter to the call to Enumerable.Where. The other parameter to the Where call is the predicate char.IsLetter. The result of this operation is then used as the first parameter to a call to Enumerable.Distinct and the result of that goes to String.Concat.
As with many things programming it’s horses for courses and this is not the right horse. It’s useful however to know that it can be done like this.

Over to you…

Naturally this isn’t an exhaustive list – it’s just the ones I happened to think of, if you can think of another way let me know!

Reading Time: 3 minutesSQL Servers are great general purpose tools. They’re popular because they perform OK at just about everything you could want to do with data.

If you want to do something specialised however then there are almost always better techniques. Recently I needed to get medium sized sets of records identified by key, dispersed at random over an essentially read-only data set.

MS SQL Server is OK at this, but there are a bunch of products out there that are far better at this task. At a basic level though it’s not actually a difficult thing to do. It’s just a key-value lookup after all. So I wondered if in a few lines of code I could out-perform SQL Server for this one specific task.

Essentially all I did was to dig up the ISAM file. All that really means is that the main data is serialized to a file. All we store in memory is the ID (that we search for) and it’s location within the file. I then implemented a balanced b-tree to store the relationship between the ID and the file location. You could use a Dictionary<Key,Value> for the latter, it would be a bit slower but it would work. You really don’t need to write a lot of code to implement this.

So in ISAM vs. SQL Server I have on my side;

• A simple, really fast search algorithm
• No need for IPC.
• No need to parse the query
• No need to marshal writes, manage locks or indeed worry about concurrency at all

SQL Server has on its side;

• Huge expertise and experience in optimising queries and indexes
• Super-advanced caching meaning it doesn’t face the same I/O problems

Before the days of the SSD I’d say attempting this was pretty insane. The main problem with the performance of a mechanical disk is the head seek time. You can get data to and from the disk pretty fast provided that the head is in the right place. Moving the head however is massively slow in comparison. So if you have a large file and you need to move the head several times then you’re going to lose a lot of time against a system that has an effective cache strategy.
Half the battle with an ISAM file was to get the records that were going to be read together physically close to each other in the file so that the heads didn’t move much and the likelihood of a disk cache hit was much higher. It’s this kind of management where complex RDBMSs will always have the upper hand.

However SSDs have no head seek time. It doesn’t matter if you need to read from many locations. This rather levels the playing field.
Interestingly I did notice a small speed improvement when I implemented a parallel ID lookup and sorted the results so the FileStream read in sequence from first in the file to last. I’m guessing that this is down to the read-ahead cache (which is still faster than the real storage media).

So did I make it? Can a simple technique from yesteryear embarrass today’s latest and greatest? I wrote a little console app to test…

Not bad, SQL Server is 15% faster which is a pretty reasonable result. After all I couldn’t really expect to beat SQL Server with just a few lines of .NET, right?
Yeah, I’m not beaten that easily. So I cracked open Visual Studio’s Performance Explorer and found an obvious bottleneck. It turns out that .NET’s BinaryFormatter is dog slow. I tossed it out and replaced it with ProtoBuf-Net.

Oh yes, that’s much better. This is pretty consistent too. I was using SQL Server localdb which actually seems to be a reasonable analogue. It might be interesting to see it against an enterprise level installation. I suspect that the additional network latency will compensate for the additional performance at it will perform about the same.

So in other words, yes if you have a very specific task and you know what you’re doing then you can outperform the big boys.

So I suppose I should publish the source – but right now it’s horrible. Its only purpose was to prove that I could do it. I’m intending to tidy it up and post it in a different article.

Reading Time: 3 minutes

I’ve just been bitten. It’s one of the standard strap-lines of software development teaching – do it properly first time, don’t hack it and think you can fix it later. When the deadline is looming however sometimes we have to make a business decision;

• Do we drop the feature (or some other feature)?
• Do we miss the deadline?
• Do we hack it in the full knowledge that we may be bitten later?

Fortunately for us software developers, technical debt is beginning to be understood by business managers. I explain it to non-techies like this;

I could hack it and get the feature in, but if I do that might be storing up problems for the future. Next time we want to add a feature in that area or even fix a bug we might have to unpick the hack and implement it properly before we even start on new work.

Hacking it now is borrowing time from the future – at some point we’ll have to pay it back and we’ll have to pay a wedge of interest too.

A few years ago I spotted a design problem; we’d assumed that something was always related to a location. In some circumstances however it could actually be related to a vehicle.

We should have changed the database to account for this, but that meant changing pretty much every component of an n-tier system and the deadline was busy whizzing past our ears.

There’s no way we could drop the feature so I had to find a solution. What I noticed was that I could cast a vehicle ID safely into a location ID without any possibility of coincidence. We could then detect in the DAL if it was actually a vehicle and generate a fake location record that actually contained all the information we needed.

A few years later and now we’re noticing that customers are putting a whole load more records in the location table than we’d initially thought and the indexing performance of the ID type is poor. So we want to change the type to something that indexes better. None of the options available allow us to cast a vehicle ID to a location ID any more.

Since those early days the problem has been exacerbated by the fact that more and more code has been piled in on the assumption that you can safely cast a vehicle ID to a location ID.

So we’ve now got a considerable amount of code to unpick, we’re going to have to do the redesign and reimplementation work that we (perhaps) should have done in the first place, then we can start looking at improving the index performance.

In situations like this it’s easy to look back and curse the past decision. Go into any software development office and you’ll almost certainly find a developer (or several) swearing at the screen, decrying previous decisions as “stupid”, “short-sighted”, etc.

I know I’ve made mistakes, I know there are situations where I’ve chosen to incur technical debt where there were better alternatives available. On this occasion though I made the right decision – we have time to repay this debt now. Back then we most definitely did not.

So “never hack it” isn’t a rule, it’s a guideline. But you have to be aware of the business consequences if you choose not to follow it.