All Science Requires Janitor Work

I read an interesting article in the New York Times that said For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights. For anyone that has done any sort of science this should not be a surprise. Science is the product of a relentless sort of focussed stupidity combined with intelligent insight and this sort of basic work is essential to the first part of the process.

Anyone who has done a science experiment, even a simple high school one, will understand that the act of data gathering and sorting is a menial task that cannot be simply automated. As a scientist, you are often playing the role of a highly trained monkey repeatedly doing the same tasks. Normally this sort of activity would be one that could be quickly automated but there turns out to be so much art in the science that you are often the only trained monkey who can do it.

Whether, as in the article,  it’s different words meaning the same thing  for a company providing  information on drug side effects needing to know that “drowsiness,” “sleepiness” and “somnolence” all meant the same thing or whether it’s a researcher analysing EEG patterns determining whether the activity is real or just an artefact, both situations are remarkably resistant to complete automation. Even if you could automate everything, it’s only when you’ve ploughed through the dataset that you start to have an idea of what’s going on – and that’s ignoring the green jelly bean problem of post hoc analysis.

For my PhD I manually compared 1.1 million EEG patterns and that was after the computer had processed and tentatively categorised the data.  I’d go to sleep seeing wiggly lines across the back of my eyelids but that was the price to pay for correctness that no computational analysis could match.

It’s fantastic that there are start-ups out there looking to solve this problem of data wrangling (as not everyone has an army of PhD students, postdocs or other slave labour to do this for them) but it’s an exceptionally tough problem that I don’t expect to see solved any time soon.

Bad Java, no cake for you! – Java 8 default methods versus Scala traits

Java 8 was released recently and one of the much heralded features is Default methods. Simply put, these allow you to inject functionality to classes via interfaces. This is a new feature in that prior to Java 8, Interfaces could only have method signatures and not have the implementation and only Abstract classes could have method implementations but that has now changed.

Here’s a simple example that prints out “In InterfaceOne”

In Scala, another JVM language, you can do something similar but more awesome with Scala’s traits (its version of interfaces) you can override similar implementations and have a version of multiple inheritance. This example prints out “In TraitTwo”

The secret that avoid that diamond of death is that whichever trait is declared last wins. If I’d swapped around the order that the traits were declared then it would have printed “In TraitOne”

What is even nicer in Scala is that you can declare your class with traits when you instantiate it. You don’t have to declare it at the compile time of the class. This means that you have a powerful way to extend functionality of classes but without the insanity of monkey patching. ThisThe below example also prints out “In TraitTwo” but the class does not extend any trait. This of course is Scala’s cake pattern where you can mix in the traits.

What is slightly disappointing with Java 8 is that you cannot mimic this behaviour. If you try to do this in Java 8, you get a nice compile time error telling us the class inherits unrelated defaults.

I wonder why they chose not to support this. The cake patterns seems like a feature that adds flexibility without being able to shoot yourself in the foot too much.

 

DuckDuckGo

Just as an experiment I thought I would try using DuckDuckGo for searching. In an increasingly big brother world I like that they don’t track you and I like that they only serve requests securely (requests to http://duckduckgo.com get a 301 permanent redirect to https://duckduckgo.com). They also offer a proxy service and the search results seem to be pretty good.

I might give it a month and then report back.

Are Exchanges Just Glorified Chat Applications?

I came across an interesting post here on why Betfair should fear whatsapp. Strange bedfellows for an article but the general thrust of the piece is that a chat application is similar to a financial exchange, in this case Betfair, and could theoretically disrupt its business model. I have to be honest and say it’s something I’ve never thought of.

If you read what’s said then you might think that this is so. The author goes into some detail about the similarities and given the depth of knowledge in some of the other posts on his site you might think that he’s got a point. He even says straight out that “A betting exchange is really just a chat application – in place of messages, you have bets; in place of chat groups, you have markets; and in place of the chat transcript, you have the order book“.

So that’s a fair comparison isn’t it?

Ummm, nope and here’s why. Continue reading

Checking Logback based Logging in Unit Tests

I wrote a simple post a few years ago on unit testing your logging. Checking my logs I’ve seen that it’s always been a popular post as it appears to be something that lots of people want to do.

My previous version was based on log4j and since many people have moved on to logback I thought I would update it. So here it is :-)

And that’s it.

Tone and Body Language in Communication

I did some training recently that covered the importance of tone and body language in communication and I was reminded of this video of a patient suffering from an extreme form of fluent Aphasia.

This is caused by a stroke affecting part of the temporal lobe that deals with the conversion of “words in your head” to actual speech but what is incredible to me is the way that the gentleman uses his intonation, the pitch of his voice and his body language to communicate when he can only say the word “tono”.

What’s more amazing is when he is asked to count to 20 about 1m45s in. What did you think would happen?

Seven Red Lines And What You Can Do About It

If you haven’t seen this awesome parody video already, you need to. It’s a great example of what it can sometimes feel like as an expert in a meeting with non-experts.

Buzzword bingo, massive management overhead, ignorance of implementation details, solutioneering, management shutting down needed discussion, condescension, it has pretty much the full package of the worst the world can throw at you.

Of course it’s a taken to the furthest extremes otherwise it wouldn’t be funny but if you’re don’t think there’s a grain of truth to it, ask any technical person if they’ve ever been in this sort of situation and see what they say. It’s not that techies are geniuses and everyone else is an idiot, it’s just that without sufficient understanding almost any area of technical knowledge can be over-simplified to a level that defies reality. This is true for science as much as it is for technology.

The thing is, there another message other than experts have to suffer dealing with idiots all the time; it’s that experts need to be better at teaching and communicating to stop this scenario from happening in the first place.  Continue reading

Hello Again

After a brief hiatus (who are we kidding, it’s been nearly a year!) I’m going to be picking up the writing again.

Moving to Australia has been slightly more involved than I thought and that along with a new role has kind of eaten up my time. There have been so many cool thing going on recently that I’ve wanted to write about and now things have calmed down a touch on the home front I should be able to post a bit more often.

Hope to see you around.

Inform, Educate and Entertain

For all the reams of advice written on how to “do” social media successfully, I’ve read nothing that beats Lord Reith‘s description of what the BBC was created to do

“Inform, Educate and Entertain”

Okay, so you might not be running one of the world’s greatest broadcasting networks but if you want to engage an audience, any audience, follow this maxim and you won’t go far wrong.