I read an interesting article in the New York Times that said For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights. For anyone that has done any sort of science this should not be a surprise. Science is the product of a relentless sort of focussed stupidity combined with intelligent insight and this sort of basic work is essential to the first part of the process.
Anyone who has done a science experiment, even a simple high school one, will understand that the act of data gathering and sorting is a menial task that cannot be simply automated. As a scientist, you are often playing the role of a highly trained monkey repeatedly doing the same tasks. Normally this sort of activity would be one that could be quickly automated but there turns out to be so much art in the science that you are often the only trained monkey who can do it.
Whether, as in the article, it’s different words meaning the same thing for a company providing information on drug side effects needing to know that “drowsiness,” “sleepiness” and “somnolence” all meant the same thing or whether it’s a researcher analysing EEG patterns determining whether the activity is real or just an artefact, both situations are remarkably resistant to complete automation. Even if you could automate everything, it’s only when you’ve ploughed through the dataset that you start to have an idea of what’s going on – and that’s ignoring the green jelly bean problem of post hoc analysis.
For my PhD I manually compared 1.1 million EEG patterns and that was after the computer had processed and tentatively categorised the data. I’d go to sleep seeing wiggly lines across the back of my eyelids but that was the price to pay for correctness that no computational analysis could match.
It’s fantastic that there are start-ups out there looking to solve this problem of data wrangling (as not everyone has an army of PhD students, postdocs or other slave labour to do this for them) but it’s an exceptionally tough problem that I don’t expect to see solved any time soon.
Java 8 was released recently and one of the much heralded features is Default methods. Simply put, these allow you to inject functionality to classes via interfaces. This is a new feature in that prior to Java 8, Interfaces could only have method signatures and not have the implementation and only Abstract classes could have method implementations but that has now changed.
Here’s a simple example that prints out “In InterfaceOne”
In Scala, another JVM language, you can do something similar but more awesome with Scala’s traits (its version of interfaces) you can override similar implementations and have a version of multiple inheritance. This example prints out “In TraitTwo”
The secret that avoid that diamond of death is that whichever trait is declared last wins. If I’d swapped around the order that the traits were declared then it would have printed “In TraitOne”
What is even nicer in Scala is that you can declare your class with traits when you instantiate it. You don’t have to declare it at the compile time of the class. This means that you have a powerful way to extend functionality of classes but without the insanity of monkey patching. ThisThe below example also prints out “In TraitTwo” but the class does not extend any trait. This of course is Scala’s cake pattern where you can mix in the traits.
What is slightly disappointing with Java 8 is that you cannot mimic this behaviour. If you try to do this in Java 8, you get a nice compile time error telling us the class inherits unrelated defaults.
I wonder why they chose not to support this. The cake patterns seems like a feature that adds flexibility without being able to shoot yourself in the foot too much.
Just as an experiment I thought I would try using DuckDuckGo for searching. In an increasingly big brother world I like that they don’t track you and I like that they only serve requests securely (requests to http://duckduckgo.com get a 301 permanent redirect to https://duckduckgo.com). They also offer a proxy service and the search results seem to be pretty good.
I came across an interesting post here on why Betfair should fear whatsapp. Strange bedfellows for an article but the general thrust of the piece is that a chat application is similar to a financial exchange, in this case Betfair, and could theoretically disrupt its business model. I have to be honest and say it’s something I’ve never thought of.
If you read what’s said then you might think that this is so. The author goes into some detail about the similarities and given the depth of knowledge in some of the other posts on his site you might think that he’s got a point. He even says straight out that “A betting exchange is really just a chat application – in place of messages, you have bets; in place of chat groups, you have markets; and in place of the chat transcript, you have the order book“.
I did some training recently that covered the importance of tone and body language in communication and I was reminded of this video of a patient suffering from an extreme form of fluent Aphasia.
This is caused by a stroke affecting part of the temporal lobe that deals with the conversion of “words in your head” to actual speech but what is incredible to me is the way that the gentleman uses his intonation, the pitch of his voice and his body language to communicate when he can only say the word “tono”.
What’s more amazing is when he is asked to count to 20 about 1m45s in. What did you think would happen?
If you haven’t seen this awesome parody video already, you need to. It’s a great example of what it can sometimes feel like as an expert in a meeting with non-experts.
Buzzword bingo, massive management overhead, ignorance of implementation details, solutioneering, management shutting down needed discussion, condescension, it has pretty much the full package of the worst the world can throw at you.
Of course it’s a taken to the furthest extremes otherwise it wouldn’t be funny but if you’re don’t think there’s a grain of truth to it, ask any technical person if they’ve ever been in this sort of situation and see what they say. It’s not that techies are geniuses and everyone else is an idiot, it’s just that without sufficient understanding almost any area of technical knowledge can be over-simplified to a level that defies reality. This is true for science as much as it is for technology.
The thing is, there another message other than experts have to suffer dealing with idiots all the time; it’s that experts need to be better at teaching and communicating to stop this scenario from happening in the first place. Continue reading →
After a brief hiatus (who are we kidding, it’s been nearly a year!) I’m going to be picking up the writing again.
Moving to Australia has been slightly more involved than I thought and that along with a new role has kind of eaten up my time. There have been so many cool thing going on recently that I’ve wanted to write about and now things have calmed down a touch on the home front I should be able to post a bit more often.
If you know me it’s no secret that over the last few months I’ve been looking for a new challenge (stay tuned for more on that later!). While I was doing this I was struck by just how many organisations still don’t have blogs written by their technical department.
At a time where social media is EVERYWHERE this seems almost perverse. With the dependence on the web and mobile almost every company is now an internet company to some degree. An airline is in the business of moving people around in flying boxes but they have an enormous technical infrastructure. Banks are in a similar position and so are pretty much every company that sells something. Okay, not everyone is going to have a large development organisation but almost nobody has zero and because of this all companies will suffer similar issues around:
How do you attract quality technical staff?
How do you communicate complex technical issues that have a fundamental impact on the business with a wider audience of laypeople?
How to inspire and fulfil technical staff?
These are notorious hard problems to solve. High quality tech staff are far more productive than average tech staff and you only want to hire the best. Technologists are notoriously bad at communicating their domain to non-technical people, both internally and externally, and also tend to be driven by different motivations.
So why does having a technical blog help?
It’s free advertising for recruiting technical staff
Transparency drives quality both in the product and in the engineering culture
You are upskilling your technical staff for very little cost and can produce content for talks at conferences/whitepapers
You are engaging with your community and raising your profile which is turn means free brand marketing
Admittedly it’s only my experienceand not a rigorous survey but there does seem to be a fairly strong correlation between companies with a good engineering culture, i.e. places I would want to work at, and having a good tech blog. If your technical staff don’t have a voice what are we to think? That you don’t value or trust them or that the standard of your technology division is fairly low and you don’t want to expose just how bad they are? Either way, my interest in working there is going to be non-existent.
It might sound harsh but if you are in a technical leadership position you owe it to your company to have a blog written by your technology staff. I would go as far as to say that every technical person should have “write a blog post” in their yearly objectives.
If your technical department does not have a blog you might as well be announcing to the world that you are second tier and are happy to stay that way.