Data

datalogoIt doesn’t take much to realise how the information age has effected us all from I-Pods to I-Pads, from Web forms to data protection. Information (ie the meaning we can derive from data) has been at the center of organisational processes for well over 30 years now, and whether we like it or not, our personal life is going the same way, utilising the Internet data, personal directories, modelling our retirement fund, paying parking fees online, managing tax returns via the government secure gateway. Even our daily shopping provides supermarkets like Tesco with unprecedented insight into each and every one of our behaviours (unless you pay everything in cash). The dynamic of resources, the value we give them are now global data sets, and perception of data (information) decides where that value of resources is heading.

Yet IT is still somewhat immature in regards to data, data protection, data processes, data profiling, data transformation and so on. Go into any company and ask where you can view online all the data held on you, and I’ll bet you will not find it. yet a company promises you that your personal details are safe, their data protection compliant and they are probably right there too.

I have worked with data, and data issues now for close to 20 years, and the constant is that we (both organisations and individuals) have not got our heads around how data should be treated. We still don’t understand when data becomes significant to be information, and we certainly have not been able to understand how value is generated in this space. Yet we all can “feel” that the effort is important. We all know when it goes wrong, and data is wrongly used (illegally or leading to the wrong decision).

I am currently in the process of writing a number of papers on the subject which will hopefully add to the discussions on this topic area. The main aim is to look at the complexity of data in all its forms (but mainly for individuals and organisations), translate some of the practices into more transparent terminology, so that interested party can look at this space and see some of the more complex elements involved in data, data management, data processing, data security, data protection and why all of it needs to change in line with technical capabilities.

Lets start with probably one of the more complex subjects in regards to data. Assuming you have data, and it is reasonably accurate, how easy is it to get something out of it. I alswys thought statistical analysis software, deep analytics and predictive data mining was out of the reach of mortals with a very limited budget (Software packages such as SAS can set you back millions). Have a look at “R” and be amazed that a free (yes, free to download and run) package can do just as much and more. For an introduction to “R” read this document we have been writing :)

“R” introduction to a new way to exploit data