Friday, December 11, 2009

Necessity of Automated Data Mining

The main reason for the necessity of automated computer systems for intelligent data analysis is the enormous volume of existing and newly appearing data that requires processing. The amount of data accumulated each day by various businesses, scientific, and governmental organizations around the world is appalling and it resulted in data mining process. According to GTE research, scientific organizations store about 1 terabyte of new information each day and managing the database is a real problem. It is impossible for human analysts to cope with such overwhelming amounts of database.

Two problems that surface when human analysts process data are the inadequacy of the human brain when searching for complex dependencies in data, and the lack of objectiveness in their analysis. Therefore, one of the benefits of using automated data mining systems is that this process has a much lower cost than hiring an army of highly trained and paid professional statisticians for data mining.

Although data mining does not completely eradicate the need for humans, it allows an analyst who has no programming and statistics skill to extract knowledge from databases.

Thursday, December 10, 2009

Data Mining - Automation or not

Automated data mining gives marketing managers a tool to perform analyses that otherwise would need to be handled by a highly trained researcher. This is accomplished by establishing a predetermined analysis methodology. An algorithm is developed that attempts to mirror the step-by- step decision-making process that a trained modeler would follow.

At each step in the process, preset criteria are used to select analysis options. Because experts have programmed these criteria, the results should be on par with an expert analysis.

However, in data mining and modeling there are many parts of the process that cannot be automated, including choosing a methodology to match a business problem, selecting a data set, quality checking and preparing the data for analysis, choosing among the available options within the analysis process, and interpreting and presenting the results.

An analysis could be automated when:

• The data being used is from a familiar source,
• The analysis has been used before in the same context,
• The variables included have been used before in the same type of analysis, and
• The results of the analysis will be interpreted and used in an established manner.

Within these guidelines, automation can provide huge advantages in time and cost. One clear example is real-time data mining and decision support for Web marketing applications. Rather than establishing fixed business rules, data can be analyzed in real time to evaluate and optimize custom content delivery systems.

The applications do not have to be this high tech to be valuable. Direct marketing campaigns may need to be checked repeatedly for performance against a number of targeting dimensions, such as demographics and model scores. Automating a model that identifies high-performing segments would allow the marketer to quickly evaluate and modify marketing strategies. It also allows the technical analyst to focus on more complicated and challenging projects.

The advantages of this automation are even greater if the data mining software is integrated with the campaign management and tracking tool.