Big data comes with big promises. When used to support decision making, big data creates a realistic picture of the market, annihilates human biases, and instills true confidence. But when big data becomes a swamp, its value decreases to zero.
Gathering data was never a problem for one of the largest European banks, but they were unable to explore, understand, and use it. Drawing on our knowledge of algorithm development and data mining, Softaria set about developing a tool to modify the data so that it could be properly used.
We created a technology solution that brings an existing database, as well as new entries, to a unified format. At the core of the solution is a scalable clustering algorithm paired with a complex analysis algorithm – both applied multiple times to a provided dataset for testing purposes. The effect of data cleansing was continuously evaluated before the final implementation of a repeatable data mining process took place. When the tool was finally deployed, 10 million entries containing numerous duplicates and errors were converted into a clean, continuously updated, and highly usable depositary of customer data.
Robust Scientific Approach for Impeccable Data Quality
What we did:
Developed advanced algorithms for data analysis and splitting the data into clusters.
Minimized time needed for optimal algorithm performance considering database inactivity requirements imposed by the business.
Conducted a series of continued tests on provided datasets for each algorithm modification for evaluation purposes.
Managed a complex interactive algorithm development and testing process consisting of many prolonged iterations.
Enabled the client to clean up a massive database of 10 million entries, remove duplicates, and fix data inaccuracies.
Equipped the business with a long-term data cleaning solution to process existing and new entries.
Scientific Data Processing
Our scientific data processing team is available for engagement in complex research-based development projects. We deliver software solutions on the cusp of science, business intelligence, and innovative technology. Get in touch to inquire about our approach.