The Unstructured Data Structure

By: Nancee Ruzicka

In a 2013 report, FICO cautions “80% of big data is unstructured.” If that’s the case, then why are we spending so much time and energy on structured data? The answer, it seems, is that the data currently most valuable to service providers is structured. Service providers collect data from the network for important tasks like billing, revenue assurance, policy management, service assurance, and fulfillment. For the majority of service provider operations, structured data captured from customer transactions and network elements reveals everything they need to know. Structured data is continuously collected and saved, so data bases and warehouses are readily available to be searched and analyzed to uncover trends and spot opportunities. In fact, service providers are required by law to capture and store customer transaction data and they often maintain many years’ worth of data for each customer.

While revenue assurance and fraud management rely almost exclusively on transaction data and network factors to analyze user behavior, find lost revenue, and determine the sources of fraudulent activity, accurately analyzing that same data to determine marketing behavior and evaluate the customer experience requires the inclusion of unstructured data. Traditional solution providers, such as cVidya, that use customer transaction and network data for fraud management and revenue assurance, are now expanding into the marketing analytics space. This new line of business builds upon existing solutions; however, these vendors also recognize that access to the unstructured data relating to customer behavior and attitudes is critical to service providers as they define CEM (Customer Experience Management) strategies to reduce churn and increase up sell. Other vendors, such as AsiaInfo, are providing "contextual awareness" to integrate both static and dynamic data sources to provide service providers with valuable customer insights.

Incorporating unstructured data is producing results. Several service providers report success in identifying customers likely to churn by using advanced analytics and integration with unstructured data. One of these service provider identified 75% of the customers that were likely to churn by targeting only a specific 10% of its total customers as indicated by the analysis. That segment is now the focus of offers and support designed to improve their unique customer experience.

Complex events require complex solutions

One of the most difficult challenges service providers face is data volume. Service providers collect massive amounts of data and that in and of itself can be one of the biggest challenges facing them today. Vodafone India, for example, collects 32 billion records per day and the largest service providers regularly collect 40 million records per second!

Much of the data that is readily available to service providers is structured. Data about billions of transactions, millions of customers, thousands of devices and hundreds of products are all captured or calculated in a consistent manner. However, what service providers are missing, and what is becoming valuable to marketing and product management functions, is the unstructured data regarding perception, problems, and desires being expressed by customers during calls to customer care, online during chat sessions, or in online and social networking communities like Facebook and Twitter. There are also additional clues that can be found in videos and promoter index scoring that could be valuable to a service provider if properly decoded and analyzed.

Processing billions of user transactions, correlating and distributing that data to multiple OSS and BSS solutions, ensuring partner settlements, eliminating fraud, and improving the customer experience are all drivers for implementing big data management solutions like Hadoop. But there’s more to it than data storage and computing power. Managing the volume of events, processing customer transaction records, and correlating unstructured data requires complex event processing that collects and analyzes every scrap of data available via network elements, probes, and partner elements.


Latest Updates

Click to Discover>

Subscribe to our YouTube Channel