There are shadows on the horizon for data and analytics teams.

Data and analytics teams worldwide have faced a variety of market changes in the past decade.  Those changes have generated growth in the analytics field and created unique challenges for its practitioners. Thanks to rapid evolution in the field, problems once thought impractical can be revisited with fresh insights and tools.

The market now poses a host of bleeding-edge questions while simultaneously offering bleeding-edge answers in the form of technology innovation. The market forces today’s business leaders to approach this bleeding edge quickly, and awareness can make all the difference.

The market has seen a significant boom in available computing power for data analytics teams. Moore’s Law, coined in 1965 by Gordon E. Moore, co-founder of Intel, states that computers will continue to double their computing power about every two years while the costs of machines continue to decrease. This means that computing growth is, and likely always will be, exponential. It’s no wonder technology companies tend to focus on innovation rather than standards. 

As a result of exponential growth, data is available more now than ever. Another advancement that feeds these conclusions is the advent and proliferation of the Internet of Things (IoT) and mobile or wireless devices. Information previously unavailable or unquantifiable — such as biometric and logistical tracking data, location, and in-depth usage metrics —  is now often collected automatically. This massive influx of new information has created repositories too large for analysts to query alone, giving rise to automation practices like machine learning (ML) and artificial intelligence (AI). Data analysts can leverage these innovations and shifts to transform not only their workflows, but their roles within the business.

One example of the consequences of increased computing power for data and analytics is found in the field of genetic analysis. Since the first gene was sequenced in 1972, gigantic leaps have been made in the ability to study genetic forces that dictate human biology. Recent advances have been particularly remarkable, especially in terms of accessibility. 

Bioinformatics tools like EDGE have bridged the gap between the rapid availability of low-cost DNA sequencers and the comparative scarcity of knowledge required to make sense of the data. While the tools are widely available, the ability to interpret this data at scale will still fall to data and analytics teams. This will require teams to adapt to the demands of interpreting large amounts of data. While these innovations will add substantial value, they will need data analytics teams to become agile — a challenge that is best met prepared.

Nimble Workspaces Will Help Analytics Teams Meet Challenges Head-On

Most industry professionals work in a siloed structure, and most analytics teams use a linear approach similar to the Waterfall methodology. However, several factors in the market today make transitioning to a more dexterous project management mode a good idea for analytics teams. 

As analytics teams face more significant amounts of data to process and higher computational technologies become more available and ubiquitous, analytics teams that want to create solutions (rather than outsource them) should consider software development processes as an integral part of their workflow. 

There are many advantages to adapting analytics teams’ workflows to match the software development team’s standards. Perhaps one of the biggest benefits is that it allows the analytics team’s continuous integration and delivery (deployment) across their projects. This is a vital endeavor when considering other shifts in the data and analytics marketplace and how they will affect a team’s workflow and dynamics. A methodology that breaks a project into workable parts, involves constant communication between teams and stakeholders, and focuses on continuous improvement throughout every process stage would greatly benefit this transition. 

Analytics teams who shift their project management mentality will be able to facilitate faster product evolution and tighten up the innovation cycle, both of which facilitate a lean startup mentality. Teams will be competitive and drive value within the business. As Eric Ries observes in his book The Lean Startup, “the only way to win is to learn faster than anyone else.” 

While any industry leveraging data and analytics to make business decisions can benefit from shifting to a more agile-focused workflow, an apt example of this shift is the logistics industry’s technology adoption that enables tracking and route optimization. Google now provides logistics companies and teams with a Cloud Fleet Routing API, which leverages the massive computing power of the cloud to perform route optimization at a mass scale. 

Logistics, an industry historically plagued with unforeseeable obstacles and complicated management, can now use real-time streaming analytics to improve the time and effort involved in fleet management substantially. In addition, because Google offers this service as an API, fleet owners and drivers can seamlessly integrate route optimization with the mapping and directions applications they already use. Almost any logistics company can use the feature agnostically without purchasing proprietary software.  

How Cloud Computing Has Changed the Landscape

The examples above lead us to another inevitable conclusion about the impending ubiquity of cloud computing. Significant advances in computational power stem not only from hardware innovation but even more so from the advent of cloud computing. Off-premise and outsourced database management tools have enabled data and analytics teams to solve problems that were previously unsolvable or not even imagined. In short, the resources available to analytics teams are many, and transitioning to a cloud environment is an enormous advantage on several levels. 

Cloud-based technology is already the standard across most enterprises. The question is no longer if but when the technology that analytics teams rely on will move entirely to the cloud. Cloud-only offerings among SaaS providers have already become the standard across many industries. Forbes reported that in 2021, 83% of enterprise companies’ workload was stored in the cloud.¹

The COVID pandemic accelerated the already pervasive trend, as data analysts and CIOs frequently don’t have another choice regarding technology solutions. The property and casualty insurance industry is an apt example of this evolution. Guidewire, an insurance industry standard in data and analytics platforms, began offering its users hybrid tenancy options in 2021. While the new system is still compatible with legacy systems like Microsoft SQL and Oracle, Guidewire now operates on AWS’s cloud platform. Their Cloud Data Access whitepaper demonstrates new challenges associated with this transformation.

Because industry stalwarts like Guidewire’s cloud-service offerings are built on AWS, APIs must be used to access data previously housed on site. To further complicate matters, APIs generally return information in JSON format instead of the standard line-item-based table familiar to many analysts. Translating JSON packages into actionable insights may require an analyst to interact with and build other APIs or add software to the stack to automate and continue familiar processes. 

While both out-of-the-box and proprietary versions of this software are available, cost, risks, and implementation lag time should be considered. In the past, data analysts and data scientists have held markedly different roles in a corporate structure, with the latter’s skill set including far more programming and development than the former. This will soon be a thing of the past, as cloud-based technology will force all data professionals to learn coding beyond query languages and program structure to maintain their data expertise. 

The Future is Now: Adapt to Agile and Cloud Technology 

A quick scan across industries reveals that the data and analytics market is changing rapidly. Increased computing power has redefined problems data analysts are equipped to solve, creating the potential for new challenges previously outside the realm of imagination. Adapting to the wealth of new resources and data by becoming nimbler and more embedded in the development lifecycle will allow data analytics teams to leverage all the tools at their disposal to substantially increase their output and add significant value to their work. 

Data analytics teams have a unique opportunity to set the standard for developing a cloud strategy that significantly enhances the overall ROI of cloud service offerings throughout the business and to lead the business transformation from a data-driven perspective. Embracing the data and analytics teams’ demands to compete in this new market is no longer an option but a requirement that will set the course for the industry’s future. 

According to a 2021 Gartner report, only 7% of executive leadership feels they have helpful, mission-critical company information.² However, we have seen that this is not because the data or methods of obtaining it are unavailable, nor is it because the opportunities to scale effectively do not exist. On the contrary, executives are waiting on data and analytics teams to lead the way in becoming the hub for mission-critical information within their company. 

Data and analytics teams have a tremendous opportunity to create valuable and profound insights for the businesses they are embedded in by embracing the direction the market is moving and integrating a future-focused strategy. More than that, analytics teams can redefine the importance of their discipline across industries and enterprises by implementing agile and cloud technology solutions.

¹ Louis Columbus, “83% Of Enterprise Workloads Will Be in the Cloud by 2020,” Forbes (Forbes Magazine, January 25, 2018)

² Mary Baker, “Gartner Survey Shows Only Half of Business Leaders Feel Confident Leading Their Teams Today” (Gartner, July 23, 2019)

About the Author

Jeff Kanel is a Director of Data & Analytics for Centric Consulting. Through his work as a consultant, he challenges clients to see new possibilities from data. Jeff regularly works with executive teams to help them create practical data strategies and create a data-driven culture.  

With 30 years’ industry experience, Jeff has a rock-solid foundation in business process maturity, machine learning, business intelligence and modern analytics.  He is the author of “The Intelligent Enterprise” business maturity model for data.  

Outside of work, Jeff enjoys spending time with his wife and children, playing in a band, and exploring innovations in data.