As gen AI moves from the theoretical to the practical, it can be leveraged across practically all functions and departments, provided IT leaders can see the path ahead—and execute effectively.
Niranjan Ramsunder, chief technology officer at UST, a global IT services firm, has the necessary practical tips to make the most of implementing large language models within your organization.
Which internal stakeholders should CIOs engage with in order to best define the role of new AI in transforming their organizations?
For the last year or so, new AI has been mostly driven by the increasing availability of sophisticated large language models. What has been exciting is the availability of a meaningful natural language-based interaction with huge amounts of unstructured data like documents and audio sources.
Another area that has changed is information retrieval. This has been around in AI for years, but now is more accurate and larger in scale due to the supporting tools in gen AI, such as vector database and fine-tuned large language models like OpenAI and Llama.
With that context, any work in an organization that is driven by the need to understand large amounts of text becomes an immediate target area. Some examples:
- Accelerating call center responses from large documents like insurance plans and fine prints. Stakeholders include heads of call center operations and customers looking for self-help.
- Finding technical answers from large manuals. Stakeholders include heads of technical support teams.
- Summarization of documents for insightful summaries—for example, reports based on annual reports of companies. Stakeholders include financial analysts.
- Accelerating all areas of software development, from requirements for code generation, code certification and testing to development of automated scripts for deployment. Stakeholders include heads of application development and application support.
- Getting insights from unstructured feedback on the services and products of the enterprise. Stakeholders include CMOs and chief digital officers.
- Getting insights from large amounts of data to drive analysis in areas like churn of customers and fraud identification. Stakeholders include chief data and analytics officers.
- The need to engage with legal, copyright and IP departments to ensure responsible and compliant use of AI. Contract teams can be given summarization of key changes compared to the last period’s version for supply contracts and other legal documents.
Where and how should CIOs start generative AI-led transformations?
The best place could be call centers given the current shortage of experienced talent for technical support functions. More specifically, gen AI-led transformations should start in areas within call centers where gaining expertise is time consuming. Secondarily, another starting point is an area under direct control of the CIO, for example in the acceleration of software development.
Quick ROI—in less than eight weeks—is crucial to drive further adoption and scale. Tracking competitors’ use of gen AI is also crucial.
The ability to leverage the expertise of outside technology companies is more important than ever for CIOs. How do CIOs ensure they’re receiving value from the relationship? Is value-based billing the answer? What are the right metrics?
It is crucial to get help for general purpose solutions from hyperscalers, as well as from leaders like OpenAI and from niche providers and research organizations, like Stanford. While using them is an imperative, there are three important “must haves”:
- Have the ability to replace any provider and AI model and swap in any new models at any time with minimum impact to the work that is already AI-powered.
- Have the ability to use smaller models in terms of number of parameters used, to solve specific problems—like a call center copilot as against a test data generation co-pilot.
- Have control over the final fine-tuned models either through partnerships with providers of unique solutions to your enterprise, or have some other exclusive arrangements for AI models which meet your outcome goals, such as ROI or ease-of-use or time-to-production for a new feature.
This can be supported by well-defined outcome metrics and shared risk contracts. Outcome metrics include the improvement in productivity for resolving calls, like average time to close tickets or calls successfully resolved—including auto-resolved issues—self-help-based resolves and faster response closures.
There is also the measurement of various quality metrics like time taken to develop code from requirements, or time taken to deliver tested code with no defect leakage in a continuous A/B testing mode.
In working with many CIOs, as well as CTOs and CISOs, what are the keys to a successful engagement with outside technology services firms?
The key is a clear understanding of the criteria for success for any AI-driven engagement. This means it may not be enough to deliver an AI model for augmenting a business process—you may also need support to get internal legal and external regulatory approvals. The time defined for a deliverable may not be absolute, but could be relative compared to competition.
There needs to be clearly defined “shared skin in the game” for meaningful measurable business outcomes. Having a common shared understanding of the underlying “why”—the business, external and competitive needs—and how success will be measured, as well as implications for the current way of work is performed and ensuring adequate retraining are all crucial for success.
