Integrating Generative AI: Questions to Answer for an Existing Product
Generative Artificial intelligence (Generative AI), like ChatGPT, is the hammer that nearly every product person is asked to swing looking for nails as of the writing of this article. There are good reasons for the interest in Generative AI. Still, this post seeks to break things down from the perspective of a product leader or manager of one or more existing products on how to approach and evaluate Generative AI for those existing products. In a future post, there will be a discussion about leveraging Generative AI from the new product development perspective.
As a product leader, manager, or owner who has one or more existing products and is looking to leverage ways to one-up the competition, Generative AI might be that leg up. Still, it can also be a waste of time and resources. Here are some questions you should ask and consider when evaluating whether Generative AI is the correct hammer for your product:
Is there an existing client need that might be fulfilled better or faster by leveraging Generative AI?
Don't be that hammer-hitting screws with AI. Make sure there are one or more customer pain points where you hypothesize Generative AI can add value and with better value overall than other options.
To be able to do this, you need to understand what Generative AI is good at and what it is not. This is a constantly evolving space, so whatever here may change, and not all Generative AI tools are equal. Some areas that Generative AI is good at include:
Summarizing information, such as synthesizing a paper, podcast, or video for key details and takeaways
Content creation, whether text, images, audio, video, etc.
Information extraction from documents, audio, and video
Data analysis and interpretation and the ability for users to ask questions related to their data
Writing code, including rapid prototype creation
Customer service and support in the form of text and audio chatbots
Compliance as a second set of eyes, whether it is document editing, code evaluation, or any other area where Generative AI has significant background knowledge
This is a small list of areas where Generative AI can play a key role in leveraging in your product, assuming that your customer has a pain point where the solution might be one of the items above.
Are there other ways to implement a solution for identified customer problems, and is Generative AI better?
Generally speaking, there are ways to accomplish the same objective without Generative AI. However, that may not always be the case. There are some other reasons why you may leverage Generative AI even when there are other options available, including:
Quicker to Market: Sometimes problems that can be solved by Generative AI can be solved by other solutions, but implementing Generative AI may be quicker. This can be a good reason to use Generative AI.
Lack of Talent or Resources for Other Solutions: Sometimes there are alternatives, but the talent doesn’t exist in-house to leverage it or isn’t available to implement it. This is again another good reason to use Generative AI.
Marketing Angle: Sometimes, it is simply a marketing angle of being able to indicate that you are using cutting-edge technology, and you know that clients want to be able to say that they are using products with cutting-edge technology. Yes, this is a great reason to use Generative AI.
Enhance Team Generative AI Capabilities: Sometimes teams will simply be looking to leverage Generative AI as part of a solution so the team becomes familiar with it, its capabilities, items to consider implementation and help you going forward in other cases that might not be so obvious. This reason is completely legitimate as long as the expectations around it are indeed this and the measurement of success of an effort ties in with these goals and objectives.
The key here is that even though there might be other options than Generative AI, there still may be good reasons to choose a Generative AI approach. Below, you will see a number of factors to consider around economics, performance, and more in additional questions. These will factor into the business case for Generative AI. One thing people forget when evaluating Generative AI is to consider its practicality in a production setting and how it fits in strategically within an organization and not just a prototype space.
Are you looking to implement a solution as a me-too or a differentiator and build a moat?
Many Generative AI use cases are not really being done out of a competitive advantage in that other companies cannot recreate the solution. Instead, they often are done as part of a broader product offering that meets a series of clients’ needs. This is completely legitimate and all right.
However, if the strategy is to leverage Generative AI in a way of building a moat, then there are some key things to consider:
Data moat as a differentiator: Just like AI generally, Generative AI differentiation is mostly based upon the data moat. What data do you have or have access to that is different from your competition? This includes the breadth, depth, and quality of the data you have access to. The more you can build up a data moat from your competition for meaningful data in your space, the easier it will be to differentiate Generative AI solutions.
Unique model as a differentiator: For nearly all companies, the uniqueness of their model is based upon their data and investment in talent and processing resources to build the models they use. Generally speaking, the differentiator of the model is really the underlying data, so while calling this out as a separate item for most companies, it really is restating the true differentiator is the data moat.
Speed to market as a differentiator: A team that knows how to quickly leverage and implement Generative AI in a segment may simply have speed as a differentiator in that they will be consistently faster to market with solutions. Tied with this speed to market is a reduced cost to get to market.
How does data privacy play a role in Generative AI solutions?
Data privacy is really important no matter the industry, but certainly, in industries like healthcare and finance, there is special attention. Accordingly, it is really important to understand the agreements and environments of partners and your agreements with those partners. Further, it is important to not use key identifying information of individuals in training models. The same concerns you have with data privacy and analytics generally should be the same in Generative AI. However, you should realize that some leaders may have some extra concerns here based on some negative stories that had come out originally as Generative AI came onto the market. Understanding this, you want to make sure to clearly communicate how you make sure a customer’s data is maintained and used safely and with the customer in mind all the time.
Does the economics of Generative AI make sense?
Generative AI tends to be expensive in its pricing model for proprietary solutions. Now that pricing is coming down, my prediction is that this will not be discussed as an issue much in a couple of years from now, but for those looking to implement something in an existing product that has a significant number of users, then this is an important item to evaluate closely today. Note there are open-source alternatives that can be used, including some high-performing smaller models that may meet your needs and be an alternative to these pricey proprietary models.
Does the speed of Generative AI make sense?
Generative AI tends to respond in multiple seconds (sometimes 10’s of seconds) versus milliseconds. Speed is a factor of equipment and model usage, so Generative AI can be enhanced by leveraging higher-performance equipment or using models that are performant. This is an area of concern now still and something to consider, but over the next couple of years, it is my belief this will become a non-issue for most Generative AI use cases as models and equipment will be faster.
How important is Generative AI response consistency?
Generative AI tends to respond in a stochastic manner where different responses will be given for the same item. There are ways to counteract this, some leveraging the ‘temperature’ of models to make it a deterministic model, but that can also take away some of the perceived benefits of generative AI. I mention this item for awareness, but I really do not think this should be a concern for product leaders and managers.
How big of a concern are hallucinations for your use case?
Generative AI ‘hallucinations’ represent when generative AI simply makes up things that are not true. There is a recent classic case in which some lawyers were leveraging generative AI to write a court response that the generative AI used simply made-up case law and cited it in their brief. See Reuters. This is obviously a massive no-no for the use case for those lawyers, and now they face disciplinary action.
Before getting too concerned here, it is important to know that there are some good ways to counteract hallucinations. Implementing system prompts that require only having generative AI response when it absolutely knows a response, for example. Another approach is leveraging a secondary generative AI ‘auditor’ that filters anything being responded to as a truth filter is another approach. There are additional ways to counteract hallucinations, but even the raw models themselves are getting better each day in minimizing hallucinations. However, depending on your product and the use case you are seeking to leverage Generative AI within it, the possibility of hallucinations may be a greater or smaller concern and, in some cases, may completely eliminate potential use cases.
How important are product performance and uptime?
One challenge Generative AI has is that it is currently slower than a lot of other software performance metrics that people are used to in products. Instead of 10’s of milliseconds to provide a user response, it may be seconds or 10’s of seconds. Depending on the use case, this may be all right, but it is something to be aware of. The benefit is that more and more smaller models exist, and these smaller models are faster with the same computer hardware and can help improve speed. My view is that in a year or two, this will be a de minimis issue, but for now, if going live in an existing product, make sure to be aware of and test.
Related to performance is uptime. Our customers expect our products to work. Some Generative AI proprietary models have had some mixed performance in uptime history. Or even when up have had some significant degradation in performance. If leveraging a proprietary solution, then it is important to look into the performance and uptime consistency historically.
What are the success metrics for any Generative AI initiative into an existing product?
Making sure upfront how to measure value and success is important. Define that early, along with identifying how you will measure this when put in place, will make sure that you not only communicate to leadership the value provided but also make it easier for the next time you’re seeking to invest resources into enhancing existing products with Generative AI or create new products where Generative AI is a key component.
Conclusion
Leveraging Generative AI in your product can be a great way to improve the user experience, increase customer engagement, and drive business growth. However, it's important to carefully consider all of the factors involved before making the decision to invest in Generative AI. By following the steps outlined in this blog post, you can increase your chances of success when leveraging Generative AI in your product.
Happy experimenting!