ChatGPT isn't the answer Enterprise is looking for
Amidst all the hype, ChatGPT has been touted as the Google Search killer.
Search indeed is broken and ripe for disruption. The current search experience is far from perfect, with users frustrated by pages of irrelevant ads and low-quality sources "optimised" for SEO. Even less when you are an enterprise user, and willing to pay for a non-ad experience optimised to deliver results relevant for you.
And ChatGPT is brilliant, and will solve for a lot of creative and marketing use cases, but it is not the solution that enterprise users or knowledge workers have been waiting for. For any information-retrieval, or search system to be worthy of enterprise use, it needs to meet three core criteria:
1. Comprehensive
The system must be able to look for and find information from all the sources where the user would look, and more. ChatGPT is not capable of searching the internet for information, nor does it have access to the vast amount of underlying data available through Google or other search engines - it is not able to provide information that is not already included in its training data. The information that it is trained on may become obsolete soon, and ChatGPT is not able to learn new information, so it will not provide accurate information on a wide range of topics, including those related to recent events or developments.
For example, ChatGPT cannot answer questions related to "Inflation Reduction Act", likely announced after the model was trained
Here another example, a question that a knowledge worker in the investment management, legal or sustainability industry is likely to research
Do any suppliers of apple have exposure to forced labor practices or violate the modern slavery act
While ChatGPT doesn't have any information on the topic, note that Google Search response isn't enough by itself. It is comprehensive, but is all of it useful? As an equity analyst, will you take information from rollcall.com or theinformation.com for face value? Which raises the second point.
2. Trustworthy
For an enterprise system, the accuracy of the information is essential, especially in critical decision-making situations. Here ChatGPT fails miserably. While ChatGPT may be able to generate responses that are similar to those a human might provide, it is not capable of fact-checking or verifying the accuracy of the information it provides. Even worse, it can turn unreliable data into convincing answers and sound right even when it is "making things up".
Case in point is Andrew Ng's hilarious post, where chatGPT explains why an abacus is faster than a GPU!
Andrew Ng
Stack Overflow imposed a temporary ban this week on its users sharing answers from ChatGPT for the same reason “The primary problem is that while the answers which [it] produces have a high rate of being incorrect, they typically look like they might be good,”.
Like we saw above, Google doesn't do too well in this department either.
3. Actionable
The entire point of finding information in enterprise usecase is to take action - draw conclusions / make decisions etc. Which means the output information has to be relevant, useful, material / impactful, and complete enough to allow the user to take the next action.
See the snippets below - while ChatGPT provides some useful prompts in answer to Tesla's impact on biodiversity, the answer is far too vague and unspecific to be actionable.
Similar story, when asked about the impact of global chip shortage on a variety of companies like Volkswagen and TSMC. In both these cases, the user will almost 100% have to result to an old fashioned Google Search to find specific information, and then weed out noise to get to the actual, actionable answer.
While ChatGPT itself is not a viable alternative to traditional search engines for enterprise users because of these fundamental shortcomings, it sets a precedence for disruption of search, and is a very strong statement to the power of large language models.
An ideal enterprise search solution will return the most material / actionable results for a user (and not the ones with optimised SEO), with the least amount of noise, from all possible data sources where relevant information can exist, be it external or internal. The results will be ranked based on the user's professional preferences, such as their industry or function, and an understanding of what helps them do their job better (and not what maximises clicks or screentime). Ideally, this solution would significantly help users in decision making even in complex problems (like investing), by automatically recommending new topics or information, the "unknown unknowns", even when they don't know what to search for.
PS: If you are frustrated by the lack of an enterprise search system, Auquan has something really cool in the works. Keep watching this space for more!
PPS: Need more convincing? Sam Altman raised some of these points himself (tweet below)
ChatGPT is incredibly limited, but good enough at some things to create a misleading impression of greatness.
— Sam Altman (@sama) December 11, 2022
it's a mistake to be relying on it for anything important right now. it’s a preview of progress; we have lots of work to do on robustness and truthfulness.
And chatGPT itself agrees:
"Google search would perform better than chatGPT in tasks that involve retrieving information from the internet. As a search engine, Google is designed to quickly and efficiently search the vast amount of information available on the internet and provide relevant results, such as looking up a fact or finding the price of a product. In contrast, CHATgpt is a language model that is not capable of searching the internet for information."
(Not bad for a language model!)