Proof-of-Concept, Graveyard of Ideas

It used to frustrate me when my daughters told me their homework had to be a certain number of words. They’d stress over the volume to be written rather than the content of the answer. Anyone can pad out an essay with repetition. That’s quantity over quality.

The same thing is happening in science. AI makes it much easier to churn out papers seeking citations rather than performing valuable research. There are important lessons here for when to automate business processes and whether to use AI.

Publish or Perish 

Sayash Kapoor and Arvind Narayanan, whom we met in “What AI Can and Cannot Do”, point out that while the quantity of scientific research is accelerating, the quality judged by genuine breakthroughs, is not. One reason is the publish-or-perish incentives that determine funding and tenure.

The way to increase output is to work on tweaks to well-known theories. There is a captive audience ready to cite your work, especially if you cite them first. It’s a mutual backscratching that benefits scientists, but not science as a whole.

Kapoor and Narayanan give the example of biomedicine. Researchers working on known molecules will likely do work that justifies a paper and generates citations. In contrast, it might take ten years to do important work on new molecules and end up with only a single paper to show for it.

AI increases the output of those working within the confines of what is already known. Extrapolations, interpolations, and application of theory to new data are all in its wheelhouse. Speed is its superpower, understanding is not.

This means that AI reinforces bad theory at breakneck pace. Kapoor and Narayanan discuss the geocentric theory of the universe, which is the idea that everything revolves around the earth. The observations that call this theory into question can be smoothed out mathematically with ever more complicated models. Only when the heliocentric theory emerged and was a neater and simpler explanation, did opinion start to change.

The authors argue that AI would have handled the increasingly complicated maths and an outcome-based assessment would have concluded it was correct. We might still believe the earth was the centre of the universe. This is a version of the AI makes us dumb argument we saw in Smarter Workers, Shallower Students.

The Limitations of AI Agents 

In business, if we use AI to automate a bad process then we increase the speed at which we can get into trouble.

This might mean low quality marketing content, repeating incomplete financial reconciliations, or deploying code without the checks and balances a professional developer would use.

It turns out this last one is common among scientists and is a growing problem among people using no code and low code applications. Automated testing, version control and design guidelines are non-negotiable in software development, but unknown to most people outside of this world. Applications that allow us to play at being developers can leave us with serious problems.

Gartner estimates that at least 30% of AI projects will be dropped after pilot. Jing Hu of the 2^nd Order Thinkers Substack says 46% don’t go beyond proof of concept. This is a positive in my book, given all the risks involved in automating bad practice, but it is a collosal waste of money.

It’s worth remembering that large language models have no memory, will produce different responses to the same question and may be unable to correct an error.

This means before putting any automation into production, you should follow these steps:

1. Ensure the process is necessary, clear and well understood

2. Test the results by changing both inputs and required outputs

3. Cross-check any code using software best practices

At MSBC, we have launched Build AI. This is a 90-minute consultation to prioritise tasks for automation. We then recommend the appropriate software to use, which is not always AI-enabled, and implement one workflow for you. If you’d like to know more, check out this link.

Flesh and Blood

My daughters no longer have school essays to write each week. They moved onto university and one graduates today. They tell me their project work is still governed by quantity targets, but they are in an environment that rewards original thought.

That is where we want our businesses to be. AI does not help us with this, for the simple reason that we do not know, and hence cannot code, how humans create. For now, that remains flesh and blood’s advantage over silicon.

Questions to Ask and Answer

1. Do I understand how the process I am automating works?

2. Have I given the AI model all the information that it needs?

3. What steps am I taking to check AI answers for accuracy?

Find out more. Hit reply to this newsletter and ask about “Build AI”.