By Herb Greenberg, Empire Financial Research

The big question in the early innings of conversational and generative artificial intelligence (‘AI’), such as ChatGPT, is… how good are its responses?

CNBC recently had a great conversation about this with longtime tech investor Roger McNamee, whom I used to quote decades ago when he was a fairly unknown tech analyst for mutual fund company T. Rowe Price (TROW).

What I liked about Roger was that he was always cogent, easily untangling complicated ideas and companies, and willing to publicly take the unpopular view. (He still is today!)

He was on CNBC the other day, and I was fascinated with his comments about ChatGPT, which as you may have noticed has swiftly become a well-worn (dare I say, overdone) theme in recent Empire Financial Daily essays.

Roger’s view was that since much of what ChatGPT does is “stuff scraped for free” off the web, the result is low-quality content. “Garbage in, garbage out,” he said, adding: “B.S. artist in code.”

His comments on the web scraping are actually seriously significant, but not widely discussed… probably because we’re still in the “gee whiz” phase of absorbing everything the release of ChatGPT has unleashed.

But as one friend who is steeped in all of this explained to me…

Oh, the scrape problem is a deep liability issue. Not just because of indifferent quality, but because it’s [a] massive Napster-like copyright violation, whether art, software, code, essays, or music. Massive lawsuits [are] working through various courts, and Google and Microsoft have huge liability.

The idiot AI engineers ingested massive amounts of content, compensating no one, and it’s now being sold. We put Napster out of business for less.

As it turns out, even ChatGPT creator OpenAI seems to agree with that concern…

Earlier this week, the company launched a new “AI classifier,” which it says is “trained to distinguish between AI-written and human-written text.”

According to OpenAI…

We’ve trained a classifier to distinguish between text written by a human and text written by AIs from a variety of providers. While it is impossible to reliably detect all AI-written text, we believe good classifiers can inform mitigations for false claims that AI-generated text was written by a human: for example, running automated misinformation campaigns, using AI tools for academic dishonesty, and positioning an AI chatbot as a human.

But there’s a catch, as OpenAI goes on to explain…

Our classifier is not fully reliable. In our evaluations on a “challenge set” of English texts, our classifier correctly identifies 26% of AI-written text (true positives) as “likely AI-written,” while incorrectly labeling human-written text as AI-written 9% of the time (false positives). Our classifier’s reliability typically improves as the length of the input text increases. Compared to our previously released classifier, this new classifier is significantly more reliable on text from more recent AI systems.

We’re making this classifier publicly available to get feedback on whether imperfect tools like this one are useful. Our work on the detection of AI-generated text will continue, and we hope to share improved methods in the future.

This, of course, gets back to “garbage in, garbage out”… but also the sheer ethics of some of this.

Seems the classifier should’ve been introduced at the same time as the original ChatGPT, no?

There are simply so many angles to this… and we’re barely into the first inning!

The good news, I guess, is that this isn’t two years ago when by now we would have seen a slew of initial public offerings (“IPOs”) – many of them garbage – and even chat AI-themed exchange-traded funds (“ETFs”).

Not that they still aren’t percolating under the surface, after all ChatGPT was the fastest uptake of any online tech service.

Meanwhile, Empire Financial Daily readers have no shortage of opinions on ChatGPT, in response to my question about whether they have tried it and what they think of it…

Among those responding were a former software engineer at Lockheed Martin (LMT) and Honeywell (HON) who knows a thing or two about coding…

“Good morning, Herb. I got curious about ChatGPT about two weeks ago due to the ‘buzz’ surrounding it so I decided to try it out.

“At first, I tried simple things like, ‘Who was the first president of the United States?’ It came back with the correct answer. ‘Not impressed,’ I said to myself. Eliza could’ve answered that (Eliza was a conversational program that mimicked a conversation back in the 1970s).

“I tried something a bit more challenging. So I typed, ‘wire up a 9 volt battery to an LED with an appropriate limiting resistor.’ To my surprise, it gave me a description of the series circuit with a suitable value for the limiting resistor. One thing was wrong though, it did not describe the necessary ground all electric circuits require.

“I told it so and it apologized and thanked me for the correction. I then asked it to write a program for a microcontroller that initializes most of the peripherals. It did that and told me that it could not fully comply because some features compete for the limited resources on the chip so it gave me the most it could. I copied the program, loaded it on the chip and, voila it WORKED! Mind you, I gave it a one sentence description and it figured it out.

“In my opinion, this chatbot does a better job of giving useful information than Google’s search engine. In comparison, Google is now where Yahoo’s search engine was back in the day. I can see Google easily losing part of its revenue to this type of technology.

“One last thing I tried because I could not get any useful answers from Google (so did two other engineers at NASA). I asked it to describe how to configure a video player to do a very specific thing. It kept telling me incomplete answers at first so I had to ‘coax’ or ‘guide’ it to fully understand what I was after. Two hours later of collaborating with it, it gave me a solution that worked and my team and I will be using [it] for a team demo this week.

“In my opinion, this technology will be useful to anyone so long as the person has a full understanding of what they are looking for as an ‘appropriate and verifiable answer.'” – M.P.

On the other hand…

“I tried to ask ChatGPT a few questions about politics and found that it had no answers. The questions had a skew towards the conservative side and my suspicions rose about who was writing this software. I then asked a few other questions that were neutral and should have been easily answered. I got a big nothing back again…

“It was weeks ago when I did this so I do not remember the questions I asked and probably should have saved them. I was not impressed with it AT ALL.

“ChatGPT has a LONG LONG LONG way to go to be useful for anything in my opinion.” – Tom P.

Among other responses…

“Herb, thank you for asking. I spent a little time playing around with ChatGPT. It seems just like a sophisticated tool, that is only as good as its database and how it has been programmed to answer a particular query. If it was really an ‘A.I.’ system, it would recognize that it may be lacking data, that it has a bias, and then go look for additional data. It cannot do that it appears, so far. Unless one is aware of ChatGPT’s weaknesses, one might rely on very inaccurate and biased answers, and not know it.

“So, attached is my interaction asking about ‘Climate Change.’ You will find it maybe a little informative, but very interesting in the responses. It appears to me to only give the accepted narrative, neglecting a lot of real science, and ignoring statistical data it should have available, or be able to calculate! It is really good in talking in circles and in some cases not answering my question.” – Steve B.

Herb comment: In other words, Steve, it’s like a politician! Thanks for sending along the conversation. It was too long to read in detail, but I’m impressed you and a computer could keep the conversation going for that long.

“ChatGPT is a welcome news to me, those that oppose it [have] some selfish motive. The cost of College/Education will have nowhere to go but down. Colleges should start now to think of what to do with all those DORMS . Cost of college books will go down. What sense does it make to live in the Dorm and deal with all the safety issues and cost when all you need is a laptop and internet connection and a curriculum. The younger generation [is] definitely blessed. ” – Angela O.

“I believe if not regulated in school assignments, it will cause our students [to lose] their ability to think and forget to be creative.” – Catherine M.

“I read your email with great interest as just today I attended a faculty meeting at U Michigan Ross School of Business and ChatGPT was the topic. I come from corporate and started teaching undergraduate and graduate students at Ross last year.

“The faculty conversation was robust and interesting. Some professors/admin fear that students will use it to cheat. Some talked about the need to test students with paper and pencil. Others feel we should look at it as another tool in the toolbox for students.

“I have seen the advent of the calculator, internet, Google search, spell check, stock photography/video and other disruptors. To me they are all tools that should be understood and optimized. Not feared.

“Thanks for your interest in the topic and look forward to hearing more from you.” – Susan M.

Herb comment: Angela and Catherine, the full effect on higher education – or all levels of education – remains to be seen.

And Susan, I’m assuming we’re contemporaries, and I agree. When Twitter first launched my gut reaction was, “Oh no…” And then I figured I would embrace it. Twitter has since become a useful tool, though sometimes a nuisance. That was 2008.

“Sounds like something with a large customer help line requirement like the IRS could greatly benefit. Probably even though it makes mistakes probably less than the current human response if you can reach them at all.” – Michael T.

Herb comment: Assuming, Michael, that the bots get it right!

“Yes [I] tried it. I was amazed at its ability to do several tasks, such as generate a basic resume letter that can be quickly amended for my specific details etc.

“I do worry that it can be manipulated to only respond with answers/documents that reflect political or current social ideology. In writing this I realize I should test it by asking for a pro-con response to a highly emotional topic such as ‘climate change’ etc.” – Edwin D.

Herb comment: You aren’t alone… See Jerrel’s comment below.

“I’m 89+, born in the dust bowl, have an engineering degree, 2700 USAF flying hrs, 32 yrs retired from DE mgmt. positions. Technology over the last 40 or so years, esp. the internet, has overrun the intelligence and/or education & interests of the majority of human population. The instantaneous distribution of false voluminous information by illegal/uninformed sources overloads our minds, and the outdated second amendment is horribly abused, among others certain aspects of our constitution, and political system were never designed for today’s technologies, and I fear will prevent the necessary solutions to save our relatively free society. Politics has always had its weaknesses and failures, but none so bleak as nowadays. Same goes for our various media. Who & what [does one] dare believe anymore? Instant falsehood communications exaggerated and instantly broadly dispersed w/o controls. I fear for my many grand and great grandchildren.” – Jerrel L.

Herb comment: Jerrel, I have grandkids and I’m as scared for them as you are for yours.

The only thing that gives me comfort is that each generation has feared for the next… and somehow, we’re still here. At the risk of sounding naïve, but really just trying to be optimistic: Hopefully, on this topic at least – and for everybody’s sake – history continues to repeat itself.

Thank you for reading! Have some feedback for us?
Contact the 24/7 Wall St. editorial team.