💡Building Better Prompts
Buried in the news that GPT-4 aced every accounting certification, was the fact that chain-of-thought prompting yielded a 16% better score on the CPA exam.
Simply by changing how the prompt was written, it scored 16% better. The study is a fascinating read, and a great advertisement for the value of prompt engineering. In their tests vanilla GPT-4 scored 16% better than GPT-3.5, and with some prompt engineering they got GPT-4 to score another 16% better.
We discussed it on my daily pod today, and if your initial reaction is I’m not going to that level of effort to work with a machine, consider the efforts we go through with our human colleagues. The time we spend with interns, the cross-training, the effort that’s poured into every individual.
I’ll admit, a positive culture that grows fellow human beings has a different appeal than becoming a robot-whisperer, but who says we can’t have both!
I’ve built a swipeable prompt that uses chain-of-thought prompting for research. It could be tax research, software research, any question you’d normally Google. My initial results have performed higher than any other research utility I’ve used.
I’m using ChatGPT Plus, specifically the new Bing Search GPT-4 model (select GPT-4, Browse with Bing). I tried the same with the free version of Bing and the results were much more sporadic. ChatGPT broke around 25% of the time, and Bing gave up 75% of the time 🤷
Here’s a link to swipe the prompt, but you:
Paste the full prompt into ChatGPT
Provide it with an explicit URL to research from. This ensures it’s only referencing up-to-date, authoritative source material. Currently this doesn’t work with PDF files, so in the below example of an IRS publication (I know this isn’t authoritative you tax nerds) I linked it to the HTML version.
Provide it with a question.
Rather than simply asking a question, the prompt has it fetch the 10 most relevant excerpts from the source material, then based on those excerpts forms an answer to the question. This basic example of chain-of-thought prompting focuses the model on the most relevant material, and provides a more logically-coherent answer.
💎This Week's Sponsor
Have a product or job opening to advertise? Reply to this email for more info.
📅 This Week
Lots of announcement from Microsoft Build 2023, but most interesting to me is Windows Copilot, a chat assistant built into Windows
It works with the programs on your screen, and as I’ve expressed in the past a chat assistant that works with the apps on your desktop will be game-changer that opens the door to automating legacy systems like tax software
I’ll be doing a keynote at the Pronto Tax Conference in San Diego 7/25-26 and I’ve got 4 tickets to give away, use code ‘jason2023’
I made a video running through my #1 productivity unlock that I still don’t see most firms leaning into
BetterLegal launched an assistant to demystify your contracts
Digits teased Finance AI - join the waitlist here
I’m not generally an AI doomer but this talk is a good overview of how problematic AI is today (credit: Hector Garcia)
We’re closing in on 50 episodes of the daily podcast, with over 1k accountants tuning in every day now 🤯 this week we covered how to make your website better, why ChatGPT plugins will change how we work with software, my thoughts on reporting apps, and kicked off ‘Olde Thymey Tuesdays’ where we tackle legacy accounting tropes with practical advice for accountants in those legacy situations
😍From The Community
My private peer networking community Realize for firm leaders, now capped at 500 members.
Upcoming Events:
05/31 🚜 The Content Farm - Publishing For Audio with Jason Staats
06/01 🐠 EOS-Ish - Scorecards with Rachel Fisch
06/06 💘 Swiping Right - Tech & Client Education with Kellie Parks
06/07 🥳 AICPA Engage - Realize Cabana Party with Jason Staats
06/20 💻 Demo Day - Copilot with Jason Staats
06/20 🐠 EOS-Ish - Where To Go From Here with Rachel Fisch
06/21 🏫 Hall Pass - Sales System with Brandon Hall
06/21 🎤 Who Runs The World - “Feelin’ good as hell” with Twyla Verhelst
06/26 🕸️ Into The Chadiverse with Chad Davis
▮▮▮▮▮▮▮▮▮▮▮▮▯▯▯▯▯▯▯▯ 308/500 members