We are training our model to help optimize interest rate decisions, and a key component of that is to audit previous decisions. If we can easily audit previous decisions, it can help us make smarter ones going forward.
The Test: Fixed vs Floating Since October 2023
Here’s the scenario I was working on this weekend: October 2023 through today. $50mm IO loan. 5 year hold. Fixed vs floating. Here’s a graph of T5 vs SOFR over the last two years to help illustrate the analysis.
I used the Pensford’s historical floating resets and Treasury yields (https://pensford.com/forward-curve). I loaded those up into a Claude and added some instructions.
Then I asked it to compare fixed vs floating over the last two years.
Claude response
Those numbers are pretty close, but didn’t match mine precisely - about $100k off on each. Close enough for back of the envelope stuff, but not if I needed precision.
Same Question, a $1mm Different Answer
I closed the conversation and started a new one. I asked the exact same question. I should get identical responses, right? They weren’t - they now differed by more than $1mm from the conversation I just had.
Remember, those numbers should be around $7mm!
Even worse, the interest rates themselves would easily pass most sanity checks. I do rates for a living…would I know that a floating WAIR of 6.71% is wrong when the right answer is 6.83%? No. Those are close enough to trick me into believing they are correct.
I would then assume the actual interest was accurate because that’s basic math. “Of course AI will get that part right.” What if I was doing this analysis for real and had gotten the second answer? Now all my analysis is based on a faulty premise.
In fact, the only reason I double checked was because I was using it as a topic for this newsletter. It presented wrong math with total confidence.
When I pushed back, it simply apologized and said my numbers looked right. I said that wasn’t good enough - tell me how you arrived at your answer! It said it didn’t know how it had arrived at those numbers.
The AI Companies That Are Just Wrappers
Tech start ups raised a lot of money in 2021 and 2022. They still have money in the bank even though their business model failed, so they pivoted and added “AI” to the end of their company name. JamesFranklinAI…SarkisianAI…KevinPatulloAI…
These shops are just leveraging an existing AI model (Claude, ChatGPT, Gemini, Grok, etc) and then plugging it into one of their UI screens so it feels like it’s their software.
Business owners like me are so excited by the potential of AI that we are vulnerable to an AI sales pitch in a way that we woudn’t be to other sales pitches. We are susceptible to being duped by these “AI” companies. They hope to sell enough business owners that this is real AI, get MRR up, and sell the company before anyone catches on that it’s just a ChatGPT wrapper.
How to Vet an AI Vendor
If you are considering using one of these companies, I have a great way to vet them. Before the demo, run a scenario like I just described. Use complex math that would matter to you. Take the real answers before the call. Once the demo starts, ask them to run that same scenario. Twice.
How do their answers compare to what Excel told you? If it works perfectly, please call me and tell me who it was! If it doesn’t match your answers, it’s not a viable solution.
Save whatever money you are about to pay them and just do it yourself in an enterprise ChatGPT or Claude account. You’ll have the same number of incorrect answers. Inconsistencies. Hallucinations. But at least you saved the SaaS fee for a smoke and mirrors product.
Beware the AI sales pitch..it’s probably not what you are after, even if you really really really want it to be.
Next week, I’ll fill you in on part 2 of this case study; with even more twists and turns.