My dad has been instrumental in my career in technology. He was the “great tinkerer”, and would encourage me to be the same. He would teach me the basics of a computer, and encourage me to be curious. I would make lots of mistakes, and my dad would direct me to the correct answer and I’d learn from my mistakes. It would sometimes take longer to teach me than it would be for him to do it himself, but patience was key. The more I could do by myself the better I got, the quicker I learned and the more capable and confident I got in my solutions. I got really good at what I did to the point where he said I passed him up in his knowledge and capabilities pretty early in my career, even though he had spent his career in technology.

For AI, that’s not the case in my experience. Maybe I haven’t figured out the technology, or maybe its because I have to create context for the multiple different AI’s I use. Its feels like it’s akin to me training hundreds of different people and trying to keep track of which tool, or even what version of which tool learns best based on what context I provide. Then I have to give it that context to provide an output that will get me what I want without much more back and forth.

The Project

Let me share a story of how I felt this in practice when working on an Excel problem.

I had to compare data across 4 different data sources. I could pull the data and essentially the “ID” to compare across all data sources, so thought this would be a quick and easy spreadsheet with a couple VLOOKUPS, TRIM, and some IF statements, and de-dupe features to make sure the data was clean. Easy right?

I thought this would be a perfect opportunity for AI to shine. Pretty simple output, just show me where in each of the data sources things show up and not show up, combine in a pretty master sheet, so I can compare the final output and cross-check the data across multiple sources.

I spent the first hour thinking it was amazing, producing outputs that worked well, and checked out, getting me the data in an hour in an easy to review format.

Then the cracks started to form.

I had 4 sheets for the 4 data sources, a master source of truth sheet with all 4 data source IDs, combined with some data validation (trim, confirming data is in the cells it was referencing, etc.). It then oddly created a master list as a static list (meaning if I changed any of the data across the 4 data sources, it would not update the master list). That meant any add/delete I did on the data sources wouldn’t be reflected in the master list.

I went back and forth with it for a couple more hours since the single source of truth sheet kept being wrong due to the formula not catching edge cases. Then I was trying to find out how best to update (both add & delete) from the master sheet without causing issues with other linked data, where if the exported data deleted a row, it would update in the master static list while also de-duplicating the data across all sources, and making sure any new data was added, it would add to the master list.

For a while there, I was making progress, getting things solidified, it wasn’t going the approach I would take, but it was working. Maybe it had found out a better way to the problem I was presenting to it.

Then the AI went down this weird python spat for a bit. It was trying to create LibreOffice formulas which wasn’t working in Excel (obviously), and then giving up and writing a static spreadsheet that would require me to run this script every time I wanted to update the data. It was continuing to ignore the section where I said I would have to add data in myself to the final spreadsheet due to limitations of the exports.

At that point it completely lost the plot. I wanted something somewhat simple and updatable, without so many additional steps to enter in variables that would cause this data to be useless, or even worse, wrong.

At that point, I gave up and took its basic work, and massaged it into something that worked for me, removing the duplicated sheets that was just a reference sheet of referenced data and simplifying the output.

This additional work added at least another day for me, verifying the work it had, confirming the data lined up with the final product, etc. To me it felt like 8 hours with AI. 8 hours without AI. Which would be better in the long run? I kept thinking, throughout the entire process, should I take over now? Perhaps now I should stop and re-think this workflow. I couldn’t reset the chat window, otherwise I’d have to start new trying to describe what I wanted and what it had learned already.

So perhaps it was a limitation of what information I was giving it, a bit of frustration with it not seeing the vision I was casting for it, and this weird spat where it went off to solve it in a way that was completely unusable for my needs.

What It Actually Cost Me

At first it was great. It really helped me create formulas that I could vet without having to spend a lot of time thinking about the complete workflow mechanics. Excel can be like that, spend an hour on a formula thinking through the edge cases, and how the formula works. Semantically, I didn’t have any errors. Logically, I had plenty. But at the time, it did help me divert my time to other tasks while it was thinking.

Incredibly helpful.

But, in doing so it kept me from finding critical flaws in its logic that I then had to re-learn its approach to determine if it was any good, then decipher and guide it to a better solution, if it was chasing down a logically flawed method.

It ended up in some cases doubling my mental overhead, because I now didn’t have the core pathways created in my head about what data was going where, because it created those pathways, but also required me to reengineer those that did work while keeping those that didn’t.

It was like my dad would say “it would be easier if I did it myself, but I want to teach you how to do it right”.

The difference is AI has the memory of a goldfish. Those paths (the context), unless recognized by itself and added explicitly, requires me to recognize for it (essentially learning how AI learns) and putting it in a format that it can digest and implement itself based on the grand knowledge it has.

Essentially, I have to learn how the AI learns then write up how I can teach it to learn (my context) because it’s not gonna remember anything from this session as soon as I close the window. Then I have to remember to inject that learning into the context of the next window so it can output something better next time… as long as the training data and approach it takes doesn’t change again.

That's an incredible amount of overhead, and for a lot of people it's exactly why they give up. Even if you've put in the time, learned how AI learns, built context around it so it produces something usable, you'll still hit cases where it's just faster to do it yourself. Especially the one-offs, where it makes no sense to teach it to learn, then learn how it responds, then rework your whole approach every single time you open a new chat window, because its a new AI version or, in some cases a new AI completely which produces drastically different results.

Moving Forward

Now, don’t get me wrong, I’ve had some really unique and great resolutions to issues from AI. But that is dependent on a myriad of reasons, including even how tired I am and how good I am at explaining what I want.

It brings me back to my dad, and the roadblocks I think AI training is experiencing. It doesn’t have the single or small circle of teachers, so we end up with a million “correct” ways to do something and the outcome is either sterile, or based on the methodology of multiple “cooks in the kitchen” that isn’t exactly what I wanted. Those that are successful in getting AI to give them the “wow” factor are able to inject their methodology into the AI to better produce responses that align more with what they want out of it.

For human-to-human training, we don’t have to do that. We can just reiterate the sections that we can see are problematic and never really touch on that issue again. We learn how they learn, they learn how we teach, and eventually get into a rhythm that works for both sides. I think of what I learned in college, you have Forming, Storming, Norming, Performing when starting with a new team. With AI it seems to be dropping me back into forming every single time I open a new window, even if I have given it broad context on what I’m hoping for.

That’s why, for many (including myself sometimes), it turns into a “just let me do it, because its faster” outcome instead of being able to teach it HOW to remember, how to respond to my questions, and how I expect a result.

Sometimes it’s easier for us to do it “the old way” because we still haven’t figured out what it needs to answer the questions clearly. Perhaps the better way is to start with questions that may take longer to get to the final solution, asking us what we are trying to achieve rather than assuming what we want because the smorgasbord of everyones ideas in one box, says we may likely want this answer.

This experience was a failure in my book. The outcome of the document I tried having AI create did not win me over because of its repeated failures. But in the end, AI is still early in its development. I'd classify it as an exuberant college graduate, trying to impress but still needing a fair bit of guidance. Until context improves and the walls come down in giving access to our life data thats been contextualized and collected for a decade or more, most people will still be weighing whether it's faster to just do it themselves with the tools we’ve been given for the last 20 years.