“That thing you built is down - how do we fix it?”
Despite the pure bliss of shipping something you’ve built with AI, if you’ve actually launched a few things on your own, you probably know the feeling in the pit in your stomach when something breaks. Or just the looming sense of dread when something you’ve created starts to expand and you’re not really sure how to handle it.
Yikes.
If you come from a non-technical background, this can be fatal for your momentum. There’s so much possibility for what LLMs can help us create, but most AI tools are not designed to teach you how the underlying infrastructure works. So when things fall apart, it’s not always clear what happened or why.
Just to be clear: You don’t need to feel bad about this! Jumping in to start building is often the best way to learn. You’re probably trying to solve problems that you didn’t previously know existed, and you likely don’t even have a name for what you’re trying to do.
That’s where I’d like to help. I’m currently a marketer, but I came up through the world of software engineering, getting my degree in CS and working as a dev for years before making the switch. So I have a unique perspective on where this stuff is going wrong. And this particular challenge - where you build something, but it breaks down the road - is a perfect example.
Because this isn’t a new problem for software builders. In fact, it’s a classic. We even have a name for it: This, my friends, is a Day 2 Problem.
What’s that, exactly?
Day 2 Problems are the long-term issues and challenges that show up when growing and maintaining a product, after the planning (Day 0) and initial building work (Day 1) are done.
If you’re having trouble imagining what I mean, here’s an example:
The story of Dave’s really bad week.
Meet Dave. He's on the finance team at a 100+ person company and has a can-do attitude. Everyone loves Dave. He’s not an engineer, but he’s always trying out new software and he’s already showing other people on the team how to build things with AI.
This week, things went wrong. Very wrong.
Monday
Dave’s team is drowning in invoices. Vendors send in PDFs by email, but then someone has to manually type the information into the system by hand. It takes forever, so everyone is several weeks behind.
So after thinking about it over the weekend, Dave hits the ground running on Monday morning. With the help of an AI tool, he’s built an automation: Invoice comes in, AI scans the PDF, key information is extracted and uploaded. It even flags if something looks off.
Hurray, Dave! Everyone is so impressed and his boss tells him, “This is the kind of thing we can use to help you land that promotion we’ve been trying to get for you.” Dave is practically dancing in his car on the commute home.
Tuesday
First thing in the morning, the accounts payable team pings Dave: 3 invoices posted with the wrong amounts. And not by a small amount. Dave scrambles to figure out what happened: Did the AI misread the PDFs? Did it not properly upload it to the system?
He opens his automation and… there’s no way to know. There’s no record of what the AI did for each step - it just says “Automation complete.”
So Dave starts poking around. He tweaks a few things and finds the issue. Fortunately, it’s minor, so he fixes it and updates the automation.
Once it’s live, he does a test run. … and it doesn’t work. Uh oh. This is worse. Time for Control-Alt-Delete! He goes back to the system to try to go back to the previous version, but that’s not a thing. He edited the only version and there’s nothing else there.
Dave spends the next few hours rebuilding his original version, and then adding in the new fix. Then the rest of the afternoon triple-checking to make sure it all works before updating right before going home.
Hopefully everything is fixed now?
Wednesday
Dave is stuck in back-to-back meetings all afternoon. At 2pm, the automation program requires an update. It just needs to be reset, the error message says. His teammate Marco offers to jump in.
But… where? Dave is the one who always does it. How do you even get to the editor? There’s no documentation. And when Marco finally finds someone who knows, the login credentials are tied to Dave’s email and there’s a password that is not Password123.
Marco texts Dave. Dave, who is currently pretending to pay attention in a quarterly planning meeting, tries to text back instructions under the table. It doesn’t go well. The invoices sit until Dave is free at 4:30.
At least the problem is resolved by 4:31!
Thursday
Dave’s boss still believes! She sees the vision and knows that it’s already saving their team significant time. So she asks the natural next question: “Can we roll this out to the Austin office, too?”
Dave pulls up the automation and his stomach sinks. Everything is hardwired: one email inbox, one set of vendor formats, one connection to one accounting system. To make it work for Austin, he’ll have to do more than just tweak some settings. He’ll pretty much have to rebuild it from scratch. Again.
He tells his boss, “Sure, no problem!” And cancels his after-work plans to get it done.
Friday
Smooth sailing! After all the bumps, everything seems to be running correctly.
Until 3pm, when a vendor calls: Their invoice hadn’t been paid this week. Dave checks: the automation never processed them. It couldn't parse the format, so it just… skipped them. No error. No flag. Nothing.
Even worse: That’s not the only one. Dave starts digging and finds 11 more invoices from the week that were silently skipped the same way. Every day this week, while he was troubleshooting the problems he could see, there were more problems that he couldn't.
His boss stops by. Dave does not feel like dancing on the drive home.
Dave had a lot of Day 2 Problems
Now, most “Day 2” problems don’t literally happen on the second day. But the experience of things breaking in unexpected ways, or requiring an update after the initial launch is unavoidable.
Here’s the bad news: There are way more things that could have gone wrong. What about security? Or the AI model getting updated and suddenly giving very different outputs? The longer the system runs, the more vulnerable it is to falling apart.
But fortunately, I also have very good news for you: This is not a new problem! Dealing with Day 2 issues is something software engineers have been dealing with for decades. You can learn from their experience and save yourself a lot of heartache.
So how do you do that?
My #1 piece of advice for all my friends, coworkers, and internet strangers is this:
You solve Day 2 problems by asking yourself the right questions on Day 0.
How do we solve this? By asking questions.
The real underlying issue with Day 2 problems is lack of awareness.
Most of Dave’s problems weren’t even that hard to fix. He just didn’t realize they even could be problems until after they happened and he was scrambling to fix them. Like I said at the top: I don’t blame Dave, and I wouldn’t blame you, either. You don’t know what you don’t know.
What Dave should have asked on Monday
Dave and his team actually got one thing right: They knew who owned it. Before everything else, make sure that’s clear.
Back in 2006, Amazon's Werner Vogels famously introduced this principle:
"You build it, you run it."
If you’re not the right person to actually maintain and sustain something you’re building, you need to be very clear about who will? So whether it’s you, or you plus some other person who will be owning in the future, ask yourself these questions, before you have a week like Dave’s:
- How will I track what the system does at each step?
Dave couldn't tell whether the AI misread a PDF or the upload failed, because nothing recorded the inputs and outputs along the way. Any ongoing project should have some degree of traceability, so you have a record of what has happened.
(Shameless plug: This is one of the main reasons people love using n8n. A visual workflow makes it dramatically easier to see where things went wrong, and n8n records all of the inputs and outputs for every step of every execution.) - How will changes be made, and how will I undo them?
Dave edited his only copy and wasn’t able to go back. Before you launch, decide: Where are versions saved? Can you quickly roll back updates? Can you test a change somewhere that isn't live? - Who else needs access, and what should they be able to do?
Marco couldn't help because the credentials were Dave's and all the system logic was in Dave’s head. If other people will use or depend on it, figure out shared credentials, permission levels, and enough documentation that someone else can operate it without texting you. - How might this need to grow?
Dave's boss asked to expand the project and the whole thing needed to be rebuilt. Before you build, ask: Will this expand? How much? Building with that in mind from the start is far cheaper than retrofitting later. - How will I know if something is working (or not working)?
Dave's automation looked fine all week, but was secretly skipping invoices. If your system needs to run in the background, make sure there’s a way to monitor that it’s running, and ideally, will notify a human if it fails.
Beyond Dave's week
Dave got unlucky, but he also got lucky. Even in his disaster of a week, more could have gone wrong. Here are a few more questions you should consider:
- How secure does this need to be?
If someone broke into Dave's system, they'd have access to vendor payment data. Ask yourself: what's the worst thing someone could see or do if they got in? If the answer involves financial data, customer information, or anything federally protected, security can't be an afterthought. - What happens when the AI changes?
Traditional software gives the same output every time until someone changes the code. AI doesn't. Model updates can adjust how a prompt is handled, meaning that the same input gives a different result. If your product depends on consistency, you need a way to test that it still behaves the way it did last week. (The industry term for this is "evals" - Think of it as QA for AI.) - How much does this cost to run?
Maybe Dave’s system only costs 2 cents for each run, which is invisible in the first week. But if it becomes a default part of the company and scales to other locations, it may require a major budget approval.
As you’d imagine, experienced software engineers have a much more comprehensive set of areas to review.
One popular way to describe them is the “ilities,” since so many of these topics end with that suffix: maintainability, portability, scalability, etc. As I was learning all of this stuff in school, we were taught that there are 61 of them!
Knowing how deep to go really depends on what you’re trying to build:
If you were building a, well, building, what would this be: a shed, a 2-story house, or a 20-story building? If you’re building a quick, simple app, you don’t need to be so exhaustive. But if you’re trying to build something that needs to run reliably and deal with complex needs, prepare accordingly.
Maybe you need to bring in a professional? Just like some people build their own outdoor sheds, but not their own nuclear reactors or industrial bridges, there are times when bringing in an engineer is the right move.
Wait, can’t AI do this for me?
You may be asking yourself: Wait, why doesn’t AI solve this, already? Given that it’s been trained on massive sets of software engineering data, why doesn’t it answer (or even ask) these questions, out of the box?
The problem is that LLMs are optimized to give you the output you’re looking for, and if you don’t know what you want, then it doesn’t either!
So, when you say “Build me an app that does X,” it won’t assume that what you actually mean is “Build me an app that does X, with the ability to provide third-party access, and the ability to expand to adjacent use-cases, and make sure we know when it fails or does something wrong.”
You just need a bit of nudging. If you ask, “How should I handle version control?” in my experience, it will often respond with something like, “Good point! We should actually add that, like this.”
But you have to guide it in that direction. And you have to know what you actually want!
Not sure where to start? Just use a prompt like this to give the AI the correct context and get it to ask you what you really need:
“I'm not a professional software developer. I want to make sure this project follows best practices for software engineering: things like maintainability, security, observability, and scalability. Interview me about my project and help me identify what I should be thinking about before I launch.”
Building is a process, not a project.
So Dave learned his lesson. Building software is not something you do one time and then forget about it. Once the thing is live, there’s just as much work keeping it going. It turns out, Day 2 is the longest day!
But that’s not a bad thing: It’s actually kind of a mark of honor. Products only make it to Day 2 if people enjoy using them or find enough value to keep coming back.
So if you’re going to start building, give your product a chance at a long and happy life.
Prepare for Day 2 by asking the right questions on Day 0!