#233 – Chris Reynolds on Building Trust With Project Contracts for Reliable AI Development

[00:00:19] Nathan Wrigley: Welcome to the Jukebox Podcast from WP Tavern. My name is Nathan Wrigley.

Jukebox is a podcast which is dedicated to all things WordPress. The people, the events, the plugins, the blocks, the themes, and in this case, building trust with project contracts for reliable AI development.

If you’d like to subscribe to the podcast, you can do that by searching for WP Tavern in your podcast player of choice, or by going to wptavern.com/feed/podcast, and you can copy that URL into most podcast players.

If you have a topic that you’d like us to feature on the podcast, I’m keen to hear from you and hopefully get you, or your idea, featured on the show. Head to wptavern.com/contact/jukebox and use the form there.

So on the podcast today we have Chris Reynolds.

Chris has been immersed in the WordPress world for around 20 years, with a career spanning agency work, and now leading the developer relations team at Pantheon. His deep WordPress experience is paired with a playful approach to emerging tech, making him a familiar voice at events like WordCamp US and across the wider community.

In today’s episode, the discussion explored the fast changing landscape of AI assisted development. Getting into how large language models, like Claude, are not only changing the way developers build for the web, but also how they’re collaborating with these tools whilst mitigating risks.

Chris shares insights from his recent talk at WordCamp US, Parenting Claude: Guardrails for AI Assisted Development, where he introduces the concept of a project contract. Essentially, a structured set of rules and checklists built directly into the coding workflow to ensure AI generated code meets standards for testing, accessibility, security, and design.

We discussed the new found playfulness in web development. Chris makes the case that, thanks to AI, the traditional barriers to entry are eroding, enabling even those without a formal developer background to build an experiment much more easily.

At the same time, we need to be mindful of the importance of establishing robust guardrails, like automated reviewer agents, and pre-commit hooks, so the AI doesn’t take shortcuts or accidentally introduce chaos, or secrets, into your code base.

The conversation focused not only on technical solutions, but also got into bigger questions about trust, how much oversight AI really needs, and the evolving role of developers.

Chris emphasised that while these AI tools can accelerate and democratise coding, they can also pose a challenge to conventional wisdom around code review and software reliability.

So these new frontiers are both exciting and a cause of anxiety for developers who can sense the industry move from the deterministic, step-by-step processes, to the unpredictable and occasionally wild world of non-deterministic AI models.

If you’re interested in how AI is transforming web development, from practical workflows to philosophical ramifications, and how you can implement AI without losing control of your projects, this episode is for you.

Just before we begin, the manner in which this episode was recorded was not my usual process, and so the audio is not quite as good as I’d usually expect to deliver.

If you’re interested in finding out more, you can find all of the links in the show notes by heading to wptavern.com/podcast, where you’ll find all the other episodes as well.

And so without further delay, I bring you Chris Reynolds.

I am joined on the podcast by Chris Reynolds. Hello, Chris.

[00:04:11] Chris Reynolds: Hey, Nathan. Is this the first time we’ve met in human forms?

[00:04:14] Nathan Wrigley: Yes, it is the first time we’ve met. I’ve seen you on a screen.

[00:04:16] Chris Reynolds: I know I’ve seen you on a screen too.

[00:04:18] Nathan Wrigley: You’re joining me on the podcast today so that we can have a chat about the bits and pieces that you do in the AI space. But before we get into that, you are doing a presentation or have done, how did it go?

[00:04:28] Chris Reynolds: It went really well.

[00:04:29] Nathan Wrigley: Okay. And it was all about LLMs and constraining them and making them do your bidding as opposed to them doing whatever it is that they like to do. Do you want to just tell us a little bit, first of all, let’s do your bio. Just tell us about who you are, where you’ve worked, where you currently work, what is it you do in the WordPress space? Let’s begin there.

[00:04:46] Chris Reynolds: Well, I’m Chris Reynolds, I’ve been in WordPress for a really, long time, 20 ish years I’ve been using it. I’ve done kind of everything you can possibly do with WordPress in different forms. Of course, over the course of that 20 years.

Most recently edging into agency world with WebDevStudios and Human Made, before joining Pantheon, about five years ago. Joined Pantheon as a software engineer, working on the platform, and the sort of like space between where the platform ends and where, like your application, your WordPress site and whatever begins.

And then about two years ago, actually almost exactly two years ago, I joined the devrel team as a developer advocate, became a senior developer advocate. Now I’m a senior manager, developer relations. So I’m leading the developer relations team. And yeah, really love doing this stuff, because this stuff is really fun and exciting. I get to play with things more. And I get to like do the things that like our developers in our community are doing, and then come up with like what the sort of best practises for those things are.

Which is one of the reasons why I gave this talk, but also did the things that led up to giving this talk. Because one of the things that I realised very early on, like very early this year was that AI was going to be the thing that we’re going to be talking about in 2026. And so Pantheon and therefore me, needs to have an opinion about what that looks like. What is the future of that? And if I’m doing my job as a developer relations, I should have an opinion about how we can interact with AI in a way that doesn’t steal our jobs.

[00:06:13] Nathan Wrigley: I’m going to segue a little bit because about a minute ago you said the word playful, and that just set my mind going off in a completely different direction because it feels like technology for many years has been the workspace for people who have a fairly, how to describe it? They’ve got a certain mind, if you’re a developer, you probably have a certain way of approaching your code and you’ve got to be very regimented, and you’ve got to do things in a methodical way, should you wish to ship.

But it feels like last few years, like all bets are off with that. And creativity and playfulness has suddenly become a thing which is available to, going to do air quotes, vibe coders. So thoughts on that? Really? Is it a more playful industry than it was just a few years ago?

[00:06:55] Chris Reynolds: I think it depends. I don’t think that ever really went away. But I do think that there is less sort of incentive or focus to play with technology, particularly web applications, which is what we are in the business of doing. And I think there is this perception that I can’t do that ’cause I’m not a developer, but like with AI and vibe coding and Claude Code and things like that, like it opens the door to people who wouldn’t have been able to do those things before, to start experimenting and realise, oh, actually I can do that thing.

And for myself, one of the things that I, I wrote this on my blog a couple months ago. I wrote a blog post that’s talked about like how I don’t miss not writing the code myself, because for me, and it’s not the same for everyone, but for me, the practise of writing the code was never the part that was fun.

[00:07:50] Nathan Wrigley: Okay.

[00:07:50] Chris Reynolds: The part for me that was fun was building stuff. And I feel like I can let the robot do the coding and I can build the things and be satisfied with the output. And I don’t necessarily care what’s under the hood.

For my own stuff it doesn’t matter. But you know that obviously if you’re scaling that to a, you know, enterprise or whatever, you need to have more checks and balances than like my little nifty side project over here. But it does mean that like, I do a lot more weird stuff than I felt compelled to do before. So it has for me, revitalised this, sense of play, in a way that maybe before I was too burned out or too busy or too whatever, to have the time or head space to even think about.

[00:08:33] Nathan Wrigley: Yeah, because it feels like people with a non-developer background have suddenly kind of the constraints are off, and they can achieve things that previously it would’ve been 10 weeks work for a panel of amazing developers. It feels like now you can perhaps achieve that in an afternoon and a few sentences. I’m sure all the constraints around whether or not it’s ready to ship for enterprise that is all still understood. But just to get something on the screen which satisfies you and makes you happy. And that’s profound. Really quite an amazing thing.

And here’s my second and final pivot because you said that you were, a devrel. And it strikes me that maybe some of the audience don’t know what that means. So what do you do at Pantheon? What is the interface between rel and dev? What’s going on?

[00:09:16] Chris Reynolds: Developer relations is broadly, the practise of connecting with developers, and relating to them. It’s dumb. so, a lot of tech companies, that if you have a product, that is a developer centric product, or a product that developers work with, you might want, or it might be a good idea, to have a developer relations team to help your programmers who are building this tool for developers, communicate with the developers who are going to consume that tool, and understand how to use it.

So in Pantheons case, we are a hosting platform, but it’s a very developer centric hosting platform with built in environments and multi dev and all this other stuff. And that means that we benefit from having developer relations around so that we can help our developer community understand how to make the most use of all the things that we do for them.

[00:10:06] Nathan Wrigley: So you’re creating something simple to understand from something which previously was complex to understand.

[00:10:10] Chris Reynolds: Yeah. And, so that comes in many forms. It can be documentation, it can be blog posts, it can be YouTube videos. It can be live streams, it can be going onto podcasts, like giving presentations, whatever, or out in the community doing stuff.

[00:10:22] Nathan Wrigley: I suspect you’re going to have to answer this question with the company line, but do you enjoy that work?

[00:10:26] Chris Reynolds: I do. I actually do. Yeah. Like I was doing, I have said this a couple different times in different places. I was doing developer relations before I was in developer relations. And that’s not actually uncommon for a lot of people in developer relations. We were often like the people that are drawn to it.

The secret superpower, it’s not really secret. The superpower of developer relations people who succeed in it are the people that are technical. So they maybe were developers themselves, but also understand how to communicate with other humans. And can do the translation in their brain so that I can take geek speak over here, and translate it into actual English. And I’ve been able to do that for, that’s been a skill that I’ve had for a really long time. So I was kind of doing developer relations for all of that time.

But the way that surfaces and manifests is in things like giving presentations. Like anybody who’s giving a talk here at WordCamp US is doing devrel of some kind, because they’re a developer that is sharing an idea in a format that other people are intended to consume. How effective they are, because not all speakers are, equal. Not all developers are equal. And there are a lot of developers who don’t have really good communication skills and are not very good at public speaking. They can learn that as a skill, but there’s a broad range and the space that we live in.

[00:11:43] Nathan Wrigley: So, your presentation, do you just want to run us through what it was? I think if I was to give a summary of it, I would probably butcher it. Do you just want to tell us what the main premise behind your presentation, which you said you’d already done. Just tell us what it was about, what was the main points you were trying to get across?

[00:11:57] Chris Reynolds: So it’s called Parenting Claude: Guardrails for AI-Assisted Development. And the fundamental like underlying idea is that the way the LLMs are designed, the way that they function, is to try to find the fastest path from point A to point B. You give it a prompt, you give it a question, and it’s going to try to answer that question, but not necessarily with all of the data and statistics and whatever taken into account. It’s just going to try to find the quickest path from point A to point B.

And a lot of times what that means is circumventing, like just not looking at data. Because one of the examples that I give, and you’ve heard it in the news and whatever, like there, there are AI systems that were added to hiring systems, right? And the AI is prioritising these candidates over here, and deprioritizing these candidates over here. And when you look at the data, oh, all the people that are in the deprioritized pile, they’re all female. Because the company as a whole was hiring more men. So it’s looking at the data and it’s looking at who’s been a successful hire. And it’s oh, zip here is point A to point B, right?

We’ve historically mostly hired men for this role, so I’m going to look at the men and because they must be better candidates. That’s what, that’s the AI logic. It’s not that like we’ve necessarily intentionally built internal biases into the LLMs. It’s that it is able to surface those things, and if we’re not looking for them, or, and we’re not building guardrails around them, it’s going to just give that stuff to us.

And it does this in all sorts of forms, and it does this with tech, it does this with like data. It does this with everything, right? So you need to like, have ways of steering it into the right direction and forcing it into a shape so that it will do the stuff that you want it to do.

And what I was finding, in my role as developer relations, we have this Next.js launch coming up. It’s already happened, but when I was conceiving of this idea, we had this Next.js launch coming up, where we’re going to be hosting, or we are able to host Next.js sites as well. So I was like I should have a Next.js site. I’m a developer advocate I need to know how to do this thing. And I know nothing about Next.js, so I’m going to have Claude vibe code my personal website in Next.js hosted on Pantheon, and use as an experiment both to learn about Next.js, but also to learn about how Claude works.

And found out a whole bunch of things about like how to steer it in different directions. I want it to be, I want the code to be clean. I want the code to be tested. I want it to be accessible. I want it to be like all of these different things. And if I don’t have a way to force it to follow the rules, it’s not going to. It’s just going to say, oh yeah I did that, but it didn’t actually. Or maybe it won’t even say that it did, and then it’ll be like, oh I’m sorry I didn’t do it. Like whatever.

[00:14:40] Nathan Wrigley: Yes.

[00:14:40] Chris Reynolds: So the whole point, was to find a way to parent it, to guide it, without me having to be the one that is, did you do this thing? Did you do this thing? Did you do this thing? And going back and forth and have to yell at it all the time. Because yelling at AI is a fun pastime sometimes, and is a frequent pastime, I think of a lot of us that are using AI. But it gets pretty tedious after a while.

[00:15:03] Nathan Wrigley: But it’s become the new way. What’s curious though? You used the word parenting there.

[00:15:08] Chris Reynolds: Yes.

[00:15:09] Nathan Wrigley: But it, we’ll, swap that word no doubt during the podcast but, also you’ve called it a project contract. And, it feels from what you were saying as if, okay, let me rewind the clock a little bit. If I’m talking to a human being, let’s say that they’re an 8-year-old human being, I have this expectation that they’re going to behave like an 8-year-old human being, because they’re full up of eight years of things that humans do. And so I ask the 8-year-old a question, I have this anticipation that I’m going to get an 8-year-old answer back, and there’s all the boundaries there.

Ask a 15-year-old, same sort of thing, ask a 30-year-old, same sort of thing. And, I’ve got this notion that I ask you a thing, there’s a normal set of boundaries about what you are going to give back to me. But it sounds like in the AI space, it feels like there’s a potential for you to be basically talking to an empty box.

You ask it a thing and you really don’t have the faintest idea what’s going to come back unless you build the guardrails and tell it, I’m going to ask you a question, can I have an answer like this? Or, can you do these things in order to prepare the answer? Does that kind of sum it up? Is it a question of just shepherding it so that you’ve got a reasonable expectation like you would with a 15-year-old human of what’s going to come back at you?

[00:16:17] Chris Reynolds: I think, first of all, talking to AI, you’re talking to a 3-year-old because it’s only been around for three years. That’s right.

[00:16:22] Nathan Wrigley: But curiously, a 3-year-old with the entire corpus of scientific knowledge and the internet.

[00:16:27] Chris Reynolds: It’s true.

[00:16:27] Nathan Wrigley: So it’s a profoundly clever but at the same time, remarkably silly contrivance. Okay. Sorry, I interrupted.

[00:16:34] Chris Reynolds: Yes. Because of the way that it is trying to find the fastest path, it’s going to circumvent, it’s going to try to find loopholes in what you asked it to do. Because it’s looking for the fastest answer. It’s like your kid, it’s clean your room. “I’m going to shove all my stuff under the bed.”

[00:16:50] Nathan Wrigley: “Are we there yet?”

[00:16:50] Chris Reynolds: Yeah. Because now, and now the room is clean. You didn’t actually clean the room, you just shoved all the stuff under the bed. But it did accomplish the task that you asked it to do.

[00:16:59] Nathan Wrigley: Okay. That’s really an interesting analogy. So the task is achieved, but there’s chaos elsewhere.

[00:17:05] Chris Reynolds: Yes.

[00:17:05] Nathan Wrigley: It’s just hidden somewhere.

[00:17:07] Chris Reynolds: Yes.

[00:17:07] Nathan Wrigley: Okay. That’s really, okay. Let me segue, number three. Is that the intention of these large language models? Were they built with the idea of path A to B as quickly as possible? Or is that just a coincidence of how they evolved?

[00:17:22] Chris Reynolds: It is, it’s a side effect. It’s a side effect because what they’re trying to do is be helpful. Most of them are trained to be helpful. But they’re still computers, and when I tell a computer to do a thing, it’s going to do specifically that thing.

The thing about LLMs is it’s non-deterministic, so you never really know the path that it goes on to do that thing. But, and that’s where it meanders. But it’s still going to try to find the path of least resistance. And if the path of least resistance is, I don’t know, hacking the IRS or something that. It’s going to do that thing, right? It’s just going, it’s not going to care about the rules and guardrails on the other side, because the thing that you asked it to do is over there. It’s just going to try to get there as quickly as possible.

[00:18:05] Nathan Wrigley: So this non-determinism. So we’re in the year 2026 when we’re recording this, have that in your mind when you’re listening to this podcast episode. How does that sit with you? Because you’ve probably grown up in an era where determinism was the thing. You built, you write the code such that you put in A, B is what always comes out. If A goes in, B comes out. If you put in D, E comes out a hundred percent of the time.

And now we’re in a world where you put in E and I don’t know, a bunch of flowers comes out the other end. What? How? Like, how does that sit inside your developer brain? Is it alarming? Is it wonderful? Is it just curious what?

[00:18:41] Chris Reynolds: It’s somewhat alarming. Uhhuh. I think. It’s one of the things that I mention in my talk is, that this is the sort of thing that I think we as developers struggle with because LLMs are a black box. You never really know what it’s doing, even when it tells you a little bit of what it’s doing. Because you can expose some of the thought process and whatever, and sometimes it exposes its own thought process. It’s still doing things that it’s not exposing, that are like underneath somewhere where it’s like thinking what? It’s not thinking you’re a robot. Like you know, you’re not even a robot. You’re just like a bunch of like lines of like.

[00:19:14] Nathan Wrigley: There’s some electrons moving around.

[00:19:15] Chris Reynolds: Right? That’s, you’re not, thinking, there’s no thinking, but there’s something happening and it’s not telling you what that’s happening. And so it’s really hard to know what to anticipate, which is the other reason why, my talk, my opinion, like you need to build guardrails around it, to force it into doing things and then make it so that it can’t escape.

I don’t think anything that I was saying, or presenting or talking about is particularly unique, especially now. But the key thing that I found for me in my workflow was there needs to be a forcing function, needs to be some kind of a forcing function to make sure that it has to do the thing. It cannot complete the task without doing the stuff that you told it to do.

[00:19:53] Nathan Wrigley: So it’s literally like task police?

[00:19:55] Chris Reynolds: Yes.

[00:19:56] Nathan Wrigley: So, the idea is that whatever you ask it to do, it does it, but prior to giving you the final output, it runs through this project contract, as you call it. And hopefully that will then shepherd it in a certain way so that it’ll produce something which is adjacent to what you want to come out.

[00:20:13] Chris Reynolds: Yes.

[00:20:14] Nathan Wrigley: Okay. So how does that work? Where does this project contract live? How do you in, I’m guessing English in your case, how is it that you write that in a way which is, you’re codifying everything, you make sure that all the T’s are crossed and the I’s are dotted. Just run us through the process of producing a project contract. And obviously, the nature of the topic itself would I’m sure define that, but just give us some rough ideas.

[00:20:40] Chris Reynolds: There’s a lot of markdown files. Fundamentally, like Claude is supposed to read, at the beginning of each new session, the contents of a Claude.md file that is in the project directory. Wherever you’re running Claude, you can give it a Claude.md and it will, that will give it an understanding of whatever, how it should interact with this project.

Within that and beyond that, you can have a bunch of other different files. You can tell it about, you have a docs folder and you have more documentation there. There’s also an agents.md file, which you can use in various different ways. It’s another sort of markdown file that’s only for agents.

And, in my case, what I was talking about, is there’s a reviewer agent. So it’s specifically like you talk to Claude Code, you say build this thing, and then after it’s done building the thing, a reviewer agent is spawned to check the work of the first agent. Make sure that it did a whole bunch of stuff, and then if it didn’t reject the code.

And so there’s a bunch of files in my repository that are about how the reviewer agent works so that Claude understands what to expect. And then there is a checklist that has, these are the specific line items that I want every single commit to pass before this is an acceptable commit.

And it’s the whole gamut of everything, right? It’s like, does the build pass? Does linting pass? Does the test pass? There’s accessibility pass? All these things. Did you accidentally, or intentionally, commit like secrets to the repository? Stuff like that. It needs to go through this whole checklist. And then, if it passes, it writes a file to the file system that says, okay, approved. And this is the timestamp. And then that allows, if that file exists and the timestamp is valid, then Claude can actually make a commit, and it will actually commit the code. If it does not have that file or the timestamp is too old, then it will be rejected.

And I have this hooked into a pre-commit hook. So if it doesn’t trigger the reviewer agent on its own, if I tell it to commit the code, it’s going to hit that gate, and force the reviewer agent to spawn. So it can’t get around it. And that’s the forcing function.

A couple people are asking about other workflows and whatever, and how to do it in different environments where maybe you’re not using a Git repository or something. And, the key is to find a forcing function. Find something that can’t actually weasel its way out of it, because it’s going to try to weasel its way.

I had to write in a bypass for me to personally be able to commit code, because otherwise it’s going to ask for the reviewer agent. And it will try to use that bypass that I wrote for me because again it’s an unruly child. So it’s going to try to do that thing and, then you yell at it and tell it not to do that, and it says, oh, I’m very sorry.

So it’s going to try to find these loopholes and circumvent everything that you’re trying to do. So the way that you force it to do all the stuff in this long checklist that I have is by creating something that it just can’t get around.

[00:23:27] Nathan Wrigley: So I can imagine that the people listening to this, the sort of AI doomers saying, “But Chris, it’s just AI all the way down. You’ve got AI checking AI.”

[00:23:34] Chris Reynolds: Yes.

[00:23:35] Nathan Wrigley: Which on the face of it seems really peculiar. And also the fact that the guardrail is at the end of the process as opposed to, you would kind of hope that it would get it right at the beginning, but then you might just have to reject whole things based upon it. And then the fact that it sounds like from what you were saying, as if AI has almost like a predilection to misbehave.

[00:23:57] Chris Reynolds: I assume it has a predilection to misbehave.

[00:23:59] Nathan Wrigley: But does it feel like that? Is this predictability to, if I don’t put the guardrails and I don’t put this sort of corralling function in, what I’m going to get out is just going to be aberrant and useless and just who knows what’s going to come out?

[00:24:12] Chris Reynolds: I think that’s a little bit extreme. I said that I assume that it’s going to misbehave because if you don’t assume. Basically if you don’t prepare for the worst, it’s going to bite you in the ass.

[00:24:21] Nathan Wrigley: Yeah.

[00:24:22] Chris Reynolds: So you have to assume. And one of the things I’m sure that people who are listening who have used LLM coding tools specifically, will be able to relate to, is oftentimes, this is one of the first discoveries that I had when I started using this stuff. You ask it to build a thing and it builds that thing, but there’s one thing wrong with it. So you ask it to fix that one thing and it refactors the entire thing.

And in the course of refactoring the entire thing, it lost some other thing that was there before that you liked that you want to have back. And so you tell it to bring that back and it rewrites the whole thing again, right?

That’s how it works, because a lot of times it’s not able to like even retain the actual memory of the original state. So it just writes things from scratch every single time. Which from a human perspective is like bonkers, right? But from a computer standpoint, it’s oh this is a new interaction, so I’m doing this fresh.

So the way that you solve for that problem is okay, if I force you to write tests for every single thing you’re doing, then you can’t go and revert a bunch of stuff or refactor stuff without those tests failing.

[00:25:31] Nathan Wrigley: Right?

[00:25:32] Chris Reynolds: And again, starting to like build things around it to force it into a shape so that it’s actually the thing that is closer to what you want. What I’m hearing is anxiety about the uncertainty that non-determinism, the like chaos, you never know, I don’t know. It doesn’t bother me. And I don’t know if that’s a character flaw of mine that it doesn’t bother me, as much. Or if it’s just like being comfortable with the chaos, but also understanding the chaos enough to know what to expect.

Because the other thing I think that we’re seeing right now, in this moment in August of 2026, in Phoenix, Arizona, is that a lot of people are looking at this stuff for the very first time. A lot of people are just now coming to Claude Code, or have only started using it in the last couple months because it got really good around like November, December last year. Like, all of a sudden they did an update and it was just phenomenal, and it was able to do way more. And not just Claude, but like all the LLMs, they got really good at code. All of a sudden, and so now it’s a big thing, and so now more people are using it.

But I’ve been playing with LLMs since the very first Chat GPT model. I say that because there’s a lot of stuff that I learned over the course of the process, and I wasn’t using it for work necessarily. I was just using it for play and for, Dungeons and Dragons and whatever. But a lot of those things that I learned over the course of using it in those contexts, those non-work contexts, led me to understand the limitations, and where it falls down, and where it does well and the things to expect. So that when we got to this point, I already had an understanding of this is how it works, so these are the things I need to be looking for.

[00:27:15] Nathan Wrigley: Yeah. So that’s really curious. So the history that you have with it has given you a real deep understanding of what the expectations are for “Okay, this is typically where it doesn’t perform well or these are the things that it typically can do well.”

[00:27:26] Chris Reynolds: Yeah.

[00:27:27] Nathan Wrigley: Okay. So with your guardrails built in, so we’ve got your project contract that you buried inside the project somewhere, and it’s doing all these checks and balances. Where does that take you in terms of confidence?

I don’t know if you want to put a number on that, like a percentage. How does it take you from zero to 99% confident in what’s coming out? Is it more like 80%? Is human review needed in every single circumstance, or. Just run us by the confidence that you get from producing that document?

[00:27:54] Chris Reynolds: I don’t review line by line the code that AI produces.

[00:28:01] Nathan Wrigley: Just curiously, at what point in the calendar did that moment occur where you suddenly said to yourself, you know what, there’s just no, I’m just reading everything and everything’s great. There’s just no point in me checking everything now.

[00:28:13] Chris Reynolds: I mean, probably way earlier than it should have, if I’m honest. Yeah, I don’t, and one of the arguments that I’ve actually heard some of the engineers that I work with, at Pantheon, say is the fact that no one is actually reading all of that code generated by Claude or whatever.

And, his argument is like, basically no one’s going to do that because it’s so much, and it’s here and whatever. So you can say that you’re doing it, but you’re probably not. And if you say that you’re doing it and you’re not, you’re just lying to yourself. So it’s better to be honest and be like, yeah, no, I don’t actually, than to pretend like that’s not the case. So I don’t actually review a lot. And not like intentionally, like I will look at it. The demo, I gave a live demo yesterday, which is exciting because it’s a non-deterministic thing.

[00:29:02] Nathan Wrigley: Oh yeah. Good luck.

[00:29:04] Chris Reynolds: But what was interesting about that is because the scope of the task was very small. It was like, there’s a horizontal bar on mobile devices on this specific page on my site. Very specific thing. And I, when I practised it, half a dozen times or more, it got to the same conclusion every single time. When I demoed it yesterday, it got to the same conclusion.

And I can look at the code and okay, there’s the tests. I’m not going to read the test and how it wrote the test, but I can see what the change in is. It is just adding a class to a div. And it pretty much was the same class to the div every single time. So like, I can review that and be pretty confident with that. That’s a hundred percent confidence.

Also extremely small change, but that’s also at the end of yeah, it went through the 45 point checklist. It did all of the things. It wrote the accessibility, it did the, like everything is following these rules. And because I have all of these rules, and because I’m fairly confident that it’s actually going through that list, that also makes me feel like, okay, I can be a little bit, it’s a little bit more trustworthy, when it produces this output, then it was before it had all that stuff in place.

[00:30:10] Nathan Wrigley: Yeah. So in my head, there was a period of time where I didn’t trust that if I said to my son, what’s three add seven, that 10 would come out. And then there was a moment where I don’t need to ask him that question anymore. I have complete confidence that if I were to ask him that question, he would say 10.

[00:30:26] Chris Reynolds: Sure.

[00:30:26] Nathan Wrigley: And it would be 10 every single time.

[00:30:28] Chris Reynolds: Yeah.

[00:30:28] Nathan Wrigley: And it I think, I feel like it’s a bit like that. Over time you’ve just grown accustomed to, it can do this kind of thing. I now trust it in that sphere. Does that sort of sum it up? You, personally, you have built up that level of trust, but it wouldn’t necessarily be the same for me if I started my journey. I would have to go through the building up of trust in the same way that you did.

[00:30:49] Chris Reynolds: Yes.

[00:30:49] Nathan Wrigley: Okay.

[00:30:49] Chris Reynolds: But I do think that, like the reviewer workflow that I built can help you get there. Because if you put that into your project, and customise your list of, your checklist for all the things that you want to have running on everything that you are running. The goal is to make it more trustworthy, and to make it more do the thing that you expect and wanted to do.

[00:31:10] Nathan Wrigley: Could you just tell us some of the things that are in that document? So I think you said 45 and maybe that’s subject to change. Maybe next week it’ll be 49 or what have you. And maybe some of the more curious ones that wouldn’t necessarily drop out. I can imagine that you would ask it to do certain things, de rigueur, every time that, but some of the more curious ones, perhaps ones that you’ve come up with lately that our listeners might think, oh, okay. I’ve never thought of that.

[00:31:31] Chris Reynolds: I have it use test driven development or I tell it to use test driven development. And the reason for that. So test driven development is when you write the test and then you write the code to pass the test.

And it turns out that AI can do this far better than humans, because humans don’t want to do this, because it’s like, it breaks your brain. But an AI can do it, and if an AI can do it, then I’ll let the AI do it. And by having the AI write the test that the code needs to pass in order to accomplish the task that you’re asking it to do first, then that means A, again, you’re never going to have regressions, because the test is already there. And B, it already knows how to fix the thing, because it wrote the test to make the thing fail. So that’s a sort of a important nuance, in there, that isn’t necessarily, like you can write unit tests, whatever, but starting with the test is really valuable.

There’s a couple things that I have in the checklist specifically. The big one, I think that is maybe curious is making sure that it doesn’t write secrets to the file system.

[00:32:33] Nathan Wrigley: Even saying that out loud is so peculiar.

[00:32:36] Chris Reynolds: Because it will do that. And the reason why that rule has to exist is because there was a point in time when that rule didn’t exist and then it wrote this. And that’s even like secrets in, WordPress has a weird concept of what a secret is anyway, because we have API keys in our wp-config files and stuff like that. And theoretically we shouldn’t be doing that either. But we don’t have a really good universal form of like, secrets management. So it gets wibbly wobbly. But yeah, so not writing secrets to the file system is pretty important.

Yeah, honestly it’s really, it’s not super wild stuff. It’s like making sure tests pass, making sure you have accessibility checks. I guess one thing that is maybe unique So I designed this site with Claude too. Which is also an experiment because it’s like, I have no idea how AI is going to approach design. And so one of the things, I know like at my workplace, we use Storybook, for our design system. And I thought, that seems like a good tool. Maybe I can have AI build a Storybook.

And so I basically built this system so that it will, when I want it to build a new component, it builds the component in Storybook so that I can review the design first, and go back and forth there, and then implement the component into the thing. So there’s stuff in there about, if you’re introducing any new design component, then it has to be done in Storybook first.

[00:33:56] Nathan Wrigley: Okay. Yeah. Yeah. That, some interesting stuff in there, but I guess everybody’s going to have a sort of different file.

[00:34:02] Chris Reynolds: Yeah.

[00:34:02] Nathan Wrigley: Okay. So I wonder if you have any intuitions as to the process that you’ve just described and that you’ve gone through, August, 2026. Do you have an intuition that in the next few years, this kind of thing, the AI companies will solve this problem, that it will become more truthy by default? Or do you see it, that this is a thing that we’re going to have to wrestle with from now on? That our systems, our AIs, there’s always going to be this strange tendency to hallucinate, to make things up, just to go off the rails and do peculiar things.

Now obviously I’m asking you to stare into a crystal ball, but I do wonder if there has been a more truthful output over time, or if it’s just getting wilder and wilder.

[00:34:42] Chris Reynolds: I think that what we will see more of is more integrated like agent orchestration. So what I’m doing is essentially like a form of agent orchestration. I’ve got one agent doing the one thing, I’ve got another agent that reviews the work of the first thing.

And even like the keynote yesterday, she was talking about a not very dissimilar process, where like there’s five different agents and reviewer was one of them. There’s like a planner, there’s a coder, there’s a whatever. And I think that is a thing that we’ll probably start to see more of and then maybe at some point. when you talk to Claude, when you write a prompt into Claude, you’re talking to a thing. But underneath the hood, in the black box of Claude’s brain, there’s actually five different agents that are running concurrently that are processing this thing and doing their different tasks or whatever. So when it’s thinking, then it’s doing five different, like it’s going through that process. I think maybe that’s the thing that might happen.

[00:35:40] Nathan Wrigley: I’m just so fascinated by how, by now the year 2026, we’ve just all accepted that this is how it is. But if you could put 2021 Chris in that chair, you are sat over there somewhere listening to you talk, you’d be, “No, this is Star Trek. This is literal science fiction. I haven’t the faintest idea of what he’s talking about. None of this adds up, computers can’t do that. They’ll never do that.”

And yet here we are, we kind of blew through the Turing Test, like it was nothing. And that felt for the longest time, like that was going to be an important milestone. And everybody checked their watches like, “Oh goodness, that happened.” It is remarkable how quickly the landscape is changing.

[00:36:17] Chris Reynolds: What’s really interesting is one of the first people that I read about AI stuff, she’s an AI researcher, and she specifically focuses on small language models like the very old models that existed before GPT and whatever. And fundamentally how these things all work is very very similar. But like back, so this is getting into 2021, 2022 era of stuff, where like, it exists but it’s really bad. I don’t know how many people still remember, Horse eBooks, the Twitter account.

[00:36:49] Nathan Wrigley: No.

[00:36:50] Chris Reynolds: But it was like this bizarre account, that was a bot account, that was just like an AI or machine learning or something, some robot just like spitting garbage into Twitter, that just made no sense at all. And was, it’s, hilarious. And you also remember, in the not terribly distant past, when we would get like, weird phishing emails, but it would have different tone changes in voice. Like it was very, obviously written by a robot or like spam messages and stuff like that. And that was just what existed at the time.

None of the things that exist today are like, they all come from the same root, but for sure, the flip from this is garbage, to this is genuinely useful, or at least sounds humanlike, was it, felt like overnight. And it felt like a wild flip. When Chat GPT first became available for the public, that was how is this even? And also like there is the whole thing, very shortly after that when Microsoft released the Bing Chat bot.

[00:37:53] Nathan Wrigley: Yes.

[00:37:53] Chris Reynolds: There was the whole thing about like Bing wanting to marry this reporter. And that goes to show how little we even know about the things that we’re building. Like, none of the AI companies really understood, or even still I think understand? Why it works the way it does.

[00:38:11] Nathan Wrigley: That’s quite an alarming admission actually.

[00:38:13] Chris Reynolds: It’s just, it does.

[00:38:14] Nathan Wrigley: Yes.

[00:38:14] Chris Reynolds: So we’re just going to keep going. Which is also, I find that alarming from the perspective of, now they’re asking for like huge new data centres across everywhere. I’m sure anybody listening to this is angry about data centres, and I’m angry about data centres. But like, that’s with the belief that those data centres are going to matter meaningfully. And I don’t know if they will.

Like we don’t, because we didn’t know how we got here. And it’s doing the thing now. I can use it to code my site. Like I can do that today. Do I need 50 new data centres across the United States to make that better? I don’t know.

[00:38:55] Nathan Wrigley: You’ve almost described the perfect dystopian science fiction plot there, haven’t you in a way? This stuff just all happened. It gave a really beguiling approximating humanlike responses that got more human over time to the point where we couldn’t.

[00:39:08] Chris Reynolds: Did we just invent Skynet?

[00:39:09] Nathan Wrigley: Yeah. We, can’t detect that it’s not a human anymore. It really does feel like, somewhere hidden from our view is an evil overlord stroking a cat or something like that.

And of course, we don’t really know how we got here, but I am so beguiled by it. I’ve got the same concerns as you. Data centres, the environment, all of this wasting of the Earth’s resources. But at the same time, that doesn’t seem to stop me from being curious about it and trying it out and the AI is just wrong for everybody else to use. But when I use it, oh, it’s perfectly fine. But, you know? That’s a thing.

[00:39:41] Chris Reynolds: Yeah.

[00:39:42] Nathan Wrigley: Okay. I think we’ll knock it on the head there, as we say in the UK. Is there anything you wish that we had covered just before we end it? Or do you think we got through most of what we needed to.

[00:39:50] Chris Reynolds: I think one of the things that gives me pause or concerns me most about AI, is the sort of like handing over that amount, or an amount or any amount, of sort of cognitive load to AI. And therefore just letting those parts of our own selves, and our own brains like dwindle and wither.

And I think about this from the perspective of just me personally, my kids, my family, whatever using it and then like becoming less skilled ourselves. But I also think about it from the perspective of, if we’re in engineering, in development, in the world that we are working in. If we are telling everyone, “oh your job now is to be essentially a senior engineer reviewing the work of a junior developer, that is the AI,” how do senior developers exist?

[00:40:41] Nathan Wrigley: You’ve removed that bottom rung of the ladder, haven’t you? Yeah. You’ve taken away the capacity for somebody coming out of college to go on the first step. And then the second and then the third. Because the AI did all the first, second and third step.

[00:40:52] Chris Reynolds: Like I understand that there’s a different set of skills, and a different set of rules and whatever, things that are going to be part of your job, to be that reviewer, to be that new engineer that uses AI to be the developer. But there’s certain things I think, that are part of that learning journey that got me to where I am, that got other senior developers to where they are, to understand the context, to be able to guide the AI that I don’t know that you can circumvent that. I don’t know that you can bypass that.

[00:41:22] Nathan Wrigley: I think you’d be curious if the AI company’s employed like a really credible team of philosophers. You know what I mean? Like for every a hundred developers, we have 10 philosophers, who are just going to think and worry about what the heck’s going on. It sounds ridiculous, but maybe that’s what we need because, some of the things that you’ve described today, they’re just so curious that you need to shepherd it, you need to constrain it, you need to worry about the bizarre things that it can get up to. We need to worry about the future, the data centres and all of this.

And, on some level it does feel a bit like we’ve got a blindfold on. We appear in our own minds at least we appear to be going in the direction that we want to go in. But I don’t know, I guess only time will tell. If we’re all here in 2027.

[00:42:05] Chris Reynolds: That will be the thing.

[00:42:05] Nathan Wrigley: Then we succeed 2029, who knows? All bets are offs.

[00:42:09] Chris Reynolds: Yeah. I can count three years in the future that my job is probably not going to go away. But beyond after that, beyond that, I don’t know.

[00:42:15] Nathan Wrigley: Okay.

[00:42:16] Chris Reynolds: I’m pretty sure, I’m pretty sure my job is okay because I do think, developer relations at all about talking to humans, but maybe there’s not going to be any humans to talk to, so maybe I’ll just be talking to AI.

[00:42:24] Nathan Wrigley: Okay. WordCamp US 2029. It’s a date. We’ll be back at that point.

Chris Reynolds, thank you for sharing all of those thoughts and, yeah, that’s been really educational. Thank you.

[00:42:34] Chris Reynolds: Thank you for having me.