I've been building software products for over twenty years. That experience taught me one useful thing about new tool waves: break them early, and figure out what actually holds up.
When AI coding assistants arrived, I did exactly that. Early versions were fine for autocomplete. Anything requiring system-level judgment fell apart fast. Context was shallow. Reasoning broke down. A senior engineer could spot the failure before the model finished its suggestion.
So I parked AI in the "interesting, but overhyped" category and moved on.
That changed in early 2026. I tested the newer models across three focused weekends on real work. One weekend: a working cross-platform desktop app, without me writing a line of code by hand. Another: a full-stack web application (Rails, PostgreSQL, React, authentication, document processing) in a few days.
On the third weekend, I applied the same approach to a mature production codebase. It collapsed. The AI could read the code but couldn't understand the architecture or the decisions behind it.
That gap isn't a reason for skepticism. It's the entire point. The question was never whether AI "works." It's where it helps, where it doesn't, and whether your team can tell the difference in real time. That ability is what I'd call AI fluency - and it's the biggest skill gap I see across engineering teams right now.
What it actually took to make AI a real instrument
At Railsware, we have around 200 people, roughly half engineers, the rest product managers, designers, marketers, and recruiters. Everyone started experimenting with AI at roughly the same time. And for a while, it went nowhere in particular.
The turning point wasn't the tools. It was knowledge sharing. We added Slack channels for AI discoveries, internal webinars from people who'd actually cracked something useful, shared prompt libraries, and internal tools built by engineers who got tired of explaining the same thing twice.
What's worth highlighting: we treat AI the same way we treat Figma or GitHub. A professional instrument, not a novelty.
Then we did something that forced the issue: we put AI fluency into our competency model. We defined what it looks like at junior, senior, and team lead levels, how to evaluate it, and how it factors into hiring. That made it real.
How we evaluate it
AI fluency isn't binary. It develops in layers, and it looks different depending on your role and what you're accountable for.
At the individual level (mostly juniors), the work is about your own output. Engineers learn to give AI enough context to get useful code instead of generic scaffolding, and more importantly, to recognize when the output looks right but doesn't fit the system. Product managers learn to describe flows precisely enough to get prototypes worth testing. Content people learn what to trust in a draft and what to rewrite entirely.
The rule is the same across roles: you're the last check before anything leaves your hands.
At the team level (seniors), fluency shifts from personal output to others' output. Senior engineers build shared workflows, review AI-assisted pull requests, and catch patterns that are technically correct but architecturally wrong. Product managers use AI to stress-test specs before they reach the engineering team, cutting wasted cycles on things that won't survive contact with the codebase.
At the organizational level (team leads), it becomes about decisions: where AI belongs in the workflow, where it needs guardrails, what data can and can't go into a prompt. Speed gains are easy to measure. Quiet quality degradation is not. That's the job at this level.
Across all three, we evaluate four dimensions, just with increasing scope:
- Effective application - turning problems into structured AI interactions. For a junior, that's a well-contextualized prompt. For a senior, it's a reusable workflow the team can build on. For a team lead, it shapes how processes are designed around the tool.
- Critical validation - knowing when to trust the output. Juniors check facts and review generated code line by line. Seniors define which use cases are safe and which aren't. Team leads set governance boundaries for the whole team.
- Ownership of output - treating AI output as a draft, not a result. Juniors review everything before it ships. Seniors teach the team to recognize what wasn't properly reviewed. Team leads enforce quality standards regardless of how the work was produced.
- Responsible use - handling data and risk deliberately. Juniors don't paste sensitive information into a prompt. Seniors write guidelines and provide peer oversight. Team leads build compliance structures.
The framework stays consistent across levels. What changes is the scale of what you're responsible for.
Building with AI still requires builders
At some point, every sufficiently useful tool stops being discussed and starts being assumed. The same way electricity became part of every industry. We stopped debating whether to use version control. We'll stop debating whether to use AI.
But having access doesn't guarantee better output. System design, product thinking, and navigating a codebase with years of decisions baked in - these still require people who understand what they're doing and why. AI doesn't change that. It just makes the gap between teams that have that depth and teams that don't more visible, and faster to widen.
The companies that will stand out aren't the ones using AI everywhere. They're the ones that built enough fluency to know when to use it and when not to.