Back in 2021, Ford’s ML teams had a problem that’s a little embarrassing to write down. Engineers spent more time hunting for data and rebuilding it than actually training models. A development cycle took six months. People kept re-creating the same features, not because they were lazy but because there was no way to know someone three teams over had already built the exact thing. Training data got computed one way, serving data another, and models that looked great in a notebook quietly fell apart in production.
We decided to build a feature store. I went in thinking I understood the problem. I did not. I learned more from this project than almost anything else I’ve worked on.
The decision nobody warned us about
Before writing any code we looked at what already existed: Feast, Tecton, Hopsworks. All real, all solving a real version of our problem.
We nearly went with Hopsworks. Technically it was the closest fit. Then we looked at the authorization model and hit a wall. Access control was at the store level, not the feature level. At Ford that’s a non-starter. You can’t hand a marketing team access to vehicle diagnostic data just because they want one feature sitting next to it. Per-feature access control wasn’t a nice-to-have. It was a compliance requirement.
Feast and Tecton had their own gaps, and all three shared one: we couldn’t push Ford’s data to someone else’s cloud. It had to live on-prem.
So we built it ourselves.
That added months and a lot of complexity. I still think it was the right call. But I want to be honest about the cost, because “build vs. buy” makes it sound like a clean decision you make once. The real work is everything that comes after, when you have to live with it.
The authorization problem we underestimated
The hardest thing we hit wasn’t storage or query performance. It was auth.
Ford runs Apache Ranger for access control, which is a sane enterprise choice. The catch: the Ranger API had rate limits that made real-time permission checks impractical at the volume features get queried. We couldn’t call Ranger on every access without the whole thing grinding to a halt.
What we landed on was ugly but it worked. Nightly exports of Ranger policies as JSON into HDFS, event notifications to catch policy changes, and an authorization service we wrote that mapped all of it onto our feature metadata. We wired it into Ford’s internal access-request system so people could ask for feature access without leaving the platform.
The cost: we were always running on a slightly stale view of permissions, refreshed nightly instead of live. For our use case that was fine, but it took weeks of arguing to get comfortable with it, and it added operational surface area we hadn’t planned for.
The lesson stuck with me. Authorization in enterprise ML is mostly a compliance and political problem with a technical piece bolted on. Treat it that way on day one.
The discovery problem we definitely didn’t anticipate
This one still stings.
When we designed the feature catalog, the searchable index of everything in the store, we assumed the number of features would stay manageable. So we built basic search. Keyword matching, a few filters. Fine for a few hundred features.
We were wrong about scale. As more teams onboarded, the catalog got genuinely hard to navigate. Engineers couldn’t find features that already existed, so they built duplicates. The exact problem we’d built the whole system to kill was sneaking back in through a search box that couldn’t keep up.
We should have gone with Elasticsearch from the start: semantic search, faceted filtering, descriptions that actually showed up when you needed them. We knew it, too. We’d talked about it early and pushed it down the list. That decision got more expensive every month the catalog grew.
If I built it again: feature discovery is a product problem, not an infrastructure afterthought. The best feature store on earth is useless if nobody can find what’s in it.
What we actually shipped
After all of it, the build call, the auth workaround, the discovery gap, the thing worked. Development cycles went from six months to one week. Teams stopped rebuilding the same features in parallel. Training-serving skew stopped being a recurring source of production surprises.
The performance targets held: under 500ms retrieval for exploration, under 50ms for real-time inference, 99.99% uptime on serving. The hybrid approach, computing cold features on demand and pre-materializing the hot ones, turned out to be right for both cost and latency.
700,000+ engineering hours saved a year across Ford’s ML teams. That number still feels made up when I say it out loud.
What I’d tell someone starting this today
Don’t buy a feature store until you’ve checked feature-level authorization against your real compliance model. Most off-the-shelf options treat access control as an afterthought. Build, but scope it hard and add complexity only once the pain of not having it is undeniable.
Invest in discovery earlier than feels reasonable. For most users the catalog is the feature store, the part they actually touch. The infrastructure only matters when it keeps those entries accurate and findable.
And training-serving skew is easy to miss and can ruin otherwise strong models if you don’t treat consistency as a first-class requirement from the start. We did. It paid off.