AI on my own machine
Some of the AI in my tools runs on the same Mac as the tools.
I use a local model for small decisions where language matters: choosing a diagnostic branch, ranking relevant context, or classifying an item. The configured setup described in my tooling uses Qwen3-4B through MLX on Apple Silicon. A local service makes the model available to the command-line tools.
The point is to make a bounded judgment close to the program that needs it. Ordinary code still parses the response, checks the option, and controls what happens next.
- CLIA question, evidence and allowed options
- Local serviceMLX inference on the Mac’s loopback interface
- ResponseAn allowed label or an explicit unresolved result
- CallerValidation, permissions and any authorized action
What does “local” establish?
These examples explain the setup’s contract. They are not a live check of my machine.
The model answers on my Mac
The inference service listens on loopback, which makes it reachable from that machine rather than from the network. The launch configuration sets offline mode for the model libraries. Model files have to be downloaded and installed separately.
That describes the inference path. A CLI that collects data from another service can still make network requests for that collection. Local inference does not make an entire workflow offline.
The model cannot answer in time
The caller has a bounded timeout and must preserve an unresolved result or use its documented safe fallback. Generic decisions are local-only, so an unavailable local model does not silently send that decision to a hosted provider.
I keep prompts small and avoid parallel large requests. A canceled client can leave work queued on the server, so adding more requests is not a reliable way to get an answer sooner.
The model returns an allowed but wrong answer
Running locally changes where inference happens. It does not make the judgment correct. I still need labeled examples, cases with ambiguous evidence, and checks for how the program handles a bad choice.
The decision tooling’s notes still record a temporal-relevance mistake in a small evaluation set. I use that as a reason to keep testing, rather than treating local execution as a quality guarantee.
A service is listening, but is it usable?
An open port is too little evidence. My verification workflow checks the service configuration and expects a real completion. Its stronger check restarts the managed service and verifies another request afterward.
This story describes those checks; it does not claim they were run on the machine during publication.
What I keep small
A short decision with a few options fits this setup better than an oversized session transcript. The caller validates the response, keeps permissions in code, and has a useful result when the model abstains. Those constraints matter more than making every tool use AI.