Watch the video, transcript and sources
https://www.youtube.com/watch?v=HkZzHWiqzZA
+ Apple: four Macs, local inference
- Mercor: benchmark coverage gaps
- Oct 18 update: OpenCode Console breachTwo Macs can give you enough memory for a larger model while leaving each response about as fast as before. Apple's October 15 research explains the distinction, and it's worth understanding before the shopping cart gets ambitious.
MLX is Apple's open-source array framework for machine learning on Apple silicon. Apple reports Muse Spark Open generating over 36 tokens per second across four M5 Ultra Mac Studios with an 8,192-token context. That is a vendor measurement of a specific model, precision and configuration.
The model has to fit first
Apple puts the four-bit model footprint at about 493 GB. A Mac Studio configuration can have up to 512 GB, and inference also needs memory for intermediate computations. The weights nearly fill the suitcase before you've packed a sock.
Pipeline parallelism places earlier layers on one Mac and later layers on another. A generated token still follows those stages in sequence. Apple's smaller-model comparisons show decode speed very similar to a single machine. Large-context prompt processing can benefit from overlapping chunks; that is a different part of the workload.
Tensor parallelism divides work inside each layer. Multiple machines can compute together, but they must exchange inputs and combine outputs repeatedly. Fast communication matters. Apple uses Thunderbolt RDMA and two cables per peer link in its setup.
In a separate 65,536-token-context experiment, the extra cable changed the first-token wait from about 50.5 seconds to 40.6 seconds. That measures prompt-processing delay. The token-generation rate describes another phase. Keep those labels attached when comparing charts.
A local server can create an inference boundary you administer. The surrounding tools and services need their own review, and the hardware's economics depend on utilization and the model your task needs. Apple supplies no universal break-even price against cloud APIs here.
The exam paper has blank spaces
Mercor mapped 29 selected benchmarks onto the US jobs taxonomy. Its map reports 924 covered tasks out of 18,797, and estimates that 90% of working hours fall on tasks with little or no coverage in that index.
Missing coverage leaves capability unknown. It doesn't establish that models fail on 90% of work, or that those jobs are safe from automation. Mercor used AI-assisted tagging and expert review of a subset of its APEX prompts; experts accepted 76.7% of first-choice mappings across the reported judgments. The methodology has its own uncertainty.
For a buying decision, replay work you actually need with acceptance checks you can inspect. A leaderboard rarely includes the awkward bits after your demo.
An overnight Console update
This is the October 17 edition, with an explicitly labelled October 18 GMT+7 update. OpenCode's edited public notice says a security breach affected 1.4% of Console users, a subset with external providers linked, and that affected users were emailed. It calls the incident a large-scale distillation attack and says remediation is ongoing. A public root cause was unavailable in the notice checked for this episode.
Replies include people reporting vanished provider credit. Those remain user accounts of events; the notice does not establish each person's loss or compensation. Console users should read the official notice, review provider usage and follow remediation instructions if affected. The notice's specific hosted Console scope provides no evidence here of a compromise in every local installation.
Apple's article actually names OpenCode as a client for a local MLX server. That is the useful connection: one coding interface can sit in front of different infrastructure and credential boundaries.
Verdict: NEEDS REVIEW
I'd measure my real workload before buying the cluster. Check that the model fits, measure the complete task, and review every service holding a credential. An acceptance test belongs beside the purchase order.
What would you measure first before buying a local AI machine?
Sources
HN reader's precision-format objection — reader opinion; formats checked against Apple's table
Amruth Pillai's reported provider-credit loss — unverified user report
And that's the diff for today. I'm Niko from Axrisi. Merge responsibly.