How to Use Apple’s Foundation Models in a Web App with a macOS Companion

                     How to Use Apple's Foundation Models in a Web App with a macOS Companion
               



Tópico:
                     How to Use Apple's Foundation Models in a Web App with a macOS Companion
               
Categoria: Tutoriais | FreeCodeCamp Premium
Idioma Principal: Português (Conteúdo de Tecnologia)

Conteúdo do Tutorial / Guia Passo a Passo:
-------------------------------------------------------------------------
Not every AI feature needs a cloud model, with its per-token bills, network round-trips, and private data leaving your machine. If you're on a modern Mac, a capable language model is already on your disk.

Foundation Models is Apple's Swift framework for working with large language models. It's the on-device model behind Apple Intelligence, Apple's Private Cloud Compute, or another provider's server model.

This tutorial targets the on-device model: you send it a prompt and it runs entirely on the Mac's own hardware locally, free-per-call, and offline-friendly.

Paired with Apple Vision for reading images on device, that's enough to build real AI features like summaries, classification, and structured extraction without the data ever leaving your machine.

Table Of Contents

• What You Will Build

• Prerequisites

• Why a macOS Companion App?

• Foundation Models Can't Read Images Directly

• Project Structure

• Build the React App

• Check Companion Health

• Convert the Image to Base64

• Analyze Immediately After Upload

• Send the Image to the Companion

• Render the JSON Output

• Build the macOS Companion App

• Check Foundation Models Availability

• Extract Text with Apple Vision

• Ask Foundation Models to Explain the Vision Output

• Return JSON to the Browser

• Run the App

• Conclusion

• Resources

What You Will Build

You'll build Vision Bridge, a web app that sends an image to a local macOS companion. The companion reads the image with Apple Vision, reasons about it with Foundation Models, and returns structured JSON to the browser: private, on-device AI behind a plain web interface.

You can find the complete source code in this GitHub repository: github.com/03balogun/vision-bridge.

The goal isn't to build a giant product but rather to understand the architecture behind how this works.

Vision Bridge has two parts:

• A React app with a split-screen interface.

• A macOS companion app that exposes a local API.

The React app has:

• An image upload area

• An image preview

• Automatic analysis after upload

• A JSON output viewer

• A companion health status indicator

The macOS companion app has:

•

Código Selecionar Expandir

GET /v1/health
•

Código Selecionar Expandir

POST /v1/analyze-image
• Apple Vision OCR

• Foundation Models availability checks

• Foundation Models reasoning over Vision output

The final response looks like this:

Código Selecionar Expandir

{
"support": {
"visionAvailable": true,
"foundationModelAvailable": true,
"foundationModelStatus": "available"
},
"image": {
"filename": "screenshot.png",
"contentType": "image/png",
"byteCount": 1048576,
"width": 1440,
"height": 900
},
"vision": {
"detectedText": [
{
"text": "Build failed",
"confidence": 0.96,
"boundingBox": {
"x": 0.12,
"y": 0.31,
"width": 0.45,
"height": 0.08
}
}
]
},
"model": {
"summary": "The image appears to show a software build failure.",
"description": "A developer tool window is showing an error state with diagnostic text.",
"suggestedTags": ["screenshot", "developer-tool", "error"],
"possibleUses": [
"Generate alt text",
"Summarize screenshots",
"Extract document data"
]
}
}

Prerequisites

To follow along, you need:

• macOS 26 or newer

• Xcode with the macOS 26 SDK

• Node.js 20 or newer

• Basic React knowledge

• Basic Swift knowledge

• A Mac that supports Apple Intelligence

Foundation Models availability depends on the Mac, the OS version, and Apple Intelligence settings. The companion

... [O tutorial continua no link abaixo] ...