🔒 Protected Post
Please enter the password to view this content. (On Request: email Charles!)
Incorrect password.
Real-time Cross Platform Collaboration Between AVP and iPad
“With an example of the sketching interface”
- Charles Yushi Cai
- January 2026
A blog series dedicated to Apple Vision Pro and Apple ecosystem development. Tailored for beginners and designed to foster community growth (as well as myself!).
The Goal
We want to create a shared canvas where an artist can draw on an iPad using Apple Pencil, and a user on Apple Vision Pro those strokes appear instantly in mid-air. In this way we leverage the unique capabilities of both interfaces. (iPad is more sketch-friendly, while Vision Pro offers immersive 3D space in the later preview step.)
To achieve this without a hefty backend, we will use Apple’s Network framework (Bonjour) for local peer-to-peer discovery. Note that in this version, we are not required to get a device ID as part of the handshake. This smoothens the connection process.
Part 1: The Handshake
Before writing any Swift code, we need to give our apps permission to talk. This is the most common point of failure for beginners.
Both the iPad (Host) and Vision Pro (Client) need to agree on a specific Service Name.
Steps
- Open
Info.plistin both projects. - Add the following keys.
The string _my-vision-app._tcp acts as the secret handshake. If they don’t match exactly, the devices will remain strangers.
<key>NSBonjourServices</key>
<array>
<string>_my-vision-app._tcp</string>
</array>
<key>NSLocalNetworkUsageDescription</key>
<string>We need local network access to sync drawings in real-time.</string>
Tip:
If your app crashes immediately upon launch or the devices never connect, double-check that you haven’t misspelled the service string in one of the files. It must be identical.
Part 2: The Shared NetworkManager
While not necessarily need to host in one project, we do need a unified logic controller to handle sending and receiving data. Instead of writing separate code for each device, we create a single NetworkManager.swift file that we drop into both projects.
This manager handles two roles:
- The Listener (iPad): Advertises the service and waits for connections
- The Browser (Vision Pro): Scans the local network for the service
We also need a common language. Swift’s Color isn’t network-ready (it’s not Codable), so we create a SharedStroke struct using simple indices or RGB values.
struct SharedStroke: Codable, Identifiable {
var id: UUID
var points: [CGPoint]
var colorIndex: Int
var width: CGFloat
var isFinished: Bool
}
struct GamePacket: Codable {
var deviceId: Int
var stroke: SharedStroke
}
Kindly attach the code NetworkManager.swift for Create3D here: (note that not all the code is necessary, but just for reference in your case)
import Foundation
import Network
import SwiftUI
struct NetworkColor: Codable {
var r: CGFloat
var g: CGFloat
var b: CGFloat
var a: CGFloat
static func from(_ color: Color) -> NetworkColor {
return NetworkColor(r: 0, g: 0, b: 0, a: 1)
}
}
struct SharedStroke: Codable, Identifiable, Equatable {
var id: UUID
var points: [CGPoint]
var colorIndex: Int
var width: CGFloat
var isFinished: Bool
}
struct SharedTransform: Codable, Equatable {
var px: Float; var py: Float; var pz: Float
var rx: Float; var ry: Float; var rz: Float; var rw: Float
var scale: Float
}
struct GamePacket: Codable {
var deviceId: Int
var stroke: SharedStroke?
var modelTransform: SharedTransform?
var command: String?
var payload: String?
}
class NetworkManager: ObservableObject {
// This string must match the "Bonjour services" in your Info.plist exactly.
private let serviceType = "_create3D-control._tcp"
@Published var isConnected: Bool = false
@Published var connectionStatus: String = "Disconnected"
@Published var lastReceivedPacket: GamePacket?
private var listener: NWListener?
private var browser: NWBrowser?
private var connection: NWConnection?
func startHosting() {
setupListener()
}
func startSearching(deviceId: Int) {
setupBrowser(myDeviceId: deviceId)
}
func send(packet: GamePacket) {
guard let connection = connection else { return }
do {
let data = try JSONEncoder().encode(packet)
connection.send(content: data, completion: .contentProcessed({ error in
if let error = error {
print("Send error: \(error)")
}
}))
} catch {
print("Encoding error: \(error)")
}
}
private func setupListener() {
do {
let listener = try NWListener(using: .tcp)
self.listener = listener
listener.service = NWListener.Service(type: serviceType)
listener.newConnectionHandler = { [weak self] newConnection in
print("New connection received!")
self?.startConnection(newConnection)
// If only want one device, stop listening here:
// self?.listener?.cancel()
}
listener.stateUpdateHandler = { newState in
print("Listener state: \(newState)")
}
listener.start(queue: .main)
self.connectionStatus = "Listening..."
} catch {
print("Failed to create listener: \(error)")
}
}
private func setupBrowser(myDeviceId: Int) {
let browser = NWBrowser(for: .bonjour(type: serviceType, domain: nil), using: .tcp)
self.browser = browser
browser.browseResultsChangedHandler = { [weak self] results, changes in
guard let self = self else { return }
if let result = results.first {
print("Found service: \(result.endpoint)")
self.browser?.cancel() // Stop searching once found
let newConnection = NWConnection(to: result.endpoint, using: .tcp)
self.startConnection(newConnection)
}
}
browser.start(queue: .main)
self.connectionStatus = "Searching..."
}
private func startConnection(_ newConnection: NWConnection) {
self.connection = newConnection
newConnection.stateUpdateHandler = { [weak self] state in
DispatchQueue.main.async {
switch state {
case .ready:
self?.isConnected = true
self?.connectionStatus = "Connected"
self?.receiveNextMessage()
case .failed(let error):
print("Connection failed: \(error)")
self?.isConnected = false
self?.connectionStatus = "Failed"
case .cancelled:
self?.isConnected = false
self?.connectionStatus = "Disconnected"
default:
break
}
}
}
newConnection.start(queue: .main)
}
private func receiveNextMessage() {
connection?.receive(minimumIncompleteLength: 1, maximumLength: 65536) { [weak self] (data, context, isComplete, error) in
if let data = data, !data.isEmpty {
do {
let packet = try JSONDecoder().decode(GamePacket.self, from: data)
DispatchQueue.main.async {
self?.lastReceivedPacket = packet
if let stroke = packet.stroke {
print("Received Stroke ID: \(stroke.id)")
}
if packet.modelTransform != nil {
print("Received 3D Model Update")
}
}
} catch {
print("Error decoding packet: \(error)")
}
}
if let error = error {
print("Receive error: \(error)")
return
}
self?.receiveNextMessage()
}
}
}
Part 3: The iPad Host (Sender)
The iPad acts as the drawing tablet. We hook into SwiftUI’s DragGesture to capture pencil movement.
DragGesture(minimumDistance: 0, coordinateSpace: .local)
.onChanged { value in
let newPoint = value.location
if networkManager.isConnected {
let stroke = SharedStroke(...)
networkManager.send(packet: GamePacket(stroke: stroke))
}
}
Part 4: The Vision Pro Client (Receiver)
.onAppear {
networkManager.startSearching(deviceId: Int.random(in: 1...9000))
}
.onChange(of: networkManager.lastReceivedPacket) { packet in
if let stroke = packet.stroke {
self.strokes.append(stroke)
}
}
Part 5: The Environment Pitfall
@main
struct YourApp: App {
@StateObject var networkManager = NetworkManager()
var body: some Scene {
WindowGroup {
ContentView()
.environmentObject(networkManager)
}
}
}
Final Result
Reference
- [1] General: https://developer.apple.com/documentation/visionos/connecting-ipados-and-visionos-apps-over-the-local-network provides an overview of these techniques.