Real-time Cross Platform Collaboration

🔒 Protected Post

Please enter the password to view this content. (On Request: email Charles!)

Real-time Cross Platform Collaboration Between AVP and iPad

“With an example of the sketching interface”

Charles Yushi Cai
January 2026

A blog series dedicated to Apple Vision Pro and Apple ecosystem development. Tailored for beginners and designed to foster community growth (as well as myself!).

The Goal

We want to create a shared canvas where an artist can draw on an iPad using Apple Pencil, and a user on Apple Vision Pro those strokes appear instantly in mid-air. In this way we leverage the unique capabilities of both interfaces. (iPad is more sketch-friendly, while Vision Pro offers immersive 3D space in the later preview step.)

To achieve this without a hefty backend, we will use Apple’s Network framework (Bonjour) for local peer-to-peer discovery. Note that in this version, we are not required to get a device ID as part of the handshake. This smoothens the connection process.

Part 1: The Handshake

Before writing any Swift code, we need to give our apps permission to talk. This is the most common point of failure for beginners.

Both the iPad (Host) and Vision Pro (Client) need to agree on a specific Service Name.

Steps

Open Info.plist in both projects.
Add the following keys.

The string _my-vision-app._tcp acts as the secret handshake. If they don’t match exactly, the devices will remain strangers.

<key>NSBonjourServices</key>
<array>
    <string>_my-vision-app._tcp</string>
</array>
<key>NSLocalNetworkUsageDescription</key>
<string>We need local network access to sync drawings in real-time.</string>

Tip:
If your app crashes immediately upon launch or the devices never connect, double-check that you haven’t misspelled the service string in one of the files. It must be identical.

Part 2: The Shared NetworkManager

While not necessarily need to host in one project, we do need a unified logic controller to handle sending and receiving data. Instead of writing separate code for each device, we create a single NetworkManager.swift file that we drop into both projects.

This manager handles two roles:

The Listener (iPad): Advertises the service and waits for connections
The Browser (Vision Pro): Scans the local network for the service

We also need a common language. Swift’s Color isn’t network-ready (it’s not Codable), so we create a SharedStroke struct using simple indices or RGB values.

struct SharedStroke: Codable, Identifiable {
    var id: UUID
    var points: [CGPoint]
    var colorIndex: Int
    var width: CGFloat
    var isFinished: Bool
}

struct GamePacket: Codable {
    var deviceId: Int
    var stroke: SharedStroke
}

Kindly attach the code NetworkManager.swift for Create3D here: (note that not all the code is necessary, but just for reference in your case)

import Foundation
import Network
import SwiftUI

struct NetworkColor: Codable {
    var r: CGFloat
    var g: CGFloat
    var b: CGFloat
    var a: CGFloat
    
    static func from(_ color: Color) -> NetworkColor {
        return NetworkColor(r: 0, g: 0, b: 0, a: 1)
    }
}

struct SharedStroke: Codable, Identifiable, Equatable {
    var id: UUID
    var points: [CGPoint]
    var colorIndex: Int
    var width: CGFloat
    var isFinished: Bool
}

struct SharedTransform: Codable, Equatable {
    var px: Float; var py: Float; var pz: Float
    var rx: Float; var ry: Float; var rz: Float; var rw: Float
    var scale: Float
}


struct GamePacket: Codable {
    var deviceId: Int
    
    var stroke: SharedStroke?
    var modelTransform: SharedTransform?
    
    var command: String?
    var payload: String?
}


class NetworkManager: ObservableObject {
    // This string must match the "Bonjour services" in your Info.plist exactly.
    private let serviceType = "_create3D-control._tcp"
    
    @Published var isConnected: Bool = false
    @Published var connectionStatus: String = "Disconnected"
    @Published var lastReceivedPacket: GamePacket?
    
    private var listener: NWListener?
    private var browser: NWBrowser?
    private var connection: NWConnection?
    
    func startHosting() {
        setupListener()
    }
    
    func startSearching(deviceId: Int) {
        setupBrowser(myDeviceId: deviceId)
    }
    
    func send(packet: GamePacket) {
        guard let connection = connection else { return }
        
        do {
            let data = try JSONEncoder().encode(packet)
            connection.send(content: data, completion: .contentProcessed({ error in
                if let error = error {
                    print("Send error: \(error)")
                }
            }))
        } catch {
            print("Encoding error: \(error)")
        }
    }
    
    private func setupListener() {
        do {
            let listener = try NWListener(using: .tcp)
            self.listener = listener
            
            listener.service = NWListener.Service(type: serviceType)
            
            listener.newConnectionHandler = { [weak self] newConnection in
                print("New connection received!")
                self?.startConnection(newConnection)
                // If only want one device, stop listening here:
                // self?.listener?.cancel()
            }
            
            listener.stateUpdateHandler = { newState in
                print("Listener state: \(newState)")
            }
            
            listener.start(queue: .main)
            self.connectionStatus = "Listening..."
            
        } catch {
            print("Failed to create listener: \(error)")
        }
    }

    
    private func setupBrowser(myDeviceId: Int) {
        let browser = NWBrowser(for: .bonjour(type: serviceType, domain: nil), using: .tcp)
        self.browser = browser
        
        browser.browseResultsChangedHandler = { [weak self] results, changes in
            guard let self = self else { return }
            
            if let result = results.first {
                print("Found service: \(result.endpoint)")
                self.browser?.cancel() // Stop searching once found
                
                let newConnection = NWConnection(to: result.endpoint, using: .tcp)
                self.startConnection(newConnection)
            }
        }
        
        browser.start(queue: .main)
        self.connectionStatus = "Searching..."
    }
    
    private func startConnection(_ newConnection: NWConnection) {
        self.connection = newConnection
        
        newConnection.stateUpdateHandler = { [weak self] state in
            DispatchQueue.main.async {
                switch state {
                case .ready:
                    self?.isConnected = true
                    self?.connectionStatus = "Connected"
                    self?.receiveNextMessage()
                case .failed(let error):
                    print("Connection failed: \(error)")
                    self?.isConnected = false
                    self?.connectionStatus = "Failed"
                case .cancelled:
                    self?.isConnected = false
                    self?.connectionStatus = "Disconnected"
                default:
                    break
                }
            }
        }
        
        newConnection.start(queue: .main)
    }
    
    private func receiveNextMessage() {
        connection?.receive(minimumIncompleteLength: 1, maximumLength: 65536) { [weak self] (data, context, isComplete, error) in

            if let data = data, !data.isEmpty {
                do {
                    let packet = try JSONDecoder().decode(GamePacket.self, from: data)
                    DispatchQueue.main.async {
                        self?.lastReceivedPacket = packet
                        
                        if let stroke = packet.stroke {
                             print("Received Stroke ID: \(stroke.id)")
                        }
                        if packet.modelTransform != nil {
                             print("Received 3D Model Update")
                        }
                    }
                } catch {
                    print("Error decoding packet: \(error)")
                }
            }

            
            if let error = error {
                print("Receive error: \(error)")
                return
            }
            
            self?.receiveNextMessage()
        }
    }
}

Part 3: The iPad Host (Sender)

The iPad acts as the drawing tablet. We hook into SwiftUI’s DragGesture to capture pencil movement.

DragGesture(minimumDistance: 0, coordinateSpace: .local)
    .onChanged { value in
        let newPoint = value.location
        if networkManager.isConnected {
            let stroke = SharedStroke(...)
            networkManager.send(packet: GamePacket(stroke: stroke))
        }
    }

Part 4: The Vision Pro Client (Receiver)

.onAppear {
    networkManager.startSearching(deviceId: Int.random(in: 1...9000))
}
.onChange(of: networkManager.lastReceivedPacket) { packet in
    if let stroke = packet.stroke {
        self.strokes.append(stroke)
    }
}

Part 5: The Environment Pitfall

@main
struct YourApp: App {
    @StateObject var networkManager = NetworkManager()

    var body: some Scene {
        WindowGroup {
            ContentView()
                .environmentObject(networkManager)
        }
    }
}

Final Result

Reference

[1] General: https://developer.apple.com/documentation/visionos/connecting-ipados-and-visionos-apps-over-the-local-network provides an overview of these techniques.

Share this article:

Real-time Cross Platform Collaboration

🔒 Protected Post