Repository navigation
Expand file tree
/
Copy pathagent-api.html
More file actions
897 lines (863 loc) · 64.2 KB
/
Copy pathagent-api.html
File metadata and controls
897 lines (863 loc) · 64.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>iphone-use — Agent API</title>
<style>
:root {
--fg: #1a1a1a; --muted: #6b6b6b; --bg: #fbfbfa; --card: #ffffff;
--border: #e6e4df; --accent: #c0392b; --code-bg: #f4f2ee; --ok: #2e7d32;
}
* { box-sizing: border-box; }
body {
margin: 0; padding: 0 1.25rem 5rem; background: var(--bg); color: var(--fg);
font-family: -apple-system, BlinkMacSystemFont, "PingFang SC", "Microsoft YaHei", sans-serif;
line-height: 1.6; -webkit-font-smoothing: antialiased;
}
.wrap { max-width: 880px; margin: 0 auto; }
header.page { padding: 3rem 0 1.5rem; border-bottom: 2px solid var(--fg); margin-bottom: 2rem; }
header.page h1 { margin: 0 0 .3rem; font-size: 2rem; letter-spacing: -.02em; }
header.page p { margin: 0; color: var(--muted); }
h2 { margin: 2.6rem 0 .8rem; font-size: 1.3rem; letter-spacing: -.01em;
padding-bottom: .3rem; border-bottom: 1px solid var(--border); }
h3 { margin: 1.8rem 0 .5rem; font-size: 1.02rem; }
code { font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, monospace;
background: var(--code-bg); padding: .12em .4em; border-radius: 4px; font-size: .9em; }
pre { background: var(--code-bg); border: 1px solid var(--border); border-radius: 8px;
padding: 1rem 1.1rem; overflow-x: auto; font-size: .86rem; line-height: 1.5; }
pre code { background: none; padding: 0; }
table { width: 100%; border-collapse: collapse; margin: 1rem 0; font-size: .92rem; }
th, td { text-align: left; padding: .55rem .7rem; border-bottom: 1px solid var(--border); vertical-align: top; }
th { font-weight: 600; color: var(--muted); font-size: .82rem; text-transform: uppercase; letter-spacing: .04em; }
.method { font-family: ui-monospace, monospace; font-weight: 700; font-size: .82rem;
padding: .12em .5em; border-radius: 4px; color: #fff; }
.get { background: #2e7d32; } .post { background: #1565c0; } .put { background: #ef6c00; }
.ep { font-family: ui-monospace, monospace; font-weight: 600; }
aside { background: var(--card); border: 1px solid var(--border); border-left: 3px solid var(--accent);
border-radius: 6px; padding: .8rem 1rem; margin: 1.2rem 0; font-size: .92rem; }
aside.note { border-left-color: var(--accent); }
.muted { color: var(--muted); }
.pill { display: inline-block; background: var(--code-bg); border: 1px solid var(--border);
border-radius: 999px; padding: .05em .6em; font-size: .8rem; font-family: ui-monospace, monospace; }
</style>
</head>
<body>
<div class="wrap">
<header class="page">
<h1>iphone-use — Agent API</h1>
<p>Direct-to-device HTTP control for agents, MCP clients, scripts, and the browser companion.</p>
</header>
<p>
An agent drives the iPhone by talking to the <strong>already-running daemon</strong> over
HTTP. The daemon serves the phone only through its device runner: input is performed by
on-device runner actions and video is runner's MJPEG stream. It never touches the Mac's screen,
cursor, or focus, and needs no macOS permission grants.
</p>
<aside class="note">
<strong>Failures are closed, not substituted.</strong> A request that runner cannot
perform returns an explicit error; there is no other input or capture path to fall
back to.
</aside>
<aside class="note">
<strong>Control requires completed runner setup.</strong> Installing or starting the daemon
does not make the phone ready. Run <code>~/.iphone-use/setup-runner.sh doctor</code>,
<code>~/.iphone-use/setup-runner.sh</code>, and
<code>~/.iphone-use/setup-runner.sh status</code> with the pinned iPhone connected,
trusted, unlocked, and awake. Human operators can open <code>/setup</code> for the
auth-gated live connection guide; it reads status only and enables <code>/phone</code>
when <code>backend:"direct"</code> and <code>drivable:true</code>.
</aside>
<aside class="note">
<strong>Phase 1 network boundary:</strong> daemon auth protects port 44321, not runner's
unauthenticated device-side 8100/9100 services. USB <code>iproxy</code> pins the Mac
relay to a device but does not block direct LAN access to the phone. Use a trusted,
isolated network until a Phase 2 companion or controlled tunnel supplies authenticated
device transport.
</aside>
<h2>Authentication</h2>
<p>
Machine callers authenticate with a bearer token. Browser-facing reads
(<code>/agent/status</code>, <code>/agent/screenshot</code>,
<code>/agent/mjpeg</code>) and <code>/agent/mode</code> also accept a valid browser
session cookie. <code>/agent/input</code> and <code>/agent/elements</code> require the
bearer path. Bearer selection has two modes:
</p>
<table>
<tr><th>Mode</th><th>When</th><th>Token to use</th></tr>
<tr>
<td><strong>Dedicated token</strong></td>
<td><code>IPHONE_USE_AGENT_TOKEN</code> is set on the daemon</td>
<td>That token — the password is no longer accepted as a bearer</td>
</tr>
<tr>
<td><strong>Password fallback</strong> (legacy)</td>
<td><code>IPHONE_USE_AGENT_TOKEN</code> is unset</td>
<td><code>IPHONE_USE_PASSWORD</code></td>
</tr>
</table>
<pre><code>Authorization: Bearer <IPHONE_USE_AGENT_TOKEN or IPHONE_USE_PASSWORD></code></pre>
<p class="muted">
When the daemon runs with no password and no agent token (open LAN-dev mode) the agent
API is open too — the same policy as the browser cookie gate. The bearer is compared in
length-checked constant time.
</p>
<aside class="note">
<strong>State changes have a second, explicit intent gate.</strong> Every
state-changing POST must include the exact header
<code>X-Phone-Control: 1</code>, including <code>/control</code>,
<code>/agent/input</code>, <code>/agent/actions</code>,
<code>/agent/mode</code>, and POST
<code>/agent/inbox</code> and <code>/agent/inbox/drain</code>. This is a
CSRF/intent guard in addition to bearer or session authentication; it does not replace
authentication. GET requests do not need the header. The bundled browser and MCP
clients add it automatically. A missing or different value returns a self-describing
403 response with <code>required_header</code> and a retry hint:
<code>{"ok":false,"error":"missing_control_header","required_header":"X-Phone-Control: 1",…}</code>.
</aside>
<h2>Endpoints</h2>
<table>
<tr><th>Method</th><th>Path</th><th>Purpose</th></tr>
<tr><td><span class="method get">GET</span></td><td class="ep">/agent/status</td><td>runner reachability/actionability, device lifecycle, and recovery hints.</td></tr>
<tr><td><span class="method post">POST</span></td><td class="ep">/agent/input</td><td>Bearer-authenticated control with the mutation header. Routes to runner and fails closed.</td></tr>
<tr><td><span class="method post">POST</span></td><td class="ep">/agent/actions</td><td>Bounded multi-step runner execution with full-batch validation, semantic waits, one control lock, and first-failure stop.</td></tr>
<tr><td><span class="method get">GET</span></td><td class="ep">/agent/screenshot</td><td>Current target screen as PNG, read from runner. <code>?overlay=elements</code> numbers each interactive element on it.</td></tr>
<tr><td><span class="method get">GET</span></td><td class="ep">/agent/mjpeg</td><td>Live runner MJPEG stream; accepts bearer auth or a logged-in browser cookie.</td></tr>
<tr><td><span class="method get">GET</span></td><td class="ep">/agent/h264</td><td>The same live screen as H.264 (~10× less bandwidth); same auth and viewer accounting as <code>/agent/mjpeg</code>.</td></tr>
<tr><td><span class="method get">GET</span></td><td class="ep">/agent/elements</td><td>Snapshot-bound, flattened runner element list with sparse locator/state fields and screen dimensions.</td></tr>
<tr><td><span class="method get">GET</span></td><td class="ep">/agent/activity</td><td>What agents did, in plain words (<code>Tapped “Wi-Fi”</code>), newest last; <code>?since=<next></code> for newer lines. Typed text is never kept. Bearer or a logged-in browser cookie.</td></tr>
<tr><td><span class="method get">GET</span> / <span class="method post">POST</span></td><td class="ep">/agent/logs</td><td>Capture the phone's system log (<code>{"action":"start"|"stop"|"clear"}</code>) and read it with filters.</td></tr>
<tr><td><span class="method get">GET</span></td><td class="ep">/agent/crashes[/:name]</td><td>The phone's crash reports, newest first; one report's text.</td></tr>
<tr><td><span class="method post">POST</span></td><td class="ep">/agent/location</td><td>Simulated GPS position (<code>{"lat","lon"}</code>) or <code>{"clear":true}</code>.</td></tr>
<tr><td><span class="method get">GET</span> / <span class="method post">POST</span></td><td class="ep">/agent/record</td><td>Record the live view to an MP4 (<code>{"action":"start"|"stop"}</code>); GET shows the running recording.</td></tr>
<tr><td><span class="method get">GET</span> / <span class="method put">PUT</span></td><td class="ep">/agent/app-notes</td><td>Per-app markdown notes (<code>?bundle=</code>), also shown in the <code>registry</code> block when an agent enters the app.</td></tr>
<tr><td><span class="method post">POST</span></td><td class="ep">/agent/mode</td><td><code>mode=agent</code> brings runner up; <code>mode=human</code> stops runner and hands the phone back to the person holding it. It never changes the persisted target.</td></tr>
<tr><td><span class="method get">GET</span> / <span class="method post">POST</span></td><td class="ep">/agent/inbox</td><td>Legacy Shortcuts-bridge result queue. GET is always a read-only peek; POST appends and requires the mutation header.</td></tr>
<tr><td><span class="method post">POST</span></td><td class="ep">/agent/inbox/drain</td><td>Atomically return and clear the inbox; requires bearer auth and the mutation header.</td></tr>
<tr><td><span class="method post">POST</span></td><td class="ep">/control</td><td>Browser-only direct control: session cookie + mutation header, with an explicit HTTP acknowledgement.</td></tr>
<tr><td><span class="method post">POST</span></td><td class="ep">/pair/new</td><td>Signed-in browser: one-time scan-to-connect code + QR (SVG).</td></tr>
<tr><td><span class="method post">POST</span></td><td class="ep">/pair</td><td>Trade a scanned code (or a paired device token) for a session; no password.</td></tr>
</table>
<h3><span class="method get">GET</span> <span class="ep">/agent/status</span></h3>
<p>Accepts the agent bearer or a logged-in browser cookie. A healthy response looks like:</p>
<pre><code>{
"ok": true,
"backend": "direct",
"target_configured": true,
"managed_runner": true,
"managed_runner_pending": false,
"recovery_owner": "daemon",
"runner": true,
"runner_actionable": true,
"runner_locked": false,
"drivable": true,
"mode": "agent",
"device_state": "ready",
"screen_state": "waiting",
"releasing": false,
"reconnecting": false,
"released": false,
"human_handoff": false,
"hint": "",
"setup_blocked_on": "",
"setup_phase": "ready",
"setup_message": "direct runner backend and launchd supervisor verified",
"runner_build": { "state": "ready", "phase": "ready", "since": 1765000000, "age_secs": 12, "log_tail": "" },
"runner_died_reason": "",
"runner_died_at": 0,
"viewer_count": 0,
"mjpeg_viewer_count": 0,
"version": "<running version>",
"latest": null,
"update_available": false
}</code></pre>
<table>
<tr><th>Field</th><th>Meaning</th></tr>
<tr><td><code>backend</code></td><td>Always <code>direct</code> (the phone is driven over runner). Kept for client compatibility.</td></tr>
<tr><td><code>target_configured</code></td><td>Whether a persistent <code>IPHONE_USE_UDID</code> is configured.</td></tr>
<tr><td><code>managed_runner</code></td><td><code>true</code> when this daemon owns the local loopback runner supervisor and relay lifecycle; remote endpoints are externally managed.</td></tr>
<tr><td><code>managed_runner_pending</code></td><td><code>true</code> during first-run local setup when managed Direct was requested but no canonical device target has been persisted yet. In this state the daemon will not stop or bootstrap runner.</td></tr>
<tr><td><code>recovery_owner</code></td><td><code>daemon</code> for managed Direct, <code>unconfigured</code> while first-run local management is waiting for a persistent target, and <code>external</code> for a remote or explicitly unmanaged runner endpoint.</td></tr>
<tr><td><code>runner</code></td><td>runner HTTP is reachable. This alone is not proof that it can act.</td></tr>
<tr><td><code>runner_actionable</code></td><td>A real device-side action probe succeeded.</td></tr>
<tr><td><code>runner_locked</code></td><td><code>true</code>, <code>false</code>, or <code>null</code> when lock state cannot be determined.</td></tr>
<tr><td><code>drivable</code></td><td>The control-readiness gate: actionable runner with no release or reconnect transition in progress.</td></tr>
<tr><td><code>device_state</code></td><td><code>ready</code>, <code>locked</code>, <code>blocked</code>, <code>offline</code>, <code>releasing</code>, <code>released</code>, or <code>reconnecting</code>.</td></tr>
<tr><td><code>screen_state</code></td><td>Coarse transport lifecycle (<code>waiting</code>/<code>offline</code>); it is not a frame-level MJPEG health check.</td></tr>
<tr><td><code>releasing</code></td><td>The managed local runner service is being stopped after idle timeout.</td></tr>
<tr><td><code>reconnecting</code></td><td>The daemon is bootstrapping its managed runner supervisor. If <code>setup_blocked_on</code> is non-empty, follow <code>hint</code> first; otherwise wait for this to clear. In every case, require <code>runner_actionable</code> and <code>drivable</code> before acting.</td></tr>
<tr><td><code>released</code></td><td>runner was intentionally stopped — after the idle period (<code>IPHONE_USE_IDLE_RELEASE_SECS</code>, default 600, <code>0</code> keeps it up) or by a human hand-off. The parked state survives daemon restarts and logins; the next agent request brings runner back up. A completed health probe that finds a replacement/self-healed runner clears this guard; clients do not need a redundant reconnect.</td></tr>
<tr><td><code>human_handoff</code></td><td><code>true</code> while the phone has been handed back to its holder with <code>{"mode":"human"}</code>. Agent input answers <code>409 phone_handed_to_human</code> until <code>{"mode":"agent"}</code> takes it back.</td></tr>
<tr><td><code>viewer_count</code></td><td>Open live <code>/agent/mjpeg</code> streams (same number as <code>mjpeg_viewer_count</code>; kept for older clients).</td></tr>
<tr><td><code>mjpeg_viewer_count</code></td><td>Open live <code>/agent/mjpeg</code> streams.</td></tr>
<tr><td><code>setup_blocked_on</code></td><td>Fresh daemon-owned managed-Direct setup blocker: <code>warp</code>, <code>proxy</code>, <code>usb</code>, <code>trust</code>, <code>ddi</code>, <code>runner</code>, or empty. KeepAlive preserves the last blocker while the next preflight pass checks it, then clears or replaces it only after evidence; it is deliberately empty for unconfigured and externally managed targets.</td></tr>
<tr><td><code>setup_phase</code></td><td>Fresh managed-Direct helper phase such as <code>prereq</code>, <code>ddi-wait</code>, <code>building</code>, <code>serving</code>, or <code>ready</code>. During a long first build, this distinguishes progress from a silent hang.</td></tr>
<tr><td><code>setup_message</code></td><td>Bounded, JSON-escaped detail for <code>setup_phase</code>, including periodically refreshed runner build elapsed time. It is progress context, not actionability proof.</td></tr>
<tr><td><code>runner_parked</code>, <code>runner_parked_reason</code></td><td>Persistent parking and its recorded evidence: <code>exited</code>, <code>passcode_canceled</code>, <code>session_invalidated</code>, <code>crashed</code> (iOS ended the runner's test process, e.g. a watchdog kill), or null for unknown contents. Invalidated/crashed/unknown sessions remain parked without claiming <code>ended_on_phone</code>; prewarming and automatic schedules do not resume them. An explicit request may reconnect subject to ownership, privacy and handoff gates.</td></tr>
<tr><td><code>runner_build</code></td><td>Bring-up progress as <code>{state, phase, since, age_secs, log_tail}</code>. <code>state</code> is <code>building</code> (working — keep waiting), <code>failed</code> (the helper gave up — read <code>log_tail</code>), <code>stalled</code> (no status write for over 5 minutes, so the helper died without reporting), <code>ready</code>, <code>parked</code> (no build is in progress; waiting for an explicit request; see <code>runner_parked_reason</code> for recorded evidence), <code>stopped</code> (the helper was stopped on purpose, e.g. <code>setup pause</code> — no build is in progress), or <code>unknown</code> (no managed setup has run). This is the distinction <code>setup_blocked_on</code> cannot make: a slow first build and a dead build both leave it empty. <code>since</code> is the helper's last status write in Unix seconds and <code>age_secs</code> how long the state has held. <code>log_tail</code> carries the last lines of <code>~/.iphone-use/runner.log</code>, and is populated only for <code>failed</code> and <code>stalled</code> so a routine poll stays small.</td></tr>
<tr><td><code>runner_died_reason</code></td><td>Why runner last stopped being drivable, recorded at the transition and reported only while it is still down: <code>idle_release</code> (the daemon stopped it on purpose — nothing is broken), <code>device_locked</code>, <code>session_severed</code> (the runner still answers <code>/status</code> but its test session was torn down — a WARP/VPN reconnect, Mac sleep, or a lock), <code>unreachable</code> (nothing answers: runner exited, relay died, or the phone's Wi-Fi address changed), or empty. Attribution is by observed signature, not by probing the VPN — a WARP reconnect and a Mac sleep are indistinguishable to the daemon, so <code>session_severed</code> names both. It exists so a mid-session <code>runner:true → false</code> is not mistaken for a human picking the phone up.</td></tr>
<tr><td><code>runner_died_at</code></td><td>Unix seconds of that transition; <code>0</code> when runner is healthy or has never gone down.</td></tr>
<tr><td><code>hint</code></td><td>Concrete recovery guidance for locked, severed, released, or unreachable runner. When a death was observed, this is the reason-specific guidance for <code>runner_died_reason</code>.</td></tr>
</table>
<aside class="note">
<strong>Readiness rule:</strong> require <code>runner_actionable:true</code> and
<code>drivable:true</code> before acting.
</aside>
<p class="muted">
<code>mode</code> is a lifecycle field: <code>agent</code> while runner is reachable,
<code>offline</code> otherwise.
<code>version</code> is the running daemon; <code>latest</code> is the newest GitHub
release tag (refreshed daily; <code>null</code> until first fetch / offline;
disable with <code>IPHONE_USE_NO_UPDATE_CHECK=1</code>). When
<code>update_available</code> is true, upgrade with the install one-liner —
the web client shows the same hint as a banner.
</p>
<h3><span class="method get">GET</span> <span class="ep">/agent/capabilities</span></h3>
<p>
Two questions that used to be conflated: what this build can do, and whether
the phone can be driven right now. Read-only — it opens no device connection,
wakes nothing, and takes no owner lease — so it is safe to call before
deciding what to attempt.
</p>
<pre><code>{
"ok": true,
"backend": "direct",
"recovery_owner": "daemon",
"scope": "ui_control",
"supported": {
"single_step_actions": ["alert","back","drag","home","key","keyboard","launch_app",
"longpress","open_url","perform","picker","scroll","set_value",
"shortcut","swipe","tap","tap_locator","text"],
"batch_actions": ["…the same set validated by /agent/actions…"],
"perform_actions": ["increment","decrement","adjust","toggle","menu","double_tap",
"two_finger_tap","scroll_to_visible","pinch","rotate","force_press"],
"element_tree": true,
"observation": { "return_delta": true, "settle_ms_max": 20000 },
"modes": ["agent", "human"],
"lifecycle_managed_here": true
},
"available": {
"ok": false,
"blocked_by": "locked",
"evidence": "cache",
"detail": {
"released": false, "releasing": false, "reconnecting": false,
"human_handoff": false, "runner_up": true, "runner_actionable": false,
"runner_locked": true, "ownership": "free", "needs_owner_identity": false
}
}
}</code></pre>
<p>
<code>supported</code> is static for this build.
<code>modes</code> lists only values this daemon would accept — an externally
managed endpoint lists none, because it answers
<code>runner_is_externally_managed</code> to both.
<code>perform_actions</code> is the closed set checked before the device is
touched; an unlisted name is refused without reaching the phone.
</p>
<p>
<code>available</code> is a <strong>cached snapshot</strong>, marked
<code>evidence: "cache"</code>. Reading it starts no probe, so it reflects the
last completed observation rather than the phone's state this instant.
<code>ok</code> is deliberately three-valued: <code>true</code> drivable,
<code>false</code> blocked, and <code>null</code> meaning the daemon has no
evidence either way — <em>not</em> that things are fine.
<code>blocked_by</code> names the obstacle:
<code>human_handoff</code>,
<code>releasing</code>, <code>reconnecting</code>, <code>released</code>,
<code>locked</code>, <code>owned_by_other</code>, <code>not_actionable</code>,
<code>offline</code>.
</p>
<p>
<code>detail.ownership</code> is decided by running the real lease-admission
rule against a copy of the lease, so the answer cannot drift from what a
control request would get, and nothing is claimed, extended or cleared:
<code>free</code> (no live lease),
<code>self</code> (yours),
<code>refused</code> (a live lease would reject this caller — including an
anonymous one, which the admission rule rejects, so
<code>needs_owner_identity</code> then says naming yourself would change the
answer),
<code>takeover_permitted</code> (an explicit <code>X-Phone-Owner-Takeover</code>
would be admitted, but by evicting the current holder — the phone is not yours
yet), and
<code>expired</code> (a stale record that blocks nobody).
</p>
<p class="muted">
<code>scope: "ui_control"</code> is a boundary, not decoration: this catalogue
describes UI control and observation only. It says nothing about management
surfaces such as app install/uninstall over CoreDevice, and absence here is
not a statement that those do not exist.
</p>
<h3><span class="method post">POST</span> <span class="ep">/agent/mode</span></h3>
<p>
Moves the phone between remote control and the person holding it.
<code>{"mode":"agent"}</code> brings daemon-managed runner up;
<code>{"mode":"human"}</code> stops runner so the holder has the phone to themselves, and
agent input then answers <code>409 phone_handed_to_human</code> until
<code>{"mode":"agent"}</code> takes it back. Any other value returns <code>400</code>;
<code>{"mode":"mirror"}</code> returns <code>410 {"error":"mirror_removed"}</code>.
The canonical classic UDID must already be persisted as
<code>IPHONE_USE_UDID</code>; a request cannot establish or temporarily change it:
</p>
<pre><code>MUTATION="X-Phone-Control: 1"
# first require target_configured:true in /agent/status
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/mode" -d '{"mode":"agent"}'
# poll /agent/status until runner_actionable:true and drivable:true
# a different request-scoped udid returns HTTP 409 target_change_requires_restart
# give the phone back to the person holding it (stops runner)
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/mode" -d '{"mode":"human"}'
# {"ok":true,"mode":"human","released":true,"hint":"…"}</code></pre>
<p class="muted">
<code>mode=agent</code> works only when <code>managed_runner:true</code>, requires
<code>~/.iphone-use/setup-runner.sh</code> and an unlocked, awake phone, and is
asynchronous. During startup, status reports <code>reconnecting:true</code>. A concrete
setup prerequisite failure takes precedence: <code>device_state</code> becomes
<code>"blocked"</code>, <code>setup_blocked_on</code> names the cause, and
<code>hint</code> gives the operator action instead of telling clients to keep waiting.
With no blocker, <code>setup_phase</code> and <code>setup_message</code> report the
live build stage; clients must keep treating <code>drivable:true</code> as the only
permission to act.
When <code>recovery_owner:"external"</code>,
the endpoint returns <code>409 runner_is_externally_managed</code> for both modes; recover
the service on its owning host.
</p>
<p class="muted">
The MCP <code>phone_reconnect</code> tool is a constrained wrapper around
<code>{"mode":"agent"}</code>. It takes no arguments, never accepts a UDID, and never
changes the configured device. For managed runner it waits up to roughly 120
seconds for the persisted canonical target; an externally managed runner endpoint returns
an operator-owned recovery error instead.
</p>
<h3><span class="method post">POST</span> <span class="ep">/agent/input</span></h3>
<p>
Body is one JSON control message. Coordinates are normalized
<span class="pill">[0,1]</span> over the runner window
(<code>0,0</code> = top-left, <code>1,1</code> = bottom-right). Gestures are
atomic: use tap, long-press, swipe, or drag rather than
<code>down</code>/<code>move</code>/<code>up</code> streams.
</p>
<table>
<tr><th>Message</th><th>Direct/runner effect</th></tr>
<tr><td><code>{"type":"tap","x":0.5,"y":0.5}</code></td><td>Tap normalized coordinates on-device.</td></tr>
<tr><td><code>{"type":"tap","label":"新备忘录"}</code></td><td>Tap only when the exact visible label has one match in a fresh element tree. Zero, multiple, or non-actionable matches return <code>422</code> with <code>outcome:"not_sent"</code>. Optional <code>"kind":"Button"</code> narrows matches by element type; rows outside the application frame never compete with an on-screen match. Any element tap whose centre is covered by another control (fixed bar, keyboard, floating button) returns <code>409 element_occluded</code>, <code>outcome:"not_sent"</code>; a target runner reports as <code>visible:false</code> (in the tree but not drawn, such as Chrome's tab grid behind the page) returns <code>409 element_not_visible</code>. <code>"allow_occluded":true</code> overrides both.</td></tr>
<tr><td><code>{"type":"tap_locator","locator":{"identifier":"new-note-button","kind":"Button","enabled":true}}</code></td><td>Re-read the current tree and tap only when every supplied exact locator field resolves to one unique actionable element. Supported fields match <code>wait_for</code>: label, identifier, kind, value, focused, enabled, and visible.</td></tr>
<tr><td><code>{"type":"tap","element":3,"snapshot":"…"}</code></td><td>Tap the centre of element index 3 only if the current tree still matches the ephemeral <code>snapshot</code> returned by the same <code>/agent/elements</code> response.</td></tr>
<tr><td><code>{"type":"longpress","x":0.4,"y":0.6,"duration_ms":700}</code></td><td>One complete long-press; default duration is 600 ms. No matching <code>up</code> is needed.</td></tr>
<tr><td><code>{"type":"swipe","x1":0.5,"y1":0.8,"x2":0.5,"y2":0.2}</code></td><td>One complete swipe from start to end; optional <code>duration_ms</code>.</td></tr>
<tr><td><code>{"type":"drag","x1":0.4,"y1":0.6,"x2":0.7,"y2":0.3}</code></td><td>Hold then drag; optional <code>hold_ms</code> (default 500) and <code>duration_ms</code>.</td></tr>
<tr><td><code>{"type":"scroll","x":0.5,"y":0.5,"dx":0,"dy":60}</code></td><td><strong>Positive <code>dy</code> scrolls down (reveals content farther down); negative <code>dy</code> scrolls up.</strong> Positive <code>dx</code> reveals content to the right. The anchor defaults to screen centre.</td></tr>
<tr><td><code>{"type":"text","text":"你好","clear":true}</code></td><td>Send Unicode to the focused field. <code>clear:true</code> best-effort clears it first.</td></tr>
<tr><td><code>{"type":"set_value","element":5,"snapshot":"…","value":"你好"}</code></td><td>Write a field directly (clear, then type). The field is read back; a value that did not stick (web views often ignore direct writes) returns <code>409 value_not_applied</code> with <code>outcome:"no_effect"</code>. Formatting the field adds itself (separators, spaces) still counts as applied.</td></tr>
<tr><td><code>{"type":"key","name":"return"}</code></td><td>Named runner key: <code>return</code>/<code>enter</code>, <code>escape</code>, <code>space</code>, <code>tab</code>, <code>delete</code>/<code>backspace</code>, or an arrow <code>left</code>/<code>up</code>/<code>right</code>/<code>down</code>.</td></tr>
<tr><td><code>{"type":"keyboard"}</code></td><td>Dismiss the on-screen keyboard. <code>{"type":"key","name":"dismiss"}</code> and <code>hide</code> are aliases.</td></tr>
<tr><td><code>{"type":"shortcut","name":"home"}</code></td><td>Supported through runner. <code>{"type":"home"}</code> is also accepted.</td></tr>
<tr><td><code>{"type":"shortcut","name":"spotlight"}</code></td><td>Supported: go Home, then tap the SpringBoard Search pill.</td></tr>
<tr><td><code>{"type":"shortcut","name":"switcher"}</code></td><td><strong>Unsupported.</strong> runner cannot synthesize the App Switcher system gesture; the request fails instead of pretending success.</td></tr>
<tr><td><code>{"type":"back"}</code></td><td>Navigate back with an on-device left-edge swipe.</td></tr>
<tr><td><code>{"type":"launch_app","bundle":"com.apple.Preferences"}</code></td><td>Launch/foreground an app by bundle id. A limited known-name mapping is also accepted.</td></tr>
<tr><td><code>{"type":"open_url","url":"https://example.com/page"}</code></td><td>Open a link through runner (<code>POST /session/:sid/url</code>): Safari, or the app that owns the scheme. The <code>url</code> needs a scheme and at most 2048 bytes (else <code>400 invalid_value</code>, nothing sent). <code>http</code>/<code>https</code> always open; any other scheme (<code>tel:</code>, <code>sms:</code>, <code>facetime:</code>, <code>mailto:</code>, <code>itms-services:</code>, app schemes such as <code>weixin://</code>) is refused with <code>403 url_scheme_not_allowed</code>, <code>outcome:"not_sent"</code>, unless the owner lists it in <code>IPHONE_USE_ALLOW_URL_SCHEMES</code> (comma list, <code>*</code> = any). Counts as one action against <code>IPHONE_USE_MAX_ACTIONS</code>, and leaves the app, so it is allowed while a blocked app is in front. The activity feed shows the host only.</td></tr>
<tr><td><code>{"type":"picker","column":0,"value":"March"}</code></td><td>Set a runner PickerWheel value.</td></tr>
<tr><td><code>{"type":"install","path":"/abs/App.ipa"}</code></td><td>Install an <code>.ipa</code> or built <code>.app</code> from the daemon's Mac through CoreDevice (<code>xcrun devicectl device install app</code>; macOS only, else <code>501 install_needs_macos</code>). Or <code>"url":"https://…/App.ipa"</code>: https only, at most 2 GB, downloaded to a temporary folder removed afterwards. <code>"fresh":true</code> uninstalls the bundle first. The bundle id is read from the app's <code>Info.plist</code> (<code>Payload/<App>.app/Info.plist</code> in an <code>.ipa</code>); a bundle in <code>IPHONE_USE_BLOCKED_APPS</code> is <code>403 blocked_app</code>, and an unreadable one is <code>403 install_unverified</code> while any app is blocked. Answers <code>{ok, bundle, installed:true}</code>; a bad request is <code>400 invalid_install</code>, nothing run. Never batched; one action against the budget.</td></tr>
</table>
<h4><code>?return=delta</code> — the post-action observation</h4>
<p>
With <code>?return=delta</code> an applied action also attempts a bounded
observation of the resulting screen (it samples the tree repeatedly within
its budget, looking for two matching reads), returned as a <code>delta</code> against
<code>?since=</code> (or the action's own <code>snapshot</code>) when that
baseline is still cached, else as full <code>elements</code>, plus a fresh
<code>snapshot</code> token.
</p>
<p>
<strong>The action result and the observation are separate facts.</strong>
The mutation runs under the endpoint's 15-second action deadline; the
observation runs under its own, strictly shorter one. A slow or failed read
therefore never downgrades an applied action to
<code>outcome_unknown</code>, and never causes the mutation to be re-sent.
</p>
<table>
<tr><th><code>settle</code> field</th><th>Meaning</th></tr>
<tr><td><code>settled</code></td><td><code>true</code> only when the observed TREE stopped changing. Never a claim that the action achieved what you wanted.</td></tr>
<tr><td><code>reason</code></td><td><code>stable</code>, <code>budget_exhausted</code> (stability not confirmed within the budget), or <code>observation_failed</code> (the read path itself broke — the action still applied). Running out of budget is <code>budget_exhausted</code>, never <code>observation_failed</code>: a cancelled read is not a broken one.</td></tr>
<tr><td><code>waited_ms</code> / <code>budget_ms</code> / <code>captures</code></td><td>Time spent, budget allowed, and successful reads — all describing the TREE read only. The <code>alert</code> probe is separate and hard-capped at 1.5s.</td></tr>
<tr><td><code>sparse</code></td><td>Present when the tree held nothing to act on (empty, or container kinds only). Two identical bare reads are evidence the screen is unreadable, not that it settled.</td></tr>
<tr><td><code>stale</code></td><td>Present when the returned tree is the last successful read rather than a fresh one — because the refreshing read broke, or because the budget cut it off mid-sample. Never appears with <code>settled:true</code>.</td></tr>
</table>
<p>
<code>settle_ms</code> (default 15000, max 20000) sets the tree budget;
sized so two reads fit even when a single element read is slow (6–7s was
measured on one phone with a large tree). A budget shorter than one read may
come back with no sample at all — <code>captures: 0</code> — however healthy
the phone is; the answer carries the budget it used, so that is visible
rather than mysterious. Reading stops as soon as two reads match, so this is
a ceiling and not a wait. The observation's clock starts once the action is
confirmed, so a slow action does not consume the window;
<code>settle_ms=0</code> does not read the element tree at all (the separate 1.5s <code>alert</code> probe may still run). For existing callers,
<code>delta_error</code> is still emitted alongside <code>ok:true</code>
whenever the observation failed.
</p>
<p>
<code>/agent/input</code> returns <code>200</code> only after the runner action succeeds.
<code>/agent/input</code> adds no deadline field to the action payload; the server
applies a fixed 15-second total deadline from request receipt, including runner mutex
waiting and execution.
Invalid JSON returns <code>400</code>; a wrong bearer returns <code>401</code>;
auth lockout returns <code>429</code>; an unsupported atomic runner command can return
<code>422</code>; a down, missing, or unsupported runner path returns
<code>503</code> with <code>"fallback":"disabled"</code>. An idle-released device also
returns <code>503</code> plus <code>Retry-After: 5</code> while runner restarts.
</p>
<aside class="note">
<strong>Snapshot refs are intentionally short-lived.</strong>
If the element tree changes before an indexed tap, the server returns
<code>409 stale_element_snapshot</code> with <code>retry_safe:true</code> and sends
no tap. Fetch <code>/agent/elements</code> again and choose again. Recorded flows
should persist accessibility labels or another stable locator, never an element
index or snapshot token.
</aside>
<aside class="note">
<strong>Label taps are strict, not “first match wins”.</strong>
Exact-label resolution returns <code>422 element_not_found</code>,
<code>ambiguous_element_label</code>, or <code>invalid_element_target</code> before
sending an action. When a label is repeated, use <code>/agent/elements</code> to
disambiguate by identifier, kind, or state, then submit its index with the same
snapshot token.
</aside>
<aside class="note">
<strong>Destructive operation:</strong>
<code>{"type":"uninstall","bundle":"com.example.app"}</code> removes that app and its
data through <code>devicectl</code>. It is bearer-gated but irreversible at the API
level. Configure <code>IPHONE_USE_UDID</code> before starting the daemon on a
multi-device Mac; if a request includes <code>udid</code>, it must match that canonical
target.
</aside>
<h3><span class="method post">POST</span> <span class="ep">/agent/actions</span></h3>
<p>
Use this endpoint after a flow segment is understood and its stable locators and
postconditions are known. One request can contain at most 24 steps. The daemon validates
the <em>entire</em> JSON sequence before dispatching the first mutation, then holds one
runner control lock so another daemon client cannot interleave gestures. The first failed
action, expectation, read, lifecycle transition, or deadline stops the sequence; every
later action remains unsent.
</p>
<pre><code>{
"steps": [
{
"kind": "action",
"action": {"type": "shortcut", "name": "home"},
"after_ms": 300
},
{
"kind": "wait_for",
"expect": {
"present": [{"label": "搜索"}],
"absent": [{"label": "确认付款"}]
},
"timeout_ms": 3000,
"poll_ms": 100
},
{
"kind": "action",
"action": {"type": "tap", "label": "搜索"}
},
{
"kind": "wait_for",
"expect": {
"application": "聚焦",
"present": [{"kind": "TextField", "focused": true}]
}
}
]
}</code></pre>
<table>
<tr><th>Step kind</th><th>Contract</th></tr>
<tr><td><code>action</code></td><td>One supported Direct/runner input payload, including strict <code>tap_locator</code> and validated bundle-id <code>launch_app</code>. Optional <code>after_ms</code> is capped at 3000 ms and is only an animation settle. Destructive <code>uninstall</code> is never accepted in a batch.</td></tr>
<tr><td><code>wait_for</code></td><td>Polls the current element tree. <code>application</code> is an exact Application label; every <code>present</code> locator must match and every <code>absent</code> locator must not match. Locator fields are exact <code>label</code>, <code>identifier</code>, <code>kind</code>, <code>value</code>, <code>focused</code>, <code>enabled</code>, and <code>visible</code>. Timeout is 1–30000 ms (default 15000); poll interval is 50–1000 ms. The
per-wait ceiling leaves room for several polls even when an element read is
slow; the cumulative declared wait (60000 ms) and the batch deadline are
unchanged, so a longer single wait cannot lengthen a batch. An empty or container-only tree never satisfies an <code>absent</code> locator — see the <code>observation</code> block below.</td></tr>
<tr><td><code>pause</code></td><td>A fixed 1–3000 ms pause for animation that exposes no semantic state. Prefer <code>wait_for</code>.</td></tr>
</table>
<p>
Declared waits across one batch are capped at 60 seconds, the request body at 64 KiB,
and the server's total execution budget at 75 seconds. A success response includes
<code>completed</code>, <code>applied_actions</code>, and compact per-step results.
A failure includes <code>failed_step</code>, the completed/applied counts,
completed <code>steps</code>, the last failed wait <code>observation</code> when
available, <code>outcome</code>, <code>failed_step_outcome</code>,
<code>batch_outcome</code>, and <code>retry_safe</code>. A read-only
<code>wait_for</code> rebuilds one stale runner source session within its existing
deadline. <code>retry_safe</code> describes replaying the <em>whole batch</em>:
once an earlier action landed it is false, even when the step that failed was
itself never sent.
</p>
<p>
<code>outcome</code> describes the step that FAILED, not the batch, and keeps
that meaning for compatibility: a body can carry <code>applied_actions: 2</code>
next to <code>outcome: "not_sent"</code>, and both are true of different
things. <code>failed_step_outcome</code> is the same value under a name that
says which. <code>batch_outcome</code> is the batch's own verdict:
</p>
<table>
<tr><th>Value</th><th>What it means</th></tr>
<tr><td><code>nothing_applied</code></td><td>No action reached the phone.</td></tr>
<tr><td><code>partially_applied</code></td><td>At least one action reached the phone before the batch stopped.</td></tr>
<tr><td><code>unknown</code></td><td>The failed step's own outcome is unknown, so what the batch did cannot be stated. This is never downgraded to <code>partially_applied</code>, which would claim the earlier actions are settled and only the last one in doubt.</td></tr>
</table>
<p>
None of the three authorises a retry. <code>retry_safe</code> remains the only
field that does, and it is not inferable from these.
</p>
<h4>The <code>observation</code> block on a failed <code>wait_for</code></h4>
<p>
A failing step reports what was actually seen, because "the condition was not
met" and "nobody could see the screen" call for opposite responses:
</p>
<table>
<tr><th>Shape</th><th>What happened</th></tr>
<tr><td><code>read:false</code></td><td>No readable tree was ever obtained (<code>reads:0</code>, with <code>read_error</code>). <strong>Nothing here proves an element is absent.</strong></td></tr>
<tr><td><code>read:true, stale:true</code></td><td>The screen was read at least once and then the read path failed. The observation is the last valid one, not the current screen. A late failure never erases the reads that succeeded.</td></tr>
<tr><td><code>read:true</code></td><td>The screen was read and the condition simply was not met — <code>missing_present</code> / <code>violated_absent</code> say which.</td></tr>
</table>
<p>
<code>sparse:true</code> means the tree was empty or container-only.
Because such a tree satisfies any <code>absent</code> locator vacuously, each
one is listed in <code>absent_unproven</code> and the step does NOT match:
an unreadable screen is not evidence something is gone. <code>rows</code>,
<code>attempts</code> and <code>reads</code> carry the raw counts.
</p>
<aside class="note">
<strong>This is not permission to batch guesses.</strong> Discover a changing or
high-consequence flow with the normal see → one action → verify loop. Batch only the
stable segment, put semantic gates around page transitions, and inspect current state
instead of replaying a batch after <code>outcome_unknown</code> or
<code>retry_safe:false</code>.
</aside>
<aside class="note">
<strong>One-command replay without a model:</strong> the release-matched
<code>iphone-use-mcp</code> binary accepts
<code>flow validate <file.json></code> and
<code>flow run <file.json> --input key=value</code>. Flow v1 contains a name,
optional explicit string inputs, and the same guarded
step objects exposed by MCP. Validation is offline; execution first requires a Direct
daemon with <code>drivable:true</code>, submits exactly one
<code>/agent/actions</code> request, and never retries. Input values are resolved only
for the current invocation and are never written back to JSON. CLI arguments may remain
visible in shell history or process inspection, so secrets, private content, payment,
send, publish, or delete actions still do not belong in this mechanism.
</aside>
<aside class="note">
<strong>Browser recorder:</strong> the <code>/phone</code> control bar now exposes a
reviewed automation flow. It records only acknowledged actions, prefers exact
accessibility labels selected through the element panel, flags coordinate taps and
gestures as fragile, and converts typed text into a named runtime parameter without
retaining the literal recorded value. The operator fills a fresh in-memory value before
one-shot execution; downloaded JSON contains only the parameter definition. After an
action, it compares
bounded element-tree snapshots and appends a <code>wait_for</code> checkpoint only
when a new unique accessibility identifier or foreground application can be proved.
It never copies arbitrary screen labels or values into automatic checkpoints because
those may contain names, balances, or other private content. Otherwise it visibly
retains the 350 ms fallback. The operator can move an action
together with its checkpoint, remove either, download valid flow-v1 JSON, or send the
batch once. If another action could not be persisted, the download is explicitly marked
an incomplete draft and one-shot execution stays disabled. Parameterized flows remain
runnable only after every required value is filled. A later browser session can open
the saved JSON, restore its reviewable steps after strict validation, fill fresh
runtime parameters, and replay it without a model. Browser import rejects literal
typed text so private input cannot silently survive inside the file.
Complete flows still require confirmation that the reviewed steps contain no payment,
send, publish, delete, or other irreversible action.
</aside>
<h3><span class="method post">POST</span> <span class="ep">/control</span> <span class="muted">(browser)</span></h3>
<p>
The browser uses the same runner dispatcher through a separate cookie-authenticated
endpoint. Requests must include <code>X-Phone-Control: 1</code>; successful commands
return only <code>{"ok":true}</code>. Every message must also include a positive
<code>ttl_ms</code> from 1 through 2500; optional <code>issued_at_ms</code> is audit
metadata, while freshness is enforced with the server's monotonic deadline. Missing
login, missing control header/deadline, invalid JSON, runner failure, expiry, and lifecycle
transitions are explicit HTTP errors, so the UI must not clear input or show success
until the response is successful.
</p>
<h3>Delivery outcomes and retry safety</h3>
<p>
<code>/control</code>, <code>/agent/input</code>, and every mutation step
inside <code>/agent/actions</code> use at-most-once delivery semantics. A deadline
does not automatically mean the action failed:
</p>
<table>
<tr><th>When the deadline fires</th><th>Response</th><th>Caller action</th></tr>
<tr>
<td>Before the runner action is dispatched</td>
<td><code>408</code><br><code>{"ok":false,"error":"not_sent","outcome":"not_sent","retry_safe":true}</code></td>
<td>The daemon confirms nothing was sent. A caller may retry if the action is still wanted.</td>
</tr>
<tr>
<td>After dispatch, runner reports an error or transport failure but cannot prove the action did not land</td>
<td><code>502</code><br><code>{"ok":false,"error":"outcome_unknown","outcome":"unknown","retry_safe":false}</code></td>
<td>Inspect status and phone state. Treat the command as possibly applied; do not replay it blindly.</td>
</tr>
<tr>
<td>After the runner future has started, but completion cannot be confirmed</td>
<td><code>504</code><br><code>{"ok":false,"error":"outcome_unknown","outcome":"unknown","retry_safe":false}</code></td>
<td>Read <code>/agent/status</code> and then elements/screenshot before deciding what happened. Do not blindly replay text, scroll, Back, taps, or other non-idempotent input.</td>
</tr>
</table>
<p class="muted">
None of these responses carries <code>Retry-After</code>. In particular, a 502 or 504
is not permission to retry: the first action may already have landed on the phone.
For a multi-step response, <code>retry_safe</code> also accounts for all earlier
applied steps, so a later unsent failure does not make replaying the whole batch safe.
</p>
<h3><span class="method get">GET</span> <span class="ep">/agent/mjpeg</span></h3>
<p>
Proxies runner's on-device MJPEG service
(<code>IPHONE_USE_RUNNER_MJPEG_URL</code>, normally
<code>http://127.0.0.1:9100</code>) as
<code>multipart/x-mixed-replace</code>. It accepts either the agent bearer or a valid
browser session cookie. An open stream counts as activity so idle release does not stop
runner while somebody is watching.
</p>
<p class="muted">
The daemon allows at most four simultaneous live streams. A missing MJPEG endpoint
returns <code>503</code>; upstream connection/stream failure is an explicit error. Each
open stream increments both <code>viewer_count</code> and
<code>mjpeg_viewer_count</code>.
</p>
<h3><span class="method get">GET</span> <span class="ep">/agent/h264</span></h3>
<p>
The live screen as H.264. One upstream MJPEG connection per daemon is decoded and
re-encoded on the Mac's hardware encoder (VideoToolbox, Main profile, no B-frames,
<code>IPHONE_USE_H264_KBPS</code> average bitrate, default 2500); every subscriber gets
the same encoded stream, and a new subscriber triggers an IDR. Auth, lifecycle refusals,
the four-viewer limit and <code>?stream_id=</code> freshness tracking are the same as
<code>/agent/mjpeg</code>; a daemon that cannot encode answers <code>501</code>.
</p>
<p>
The body (<code>application/octet-stream</code>) is a sequence of messages:
<code>[u32 BE length of the rest][u8 flags][u64 BE pts µs][Annex-B access unit]</code>,
flags bit 0 = keyframe. Keyframes carry SPS and PPS in-band, so a client can derive the
codec string (<code>avc1.</code> + SPS bytes 1–3 in hex) and start at any keyframe.
</p>
<h3>Timing on every answer</h3>
<p>
Every <code>/agent/*</code> answer carries a <code>Server-Timing</code> header (<code>total</code>,
<code>runner</code>, and one metric per runner route such as <code>runner-get-source</code>). A JSON answer that
reached runner also gets a <code>timing</code> object: <code>{total_ms, runner_ms, daemon_ms, runner:[{call, count,
ms, bytes}]}</code>, where <code>ms</code> is the time until runner's response headers (runner builds the whole
answer before sending) and <code>bytes</code> its declared size. Live streams are not timed.
</p>
<p>
The daemon appends one line per request (status polls excepted) to
<code>~/.iphone-use/agent-timing.jsonl</code>: timestamp, <code>X-Phone-Owner</code>, method, path, query
parameter <em>names</em>, status and the same <code>timing</code> object — never request or screen content.
It rotates to <code>agent-timing.jsonl.1</code> at 20 MB.
</p>
<h3><span class="method post">POST</span> <span class="ep">/pair/new</span> · <span class="method get">GET</span>/<span class="method post">POST</span> <span class="ep">/pair</span></h3>
<p>
Scan-to-connect. A signed-in browser (session cookie + <code>X-Phone-Control: 1</code>)
calls <code>POST /pair/new</code> and gets <code>{ok, url, host, hosts, expires_in_secs,
svg}</code>: <code>url</code> is <code>http://<host>:<port>/pair?c=<code></code>
and <code>svg</code> its QR code. The host is the address the browser used, or this Mac's
best LAN address when it was opened on <code>127.0.0.1</code>; <code>?host=</code> picks
another of <code>hosts</code>. A loopback-only daemon answers <code>409 loopback_only</code>.
Codes work once, within 5 minutes.
</p>
<p>
<code>GET /pair?c=</code> is the landing page the system camera opens; it does not consume
the code and offers <code>iphoneuse://pair?u=<base>&c=<code></code> for the
app or a form post for browser control. <code>POST /pair</code> trades the code: as a form
(<code>c=</code>) it sets the session cookie and redirects to <code>/phone</code>; as JSON
<code>{"code"}</code> it sets the cookie and returns a <code>device_token</code> (180 days);
JSON <code>{"device_token"}</code> returns a fresh cookie. Device tokens are signed with a key
derived from the daemon secret and the control password, so changing the password revokes
them. Failures count toward the login lockout.
</p>
<h3><span class="method get">GET</span> <span class="ep">/agent/screenshot</span></h3>
<p>
<strong>Screens hidden from capture.</strong> When the app protects its content (PayPay and other
payment apps), iOS blanks it in every capture. If the capture's content band is one flat colour while
the accessibility tree has labelled elements there, the daemon returns the capture with the tree drawn
over it — outlines, kinds and labels under a banner — and sets <code>X-Capture-Redacted: 1</code> and
<code>X-Screenshot-Source: accessibility-wireframe</code>. <code>?raw=1</code> skips this. Captures over
600 KB are never checked (protected screens compress far below that).
</p>
<p>
Returns a validated <code>image/png</code> of the selected phone, read from runner: if runner
is down or returns an invalid frame, the endpoint returns an error. An invalid
PNG, runner error, or timeout immediately revokes cached actionability before the response
returns.
</p>
<h3><span class="method get">GET</span> <span class="ep">/agent/elements</span></h3>
<p>
<strong>Requires actionable runner.</strong> The phone's current UI is flattened to
agent-friendly rows in document order, accompanied by runner screen dimensions:
</p>
<pre><code>{
"screen": { "width": 440, "height": 956 },
"snapshot": "rQq8…",
"elements": [
{
"kind": "Button",
"label": "新备忘录",
"identifier": "new-note-button",
"rect": [369, 885, 38, 38],
"accessible": true,
"depth": 7
}
]
}</code></pre>
<p class="muted">
Prefer this over <code>/agent/screenshot</code> when labels and structure are enough:
the labels feed straight into
<code>{"type":"tap","label":…}</code>, and array indexes feed
<code>{"type":"tap","element":N,"snapshot":"…"}</code> using the token from the same
response. Failures keep a parseable JSON body but never
masquerade as a successful empty tree: transition, missing-runner, or busy states return
<code>503</code>; a failed source fetch after one stale-session retry returns
<code>502</code>. Clients must reject non-2xx responses and then inspect
<code>error</code>, <code>transitioning</code>, and <code>/agent/status</code>.
A source failure or timeout immediately revokes cached actionability, so an older
successful health probe cannot leave <code>drivable:true</code> behind.
</p>
<p class="muted">
Rows stay sparse for low-token callers. <code>enabled:false</code> and
<code>visible:false</code> appear only when runner explicitly reports a node as disabled
or hidden; <code>accessible:true</code> and <code>focused:true</code> appear only when
runner explicitly reports them true. An absent state field is therefore not a positive
assertion when an upstream source omitted that attribute. <code>identifier</code> and
<code>placeholder</code> appear only when the application supplies them.
</p>
<p class="muted">
Optional filters of the same fresh read, combinable with each other and with
<code>?scope=app|interactive|focused</code>: <code>?query=<text></code> keeps rows
whose <code>label</code>, <code>identifier</code>, <code>value</code> or
<code>placeholder</code> contains the text, case-insensitively (CJK as written; at most
200 characters, else <code>400 invalid_query</code>); <code>?limit=<n></code>
(1–500, else <code>400 invalid_limit</code>) caps the rows after scope and query. Kept
rows carry their original <code>index</code> and <code>snapshot</code> stays the full
read's token, so element taps work unchanged. The answer adds
<code>"filter": {"query", "limit", "matched", "returned", "omitted", "of"}</code>, with a
<code>hint</code> when rows were omitted or nothing matched. A <code>delta</code> answer
(<code>?since=</code>) is not filtered.
</p>
<p class="muted">
<code>?ocr=auto</code> (or <code>1</code>) adds Apple Vision text recognition of a fresh
screenshot when the tree is unusable (no interactive rows, containers only);
<code>?ocr=force</code> adds it on every full read; anything but <code>auto</code>,
<code>1</code>, <code>force</code> or <code>off</code> is <code>400 invalid_ocr</code>. The
answer gains <code>"ocr": {"rows": [{"kind":"OCRText","label","rect","depth":0,"source":"ocr","confidence"}], "source": "vision"}</code>;
<code>rect</code> is in points like every row. Rows have no element behind them: tap by
coordinates. A failure is an <code>ocr_error</code> string beside a normal answer.
</p>
<h3><span class="method get">GET</span> <span class="ep">/agent/screenshot?overlay=elements</span></h3>
<p>
The same PNG with each interactive element (the rows <code>?scope=interactive</code> keeps)
outlined and numbered with its index in a full <code>/agent/elements</code> read. The
response adds <code>X-Overlay: elements</code>, <code>X-Overlay-Marks: <n></code> and
<code>X-Elements-Snapshot</code>, the snapshot to tap by
(<code>{"type":"tap","element":12,"snapshot":"…"}</code>). It works with
<code>max_side</code>. When the tree cannot be read, the plain picture comes back with
<code>X-Overlay: unavailable</code>.
</p>
<h3>Device diagnostics, location and recording</h3>
<p>
These use the daemon's configured phone (<code>IPHONE_USE_UDID</code>) over its lockdown
services. Without one they answer <code>503 device_not_configured</code>; an unreachable phone
is <code>502 device_service_failed</code> or <code>504 device_timeout</code>, each with
<code>detail</code> and <code>hint</code>. Reads accept the bearer or a logged-in browser;
POSTs need the bearer, <code>X-Phone-Control: 1</code>, and take the owner lease.
</p>
<table>
<tr><th>Call</th><th>Answer</th></tr>
<tr><td><code>POST /agent/logs {"action":"start"|"stop"|"clear"}</code></td><td><code>{ok, capturing}</code>. Captures <code>com.apple.syslog_relay</code> into a ring of the last 5,000 lines; stops by itself 15 minutes after the last read. Lines never reach the daemon's own log.</td></tr>
<tr><td><code>GET /agent/logs?since=&process=&contains=&level=&limit=</code></td><td><code>{ok, items:[{seq,ts,process,subsystem?,pid,level,message}], next, capturing, missed?, stopped?}</code>. <code>level</code> is that level or more severe; <code>next</code> is the next <code>since</code>.</td></tr>
<tr><td><code>GET /agent/crashes?process=&since=&limit=</code></td><td><code>{ok, items:[{name,folder?,process,time,stamp?,size}]}</code>, newest first (<code>limit</code> 1–500, default 50).</td></tr>
<tr><td><code>GET /agent/crashes/:name</code></td><td><code>{ok, name, text, size, truncated}</code>, first 2 MB. <code>400 invalid_name</code> for anything but a bare report name; <code>404 not_found</code>.</td></tr>
<tr><td><code>POST /agent/location {"lat":35.68,"lon":139.76}</code> / <code>{"clear":true}</code></td><td><code>{ok}</code>. <code>com.apple.dt.simulatelocation</code> (Developer Mode and the Developer Disk Image). <code>400 invalid_location</code> out of range. One action against <code>IPHONE_USE_MAX_ACTIONS</code>.</td></tr>
<tr><td><code>POST /agent/record {"action":"start"}</code></td><td><code>{ok, recording:{path,seconds,frames,bytes,ended}}</code>, into <code><state dir>/runs/recordings</code>. <code>409 already_recording</code>; <code>503 video_unavailable</code> without the macOS live view.</td></tr>
<tr><td><code>POST /agent/record {"action":"stop"}</code></td><td><code>{ok, summary:{path,seconds,frames,bytes,width,height,ended?}}</code>, or <code>summary:null</code> when nothing was recording. <code>502 recording_empty</code> when no frame arrived. Ends by itself at 10 minutes or 200 MB (<code>ended</code>).</td></tr>
<tr><td><code>GET /agent/record</code></td><td><code>{ok, recording}</code> (<code>null</code> when none).</td></tr>
</table>
<h3><span class="method get">GET</span>/<span class="method put">PUT</span> <span class="ep">/agent/app-notes?bundle=…</span></h3>
<p>
Plain markdown about one app, written by agents or the person: obstacles, where things
are, what not to do. GET answers <code>{ok, bundle, text, updated_at?}</code> (empty
<code>text</code> when none); PUT takes the markdown as the body (at most 16 KB, else
<code>413 too_large</code>; empty deletes) and needs <code>X-Phone-Control: 1</code>.
Stored as <code>~/.iphone-use/app-notes/<bundle>.md</code>, beside the flow store.
The first 2 KB appear as <code>app_notes</code> in the <code>registry</code> block when an
agent enters the app.
</p>
<h3><span class="method get">GET</span>/<span class="method post">POST</span> <span class="ep">/agent/inbox</span></h3>
<p>
The return path of the <a href="../shortcuts/README.md">legacy Shortcuts RPC bridge</a>: the phone
(an iOS Shortcut's "Get Contents of URL" action) <code>POST</code>s arbitrary JSON here;
an agent <code>GET</code>s it. GET is always a read-only peek; the legacy
<code>?peek=1</code> spelling remains equivalent. The append POST requires agent auth
plus <code>X-Phone-Control: 1</code>. To consume results, use
<code>POST /agent/inbox/drain</code> with the same auth and mutation header; it returns
and atomically clears the queue. Bounded ring buffer (64), oldest dropped.
</p>
<aside class="note">
The inbox endpoint remains available, but launching the Shortcuts bridge from the agent
side is not a promised runner capability; treat the bridge as legacy.
</aside>
<h2>Example — a see-decide-act loop</h2>
<pre><code>HOST=http://192.168.0.190:44321
PW=<password>
AUTH="Authorization: Bearer $PW"
MUTATION="X-Phone-Control: 1"
# 1. require the configured target to be actionable
curl -s -H "$AUTH" "$HOST/agent/status"
# continue only with runner_actionable=true, drivable=true
curl -s -H "$AUTH" "$HOST/agent/elements"
# 2. see
curl -s -H "$AUTH" "$HOST/agent/screenshot" -o screen.png
# 3. act — Home and Spotlight are supported by runner
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"shortcut","name":"home"}'
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"shortcut","name":"spotlight"}'
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"text","text":"Notes"}'
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"tap","x":0.5,"y":0.22}'
curl -s -H "$AUTH" -H "$MUTATION" -X POST "$HOST/agent/input" -d '{"type":"scroll","x":0.5,"y":0.5,"dx":0,"dy":60}'</code></pre>
<h2>Behaviour notes</h2>
<aside class="note">
<strong>No Mac cursor.</strong> runner synthesizes input on the phone; the Mac's screen,
cursor, and focus are never touched.
</aside>
<aside class="note">
<strong>Unicode text.</strong> runner sends text through <code>/wda/keys</code>, so CJK
text is typed as-is rather than through keycodes and the phone's input method.
</aside>
<aside class="note">
<strong>Video and screenshots are distinct health layers.</strong>
<code>/agent/mjpeg</code> proves the 9100 live-video relay; a successful
<code>/agent/screenshot</code> proves a single runner PNG. Neither alone proves that input is
actionable — keep gating actions on status.
</aside>
<aside class="note">
<strong>System-command boundary.</strong> runner supports Home, Spotlight, Back, keyboard
dismissal, and the documented named keys. App Switcher is unsupported. Launching the
Shortcuts bridge, Control Center, and other system gestures are out of scope until they
gain a device-side implementation and hardware acceptance.
</aside>
<h2>Live acceptance boundary</h2>
<p>
Source/tests and a <code>200</code> status response are not physical-device proof. Before
shipping an integration, verify on the intended iPhone that:
</p>
<ol>
<li>the daemon runs without any macOS permission grant and never touches the Mac's screen or cursor;</li>
<li>status identifies the pinned device and reports actionable/drivable runner;</li>
<li>managed recovery reports <code>releasing → released → reconnecting → ready</code>, while an external endpoint remains owned by <code>recovery_owner:"external"</code>;</li>
<li>MJPEG updates continuously and a stopped 9100 relay is reported as failure;</li>
<li>tap, long-press, swipe, drag, both scroll directions, CJK text, named keys, Home, and Spotlight each land once;</li>
<li>Switcher fails honestly and runner loss is reported as an error, never as a stale or substitute frame;</li>
<li>lock/unlock, USB reconnect, daemon/Mac restart, and a multi-device host recover the same canonical target;</li>
<li>8100/9100 exposure is measured on an isolated network rather than assumed safe because the Mac relay is loopback-only.</li>
</ol>
</div>
</body>
</html>