# drafts

name: draft-ietf-roll-enrollment-priority
rev: 15
title: Controlling Secure Network Enrollment in RPL networks
time: 2026-02-20T15:43:37Z
group_uri: /api/v1/group/group/1730/
expires: 2026-07-28T19:17:14Z
ad: /api/v1/person/person/122823/
shepherd: /api/v1/person/email/mariainesrobles@googlemail.com/
states: ["/api/v1/doc/state/1/", "/api/v1/doc/state/13/", "/api/v1/doc/state/44/"]
categories: ["Agent discovery/reg", "Autonomous netops"]
tags: []
fetched_at: 2026-02-27T22:56:07.990704+00:00
abstract:
[RFC9032] defines a method by which a potential [RFC9031] enrollment
   proxy can announce itself as available for new RPL nodes to enroll on
   a network.  The announcement includes a priority for enrollment.
   This document provides a mechanism by which a Routing Protocol for
   Low-Power and Lossy Networks (RPL) Root can globally disable
   enrollment announcements or adjust the base priority for enrollment
   operations.

full_text:
ROLL Working Group                                         M. Richardson
Internet-Draft                                  Sandelman Software Works
Intended status: Standards Track                            R. A. Jadhav
Expires: 28 July 2026                                        Huawei Tech
                                                              P. Thubert
                                                           Cisco Systems
                                                             K. Iwanicki
                                                    University of Warsaw
                                                         24 January 2026


         Controlling Secure Network Enrollment in RPL networks
                 draft-ietf-roll-enrollment-priority-15

Abstract

   [RFC9032] defines a method by which a potential [RFC9031] enrollment
   proxy can announce itself as available for new RPL nodes to enroll on
   a network.  The announcement includes a priority for enrollment.
   This document provides a mechanism by which a Routing Protocol for
   Low-Power and Lossy Networks (RPL) Root can globally disable
   enrollment announcements or adjust the base priority for enrollment
   operations.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 28 July 2026.

Copyright Notice

   Copyright (c) 2026 IETF Trust and the persons identified as the
   document authors.  All rights reserved.






Richardson, et al.        Expires 28 July 2026                  [Page 1]

Internet-Draft                 join-metric                  January 2026


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Motivation and Overview . . . . . . . . . . . . . . . . .   2
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Protocol Definition . . . . . . . . . . . . . . . . . . . . .   4
     3.1.  Option Format . . . . . . . . . . . . . . . . . . . . . .   4
     3.2.  Option Processing . . . . . . . . . . . . . . . . . . . .   5
     3.3.  Upwards Compatibility . . . . . . . . . . . . . . . . . .   6
   4.  Security Considerations . . . . . . . . . . . . . . . . . . .   7
   5.  Privacy Considerations  . . . . . . . . . . . . . . . . . . .   8
   6.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   8
   7.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   8
   8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   8
     8.1.  Normative References  . . . . . . . . . . . . . . . . . .   8
     8.2.  Informative References  . . . . . . . . . . . . . . . . .   9
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   9

1.  Introduction

   [RFC7554] describes the use of the Time-Slotted Channel Hopping
   (TSCH) mode of [ieee802154].  [RFC9031] and [RFC9032] describe
   mechanisms by which a new node (the "Pledge") can use a nearby router
   as a Join Proxy.  [RFC9032] describes an extension to the 802.15.4
   Enhanced Beacon that is used by a Join Proxy to announce its
   existence such that Pledges can find them.

1.1.  Motivation and Overview

   Not every routing member of a mesh ought to announce itself as a
   _Join Proxy_. There are a variety of local reasons for which a
   6LowPAN Router (6LR) might not want to provide the _Join Proxy_
   function.  Some reasons include low available battery power, already
   high committed network bandwidth, and lack of available free memory
   for Neighbor Cache Entry (NCE) slots.  An NCE entry is needed in
   order to maintain communication with the Pledge nodes trying to
   enroll.





Richardson, et al.        Expires 28 July 2026                  [Page 2]

Internet-Draft                 join-metric                  January 2026


   There are other situations where the operator of the network would
   like to selectively enable or disable the enrollment process in a
   specific Destination Oriented Directed Acyclic Graph (DODAG).  In
   particular, as the enrollment process involves permitting unencrypted
   traffic into the best effort part of a network, it would be better to
   turn the enrollment process off when no new nodes are expected.

   This document describes a Routing Protocol for Low-Power and Lossy
   Networks (RPL) Destination Information Object (DIO) option that can
   be used to set a minimum enrollment priority.  The minimum priority
   expresses the inability of the RPL DODAG globally to accept new
   joins.  It may derive from multiple constraining factors, for
   instance, the size of the DODAG, the occupancy of the bandwidth at
   the DODAG Root, the memory capacity at the Root, or an administrative
   decision.  Each potential _Join Proxy_ utilizes this value as a base
   on which to add values relating to local conditions, such as its Rank
   and number of pending joins.  As explained in [RFC9032], higher
   values decrease the likelihood of an unenrolled node sending
   enrollment traffic via this _Join Proxy_. In particular, by setting
   the minimum enrollment priority to the maximum value allowed, a
   network operator can globally disable all new enrollment traffic.

   Moreover, when a RPL domain is composed of multiple DODAGs, a node at
   the edge of more than one such DODAG may not only join any of the
   DODAGs but also move between them in order to keep their relative
   sizes balanced.  For this, the approximate knowledge of the size of
   the DODAGs is also an essential metric.  Depending on the network
   policy, the size of the DODAG may or may not affect the minimum
   enrollment priority.  Therefore, since making one proportional to the
   other would be limiting their value, the current size of the DODAG is
   advertised separately in the new option.

   Updates to the option propagate through the network according to the
   trickle algorithm.  The contents of the option are generated at the
   DODAG Root and do not change at any hop.  If the contents represent
   an update that is considered important (e.g., quickly disabling any
   enrollments), the option can trigger trickle timer resets at the
   nodes to speed up its propagation.

2.  Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.





Richardson, et al.        Expires 28 July 2026                  [Page 3]

Internet-Draft                 join-metric                  January 2026


   The term 6LR means 6LowPAN Router, and is defined in [RFC6606].  It
   refers to a router that forwards packets in a 6LowPAN network.

   The terms DAO, DODAG, DODAG root, DIO, trickle timer are from
   [RFC6550].  The lollipop counter function comes from [RFC6550],
   Section 7.2.

   The term (1)"Join" has been used in documents such as [RFC9031] to
   denote the activity of a new node authenticating itself to the
   network to obtain authorization to become a member of the network.

   In the context of the [RFC6550] RPL protocol, the term (2)"Join" has
   an alternative meaning: that of a node (already authenticated to the
   network, and already authorized to be a member of the network),
   deciding which part of the RPL DODAG to attach to.  This term "Join"
   has to do with preferred parent selection processes.

   In order to avoid the ambiguity of this term, this document refers to
   the process (1)"Join" as enrollment, leaving the term "Join" to mean
   (2)"Join".  The term "onboarding" (or "IoT Onboarding") is
   increasingly used to describe what is now called (1)Join in other
   documents, and is called enrollment in this document.  However, the
   term _Join Proxy_ is retained with its (1)"Join" meaning from
   [RFC9031].

3.  Protocol Definition

   This document uses the extensions mechanism specified by [RFC6550].
   No mechanism is needed to enable it.

3.1.  Option Format

   The following option is defined for transmission in DIOs issued by
   the DODAG Root to be propagated within the DODAG.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Type = TBD01  |Opt Length = 4 |Version Number |T| Min Priority|
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Exp  |DODAGSz|
      +-+-+-+-+-+-+-+-+

   Type  To be assigned by IANA.

   Version Number  An 8-bit unsigned integer set by the DODAG root and





Richardson, et al.        Expires 28 July 2026                  [Page 4]

Internet-Draft                 join-metric                  January 2026


      denoting the version number of the contents of the option.  The
      version number is interpreted as a lollipop counter (see
      Section 7.2 of [RFC6550]).

   T  A bit indicating whether the particular version of the option is
      important in that adopting its contents should trigger a trickle
      timer reset at the node.

   Min Priority  A 7-bit field providing a base value for the Enhanced
      Beacon Join priority.  A value of 0x7f (127) disables the _Join
      Proxy_ function entirely.

   Exp  A 4-bit unsigned integer indicating the power of 2 that defines
      the unit of the DODAG Size, such that (unit = 2^Exp).

   DODAGSz  A 4-bit unsigned integer expressing the size of the DODAG in
      units that depend on the Exp field.

   The DODAG Size is calculated as (DODAGSz * 2^Exp).

   The DODAG Size can be measured by the Root based on the DAO activity.
   In such a case, it represents the number of routes not the number of
   nodes, and can thus be used to infer the load only in a network where
   each node advertises roughly the same number of addresses and
   generates roughly the same amount of traffic.

   As the DODAG Size is always a multiple of a power of 2, when the
   actual size falls between two such values, the DODAG Root is to
   always round up.

   Future work such as [I-D.ietf-roll-capabilities] will enable
   collection of capabilities such as this one in reports to the DODAG
   Root.

   In any case, the DODAG Size may slightly change between one DIO and
   the next, so the value transmitted is considered as an approximation.

3.2.  Option Processing

   The contents of the option MUST be generated by the DODAG Root.  A
   6LR MUST NOT change the option when propagating it.

   Whenever the DODAG root changes the values of Min Priority or DODAG
   Size in the option, it MUST also increment the value of Version
   Number.  Moreover, if the change is considered important (i.e., it is
   expected to propagate in the DODAG quickly), the DODAG Root SHOULD
   also set the T bit to 1; otherwise, it MUST set the bit to 0.




Richardson, et al.        Expires 28 July 2026                  [Page 5]

Internet-Draft                 join-metric                  January 2026


   Upon receiving the option, a 6LR first checks the value of the
   Version Number field in the option, _vr_, versus the value of the
   Version Number it has last adopted locally, _vl_.

   *  If _vl_ is greater than _vr_ (in the lollipop counter order), then
      the 6LR MUST ignore the received option.

   *  Otherwise, the 6LR MUST adopt the contents of the option (i.e.,
      the values of Version Number, Min Priority, DODAG Size, and the T
      bit) as its local ones.  Moreover, if _vl_ was smaller than _vr_
      (in the lollipop counter order) and the T bit in the received
      option was set, then the 6LR MUST reset its DIO trickle timer.

   A 6LR, which would otherwise be willing to act as a _Join Proxy_,
   will examine the locally adopted value of Min Priority and to that
   number add any additional local consideration (such as upstream
   congestion, number of NCE slots available, etc.).

   The maximum resulting value any 6LR can obtain this way is 0x7f.

   The resulting priority, if less than 0x7f, should enable the _Join
   Proxy_ function.

3.3.  Upwards Compatibility

   A 6LR that did not support this option would not act on it or
   propagate it in its DIO messages.  In effect, the 6LR's subtree nodes
   could not receive any telemetry.  Therefore, 6LRs that support this
   option but do not receive it via any path SHOULD assume a default
   value of 0x40 as their base value for the Enhanced Beacon Join
   Priority.

   A 6LR downstream of a 6LR where there was such an interruption in the
   telemetry could err in two directions:

   *  If the value implied by the base value of 0x40 was too low, then
      the 6LR might continue to attract enrollment traffic when none
      should have been collected.  This is a stressor for the network,
      but this would also be what would occur without this option at
      all.











Richardson, et al.        Expires 28 July 2026                  [Page 6]

Internet-Draft                 join-metric                  January 2026


   *  If the value implied by the base value of 0x40 was too high, then
      the 6LR might deflect enrollment traffic to other parts of the
      DODAG, possibly refusing any enrollment traffic at all.  In order
      for this to happen, some significant congestion must be seen in
      the sub-DODAG where the implied 0x40 was introduced.  The 0x40 is
      only the half-way point, so if such an amount of congestion was
      present, then this sub-DODAG of the DODAG simply winds up being
      more cautious than it needed to be.

   It is possible that the temporal alternation of the above two
   situations might introduce cycles of accepting and then rejecting
   enrollment traffic.  This is something an operator should consider if
   they incrementally deploy this option to an existing Low-power/Lossy-
   Network (LLN).  In addition, an operator would be unable to turn off
   enrollment traffic by sending a maximum value enrollment priority to
   the sub-DODAG.  This situation is unfortunate, but without this
   option, the situation would occur all over the DODAG, rather than
   just in the sub-DODAG that the option did not reach.

4.  Security Considerations

   As per [RFC7416], RPL control frames either run over a secured layer
   2 or use the [RFC6550] Secure DIO methods at layer 3.  This option
   can be placed into either a "clear" (layer-2 secured) DIO or a
   layer-3 Secure DIO.

   In most deployments involving wireless technology, layer 2 is always
   encrypted using a layer-2 specific technology, and so privacy of this
   option is available.

   However, a malicious node that was part of the RPL control plane
   (i.e., had been enrolled into the layer-2 security) would be able to
   see the values of this option and, based upon the observed minimal
   enrollment priority, could signal a confederate that it was a good
   time to send malicious join traffic.

   What is more, such a malicious node, being already part of the RPL
   control plane, could also send DIOs with a different minimal
   enrollment priority, which would cause downstream mesh routers to
   change their _Join Proxy_ behavior: lower minimal priorities would
   cause downstream nodes to accept more Pledges than the network was
   expecting; higher minimal priorities could cause the enrollment
   process to stall.

   The use of layer-2 or layer-3 security for RPL control messages
   prevents the two aforementioned attacks by non-participating nodes by
   preventing malicious nodes from becoming part of the control plane.




Richardson, et al.        Expires 28 July 2026                  [Page 7]

Internet-Draft                 join-metric                  January 2026


   Nevertheless, a node that is attacked and has malware placed on it
   creates vulnerabilities in the same way such an attack on any node
   involved in Internet routing protocol does.  The rekeying provisions
   of [RFC9031] exist to permit an operator to remove such nodes from
   the network.

5.  Privacy Considerations

   There are no new privacy issues caused by this extension.

6.  IANA Considerations

   Allocate a new number TBD01 from Registry RPL Control Message
   Options.  This entry should be called Minimum Enrollment Priority.

7.  Acknowledgements

   This has been reviewed by Charlie Perkins, Rifaat Shehk-Yusek, Dave
   Thaler, and Thomas Watteyne.

   Huimin She contributed text about expressing the DODAG size.

8.  References

8.1.  Normative References

   [ieee802154]
              IEEE standard for Information Technology, "IEEE Std.
              802.15.4, Part. 15.4: Wireless Medium Access Control (MAC)
              and Physical Layer (PHY) Specifications for Low-Rate
              Wireless Personal Area Networks", n.d.,
              <http://standards.ieee.org/findstds/
              standard/802.15.4-2015.html>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/rfc/rfc2119>.

   [RFC6550]  Winter, T., Ed., Thubert, P., Ed., Brandt, A., Hui, J.,
              Kelsey, R., Levis, P., Pister, K., Struik, R., Vasseur,
              JP., and R. Alexander, "RPL: IPv6 Routing Protocol for
              Low-Power and Lossy Networks", RFC 6550,
              DOI 10.17487/RFC6550, March 2012,
              <https://www.rfc-editor.org/rfc/rfc6550>.






Richardson, et al.        Expires 28 July 2026                  [Page 8]

Internet-Draft                 join-metric                  January 2026


   [RFC7416]  Tsao, T., Alexander, R., Dohler, M., Daza, V., Lozano, A.,
              and M. Richardson, Ed., "A Security Threat Analysis for
              the Routing Protocol for Low-Power and Lossy Networks
              (RPLs)", RFC 7416, DOI 10.17487/RFC7416, January 2015,
              <https://www.rfc-editor.org/rfc/rfc7416>.

   [RFC7554]  Watteyne, T., Ed., Palattella, M., and L. Grieco, "Using
              IEEE 802.15.4e Time-Slotted Channel Hopping (TSCH) in the
              Internet of Things (IoT): Problem Statement", RFC 7554,
              DOI 10.17487/RFC7554, May 2015,
              <https://www.rfc-editor.org/rfc/rfc7554>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

   [RFC9031]  Vučinić, M., Ed., Simon, J., Pister, K., and M.
              Richardson, "Constrained Join Protocol (CoJP) for 6TiSCH",
              RFC 9031, DOI 10.17487/RFC9031, May 2021,
              <https://www.rfc-editor.org/rfc/rfc9031>.

   [RFC9032]  Dujovne, D., Ed. and M. Richardson, "Encapsulation of
              6TiSCH Join and Enrollment Information Elements",
              RFC 9032, DOI 10.17487/RFC9032, May 2021,
              <https://www.rfc-editor.org/rfc/rfc9032>.

8.2.  Informative References

   [I-D.ietf-roll-capabilities]
              Jadhav, R., Thubert, P., Richardson, M., and R. N. Sahoo,
              "RPL Capabilities", Work in Progress, Internet-Draft,
              draft-ietf-roll-capabilities-09, 9 November 2021,
              <https://datatracker.ietf.org/doc/html/draft-ietf-roll-
              capabilities-09>.

   [RFC6606]  Kim, E., Kaspar, D., Gomez, C., and C. Bormann, "Problem
              Statement and Requirements for IPv6 over Low-Power
              Wireless Personal Area Network (6LoWPAN) Routing",
              RFC 6606, DOI 10.17487/RFC6606, May 2012,
              <https://www.rfc-editor.org/rfc/rfc6606>.

Authors' Addresses

   Michael Richardson
   Sandelman Software Works
   Email: mcr+ietf@sandelman.ca





Richardson, et al.        Expires 28 July 2026                  [Page 9]

Internet-Draft                 join-metric                  January 2026


   Rahul Arvind Jadhav
   Huawei Tech
   Email: rahul.ietf@gmail.com


   Pascal Thubert
   Cisco Systems
   Email: pthubert@cisco.com


   Konrad Iwanicki
   University of Warsaw
   Email: iwanicki@mimuw.edu.pl






































Richardson, et al.        Expires 28 July 2026                 [Page 10]


name: draft-contario-totp-secure-enrollment
rev: 02
title: TOTP Secure Enrollment
time: 2025-09-27T00:30:24Z
group_uri: /api/v1/group/group/1027/
expires: 2026-03-31T00:30:24Z
states: ["/api/v1/doc/state/1/", "/api/v1/doc/state/150/"]
categories: ["Agent identity/auth", "AI safety/alignment"]
tags: []
fetched_at: 2026-02-27T22:56:07.990725+00:00
abstract:
This document describes a secure key exchange scheme that extends the
   Time-Based One-Time Password (TOTP) [RFC6238] de facto enrollment
   method to prevent compromise of the non-expiring TOTP key by
   photographic capture of the QR code or by intentional or
   unintentional persistence of the key string in email, SMS, or other
   systems outside of the TOTP authenticator system itself.

full_text:
Network Working Group                                        B. Contario
Internet-Draft                                             Silent Sector
Intended status: Informational                         27 September 2025
Expires: 31 March 2026


                         TOTP Secure Enrollment
                draft-contario-totp-secure-enrollment-02

Abstract

   This document describes a secure key exchange scheme that extends the
   Time-Based One-Time Password (TOTP) [RFC6238] de facto enrollment
   method to prevent compromise of the non-expiring TOTP key by
   photographic capture of the QR code or by intentional or
   unintentional persistence of the key string in email, SMS, or other
   systems outside of the TOTP authenticator system itself.

About This Document

   This note is to be removed before publishing as an RFC.

   The latest revision of this draft can be found at https://Silent-
   Sector.github.io/totp-secure-enrollment/draft-contario-totp-secure-
   enrollment.html.  Status information for this document may be found
   at https://datatracker.ietf.org/doc/draft-contario-totp-secure-
   enrollment/.

   Source for this draft and an issue tracker can be found at
   https://github.com/Silent-Sector/totp-secure-enrollment.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 31 March 2026.




Contario                  Expires 31 March 2026                 [Page 1]

Internet-Draft           totp-secure-enrollment           September 2025


Copyright Notice

   Copyright (c) 2025 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Terminologies . . . . . . . . . . . . . . . . . . . . . .   3
     1.2.  Requirements Language . . . . . . . . . . . . . . . . . .   5
   2.  TOTP Background . . . . . . . . . . . . . . . . . . . . . . .   5
   3.  TOTP Enrollment . . . . . . . . . . . . . . . . . . . . . . .   5
     3.1.  Overview  . . . . . . . . . . . . . . . . . . . . . . . .   5
     3.2.  Hexadecimal String Enrollment . . . . . . . . . . . . . .   6
     3.3.  QR Code Enrollment  . . . . . . . . . . . . . . . . . . .   6
   4.  Enrollment Vulnerabilities  . . . . . . . . . . . . . . . . .   7
     4.1.  Hexadecimal String Enrollment Vulnerabilities . . . . . .   7
     4.2.  QR Code Enrollment Vulnerabilities  . . . . . . . . . . .   8
     4.3.  Elevated Risk for QR Code Enrollment  . . . . . . . . . .   8
   5.  TOTP Secure Enrollment  . . . . . . . . . . . . . . . . . . .   9
     5.1.  Mechanism of Operation  . . . . . . . . . . . . . . . . .   9
     5.2.  Device Enrollment Data  . . . . . . . . . . . . . . . . .  10
     5.3.  Secure Enrollment QR code Workflow  . . . . . . . . . . .  12
     5.4.  Secure Enrollment Copy Paste Workflow . . . . . . . . . .  13
     5.5.  Workflow Notes  . . . . . . . . . . . . . . . . . . . . .  14
     5.6.  Requirements  . . . . . . . . . . . . . . . . . . . . . .  14
     5.7.  Reference Code  . . . . . . . . . . . . . . . . . . . . .  16
     5.8.  Exclusions  . . . . . . . . . . . . . . . . . . . . . . .  16
     5.9.  Alternative Network Transports  . . . . . . . . . . . . .  16
     5.10. Applicability to Other Algorithms . . . . . . . . . . . .  17
   6.  Related Enhancements  . . . . . . . . . . . . . . . . . . . .  17
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .  17
     7.1.  General . . . . . . . . . . . . . . . . . . . . . . . . .  17
     7.2.  TOTP secret key exchange  . . . . . . . . . . . . . . . .  17
     7.3.  Secret key harvesting . . . . . . . . . . . . . . . . . .  18
     7.4.  Secure Enrollment Flag  . . . . . . . . . . . . . . . . .  18
   8.  Privacy Considerations  . . . . . . . . . . . . . . . . . . .  18
   9.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  19
   10. References  . . . . . . . . . . . . . . . . . . . . . . . . .  19



Contario                  Expires 31 March 2026                 [Page 2]

Internet-Draft           totp-secure-enrollment           September 2025


     10.1.  Normative References . . . . . . . . . . . . . . . . . .  19
     10.2.  Informative References . . . . . . . . . . . . . . . . .  19
   Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . .  20
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  21

1.  Introduction

   The common practice of presenting a QR Code for Time-Based One-Time
   Password (TOTP) enrollment exposes the actual shared TOTP secret key
   used by the TOTP algorithm for multi-factor authentication.  A
   fallback method of showing the key string to be typed or copied and
   pasted into a multi-factor authenticator application also exposes the
   key.  In both cases the key that is exposed is the permanent shared
   TOTP secret key that, by current standards and processes, is expected
   to never expire.

   The below process instead replaces the secret key in the QR code with
   a URL to the TOTP service for the authenticator application to
   request the secret key over a secure network transport without
   authentication.  The URL provided uses an atomic one-time use data
   token (nonce) so only one request for the secret key is honored.  If
   both the user and an attacker use the URL, only one will get the
   secret key, and only the user will have an active session to complete
   enrollment by entering the proper TOTP value derived from the secret
   key.  If the attacker gets the secret key, the user will not and the
   user will be forced to generate a new QR code with a URL to a new
   secret key until they successfully get the secret key and complete
   TOTP Verification by entering the current one-time password, meaning
   the attacker will be guaranteed to not have the same secret key.  In
   cases where the URL to the TOTP service is not accessible, the legacy
   shared TOTP secret key exchange via QR Code or text string may be
   offered as a fallback with appropriate warnings displayed against key
   exposure.

1.1.  Terminologies

   TOTP:  Time-Based One-Time Password as described in [RFC6238], and by
      itself in this document referring only to the algorithm, not the
      implementation.

   TOTP secret key:  the hexadecimal [RFC6238] string representing a
      value unique to each prover, per [RFC6238] section 3.

   hardware token:  Usually a dedicated-purpose hardware device that is
      issued to users.  In this document it refers to a device that is
      pre-set with a TOTP secret key used to display one-time passwords
      that provide prover functionality.




Contario                  Expires 31 March 2026                 [Page 3]

Internet-Draft           totp-secure-enrollment           September 2025


   soft token:  Usually a software application that provides enrollment
      and prover functionality without dedicated hardware, usually
      running on a smartphone, tablet, personal computer, or password
      manager application.  In this document it refers specifically to
      an application that stores a TOTP secret key and provides TOTP
      prover functionality.  Sometimes referred to in other documents as
      a “soft token” or “software token”, as opposed to a “hardware
      token”.

   prover:  In this document referring to an implementation of the TOTP
      protocol that provides a periodically changing one-time password
      based on the time and a TOTP secret key.  A prover may be
      implemented as an authenticator application (software token) or
      implemented as a hardware token.

   nonce:  a randomly-generated cryptographic data token that is used to
      prevent replay attacks by only allowing the value to be used once,
      somewhat analogous to a paper ticket that is torn in half or
      scanned and invalidated when presented for admission to a venue to
      prevent reuse.

   TOTP service:  A service used to enroll and verify the TOTP one-time
      password provided by a prover and associated with a specific
      authentication request.  Also referred to as a “verifier” in
      [RFC6238], as described below.

   verifier:  Per [RFC6238] section 3.R1, an authentication or
      validation server, and in this case for TOTP.

   enrollment:  also referred to as _provisioning_ per [RFC6238] section
      4.2, and _registration_ in [draft-strbac-totp2], the exchange of
      the TOTP secret key between the verifier and the prover, followed
      by the verification of the prover's one-time password by the
      verifier, and the respective secure storage and linking of TOTP
      secret key to an identity for MFA.

   URL:  used explicitly in this document to specify a "Uniform Resource
      Locator" providing a location and means of access to obtain
      information via a transfer request.  In this document expect the
      URL to require a transfer using the specified protocol to get the
      desired information.

   otpauth URI:  used explicitly in this document to specify a "Uniform
      Resource Identifier" providing information for immediate use for
      TOTP enrollment, per [totp-uri-format] and
      [draft-linuxgemini-otpauth-uri].  In this document expect the
      otpauth URI to convey the desired information in a single step
      without an additional transfer step required.



Contario                  Expires 31 March 2026                 [Page 4]

Internet-Draft           totp-secure-enrollment           September 2025


1.2.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

2.  TOTP Background

   The basis of Time-Based One-Time Password (TOTP) [RFC6238] is a
   symmetric algorithm for both the prover and the verifier to combine a
   string containing a shared TOTP secret key and the current Unix epoch
   seconds [RFC3339] which are hashed using SHA-1 or a similar
   cryptographic hash algorithm.  The result is converted to a decimal
   number and truncated, typically to 6 digits, to be used as the one-
   time password value that the prover supplies and the verifier accepts
   as valid only if it calculates the same one-time password value.

   TOTP requires that the TOTP secret key, as defined in [RFC6238]
   section 3 requirement R2, must be known by both the prover and the
   verifier.  The original implementation of TOTP may have been intended
   for use primarily in hardware tokens that were factory-issued with an
   unchangeable TOTP secret key, which was a common practice for
   hardware tokens, since [RFC6238] did not define a protocol for the
   TOTP secret key exchange.  However, increased use of mobile devices
   that synchronize their time with wireless communication providers and
   NTP servers have allowed for the proliferation of authenticator
   applications running on mobile devices, laptops, and desktop
   computers, all of which require receiving a unique TOTP secret key
   from each new verifier.  Therefore, a protocol evolved as a de facto
   industry standard for enrollment to exchange the TOTP secret key,
   described in the next section.

3.  TOTP Enrollment

3.1.  Overview

   For consistency and specificity, the remainder of this document will
   use the term “TOTP service” in place of “verifier”.

   For TOTP enrollment using an authenticator application, it is most
   logical for the TOTP service to generate the shared TOTP secret key
   using cryptographically secure random number generators to ensure
   that each prover has a unique TOTP secret key that has not been used
   elsewhere.  Therefore it is an accepted standard for the TOTP service
   to generate the key and store it securely, and for the authenticator
   application to simply accept and store the key in a secure manner.



Contario                  Expires 31 March 2026                 [Page 5]

Internet-Draft           totp-secure-enrollment           September 2025


   The current TOTP enrollment protocol was likely influenced or
   established by the [GoogleAuth] project, which provided a working
   reference system of two complementary methods for the TOTP service to
   share the TOTP secret key with the prover during enrollment,
   described as follows.

3.2.  Hexadecimal String Enrollment

   The TOTP secret key itself is expressed as a hexadecimal string of an
   arbitrary length chosen by the TOTP service.  During enrollment the
   TOTP service can display the randomly generated hexadecimal string to
   the user for them to type or paste into an authenticator application.
   Typical strengths are >= 160 bits.

3.3.  QR Code Enrollment

   To avoid typographic errors when the user enters the TOTP secret key
   into the prover that was provided by the TOTP service, the use of a
   [QR_Code] was introduced so that manual data entry was not required
   by the user.  The QR Code encodes an otpauth URI scheme, per the
   [totp-uri-format] and proposed as a draft IETF standard
   [draft-linuxgemini-otpauth-uri] with the following parameters:

   algorithm=(SHA1|SHA256|SHA512)
     OPTIONAL, defaults to SHA1.

   secret=[websafe Base32 encoded TOTP secret key, no padding]
     REQUIRED, 128 bits or more.

   digits=(6|8)
     OPTIONAL, defaults to 6.

   period=N  (totp specific)
     OPTIONAL, defaults to 30.

   The otpauth URI also includes a human-readable Label to identify the
   issuer and account name.  Some TOTP services also supply a "user"
   parameter.  Both the account name and user parameter could disclose
   the complete identity that the TOTP secret key applies to.  Below is
   a fictitious example otpauth URI that could be encoded into a QR Code
   for the user "human@domain.com" using the ExampleCorp service:

otpauth://totp/ExampleCorp%3Ahuman%40domain.com?secret=bl4gbf4ab5l3rj3d6htrxc6bqhrx3m7u&issuer=ExampleCorp

   and a urldecoded version of the text:

otpauth://totp/ExampleCorp:human@domain.com?secret=bl4gbf4ab5l3rj3d6htrxc6bqhrx3m7u&issuer=ExampleCorp




Contario                  Expires 31 March 2026                 [Page 6]

Internet-Draft           totp-secure-enrollment           September 2025


   Of particular note is that the TOTP secret key value of the "secret"
   parameter is provided in BASE32 plaintext with no cryptographic
   protections of any kind, and no expiration or method of revocation
   for the TOTP secret key if the QR Code or otpauth URI is stored,
   copied, or transmitted beyond the intended use, or if the TOTP secret
   key is discovered compromised in any other way.

4.  Enrollment Vulnerabilities

   The above two enrollment methods do not follow the recommendations
   listed in [RFC6238] section 5.1 stating:

      All the communications SHOULD take place over a secure channel,
      e.g., Secure Socket Layer/Transport Layer Security (SSL/TLS)
      [RFC5246] or IPsec connections [RFC4301].

   This results in a wide range of vulnerabilities.

4.1.  Hexadecimal String Enrollment Vulnerabilities

   Hexadecimal String Enrollment where the TOTP secret key string is
   visibly presented to the user is often vulnerable to compromise by
   the following non-exhaustive list of vectors:

   1.  a nearby observer viewing and remembering the visible TOTP secret
       key

   2.  a nearby observer photographing the visible TOTP secret key

   3.  the user copying the key to the computer's clipboard and then
       pasting the TOTP secret key into the prover during enrollment
       (which exposes the key to potentially malicious applications that
       monitor the clipboard contents)

   4.  the user copying the TOTP secret key to the computer's clipboard
       and then pasting the TOTP secret key into an email, instant
       message, or SMS message to send to another user for enrollment
       (which may persist the TOTP secret key in external or internal
       systems indefinitely)

   5.  the user copying the TOTP secret key to a computer's clipboard
       and then pasting into a file for later use (which which may
       persist the TOTP secret key to local or cloud storage)








Contario                  Expires 31 March 2026                 [Page 7]

Internet-Draft           totp-secure-enrollment           September 2025


4.2.  QR Code Enrollment Vulnerabilities

   Enrollment via [QR_Code] where the encoded TOTP secret key is visible
   presented to the user is often vulnerable to compromise by the
   following non-exhaustive list of vectors:

   1.  a nearby observer photographing the QR Code

   2.  the user copying the QR Code image into an email or SMS message
       to send to another user for enrollment (which may persist the
       TOTP secret key in external or internal systems indefinitely)

   3.  the user photographing the QR Code image on their mobile device
       for later use by the authenticator application (which may persist
       the TOTP secret key in external or internal systems indefinitely)

   4.  the user photographing the QR Code image on their mobile device
       to send to another user via email, instant message, cloud
       service, or SMS for enrollment (which may persist the TOTP secret
       key in external or internal systems indefinitely)

   5.  the user printing the QR Code image to complete enrollment later

4.3.  Elevated Risk for QR Code Enrollment

   As described earlier, the string encoded in the QR Code is a
   [RFC3986] URI that contains all of the needed information to link the
   key to the TOTP service and the user with which it is associated,
   referred to in this document as an otpauth URI.  The information
   provided in the QR Code greatly simplifies the TOTP enrollment since
   the user does not need to name the entry in their authenticator
   application or type in a hexadecimal string, so most users opt for
   the QR Code enrollment method.

   However, the tradeoff for this convenience is that the QR Code
   potentially contains all of the information needed by an attacker to
   log in to the associated target system as the associated user, except
   for the user's first factor password.  If the target system is
   internet-facing and the login page can be identified by name or other
   intelligence, and the user's identifier is an email address that can
   be found in a database of accounts with compromised passwords, then
   having the TOTP secret key, also in the QR Code, potentially gives
   the attacker all of the factors needed to log in as the user to the
   target system.







Contario                  Expires 31 March 2026                 [Page 8]

Internet-Draft           totp-secure-enrollment           September 2025


   Because of the vectors by which the QR Code could be unintentionally
   exposed or stored on systems outside of the user's control for later
   harvesting, a secure enrollment method that does not expose the TOTP
   secret key is needed.

5.  TOTP Secure Enrollment

   The following method protects the TOTP secret key via HTTPS/TLS as
   recommended in [RFC6238] section 5.1 when the TOTP service is
   accessible on a public or private network via a URL that can be
   generated by the TOTP service.

   Since this document is dealing only with TOTP implementations, the
   remainder of this document will prefer the shortened form of “Secure
   Enrollment” in place of “TOTP Secure Enrollment”, and “authenticator
   application” in place of “TOTP authenticator application”

5.1.  Mechanism of Operation

   The "secret" parameter of the otpauth URI, as described in
   [totp-uri-format], normally only accepts a BASE32 encoded hexadecimal
   secret key.  However, in a “Secure Enrollment otpauth URI”, the
   “secret” parameter MUST accept a urlencoded URL with a HTTPS protocol
   string.  This URL, referred to from here forward as the “Secure
   Enrollment URL”, is how the authenticator application connects to the
   TOTP service to get a copy of the legacy otpauth URI that contains
   the secret key.

   The [RFC3986] URL specification requires the protocol in a URL be
   separated by a colon (“:”) which is urlencoded into “%3A”.  Because
   both the “%” and “:” characters are not valid in BASE32 encoding,
   their presence in the “secret” parameter can effectively be used as a
   signal that the value of the “secret” parameter should be urldecoded
   and the resulting value used as the Secure Enrollment URL to request
   the original format TOTP otpauth URI containing the plaintext TOTP
   secret key via a network connection.

   To minimize the data that is stored in the Secure Enrollment QR code,
   all of the optional fields SHOULD NOT be included in the Secure
   Enrollment otpauth URI.  Only the “secret” parameter of the otpauth
   URI should be populated (as shown below).

   *The actual enrollment data, in the form of an otpauth URI, used by
   the authenticator application does not change.  The enrollment data
   simply gets delivered via HTTPS instead of embedded in a QR code.*






Contario                  Expires 31 March 2026                 [Page 9]

Internet-Draft           totp-secure-enrollment           September 2025


   The Secure Enrollment URL MUST include a nonce value that will be
   marked as invalid after first use by the TOTP service to insure that
   the request is only ever responded to one time only (not more than
   once).  The Secure Enrollment URL gets added to the “Secure
   Enrollment otpauth URI”, then encoded into a QR Code, then displayed
   to the user, and then decoded by the user's authenticator
   application.

   A sample Secure Enrollment otpauth URI for ExampleCorp:

otpauth://totp/?secret=https%3A%2F%2Fexamplecorp.com%2Fapi%2Fenrollmfa%2F16062671560671769238465892

   The Secure Enrollment URL portion of the URI follows the “?secret=”
   portion and decodes to:

   https://examplecorp.com/api/enrollmfa/16062671560671769238465892

   The authenticator application MUST make a HTTP POST request using the
   Secure Enrollment URL to retrieve the otpauth URI text string in the
   same legacy format used by authenticator applications prior to
   adoption of TOTP Secure Enrollment.

   It should also be noted that the use of a QR code is not required for
   Secure Enrollment.  The user’s authenticator application could accept
   the Secure Enrollment URL via QR code, copy and paste, or manual
   entry of the Secure Enrollment URL text itself.

5.2.  Device Enrollment Data

   TOTP enrollment has historically been a “blind” process because there
   was no direct interaction between the authenticator application and
   the TOTP service.  Administrators had no information to help end-
   users that forgot what device or authenticator application they to
   use to get their 6 digit TOTP codes.

   Now the authenticator application makes a HTTP POST request to the
   TOTP service during Secure Enrollment, so there is an opportunity to
   send and capture information for administrators to provide hints
   about which device and authenticator application an end user should
   use to log in.

   When the authenticator application makes the POST request it SHOULD
   send a request body with device enrollment data in JSON format using
   Content-Type set to “application/json”.  The sending of this
   enrollment data is optional.  The recommended content of the JSON
   data is as follows:





Contario                  Expires 31 March 2026                [Page 10]

Internet-Draft           totp-secure-enrollment           September 2025


   {
       "event_type":"totp-secure-enrollment",
       "time_local":"Sun, 22 Sep 2024 23:46:30 -0400",
       "time_utc":"2024-09-23T03:46:30.637Z",
       "device_model":"F990",
       "device_manufacturer":"Foo",
       "os_name":"android",
       "os_version":"12",
       "application_name":"Aegis",
       "application_version":"3.1.1",
       "location_description":"Cincinnati, Ohio, USA",
       "location_longitude":"-84.512",
       "location_latitude":"39.103"
   }

   All element values should be evaluated as strings and contain the
   following information:

   event_type:  always the literal string "totp-secure-enrollment" for
      easy identification by humans and machine

   time_local:  time of enrollment, recommended formatted as RFC 1123
      Date Time for local time of the device for human readability

   time_utc:  time of enrollment, ISO-8601 instant format recommended
      for machine readability

   device_model:  the device model name or the model number

   device_manufacturer:  the device manufacturer name

   os_name:  the operating system name

   os_version:  the operating system version in human readable form

   application_name:  the authenticator application name or title

   application_version:  the authenticator application version in human
      readable form

   location_description:  if available and permitted, a descriptive name
      for the location of the device at the time of enrollment,
      preferably showing country, region / state / province, and
      municipality

   location_longitude:  if available and permitted, the coarse longitude
      of the device at the time of enrollment




Contario                  Expires 31 March 2026                [Page 11]

Internet-Draft           totp-secure-enrollment           September 2025


   location_latitude:  if available and permitted, the coarse latitude
      of the device at the time of enrollment

   It is also optional, but recommended, for the TOTP service to store
   and/or parse the device enrollment data for future use.  The data can
   be used for both end user assistance and administrative review of the
   different devices being used for TOTP Secure Enrollment.

5.3.  Secure Enrollment QR code Workflow

   The Secure Enrollment ideal workflow will follow these general steps
   when using a QR code is viable:

   1.  *Enrollment Request*: The user initiates the enrollment process
       for TOTP for 2FA or MFA on their account.  The user experience is
       then passed to the TOTP service.

   2.  *Secure Enrollment Preparation*: The TOTP service generates a
       nonce and a random TOTP secret key, storing them both temporarily
       in a secure location with a configurable validity period
       (expiration) for human interaction, such as 5 minutes.

   3.  *Secure Enrollment Presentation*: The TOTP service generates and
       presents to the user a QR Code that contains an otpauth URI in
       the format established by [GoogleAuth], but instead of the
       "secret" parameter containing the TOTP secret key, it contains a
       urlencoded network-accessible HTTPS URL with only a nonce and no
       secret key.

   4.  *User Interaction*: The user interacts with the TOTP
       authenticator application to scan the QR Code presented.

   5.  *Secure Key Request*: When the authenticator application detects
       a “%” or “:” character in the "secret" parameter value of the
       otpauth URI, the authenticator application performs the new
       Secure Enrollment functionality by urldecoding the “secret”
       parameter and using that value in a HTTP POST request.














Contario                  Expires 31 March 2026                [Page 12]

Internet-Draft           totp-secure-enrollment           September 2025


   6.  *Secure Key Response*: When the TOTP service receives the HTTP
       POST request with the nonce, the redeem is atomic.  The TOTP
       service looks up the nonce from temporary secure storage and if
       the nonce is not expired it atomically expires the nonce and
       returns the response (over the HTTPS/TLS secure channel) with the
       otpauth URI containing the secret key in the legacy de facto
       format, the same data that would normally be provided in the QR
       Code by the [GoogleAuth] code prior to implementation of Secure
       Enrollment.  At the same time, the TOTP service optionally stores
       the JSON device enrollment data contained in the POST request
       body for future use.

   7.  *TOTP Enrollment*: The authenticator application extracts the
       "issuer", "secret", "digits", and other parameters from the
       otpauth URI and registers a new entry in the authenticator
       application.  This step is unchanged from the previous process,
       except the otpauth URI with the secret key is delivered via HTTPS
       rather than the QR Code.

   8.  *TOTP Verification*: The TOTP service presents a user interface
       that requires the user to enter the current one-time password and
       verify it before considering the user fully enrolled.

   9.  *Secure Enrollment Flag*: It is RECOMMENDED that in addition to
       the new TOTP secret key, the enrollment data stored by the TOTP
       service should include a Secure Enrollment Flag to indicate that
       this enrollment was completed over a secure channel.  This can be
       used for auditing purposes and to force users to re-enroll their
       TOTP authenticator application entries that were not completed in
       a secure manner in the past.

5.4.  Secure Enrollment Copy Paste Workflow

   The Secure Enrollment ideal workflow will follow the same general
   steps as the above QR code workflow when it is easier to copy and
   paste a text string than to use a QR code, with the exception of the
   following two steps which are slightly modified and used instead of
   the same numbered steps: 3. *Secure Enrollment Presentation*: The
   TOTP service displays in selectable text the otpauth URI in the
   format established by [GoogleAuth], but instead of the "secret"
   parameter containing the TOTP secret key, it contains a urlencoded
   network-accessible HTTPS URL with only a nonce and no actual secret
   key.  The user optionally copies the otpauth URI text into the
   clipboard or transfers it electronically. 4. *User Interaction*: The
   user interacts with the authenticator application by pasting or
   typing in the otpauth URI text and submitting it.





Contario                  Expires 31 March 2026                [Page 13]

Internet-Draft           totp-secure-enrollment           September 2025


5.5.  Workflow Notes

   In both the Copy Paste Workflow and the QR code Workflow, the final
   secret key that is enrolled is never exposed to the user, attackers,
   or other systems.  The secret key is directly exchanged via HTTPS
   between the TOTP service and the authenticator application.

5.6.  Requirements

   Secure Enrollment requires the following that is not already defined
   in [RFC6238]:

   1.   The TOTP service MUST be able to create a Secure Enrollment URL
        to an API endpoint that can accept an anonymous HTTPS POST
        request.

   2.   The TOTP service API endpoint SHOULD be configured to require
        non-deprecated TLS ciphers via HTTPS.

   3.   The Secure Enrollment URL MUST be resolvable and accessible to
        the authenticator application via internet or private network
        connection at the time of enrollment.

   4.   The TOTP service MUST generate a cryptographically secure random
        string data token to be used as a nonce to both include in the
        URL and store in secure temporary storage to be accessed by the
        API endpoint.  It is recommended that the nonce have at least 96
        bits of entropy.  Newly-generated GUID v4 or UUID v4 values with
        122 bits of entropy are acceptable nonce values at the time of
        this writing.

   5.   The TOTP service MUST set a nonce validity period (expiration)
        for completion of the human actions.  The nonce validity period
        SHOULD be reasonable for authorized users, such as 5 minutes
        into the future.

   6.   The TOTP service MUST NOT permanently store the generated secret
        key as enrolled until after the user has completed TOTP
        Verification by entering the current one-time password as a part
        of an existing user session authenticated with a previous
        factor.

   7.   The TOTP service MUST generate an otpauth URI whose secret
        parameter value is the Secure Enrollment URL, percent-encoded
        per [RFC3986] section 2.  The Secure Enrollment URL MUST NOT
        include credentials and MUST only include the nonce and
        necessary routing parameters.




Contario                  Expires 31 March 2026                [Page 14]

Internet-Draft           totp-secure-enrollment           September 2025


   8.   The TOTP service MUST exclude optional parts (per
        [totp-uri-format]) from the Secure Enrollment otpauth URI that
        is presented to the end user as a QR code or text string to
        limit data exposure.

   9.   The TOTP service SHOULD NOT issue http redirects.  Redirects
        MUST NOT downgrade confidentiality (e.g., to http).

   10.  The authenticator application MUST NOT follow http redirects.

   11.  The authenticator application MUST allow the otpauth QR code to
        exclude all optional data from the otpauth URI, including the
        Label and user/account parameter, only providing the “secret”
        parameter that is populated with the Secure Enrollment URL.

   12.  The TOTP service API endpoint MUST invalidate the nonce after
        its expiration period or atomically redeem it upon first use by
        the API endpoint, whichever comes first, using appropriate
        concurrency locking mechanisms to avoid race conditions.

   13.  The TOTP service MUST generate a new secret key any time a new
        nonce value is generated for a Secure Enrollment URL.  To avoid
        any race conditions or other security concerns, if the user
        requests a new QR code or refreshes the page containing the
        otpauth URI or QR code, a new secret key and nonce value MUST be
        generated and supplied.

   14.  The TOTP service API endpoint MUST NOT respond with an actual
        secret key unless the nonce is found in secure temporary
        storage, is unused, and the current time is within the validity
        period.  On success, the endpoint MUST return HTTP 200 with a
        body containing the otpauth URI in the same format established
        by [GoogleAuth] as described in [totp-uri-format] with the
        “secret” parameter containing the BASE32 encoded secret key.
        The response SHOULD include Cache-Control: no-store and SHOULD
        include Pragma: no-cache.  Implementations MUST NOT log the
        returned secret key or full otpauth URI.

   15.  The TOTP service API endpoint SHOULD respond with a uniform HTTP
        4xx series response error code and message whether the nonce is
        invalid or expired, preferably a 403 Forbidden response.  In
        environments with specialized security needs the TOTP service
        API endpoint MAY choose to respond with any other non-usable
        response, including a TCP disconnect, 200 success code with a
        blank response, a URL with a decoy TOTP secret key, or any other
        response that serves the purpose of thwarting an attacker with a
        false positive response.




Contario                  Expires 31 March 2026                [Page 15]

Internet-Draft           totp-secure-enrollment           September 2025


   16.  The TOTP service MAY offer a user-initiated option in the UI to
        switch to legacy enrollment.  Prior to redirecting the user to
        legacy enrollment the UI SHOULD display a localizable message
        that warns the user to protect the legacy QR Code and/or the
        TOTP secret key string from observation and photographic or
        video capture, and avoid saving, sending, or transmitting them
        in any form to prevent compromise of the TOTP secret key.

5.7.  Reference Code

   Pull requests illustrating reference implementations based on working
   open source code bases can be found here:

   TOTP Secure Enrollment authenticator application reference code
   [totp-secure-enrollment-reference-authenticator]

   TOTP Secure Enrollment server framework demo code
   [totp-secure-enrollment-reference-server]

5.8.  Exclusions

   TOTP Secure Enrollment does not claim or provide any additional
   benefit in the following non-exhaustive list of areas:

   1.  Protection of the secret key once it is stored in the
       authenticator application after successful Secure Enrollment.

   2.  If the device containing the authenticator application is
       obtained by an attacker or is compromised by malware.

   3.  Assisting in inventorying or tracking enrolled systems to
       invalidate the enrolled secret key.

   4.  Purposefully setting up the same TOTP enrollment on multiple
       authenticator applications for redundancy or shared accounts.
       This is a usage scenario for TOTP outside of industry best
       practices and incurs elevated risk.  Secure Enrollment is not
       intended to protect or facilitate this.

5.9.  Alternative Network Transports

   This specification defines Secure Enrollment only over HTTPS.  Use of
   other transports is out of scope.  Implementations MAY define private
   extensions, but interoperability is not expected.







Contario                  Expires 31 March 2026                [Page 16]

Internet-Draft           totp-secure-enrollment           September 2025


5.10.  Applicability to Other Algorithms

   Other multi-factor authentication algorithms / protocols could
   benefit by adapting this secure enrollment process to exchange shared
   secrets.  The following is a non-exhaustive list of possible
   candidates:

   *  HMAC-based One-Time Password (HOTP) [RFC4226]

   *  OATH Challenge-Response Algorithm (OCRA) [RFC6287]

6.  Related Enhancements

   There are other efforts to improve the TOTP experience and security,
   such as proposed in draft [draft-strbac-totp2].  It is believed that
   the Secure Enrollment method described here can work in conjunction
   with and enhance the enrollment / registration process as described
   in [draft-strbac-totp2] section "3.  Authentication Scheme Overview"
   to improve the security of the "TOTP2 Registration" step.

7.  Security Considerations

7.1.  General

   All security considerations for the execution of TOTP and storage of
   the TOTP secret key are covered in [RFC6238].

7.2.  TOTP secret key exchange

   The secure exchange of the TOTP secret key MUST use a secure network
   transport protocol, such as HTTPS.  The implementation of the
   protocol SHOULD follow industry best practices, such as:

   1.  TOTP service API endpoints SHOULD only allow HTTPS TLS ciphers
       that are not deprecated by the industry.

   2.  TOTP service API endpoints and authenticator applications SHOULD
       negotiate the supported ciphers in preferred order of perceived
       security by traditional and quantum standards at the time of the
       exchange, always preferring ciphers that are deemed the most
       secure against "harvest now, decrypt later" attacks.

   3.  Clients and servers SHOULD perform certificate validation per the
       HTTPS PKI; endpoints SHOULD enable HSTS; deployments SHOULD avoid
       user-modifiable TLS downgrade mechanisms.






Contario                  Expires 31 March 2026                [Page 17]

Internet-Draft           totp-secure-enrollment           September 2025


7.3.  Secret key harvesting

   Once it is common knowledge that the legacy QR codes used for TOTP
   enrollment have no encryption or safeguards, and they can be reused
   since they contain the non-expiring secret key, attackers may start
   scanning images in cloud storage or sent via MMS that contain QR
   codes to harvest the secret key, along with the other details in the
   QR code otpauth URI.  Identification of a target QR code is trivial
   once decoded into a text string since they all start with
   “otpauth://”.

7.4.  Secure Enrollment Flag

   It is RECOMMENDED in the workflow above that in addition to the new
   TOTP secret key, the enrollment data stored by the TOTP service
   should include a Secure Enrollment Flag to indicate that this
   enrollment was completed over a secure channel.  It is also
   RECOMMENDED that action be taken against user accounts that do not
   have the Secure Enrollment Flag set on their accounts.  This action
   could include, but is not limited to, one or more of the following
   options that could be happen after a given date for each option:

   1.  Manual human intervention and verification that the authorized
       user completes the new Secure Enrollment process.

   2.  Disable the user accounts until option 1 is performed.

   3.  When the TOTP service detects success for a user with the Secure
       Enrollment Flag set to false, fail the login, and send the user
       through the Secure Enrollment process.  Once they complete it,
       the Secure Enrollment Flag will be set to true and they will be
       able to log in normally using TOTP authentication.

8.  Privacy Considerations

   This protocol proposes processing enrollment data (the one-time
   Secure Enrollment URL/nonce, server-observed timing/network data, and
   optional client metadata such as device class/OS version and coarse
   location).  Consistent with [RFC6973], servers SHOULD minimize
   collection and use enrollment data only to complete enrollment and
   related security functions (e.g., abuse detection); clients SHOULD
   NOT transmit optional metadata unless the user has clearly opted in
   or a managed-device policy authorizes it; if location is sent, it
   SHOULD be coarse (no precise coordinates).







Contario                  Expires 31 March 2026                [Page 18]

Internet-Draft           totp-secure-enrollment           September 2025


9.  IANA Considerations

   This document has no IANA actions.

10.  References

10.1.  Normative References

   [GoogleAuth]
              "Google Authenticator OpenSource", n.d.,
              <https://github.com/google/google-authenticator>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/rfc/rfc2119>.

   [RFC3986]  Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform
              Resource Identifier (URI): Generic Syntax", STD 66,
              RFC 3986, DOI 10.17487/RFC3986, January 2005,
              <https://www.rfc-editor.org/rfc/rfc3986>.

   [RFC6238]  M'Raihi, D., Machani, S., Pei, M., and J. Rydell, "TOTP:
              Time-Based One-Time Password Algorithm", RFC 6238,
              DOI 10.17487/RFC6238, May 2011,
              <https://www.rfc-editor.org/rfc/rfc6238>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

   [totp-uri-format]
              "TOTP URI Format", n.d., <https://github.com/google/
              google-authenticator/wiki/Key-Uri-Format>.

10.2.  Informative References

   [draft-linuxgemini-otpauth-uri]
              Gemini, L., "The otpauth URI Scheme", I-D draft-
              linuxgemini-otpauth-uri, August 2023,
              <https://datatracker.ietf.org/doc/draft-linuxgemini-
              otpauth-uri/>.

   [draft-strbac-totp2]
              "TOTP2 Authentication Scheme", n.d.,
              <https://datatracker.ietf.org/submit/status/135578/>.





Contario                  Expires 31 March 2026                [Page 19]

Internet-Draft           totp-secure-enrollment           September 2025


   [QR_Code]  "ISO/IEC 18004, currently ISO/IEC 18004:2015 Information
              technology — Automatic identification and data capture
              techniques — QR Code bar code symbology specification",
              n.d., <https://www.iso.org/standard/62021.html>.

   [RFC3339]  Klyne, G. and C. Newman, "Date and Time on the Internet:
              Timestamps", RFC 3339, DOI 10.17487/RFC3339, July 2002,
              <https://www.rfc-editor.org/rfc/rfc3339>.

   [RFC4226]  M'Raihi, D., Bellare, M., Hoornaert, F., Naccache, D., and
              O. Ranen, "HOTP: An HMAC-Based One-Time Password
              Algorithm", RFC 4226, DOI 10.17487/RFC4226, December 2005,
              <https://www.rfc-editor.org/rfc/rfc4226>.

   [RFC4301]  Kent, S. and K. Seo, "Security Architecture for the
              Internet Protocol", RFC 4301, DOI 10.17487/RFC4301,
              December 2005, <https://www.rfc-editor.org/rfc/rfc4301>.

   [RFC5246]  Dierks, T. and E. Rescorla, "The Transport Layer Security
              (TLS) Protocol Version 1.2", RFC 5246,
              DOI 10.17487/RFC5246, August 2008,
              <https://www.rfc-editor.org/rfc/rfc5246>.

   [RFC6287]  M'Raihi, D., Rydell, J., Bajaj, S., Machani, S., and D.
              Naccache, "OCRA: OATH Challenge-Response Algorithm",
              RFC 6287, DOI 10.17487/RFC6287, June 2011,
              <https://www.rfc-editor.org/rfc/rfc6287>.

   [RFC6973]  Cooper, A., Tschofenig, H., Aboba, B., Peterson, J.,
              Morris, J., Hansen, M., and R. Smith, "Privacy
              Considerations for Internet Protocols", RFC 6973,
              DOI 10.17487/RFC6973, July 2013,
              <https://www.rfc-editor.org/rfc/rfc6973>.

   [totp-secure-enrollment-reference-authenticator]
              "TOTP Secure Enrollment reference authenticator
              application pull request", n.d.,
              <https://github.com/silentbrianc/Aegis/pull/1/>.

   [totp-secure-enrollment-reference-server]
              "TOTP Secure Enrollment reference server pull request",
              n.d.,
              <https://github.com/silentbrianc/aspnetcore/pull/2/>.

Acknowledgments

   Thank you to Lauro Chavez and Zach Fuller at Silent Sector for their
   reviews, suggestions, and support.



Contario                  Expires 31 March 2026                [Page 20]

Internet-Draft           totp-secure-enrollment           September 2025


   Thank you to Al Katawazi and Craig Tulli of Blueshift Technologies
   for reviews and feedback.

   Thank you to Mike Amundsen of Amundsen.com for guidance in starting
   the IETF draft process.

Author's Address

   Brian Contario
   Silent Sector
   Email: bcontario@silentsector.com








































Contario                  Expires 31 March 2026                [Page 21]


name: draft-liu-rtgwg-llmsync-multicast
rev: 00
title: Multicast Use Cases for Large Language Model Synchronization
time: 2026-02-12T08:52:28Z
group_uri: /api/v1/group/group/1027/
expires: 2026-08-16T08:52:28Z
states: ["/api/v1/doc/state/1/", "/api/v1/doc/state/150/"]
categories: ["ML traffic mgmt", "Model serving/inference"]
tags: []
fetched_at: 2026-02-27T22:56:07.990731+00:00
abstract:
Large Language Models (LLMs) deployments are becoming increasingly
   widespread, with inference services being the most common
   application.  This draft will discuss multicast use cases for
   inference cloud services.

full_text:
RTGWG                                                             Y. Liu
Internet-Draft                                              China Mobile
Intended status: Informational                                  Z. Zhang
Expires: 16 August 2026                                  ZTE Corporation
                                                                J. Zhang
                                                            China Mobile
                                                        12 February 2026


      Multicast Use Cases for Large Language Model Synchronization
                  draft-liu-rtgwg-llmsync-multicast-00

Abstract

   Large Language Models (LLMs) deployments are becoming increasingly
   widespread, with inference services being the most common
   application.  This draft will discuss multicast use cases for
   inference cloud services.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 16 August 2026.

Copyright Notice

   Copyright (c) 2026 IETF Trust and the persons identified as the
   document authors.  All rights reserved.











Liu, et al.              Expires 16 August 2026                 [Page 1]

Internet-Draft              Abbreviated Title              February 2026


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   2
   2.  LLM Synchronization . . . . . . . . . . . . . . . . . . . . .   2
   3.  Multicast technologies applying . . . . . . . . . . . . . . .   4
   4.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   5
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .   5
   6.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   5
     6.1.  Normative References  . . . . . . . . . . . . . . . . . .   5
     6.2.  Informative References  . . . . . . . . . . . . . . . . .   5
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   6

1.  Introduction

   With the rapid development of AI and the widespread application of
   large language models (LLMs), inference services are the most
   frequently used services.  Different users may use different LLMs,
   and the same user may use multiple LLMs simultaneously for inference
   to obtain the optimal solution.  AI inference cloud providers can
   provide large-scale real-time inference, fine-tuning, and model
   optimization services on GPU cloud platforms.  However, the GPU
   infrastructure of AI inference cloud providers may cover multiple
   cloud platforms and regions, facing significant challenges in
   deployment and application, including highly concurrent model loading
   and severe cold start latency.

   This draft will discuss multicast use cases for inference cloud
   services.

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

2.  LLM Synchronization





Liu, et al.              Expires 16 August 2026                 [Page 2]

Internet-Draft              Abbreviated Title              February 2026


                           +------------------+
                           | Model Repository |
                           +--+-----+-----+---+
                              |     |     |
                              |     |     |
             +----------------+     |     +----------------+
             |                      |                      |
             |                      |                      |
   +-------------------+  +-------------------+  +-------------------+
   | +---------------+ |  | +---------------+ |  | +---------------+ |
   | | Local Storage | |  | | Local Storage | |  | | Local Storage | |
   | +---------------+ |  | +---------------+ |  | +---------------+ |
   |                   |  |                   |  |                   |
   | +---------------+ |  | +---------------+ |  | +---------------+ |
   | |  GPU Cluster  | |  | |  GPU Cluster  | |  | |  GPU Cluster  | |
   | +---------------+ |  | +---------------+ |  | +---------------+ |
   |                   |  |                   |  |                   |
   |    GPU Cloud 1    |  |    GPU Cloud 2    |  |    GPU Cloud n    |
   +-------------------+  +-------------------+  +-------------------+

                                  Figure 1

   Highly concurrent model loading refers to the peak load and
   concurrency challenge of simultaneously downloading the same popular
   large model across dozens of GPU cloud platforms.  A single
   deployment of a model may involve 10 to 100+ copies, each requiring a
   complete copy of the model file.  Hundreds of GPU servers
   simultaneously downloading the same model (each ranging from 70GB to
   1TB in size) within a single cluster generates enormous bandwidth
   demands and creates I/O bottlenecks.

   Severe cold start latency refers to the high delay in initial model
   deployment caused by slow download speeds.  Replica startup time is
   limited by inbound network bandwidth, which varies significantly
   between different cloud providers and is typically much lower than
   the cluster's internal bandwidth.  This significantly impacts the
   download efficiency of large models in practical applications.

   Highly concurrent model downloading are typical multicast
   applications.  Such multicast applications have the following
   characteristics:

   *  Large data volume: Due to the large size of the models, typically
      a single model can reach 70GB to 1TB, placing extremely high
      demands on network bandwidth.






Liu, et al.              Expires 16 August 2026                 [Page 3]

Internet-Draft              Abbreviated Title              February 2026


   *  Transmission time: Due to cold start latency requirements, data
      transmission needs to be completed as quickly as possible.
      Transmission times exceeding tens of minutes will significantly
      impact user experience; therefore, the shorter the time, the
      better.

   Therefore, when applying multicast technology to this scenario, it is
   necessary to consider ensuring high bandwidth and low latency.

3.  Multicast technologies applying

   Considering the need to conserve network bandwidth, ingress interface
   replication technology is not suitable for this scenario.  PIM-SM, SR
   P2MP or BIER technologies should be considered instead.

   Protocol Independent Multicast - Sparse Mode (PIM-SM) [RFC7761] is a
   traditional multicast technology.  It is widely used in scenarios
   where the receivers are relatively fixed, such as IPTV systems.  When
   the network topology of the multicast tree changes, a new multicast
   tree needs to be established for each multicast stream via PIM-SM
   signaling after the BGP/IGP protocol converges.  The convergence time
   of the multicast tree is much longer than that of the IGP protocol.

   SR-P2MP (Segment Routing Replication for Multipoint Service Delivery)
   [I-D.ietf-pim-sr-p2mp-policy] is a relatively new tunneling
   technology that uses SR-MPLS/SRv6 (Segment Routing over IPv6)
   tunneling technology for multicast traffic transmission.  It requires
   the routing module of the controller or ingress node to calculate and
   determine the path of the multicast traffic.  Then, the controller or
   ingress node issues a SID (Segment Identifier) for multicast
   operations to the replication point (i.e., the multicast replication
   point) in the network.  When multicast traffic enters the tunnel, it
   is replicated and forwarded at the replication point according to the
   multicast operation SID.  When the network topology changes, the
   controller or ingress node needs to recalculate and determine the
   replication point and issue the multicast operation SID to the
   changed replication point, so that subsequent multicast traffic will
   be forwarded through the new path.

   BIER (Bit-Indexed Explicit Replication) [RFC8279] is an architecture
   that provides optimal multicast forwarding through a "multicast
   domain", without requiring intermediate routers to maintain any per-
   flow state or to engage in an explicit tree-building protocol.  BIER
   is more flexible than PIM-SM and SR P2MP.  When link failures or
   other issues occur on the multicast forwarding path, BIER can
   converge along with IGP convergence, a speed far exceeding that of
   PIM-SM and SR P2MP.




Liu, et al.              Expires 16 August 2026                 [Page 4]

Internet-Draft              Abbreviated Title              February 2026


   When considering applying multicast technology to large model
   synchronization scenarios, if the model is synchronized to the same
   destination GPU clouds each time, a multicast tree can be pre-
   established, or the SR replication path can be calculated using the
   controller, and PIM-SM or SR P2MP technologies can be used for model
   copying.

   If the destination GPU clouds for each model synchronization is
   different, pre-establishing a multicast tree or multicast path each
   time using PIM-SM/SR P2MP technologies may be inefficient because
   multicast tree establishment takes time.  In this case, using BIER
   technology is a better choice.

4.  IANA Considerations

   There are no IANA consideration introduced by this draft.

5.  Security Considerations

   There are no security issues introduced by this draft.

6.  References

6.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC7761]  Fenner, B., Handley, M., Holbrook, H., Kouvelas, I.,
              Parekh, R., Zhang, Z., and L. Zheng, "Protocol Independent
              Multicast - Sparse Mode (PIM-SM): Protocol Specification
              (Revised)", STD 83, RFC 7761, DOI 10.17487/RFC7761, March
              2016, <https://www.rfc-editor.org/info/rfc7761>.

   [RFC8279]  Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A.,
              Przygienda, T., and S. Aldrin, "Multicast Using Bit Index
              Explicit Replication (BIER)", RFC 8279,
              DOI 10.17487/RFC8279, November 2017,
              <https://www.rfc-editor.org/info/rfc8279>.

6.2.  Informative References

   [I-D.ietf-pim-sr-p2mp-policy]
              Parekh, R., Voyer, D., Filsfils, C., Bidgoli, H., and Z.
              J. Zhang, "Segment Routing Point-to-Multipoint Policy",
              Work in Progress, Internet-Draft, draft-ietf-pim-sr-p2mp-



Liu, et al.              Expires 16 August 2026                 [Page 5]

Internet-Draft              Abbreviated Title              February 2026


              policy-22, 4 September 2025,
              <https://datatracker.ietf.org/doc/html/draft-ietf-pim-sr-
              p2mp-policy-22>.

Authors' Addresses

   Yisong Liu
   China Mobile
   China
   Email: liuyisong@chinamobile.com


   Zheng Zhang
   ZTE Corporation
   China
   Email: zhang.zheng@zte.com.cn


   Junye Zhang
   China Mobile
   China
   Email: zhangjunye@chinamobile.com





























Liu, et al.              Expires 16 August 2026                 [Page 6]


name: draft-zhang-rtgwg-llmmoe-multicast
rev: 01
title: Multicast usage in LLM MoE
time: 2025-10-20T07:22:26Z
group_uri: /api/v1/group/group/1027/
expires: 2026-04-23T07:22:26Z
states: ["/api/v1/doc/state/1/", "/api/v1/doc/state/150/"]
categories: ["ML traffic mgmt", "Model serving/inference"]
tags: []
fetched_at: 2026-02-27T22:56:07.990745+00:00
abstract:
Large Language Models (LLMs) have been widely used in recent years.
   The Mixture of Experts (MoE) architecture is one of the features of
   LLMs that enables efficient inference and cost-effective training.
   With the MoE architecture, there are potential multicast use cases
   such as tokens dispatching.  This draft attempts to analyze these use
   cases.

full_text:
RTGWG                                                           Z. Zhang
Internet-Draft                                                   W. Duan
Intended status: Informational                           ZTE Corporation
Expires: 23 April 2026                                             X. Xu
                                                            China Mobile
                                                         20 October 2025


                       Multicast usage in LLM MoE
                 draft-zhang-rtgwg-llmmoe-multicast-01

Abstract

   Large Language Models (LLMs) have been widely used in recent years.
   The Mixture of Experts (MoE) architecture is one of the features of
   LLMs that enables efficient inference and cost-effective training.
   With the MoE architecture, there are potential multicast use cases
   such as tokens dispatching.  This draft attempts to analyze these use
   cases.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 23 April 2026.

Copyright Notice

   Copyright (c) 2025 IETF Trust and the persons identified as the
   document authors.  All rights reserved.










Zhang, et al.             Expires 23 April 2026                 [Page 1]

Internet-Draft              Abbreviated Title               October 2025


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   3
   2.  Use case - tokens dispatching . . . . . . . . . . . . . . . .   3
     2.1.  Intra-node multicast  . . . . . . . . . . . . . . . . . .   4
     2.2.  Inter-node multicast  . . . . . . . . . . . . . . . . . .   4
     2.3.  Dynamic Requirements  . . . . . . . . . . . . . . . . . .   5
     2.4.  Reliability requirements  . . . . . . . . . . . . . . . .   5
   3.  Multicast technologies analysis . . . . . . . . . . . . . . .   5
   4.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   6
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .   6
   6.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   6
     6.1.  Normative References  . . . . . . . . . . . . . . . . . .   6
     6.2.  Informative References  . . . . . . . . . . . . . . . . .   7
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   7

1.  Introduction

   In recent years, large language models (LLMs) have been widely used.
   Mixture of Experts Model (MoE) is one of the functions of LLM to
   achieve efficient inference and economical training.  Many LLMs
   currently adopt the MoE architecture, such as DeepSeek-V2/V3, Google
   Gemini 1.5 Pro, xAI Grok-1, Mistral 8*22B, Qwen3, etc.  During
   inference, MoE only activates a small number of parameters to
   determine each output token, which significantly reduces the amount
   of computation required by the processor, thereby reducing the
   overall computational requirements.  Therefore, the fewer parameters
   are activated, the less computation the processor needs to perform.
   In the MoE architecture, one token needs to be sent to multiple
   experts, which is a typical multicast use case.

   In most LLMs, two experts are activated during the computation: one
   is a routed expert and the other is a shared expert.  In DeepSeekV3,
   one token activates eight routed experts and one shared expert.

   When all activated experts are located on a node with multiple GPUs
   installed, only intra-node communication is required.  When activated
   experts are located on different nodes, inter-node communication is



Zhang, et al.             Expires 23 April 2026                 [Page 2]

Internet-Draft              Abbreviated Title               October 2025


   required.  Due to the bandwidth difference between intra-node and
   inter-node scenarios, communication across leaf switches and even
   spine switches is inevitable.

   This draft analyzes the multicast use case of LLM in data centers.

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

2.  Use case - tokens dispatching

               +-----------+               +-----------+
               |  Spine 1  |               |  Spine x  |
               +-+------+--+               +-+------+--+
                 |      |                    |      |
                 |      |                    |      +---------+
        +--------+      +--------------------|---------+      |
        |      +-----------------------------+         |      |
        |      |                                       |      |
      +-+------+-+     +----------+                  +-+------+-+
      |  Leaf 1  |     |  Leaf 2  |    ......        |  Leaf n  |
      +-+--+---+-+     +----------+                  +--+--+--+-+
        |  |   |                                        |  |  |
        |  |   +-------------------------------------+  |  |  |
        |  +----------------+                        |  |  |  |
        |                   |              +---------|--|--+  |
      +-+            +------|--------------|---------|--+     +-----+
      |              |      |              |         |              |
    +-+--+----+---+--+-+  +-+--+----+---+--+-+     +-+--+----+---+--+-+
    |GPU1|GPU2|...|GPU8|  |GPU1|GPU2|...|GPU8| ... |GPU1|GPU2|...|GPU8|
    +----+----+---+----+  +----+----+---+----+     +----+----+---+----+
           node 1                 node 2                   node m

                                  Figure 1














Zhang, et al.             Expires 23 April 2026                 [Page 3]

Internet-Draft              Abbreviated Title               October 2025


   During the pre-filling and decoding phases, tokens need to be sent to
   all selected experts, including routed experts and shared experts.
   The tokens dispatching can be intra-node or inter-node.  Different
   LLMs use different numbers of experts.  For example, Mixtral uses 8
   experts and activates 2 experts at a time; LlaMa 4 uses 16 experts
   (Scout) or 128 experts (Maverick), and activates 2 experts at a time;
   DeepSeekV3 uses 256 experts and activates 9 experts at a time.  The
   more routed experts there are, the more distribution there is between
   nodes.  In order to balance the experts, it is difficult to limit the
   number of experts to one node even if only two experts (one routed
   expert and one shared expert) are used.

   The tokens dispatching can be optimized.  For example, in DeepSeekV3,
   LLM first selects the node group and then selects the expert from the
   node.  By implementing the node restricted routing function, a
   maximum of four nodes are selected to reduce the inter-node
   consumption of tokens dispatching.  In addition, in order to maximize
   the usage of the high intra-node bandwidth, after the switch or GPU
   in the node receives the tokens, the switch or GPU needs to
   distribute the tokens to the experts in the same node.  This
   optimization aims to reduce the inter-node distribution, but it
   cannot avoid multicast between nodes.

   It is worth noting that even during an inference or training process,
   the experts selected by the token are not fixed.  Each token may be
   sent to a different combination of experts.  For example, in
   DeepSeekv3, each token may be sent to 9 different experts.

   Therefore, the use of multicast may be intra-node or inter-node.  The
   existing multicast implementation methods are different in intra-node
   and inter-node scenarios, and multicast management is more difficult.

2.1.  Intra-node multicast

   When tokens need to be sent to multiple GPUs in the same node, the
   GPU or the switch connected to the GPU may send the tokens in a
   multicast manner.  This requires the switch or GPU to support the
   multicast function.  This function can reduce the computational
   burden of the source GPU and reduce the bandwidth consumption between
   nodes.

2.2.  Inter-node multicast

   When tokens need to be sent to multiple nodes, Leaf switches and even
   Spine switches need to forward tokens.  Due to the limitation of
   inter-node bandwidth, the more packets there are, the greater the
   risk of congestion.  Using multicast technology can reduce the burden
   on the source GPU and reduce the risk of congestion.



Zhang, et al.             Expires 23 April 2026                 [Page 4]

Internet-Draft              Abbreviated Title               October 2025


2.3.  Dynamic Requirements

   Due to the random nature of multicast destination selection, for
   example, in the token dispatching process mentioned above, each token
   may select a different expert combination.  The selection process is
   very short, leaving no time for multicast technologies like PIM to
   establish a multicast tree.  Therefore, multicast technology that can
   meet dynamic needs is needed.

2.4.  Reliability requirements

   The transmission of all types of data (including tokens) used for LLM
   calculations requires extremely high reliability.  This means that
   packet loss, long delays or jitters, and retransmissions during data
   transmission can impact LLM calculations.  Reliability is a paramount
   requirement when applying multicast technology to data transmission.

   If reliability is insufficient, even if most data reaches its
   destination quickly, if even one destination fails to receive the
   data in time due to packet loss, long latency, or excessive jitter,
   the LLM calculation may need to be restarted, significantly reducing
   computational efficiency.

   So compared to unicast, multicast reliability technology is more
   complex.  Especially in LLM applications, it is necessary to avoid
   packet loss, long latency, excessive jitter, and retransmissions
   caused by individual multicast branches to minimize the impact on LLM
   calculations.

3.  Multicast technologies analysis

   Protocol Independent Multicast - Sparse Mode (PIM-SM) [RFC7761] is a
   traditional multicast technology.  It relies on PIM signaling to
   build the multicast tree.  When the receivers change, the multicast
   tree may need to be rebuilt.  When PIM is used for intra-node or
   inter-node multicast, the stability of the multicast tree is more
   important.  It may not be applicable when the expert combination is
   flexible.  Even in the intra-node scenario, the number of potential
   multicast trees may be large despite the limited number of GPUs in a
   single node.

   BIER (Bit-Indexed Explicit Replication) [RFC8279] is an architecture
   that provides optimal multicast forwarding through a "multicast
   domain", without requiring intermediate routers to maintain any per-
   flow state or to engage in an explicit tree-building protocol.  BIER
   is more flexible than PIM.  Experts can be numbered and can act as
   ingress or egress BFRs in BIER.  BIER header encapsulation can be a
   function defined in [RFC8296], [I-D.ietf-bier-bierin6], or



Zhang, et al.             Expires 23 April 2026                 [Page 5]

Internet-Draft              Abbreviated Title               October 2025


   [I-D.zzhang-bier-unmasked-bier].  By using the BIER function, Leaf
   and Spine switches, and even GPUs or switches connected to GPUs, can
   pre-build expert-based forwarding tables. tokens can be sent to any
   selected expert.

   Other multicast methods, such as PIM DM (dense mode) and ingress
   replication, may consume more bandwidth and may not be a good choice
   for multicast scenarios such as LLM tokens dispatching.

   Considering dynamic requirements like token dispatching, technologies
   like PIM, which require the establishment of a multicast tree, are
   inadequate.  BIER, on the other hand, allows the source GPU (similar
   to BFIR in BIER) to directly specify the destination expert group
   (similar to BFERs in BIER) and encapsulate it into the message,
   eliminating the time for multicast tree establishment.  Therefore,
   BIER is the most suitable multicast technology.  Of course, prior to
   this, control-plane negotiation between the source GPU and the
   experts regarding data transmission is required.

   While the network layer can provide multicast capabilities for
   multicast scenarios, the multicast approach needs to work in
   conjunction with the LLM software.  It may work in conjunction with
   the implementation of collective communication and NIC (network
   interface card).

4.  IANA Considerations

   There are no IANA consideration introduced by this draft.

5.  Security Considerations

   There are no security issues introduced by this draft.

6.  References

6.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC7761]  Fenner, B., Handley, M., Holbrook, H., Kouvelas, I.,
              Parekh, R., Zhang, Z., and L. Zheng, "Protocol Independent
              Multicast - Sparse Mode (PIM-SM): Protocol Specification
              (Revised)", STD 83, RFC 7761, DOI 10.17487/RFC7761, March
              2016, <https://www.rfc-editor.org/info/rfc7761>.




Zhang, et al.             Expires 23 April 2026                 [Page 6]

Internet-Draft              Abbreviated Title               October 2025


   [RFC8279]  Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A.,
              Przygienda, T., and S. Aldrin, "Multicast Using Bit Index
              Explicit Replication (BIER)", RFC 8279,
              DOI 10.17487/RFC8279, November 2017,
              <https://www.rfc-editor.org/info/rfc8279>.

   [RFC8296]  Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A.,
              Tantsura, J., Aldrin, S., and I. Meilik, "Encapsulation
              for Bit Index Explicit Replication (BIER) in MPLS and Non-
              MPLS Networks", RFC 8296, DOI 10.17487/RFC8296, January
              2018, <https://www.rfc-editor.org/info/rfc8296>.

6.2.  Informative References

   [I-D.ietf-bier-bierin6]
              Zhang, Z., Zhang, Z. J., Wijnands, I., Mishra, M. P.,
              Bidgoli, H., and G. S. Mishra, "Supporting BIER in IPv6
              Networks (BIERin6)", Work in Progress, Internet-Draft,
              draft-ietf-bier-bierin6-12, 25 August 2025,
              <https://datatracker.ietf.org/doc/html/draft-ietf-bier-
              bierin6-12>.

   [I-D.zzhang-bier-unmasked-bier]
              Przygienda, T., Zhang, Z. J., Bigdoli, H., and I.
              Wijnands, "Unmasked BIER Mode", Work in Progress,
              Internet-Draft, draft-zzhang-bier-unmasked-bier-01, 23
              February 2025, <https://datatracker.ietf.org/doc/html/
              draft-zzhang-bier-unmasked-bier-01>.

Authors' Addresses

   Zheng Zhang
   ZTE Corporation
   China
   Email: zhang.zheng@zte.com.cn


   Wei Duan
   ZTE Corporation
   China
   Email: duan.wei1@zte.com.cn


   Xiaohu Xu
   China Mobile
   China
   Email: xuxiaohu_ietf@hotmail.com




Zhang, et al.             Expires 23 April 2026                 [Page 7]


name: draft-irtf-nmrg-llm-nm
rev: 00
title: A Framework for LLM-Assisted Network Management with Human-in-the-Loop
time: 2025-09-14T07:09:00Z
group_uri: /api/v1/group/group/38/
expires: 2025-09-14T15:44:40Z
states: ["/api/v1/doc/state/2/", "/api/v1/doc/state/150/", "/api/v1/doc/state/57/"]
categories: ["Autonomous netops", "Human-agent interaction", "Data formats/interop"]
tags: []
fetched_at: 2026-02-27T22:56:07.990753+00:00
abstract:
This document defines an interoperable framework that facilitates
   collaborative network management between Large Language Models (LLMs)
   and human operators.  The proposed framework introduces enhanced
   telemetry module, LLM decision module and standardized interaction
   data models between human operators and LLM-driven systems, and
   workflows to enforce human oversight.  The approach ensures
   compatibility with existing network management systems and protocols
   while improving automation and decision-making capabilities in
   network operations.

full_text:
Network Management Research Group                                 Y. Cui
Internet-Draft                                       Tsinghua University
Intended status: Informational                                   M. Xing
Expires: 4 September 2025                                       L. Zhang
                                                 Zhongguancun Laboratory
                                                            3 March 2025


 A Framework for LLM-Assisted Network Management with Human-in-the-Loop
                       draft-irtf-nmrg-llm-nm-00

Abstract

   This document defines an interoperable framework that facilitates
   collaborative network management between Large Language Models (LLMs)
   and human operators.  The proposed framework introduces enhanced
   telemetry module, LLM decision module and standardized interaction
   data models between human operators and LLM-driven systems, and
   workflows to enforce human oversight.  The approach ensures
   compatibility with existing network management systems and protocols
   while improving automation and decision-making capabilities in
   network operations.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 4 September 2025.

Copyright Notice

   Copyright (c) 2025 IETF Trust and the persons identified as the
   document authors.  All rights reserved.







Cui, et al.             Expires 4 September 2025                [Page 1]

Internet-Draft                   LLM4Net                      March 2025


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Motivation  . . . . . . . . . . . . . . . . . . . . . . .   2
     1.2.  Problem Statement . . . . . . . . . . . . . . . . . . . .   3
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.1.  Acronyms and Abbreviations  . . . . . . . . . . . . . . .   3
   3.  Framework Overview  . . . . . . . . . . . . . . . . . . . . .   3
     3.1.  Enhanced Telemetry Module . . . . . . . . . . . . . . . .   5
     3.2.  LLM Decision Module . . . . . . . . . . . . . . . . . . .   5
       3.2.1.  RAG Module  . . . . . . . . . . . . . . . . . . . . .   5
       3.2.2.  Task Instance Module  . . . . . . . . . . . . . . . .   6
       3.2.3.  Task Instance Management Module . . . . . . . . . . .   6
       3.2.4.  Config Verify Module  . . . . . . . . . . . . . . . .   7
       3.2.5.  Access Control Module . . . . . . . . . . . . . . . .   7
     3.3.  Operator Audit Module . . . . . . . . . . . . . . . . . .   8
   4.  Data Model  . . . . . . . . . . . . . . . . . . . . . . . . .   9
     4.1.  LLM Response Data Model . . . . . . . . . . . . . . . . .   9
     4.2.  Human Audit Data Model  . . . . . . . . . . . . . . . . .   9
   5.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  11
   6.  Security Considerations . . . . . . . . . . . . . . . . . . .  11
   7.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  11
     7.1.  Normative References  . . . . . . . . . . . . . . . . . .  11
     7.2.  Informative References  . . . . . . . . . . . . . . . . .  11
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  12

1.  Introduction

1.1.  Motivation

   Traditional network automation systems struggle with handling
   unanticipated scenarios and managing complex multi-domain data
   dependencies.  Large Language Models (LLMs) offer advanced multimodal
   data comprehension, adaptive reasoning, and generalization
   capabilities, making them a promising tool for network management and
   autonomous network[TM-IG1230].  However, full automation remains
   impractical due to risks such as model hallucination, operational
   errors, and the lack of accountability in decision-making[Huang25].
   This document proposes a structured framework that integrates LLMs
   into network management through human-in-the-loop collaboration,
   leveraging their strengths while ensuring oversight, reliability, and
   operational safety.



Cui, et al.             Expires 4 September 2025                [Page 2]

Internet-Draft                   LLM4Net                      March 2025


1.2.  Problem Statement

   Network management faces significant challenges, including the
   complexity of multi-vendor configurations, the real-time correlation
   of heterogeneous telemetry data, and the need for rapid responses to
   dynamic security threats.  LLMs offer a promising approach to
   addressing these challenges through their advanced multimodal data
   understanding and adaptive reasoning capabilities.  However, the
   direct application of LLMs in network management introduces several
   technical considerations.  These include the need for semantic
   enrichment of network telemetry to enhance LLM comprehension, a dual-
   channel decision execution mechanism with confidence-based
   escalation, and auditability of LLM-generated decisions through
   provenance tracking.  Addressing these requirements is critical to
   integrating LLMs effectively into network management workflows while
   maintaining reliability, transparency, and interoperability.

2.  Terminology

2.1.  Acronyms and Abbreviations

   *  LLM: Large Language Model

   *  RAG: Retrieve Augmented Generation

3.  Framework Overview

























Cui, et al.             Expires 4 September 2025                [Page 3]

Internet-Draft                   LLM4Net                      March 2025


   +-------------------------------------------------------------+
   |             LLM-Assisted Netwrok Management System          |
   +-------------------------------------------------------------+
   |+--------------------LLM Decision Module--------------------+|
   ||                                                           ||
   ||               +--Task Instance Module--+  +-------------+ ||
   ||               | +------+  +----------+ |  |Task Instance<-----+
   ||               | |Prompt|  |Fine-Tuned| <->| Mgt Module  | ||  |
   ||               | | Lib  |  |Weight Lib| |  +-------------+ ||  |
   || +----------+  | +------+  +----------+ |  +-------------+ ||  |
   || |RAG Module|<-> +--------------------+ |  |Config Verify| ||  |
   || +-----^----+  | |Foundation Model Lib| -->|    Module   | ||  |
   ||       |       | +--------------------+ |  +-------|-----+ ||  |
   ||       |       +----^---------------^---+  +-------v------+||  |
   ||       |            |               |      |Access Control|||  |
   ||       |            |               |      |    Module    |||  |
   ||       |            |               |      +-------|------+||  |
   |+-------|------------|---------------|--------------|-------+|  |
   |+-------v------------v----+ +--------v--------------v-------+|  |
   ||Enhanced Telemetry Module| |   Operator Audit Module       ||  |
   |+-----------^-------------+ +--------------|-------------^--+|  |
   +------------|------------------------------|-------------|---+  |
                |                              |       +-----v---+  |
                |                              |       |Operator <--+
                |                              |       +---------+
   +------------v------------------------------v------------------+
   |               Original Netwrok Management System             |
   +------------------------------^-------------------------------+
                                  |
   +------------------------------v-------------------------------+
   |                       Physical Network                       |
   +--------------------------------------------------------------+

   Figure 1: The LLM-Assisted Network Management Framework

   The proposed framework is shown in Figure 1, highlighting the key
   components of LLM-assisted network management.  The human operator
   can generate a specific task instance, e.g., fault analysis or
   topology optimization, using the task instance management module.
   According to the task type, the task instance module can instantiate
   a task instance with specific foundation model, prompt, and
   adaptation fine-tuned parameters[Hu22].  The enahnced telemetry
   module improves the semantics of raw telemetry data from original
   network management system, providing supplementary information to the
   LLM decision module for more informed decision-making.  After the
   decision-making, the generated configuration parameters are validated
   against the YANG model and enforced with access control rules.  The
   operator audit module provides a structured mechanism for human



Cui, et al.             Expires 4 September 2025                [Page 4]

Internet-Draft                   LLM4Net                      March 2025


   oversight of LLM-generated configurations, and the configuration can
   be issued to the original network management system for deployment
   once the operator approves.

3.1.  Enhanced Telemetry Module

   The Enhanced Telemetry Module improves the semantics of raw telemetry
   data, providing supplementary information to the LLM decision module
   for more informed decision-making.  Telemetry data retrieved from
   network devices via NETCONF[RFC6241], e.g., in XML format, often
   lacks field descriptions, structured metadata, and vendor-specific
   details.  Since this information is not included in the pre-trained
   knowledge of LLMs, it can lead to misinterpretation and erroneous
   reasoning.  To address this limitation, an external knowledge base
   should be introduced to store YANG model schema, device manuals, and
   other relevant documentation.  The Enhanced Telemetry Module
   functions as middleware between the network management system and the
   external knowledge base.  Through its southbound interface, it
   retrieves NETCONF data from the NETCONF client of existing network
   management system.  Through its northbound interface, the module
   queries the external knowledge base for the corresponding YANG model
   or device manual.  To enhance semantic richness, the Enhanced
   Telemetry Module processes the retrieved data by simplifying
   formatted content (e.g., removing redundant or closing XML tags) and
   appending XML path and description information from the YANG tree to
   the relevant fields.  This approach ensures that the LLM has access
   to structured, contextually enriched data, improving its ability to
   analyze and reason about network telemetry.

3.2.  LLM Decision Module

3.2.1.  RAG Module

   The pre-trained LLM may not encompass specific task requirements or
   vendor-specific knowledge.  To address this kind of limitation, the
   Retrieve-Augmented Generation (RAG)[Lewis20] approach is widely used.
   This module retrieves relevant information from operator-defined
   sources, such as device documentation and expert knowledge, and
   integrates it with the Enhanced Telemetry Module to obtain YANG model
   schema.  The retrieved textual data is stored in a database, either
   as raw text or in a vectorized format for efficient search and
   retrieval.  For a given task context, the module retrieves relevant
   knowledge from the database and incorporates it into the input
   context, improving the understanding and response accuracy of LLM.







Cui, et al.             Expires 4 September 2025                [Page 5]

Internet-Draft                   LLM4Net                      March 2025


3.2.2.  Task Instance Module

   To execute a specific task, such as traffic analysis, traffic
   optimization, or fault remediation, a corresponding task instance
   must be created.  A task instance consists of a selected LLM
   foundation model, an associated prompt and fine-tuned weights.

   *  Foundation Model Library.  Operators must select an appropriate
      foundation model based on the specific task requirements.
      Examples include general-purpose models such as GPT-4, LLaMA, and
      DeepSeek, as well as domain-specific models fine-tuned on private
      datasets.  Since foundation models are trained on different
      datasets using varying methodologies, their performance may differ
      across tasks.

   *  Fine-Tuned Weight Library.  For domain-specific applications,
      fine-tuned weights can be applied on top of a foundation model to
      efficiently adapt it to private datasets.  One commonly used
      approach is to store the fine-tuned weights as the difference
      between the original foundation model and the adapted model, which
      can largely reduce storage requirements.  The Fine-Tuned Weights
      Module supports the selection and loading of an appropriate
      foundation model along with the corresponding fine-tuned weights,
      based on the selection of operators.  This ensures flexibility in
      leveraging both general-purpose and domain-specific knowledge
      while maintaining computational efficiency.

   *  Prompt Library.  For each task, it is essential to accurately
      define the task description, the format of its inputs and outputs.
      These definitions are stored in a structured prompt library.  When
      an operator instantiates a task, the corresponding prompt,
      including placeholders for contextual information, is
      automatically retrieved. operator inputs and device data are then
      incorporated into the prompt at the designated placeholders,
      ensuring a structured and consistent interaction with the language
      model.

3.2.3.  Task Instance Management Module

   The Task Instance Management Module is responsible for the creation,
   update, and deletion of task instances.  This module ensures that
   each instance is appropriately configured to align with the intended
   network management objective.








Cui, et al.             Expires 4 September 2025                [Page 6]

Internet-Draft                   LLM4Net                      March 2025


3.2.4.  Config Verify Module

   To ensure correctness and policy compliance, LLM-generated
   configurations MUST pass the YANG schema validation steps before
   being queued for human approval.  This module ensures that only
   syntactically correct configurations are presented for operator
   review, thereby reducing errors and enhancing network reliability.

3.2.5.  Access Control Module

   Although the Configuration Verify Module can guarantee the syntactic
   correction, LLMs may generate unintended or potentially harmful
   operations on critical network devices, it is essential for operators
   to enforce clear permission boundaries for LLM task instance to
   ensure security and operational integrity.  The Network Configuration
   Access Control Model defined in [RFC8341] provides a framework for
   specifying access permissions, which can be used to grant access to
   LLM task instances.  This data model includes the concepts of users,
   groups, access operation types, and action types, which can be
   applied as follows:

   *  User and Group: Each task instance should be registered as a
      specific user, representing an entity with defined access
      permissions for particular devices.  These permissions control the
      types of operations the LLM is authorized to perform.  A task
      instance (i.e., user) is identified by a unique string within the
      system.  Access control can also be applied at the group level,
      where a group consists of zero or more members, and a task
      instance can belong to multiple groups.

   *  Access Operation Types: These define the types of operations
      permitted, including create, read, update, delete, and execute.
      Each task instance may support different sets of operations
      depending on its assigned permissions.

   *  Action Types: These specify whether a given operation is permitted
      or denied.  This mechanism determines whether an LLM request to
      perform an operation is allowed based on predefined access control
      rules.

   *  Rule List: A rule governs access control by specifying the content
      and operations a task instance is authorized to handle within the
      system.

   This module must enforce explicit restrictions on the actions an LLM
   is permitted to perform, ensuring that network configurations remain
   secure and compliant with operational policies.




Cui, et al.             Expires 4 September 2025                [Page 7]

Internet-Draft                   LLM4Net                      March 2025


3.3.  Operator Audit Module

   The Operator Audit Module provides a structured mechanism for human
   oversight of LLM-generated configurations before deployment.  The
   output from the LLM Decision Module should include both the generated
   configuration parameters and an associated confidence score.  The
   configuration parameters are validated for compliance with the YANG
   model and are subject to Access Control Rules enforcement.  The
   confidence score, e.g., ranging from 0 to 100, serves as a reference
   for operators to assess the reliability of the generated
   configuration.  Each audit process must track the input context
   (e.g., input data, RAG query content, model selection, configuration
   files) and the corresponding output results.  The auditing steps are
   as follows:

   *  Result Verification: The operator verifies the LLM-generated
      output to ensure alignment with business objectives and policy
      requirements.

   *  Compliance Check: The operator ensures that the LLM output adheres
      to regulatory standards and legal requirements.

   *  Security Verification: The operator examines the output for
      potential security risks, such as misconfigurations or
      vulnerabilities.

   *  Suggestions and Corrections: If issues are identified, the
      operator documents the findings and proposes corrective actions.

   Upon completing the audit, the system maintains an audit decision
   record to ensure traceability of operator actions.  The audit record
   includes the following information:

   *  Timestamp of the audit action

   *  LLM Task Instance ID associated with the action

   *  Operator decisions, including approval, rejection, modification,
      or pending status

   *  Executed command reflecting the final action taken

   *  Operation type (e.g., configuration update, deletion, or
      execution)

   This structured approach ensures that all LLM-generated
   configurations undergo rigorous human review, maintaining operational
   accountability and security.



Cui, et al.             Expires 4 September 2025                [Page 8]

Internet-Draft                   LLM4Net                      March 2025


4.  Data Model

   This section defines the essential data models for LLM-assisted
   network management, including the LLM decision response and human
   audit records.

4.1.  LLM Response Data Model

   The LLM decision module should respond with the generated
   configuration parameters along with an associated confidence score.
   If the LLM is unable to generate a valid configuration, it should
   return an error message accompanied by an explanation of the issue.

   module: llm-response-module
     +--rw llm-response
        +--rw config?         string
        +--rw confidence?     uint64
        +--rw error-reason?   enumeration

   The LLM response YANG model is structured as follows:

   module llm-response-module {
     namespace "urn:ietf:params:xml:ns:yang:ietf-nmrg-llmn4et";
     prefix llmresponse;
     container llm-response {
       leaf config {
         type string;
       }
       leaf confidence {
         type uint64;
       }
       leaf error-reason {
         type enumeration {
           enum unsupported-task;
           enum unsupported-vendor;
         }
       }
     }
   }

4.2.  Human Audit Data Model

   This data model defines the structure for human audit operations and
   record-keeping.  It facilitates collaborative decision-making by
   allowing LLMs to generate actionable insights while ensuring that
   human operators retain final operational authority.





Cui, et al.             Expires 4 September 2025                [Page 9]

Internet-Draft                   LLM4Net                      March 2025


   module: human-audit-module
     +--rw human-audit
        +--rw task-id?      string
        +--rw generated-config?   string
        +--rw confidence?         int64
        +--rw human-actions
           +--rw operator?          string
           +--rw action?            enumeration
           +--rw modified-config?   string
           +--rw timestamp?         yang:date-and-time

   The human audit YANG model is structured as follows:

   module human-audit-module {
     namespace "urn:ietf:params:xml:ns:yang:ietf-nmrg-llmn4et";
     prefix llmaudit;
     import ietf-yang-types { prefix yang; }

     container human-audit {
       leaf task-id {
         type string;
         }
       leaf generated-config {
         type string;
         }
       leaf confidence {
         type int64;
         }
       container human-actions {
         leaf operator {
           type string;
           }
         leaf action {
           type enumeration {
             enum approve;
             enum modify;
             enum reject;
             }
           }
         leaf modified-config {
           type string;
           }
         leaf timestamp {
           type yang:date-and-time;
           }
       }
     }
   }



Cui, et al.             Expires 4 September 2025               [Page 10]

Internet-Draft                   LLM4Net                      March 2025


5.  IANA Considerations

   This document includes no request to IANA.

6.  Security Considerations

   *  Model Hallucination: A key challenge is that, without proper
      constraints, the LLM may produce malformed or invalid
      configurations.  This issue can be mitigated using techniques such
      as Constrained Decoding, which enforces syntactic correctness by
      modeling the configuration syntax and restricting the output to
      conform to predefined rules during the generation process.

   *  Training Data Poisoning: LLMs can be trained on malicious or
      biased data, potentially leading to unintended behavior or
      security vulnerabilities.  To mitigate this risk, LLMs should be
      trained on curated, high-quality datasets with rigorous validation
      and filtering processes.  Periodic retraining and adversarial
      testing should also be conducted to detect and correct anomalies
      before deployment.

7.  References

7.1.  Normative References

   [RFC8341]  Bierman, A. and M. Bjorklund, "Network Configuration
              Access Control Model", STD 91, RFC 8341,
              DOI 10.17487/RFC8341, March 2018,
              <https://www.rfc-editor.org/rfc/rfc8341>.

   [RFC6241]  Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed.,
              and A. Bierman, Ed., "Network Configuration Protocol
              (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011,
              <https://www.rfc-editor.org/rfc/rfc6241>.

7.2.  Informative References

   [TM-IG1230]
              McDonnell, K., Machwe, A., Milham, D., O’Sullivan, J.,
              Niemöller, J., Varvello, L. F., Devadatta, V., Lei, W.,
              Xu, W., Yuan, X., and Y. Stein, "Autonomous Networks
              Technical Architecture", February 2023.

   [Huang25]  Huang,, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H.,
              Chen, Q., Peng, W., Feng, X., Qin, B., and T. Liu, "A
              Survey on Hallucination in Large Language Models
              Principles, Taxonomy, Challenges, and Open Questions",
              n.d..



Cui, et al.             Expires 4 September 2025               [Page 11]

Internet-Draft                   LLM4Net                      March 2025


   [Hu22]     Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y.,
              Wang, S., Wang, L., and W. Chen, "LoRA Low-Rank Adaptation
              of Large Language Models", n.d..

   [Lewis20]  Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin,
              V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t.,
              Rocktäschel, T., and S. Riede, "Retrieval-augmented
              generation for knowledge-intensive NLP tasks", n.d..

Authors' Addresses

   Yong Cui
   Tsinghua University
   Beijing, 100084
   China
   Email: cuiyong@tsinghua.edu.cn
   URI:   http://www.cuiyong.net/


   Mingzhe Xing
   Zhongguancun Laboratory
   Beijing, 100094
   China
   Email: xingmz@zgclab.edu.cn


   Lei Zhang
   Zhongguancun Laboratory
   Beijing, 100094
   China
   Email: zhanglei@zgclab.edu.cn




















Cui, et al.             Expires 4 September 2025               [Page 12]


name: draft-liu-nmrg-ai-llm-inference-requirements
rev: 02
title: Requirements Analysis of System and Network for Large Language Model Inference Service
time: 2025-11-04T02:55:18Z
group_uri: /api/v1/group/group/1027/
expires: 2026-05-08T02:55:18Z
states: ["/api/v1/doc/state/1/", "/api/v1/doc/state/150/"]
categories: ["Model serving/inference", "ML traffic mgmt"]
tags: []
fetched_at: 2026-02-27T22:56:07.990761+00:00
abstract:
With the rise of ChatGPT, DeepSeek, and other Large Language Models,
   which is short for LLMs in the remaining part, as well as the
   proliferation of inference applications, inference serving oriented
   to large-scale users has become increasingly critical.  However, due
   to the extreme demands on computing power and communication during
   inference, the large-scale service deployment of LLMs poses
   significant challenges.  To address these challenges, different
   vendors have adopted diverse inference service architectures, such as
   vLLM, SGLang, Mooncake, etc.  This paper investigates mainstream
   inference frameworks, summarizes their core design principle and
   research question, and analyzes the challenges and requirements they
   impose on network management.  The goal is to lay a foundation for
   defining a unified LLM inference architecture in the future.

full_text:
Network Management                                                C. Liu
Internet-Draft                                                    C. Guo
Intended status: Informational                              China Mobile
Expires: 7 May 2026                                      3 November 2025


  Requirements Analysis of System and Network for Large Language Model
                           Inference Service
            draft-liu-nmrg-ai-llm-inference-requirements-02

Abstract

   With the rise of ChatGPT, DeepSeek, and other Large Language Models,
   which is short for LLMs in the remaining part, as well as the
   proliferation of inference applications, inference serving oriented
   to large-scale users has become increasingly critical.  However, due
   to the extreme demands on computing power and communication during
   inference, the large-scale service deployment of LLMs poses
   significant challenges.  To address these challenges, different
   vendors have adopted diverse inference service architectures, such as
   vLLM, SGLang, Mooncake, etc.  This paper investigates mainstream
   inference frameworks, summarizes their core design principle and
   research question, and analyzes the challenges and requirements they
   impose on network management.  The goal is to lay a foundation for
   defining a unified LLM inference architecture in the future.

Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 7 May 2026.



Liu & Guo                  Expires 7 May 2026                   [Page 1]

Internet-Draft             Network Management              November 2025


Copyright Notice

   Copyright (c) 2025 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Service-Oriented Inference Frameworks . . . . . . . . . . . .   3
     2.1.  PD Fusion Architecture  . . . . . . . . . . . . . . . . .   4
     2.2.  PD Disaggregation Architecture  . . . . . . . . . . . . .   5
   3.  Inference-related Metrics . . . . . . . . . . . . . . . . . .   6
   4.  Research Question for Service-Oriented Inference
           Frameworks  . . . . . . . . . . . . . . . . . . . . . . .   7
   5.  Challenges for Service-Oriented Inference Frameworks  . . . .   7
     5.1.  Challenge 1 . . . . . . . . . . . . . . . . . . . . . . .   8
     5.2.  Challenge 2 . . . . . . . . . . . . . . . . . . . . . . .   8
     5.3.  Challenge 3 . . . . . . . . . . . . . . . . . . . . . . .   8
   6.  Network Management Requirements for Service-Oriented Inference
           Frameworks  . . . . . . . . . . . . . . . . . . . . . . .   8
     6.1.  Efficient Load Balancing  . . . . . . . . . . . . . . . .   8
     6.2.  KV Cache Management . . . . . . . . . . . . . . . . . . .   9
     6.3.  KV Cache Transmission . . . . . . . . . . . . . . . . . .   9
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .   9
   8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   9
   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   9
     9.1.  Normative References  . . . . . . . . . . . . . . . . . .   9
     9.2.  Informative References  . . . . . . . . . . . . . . . . .   9
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  10

1.  Introduction

   Since the launch of ChatGPT in 2023, more and more product-level LLMs
   have emerged, with GPT-4o, Claude-Sonnet-3.5, Gemini, Kimi, and
   others leading the charge.  In early 2025, DeepSeek-R1 reignited the
   LLM frenzy, and Musk's xAI recently unveiled the powerful Grok3.  It
   is evident that LLMs will continue to reach new heights.





Liu & Guo                  Expires 7 May 2026                   [Page 2]

Internet-Draft             Network Management              November 2025


   Major vendors, including OpenAI, Anthropic, DeepSeek, and Google,
   have deployed their LLM applications across mobile and web platforms.
   As the field grows, daily active users (DAUs) for these applications
   are expected to surge, potentially reaching hundreds of millions
   during peak periods.  This presents significant challenges for large-
   scale inference services.  For instance, up to now, DeepSeek still
   struggles with persistent "Service Busy" issues.

   Existing large-scale inference service architectures primarily adopt
   two technical approaches: Prefill-Decoding (PD) Fusion and Prefill-
   Decoding Disaggregation, which is derived from the distinct
   computational characteristics of the Prefill (compute-intensive) and
   Decoding (memory-intensive) phases.  Efficient network management and
   hardware coordination are essential to maximize system throughput and
   minimize user-perceived latency.

   This document first introduces mainstream inference frameworks, then
   optimization metrics, and finally elaborates on the network and
   system requirements for deploying large-scale LLM inference services.

2.  Service-Oriented Inference Frameworks

   At present, there are two main technical routes of the mainstream LLM
   service systems, namely PD Fusion and PD Disaggregation.  Prefill,
   which is to simultaneously compute all of tokens of user requests,
   also known as prompts, is characterized as computational intensive,
   computing-bound, with extremely high computing force requirements.
   Decoding generates user-required content based on the KV Cache and
   first token generated by Prefill phase.  Due to the reuse of KV Cache
   of the tokens prior to the current token, it is characterized as
   memory-intensive and memory-bound, with higher requirements for
   memory in decoding phase.  A complete LLM inference procedure is
   shown in Figure 1.  Based on whether to decouple two stages with
   obviously different computing requirements, two technical routes of
   LLM inference serving system emerge, namely, PD Fusion and decoupled
   PD Disaggregation.  The rest of this section describes in detail
   about the two technical architectures.














Liu & Guo                  Expires 7 May 2026                   [Page 3]

Internet-Draft             Network Management              November 2025


  +-------------+  +-------------+ +-------------+ +-------------+
  |     LLM     |  |     LLM     | |     LLM     | |     LLM     |
  | Iteration 1 +-+| Iteration 2 ++| Iteration 3 ++| Iteration 4 ++
  +-----^-------+ |+---^------^--+|+---^-------^-+|+---^-------^-+|
        |         |    |      |   |    |       |  |    |       |  |
        |         | +--+--+   |   | +--+--+    |  | +--+--+    |  |
 <Prompt:Is apple | | KV  | <Yes> | | KV  |  <It> | | KV  |  <Is> |<EOS>
         a fruit?>| |Cache|   ^   | |Cache|    ^  | |Cache|    ^  |  ^
                  | +--^--+   |   | +--^--+    |  | +--^--+    |  |  |
                  |    |      |   |    |       |  |    |       |  |  |
                  +----+------+   +----+-------+  +----+-------+  +--+

 +-----Prefill----+--------------------Decoding----------------------+

                    Figure 1: LLM Inference Process

   Prefill: Processes all tokens in user prompts (Parallelizable,
   compute-bound, requiring high computing power).

   Decoding: Generates output tokens sequentially based on the KV Cache
   from Prefill (Memory-bound, requiring high GPU memory).

2.1.  PD Fusion Architecture

   In PD Fusion, LLM instances are deployed within a single cluster,
   managed by a global scheduler responsible for load balancing, KV
   Cache management, and resource allocation.  Most frameworks adopt
   vLLM[vLLM]'s paged KV Cache mechanism, inspired by OS virtual memory
   management.  This approach stores KV Cache into non-contiguous
   physical blocks across nodes and uses a scheduler to map logical
   blocks to physical memory.  Additionally, prefix-sharing strategies
   are employed to reuse KV Cache for prompts with identical prefixes,
   reducing redundant computations.  Remote KV Cache replication across
   nodes is also required to reduce duplicated computing of KV Cache of
   same tokens.  The architecture is shown in Figure 2.
















Liu & Guo                  Expires 7 May 2026                   [Page 4]

Internet-Draft             Network Management              November 2025


                             Request1/Prompt1
                             Request2/Prompt2
                                     |
                                     |
                        +------------v------------+
                        |                         |
                        |  Scheduler/Controller   |
              Request1  |                         | Request2
            +-----------+  *********************  +----------+
            |           |  *KV Cache Management*  |          |
            |           |  *  Load Balancing   *  |          |
            |           |  *     ... ...       *  |          |
            |           |  *********************  |          |
            |           +-------------------------+          |
            |                                                |
            |                                                |
            |                                                |
       +----v-----+  Remote    +----------+   Remote    +----v-----+
       |  Model   |KVCache copy|  Model   | KVCache copy|  Model   |
       |Instance 1<----------->|Instance 2|<------------>Instance 3|
       +----------+            +----------+             +----------+

                      Figure 2: PD Fusion Architecture

2.2.  PD Disaggregation Architecture

   In PD Disaggregation, Prefill and Decoding are decoupled into
   separate instances to optimize hardware utilization.  After Prefill
   computes the full KV Cache for a prompt, the data is transferred to
   Decoding instances for text generation.  This architecture demands
   efficient coordination between Prefill and Decoding instances, as
   well as reliable high-speed data transmission.  The workflow is
   illustrated in Figure 3.


















Liu & Guo                  Expires 7 May 2026                   [Page 5]

Internet-Draft             Network Management              November 2025


                             Request1/Prompt1
                             Request2/Prompt2
                                     |
                                     |
                        +------------v------------+
                        |                         |
                        |  Scheduler/Controller   |
              Request1  |                         | Request2
            +-----------+  *********************  +----------+
            |           |  *KV Cache Management*  |          |
            |           |  *  Load Balancing   *  |          |
            |           |  *     ... ...       *  |          |
            |           |  *********************  |          |
            |           +-------------------------+          |
            |                                                |
            |                                                |
            |                                                |
       +----v-----+                                     +----v-----+
       |  Model   |                                     |  Model   |
       |          |       Remote KVCache copy           |          |
       | Prefill  <-------------------------------------> Prefill  |
       |Instance 1|                                     |Instance 2|
       +----+-----+                                     +----+-----+
            |KV Cache                                KV Cache|
            |Transfer                                Transfer|
            |                                                |
       +----v-----+                                     +----v-----+
       |  Model   |                                     |  Model   |
       |          |                                     |          |
       |Decoding  |                                     |Decoding  |
       |Instance 1|                                     |Instance 2|
       +----------+                                     +----------+

                  Figure 3: PD Disaggregation Architecture

3.  Inference-related Metrics

   The ultimate goals of an inference system are to maximize system
   goodput which reflects the serving volume of user requests and
   minimize user-perceived latency at low cost.  For both PD Fusion and
   PD Disaggregation architectures, three kind of specific key metrics
   are defined as follows:

   Latency-related Metrics:

      TTFT (Time to First Token): The time taken by the Prefill phase to
      generate the first token.




Liu & Guo                  Expires 7 May 2026                   [Page 6]

Internet-Draft             Network Management              November 2025


      TBT (Time Between Tokens): The interval between consecutive token
      generations in the Decoding phase.

   Throughput-related Metrics:

      TPS (Tokens Per Second): The number of tokens generated per second
      by the inference system.

      RPS (Requests Per Second): The number of requests processed per
      second by the inference system.

   Cost-related Metrics:

      Cost Per Token: How much it costs for inference system to generate
      each token.

4.  Research Question for Service-Oriented Inference Frameworks

   From the user's perspective, end-to-end inference latency generated
   during a complete inference process is a key metric directly
   impacting quality of service and user experience.  From the
   perspective of llm deployment, the most critical challenge is how to
   deploy llm services at the lowest possible cost while meeting basic
   user requirements for inference latency, minimizing idle rates of
   computational resources, and ensuring systems operate at near-
   saturation levels.  Industry commonly adopts decoupled architectures
   (e.g., PD Disaggregation) or other decoupled structures to deploy and
   independently optimize distinct model components on heterogeneous
   clusters.  However, maximizing system throughput and reducing
   inference deployment costs inevitably sacrifices user end-to-end
   inference latency.  In other words, latency, throughput, and cost
   form an impossible trinity.  Synthesizing the needs and objectives of
   both users and service providers, the primary challenge for inference
   systems is: How to maximize system throughput and minimize deployment
   costs while adhering to Service Level Objective (SLO) constraints for
   inference services?

5.  Challenges for Service-Oriented Inference Frameworks

   To address the above key issue, current industry practices (e.g.,
   Mooncake[Mooncake], DeepSeek, vLLM[vLLM], SGLang[SGLang]) employ
   methods such as KV Cache prefix matching, PD Disaggregation
   deployment optimization, efficient KV Cache memory management, and
   flexible load balancing scheduling to enhance resource utilization.
   These approaches also aim to leverage fragmented, low-cost and idle
   resources, thereby increasing user throughput and reducing inference
   service deployment costs.  However, this approach faces several
   significant challenges:



Liu & Guo                  Expires 7 May 2026                   [Page 7]

Internet-Draft             Network Management              November 2025


5.1.  Challenge 1

   Whether using PD Disaggregation or PD Fusion deployment, the industry
   widely adopts KV Cache prefix matching for large-scale inference
   deployment optimization.  This technique aims to reduce redundant
   computation and storage of prefix KV Cache across different inference
   requests, thereby saving computational resources and ultimately
   improving system throughput while lowering deployment costs.  How to
   systematically manage these KV Cache prefixes within the inference
   network—including storage mechanisms, placement strategies,
   scheduling schemes, and replacement policies—remains a critical and
   urgent challenge.

5.2.  Challenge 2

   To optimize the trade-offs between inference deployment cost, service
   performance, and system throughput, implementing more efficient
   resource- and KV Cache-aware mechanisms within the inference
   network's management and control plane, alongside hardware- and
   network-aware load balancing scheduling, presents an extremely
   crucial challenge.

5.3.  Challenge 3

   If a PD Disaggregation deployment approach is adopted, ensuring
   efficient transmission of KV Cache between the separated,
   heterogeneous Prefilling and Decoding clusters becomes a vital
   challenge.  This transmission must be designed to either avoid
   impacting end-to-end inference latency entirely or minimize its
   impact as much as possible.

6.  Network Management Requirements for Service-Oriented Inference
    Frameworks

   To achieve large-scale LLM service deployment, frameworks MUST meet
   the following requirements in both control plane and data plane.

6.1.  Efficient Load Balancing

   In large-scale inference systems, the efficient scheduling and
   routing of inference requests is a critical requirement.  It
   primarily focuses on two aspects: first, the request scheduling must
   be KV Cache-aware, striving to match cached KV data to minimize
   redundant computations and storage; second, the request scheduling
   must account for hardware load and network conditions, ensuring
   sufficient resources to process inference requests as efficiently as
   possible.




Liu & Guo                  Expires 7 May 2026                   [Page 8]

Internet-Draft             Network Management              November 2025


6.2.  KV Cache Management

   In inference architecture systems, storage capacity is typically
   directly correlated with service concurrency.  In large-scale
   inference service deployment scenarios, distributed storage systems
   are often employed to manage KV Cache.  Additionally, multi-level
   storage structures within single nodes, such as HBM-CPU DRAM-SSD, are
   commonly used to cache KV data.  Beyond the placement mechanisms of
   KV Cache in hierarchical distributed storage systems, the disparity
   in communication bandwidth across cards/nodes and different storage
   tiers has become a bottleneck restricting the efficient utilization
   of KV Cache.  How to balance these communication bandwidths as much
   as possible constitutes another critical issue that needs to be
   addressed.

6.3.  KV Cache Transmission

   In large-scale inference service deployment scenarios, the separation
   of Prefill and Decode (PD) phases serves as an effective approach to
   enhance system concurrency and reduce deployment costs.  This PD
   separation involves substantial KV Cache data transmission between
   Prefill instances and Decode instances, while demanding extremely
   high real-time requirements for request servicing.  Ensuring real-
   time KV transmission between PD instances represents another
   critically important area that requires focused attention.

7.  Security Considerations

   TBD.

8.  IANA Considerations

   TBD.

9.  References

9.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

9.2.  Informative References

   [Mooncake] Qin, R., "Mooncake: A KVCache-centric Disaggregated
              Architecture for LLM Serving", 9 July 2024.




Liu & Guo                  Expires 7 May 2026                   [Page 9]

Internet-Draft             Network Management              November 2025


   [SGLang]   Zheng, L., "SGLang: Efficient Execution of Structured
              Language Model Programs", 6 June 2024.

   [vLLM]     Kwon, W., "Efficient Memory Management for Large Language
              Model Serving with PagedAttention", 2023.

Authors' Addresses

   Chang Liu
   China Mobile
   Beijing
   100053
   China
   Email: liuchangjc@chinamobile.com


   Chuyi Guo
   China Mobile
   Beijing
   100053
   China
   Email: guochuyi@chinamobile.com





























Liu & Guo                  Expires 7 May 2026                  [Page 10]


name: draft-gaikwad-llm-benchmarking-methodology
rev: 00
title: Benchmarking Methodology for Large Language Model Serving
time: 2026-01-20T15:05:40Z
group_uri: /api/v1/group/group/1027/
expires: 2026-07-24T15:05:40Z
states: ["/api/v1/doc/state/1/", "/api/v1/doc/state/150/"]
categories: ["Model serving/inference", "Data formats/interop"]
tags: []
fetched_at: 2026-02-27T22:56:07.990768+00:00
abstract:
This document defines benchmarking methodologies for Large Language
   Model (LLM) inference serving systems.  It provides test procedures,
   setup parameters, measurement specifications, and reporting formats
   for evaluating latency, throughput, scheduling, and resource
   management characteristics.  This document is a companion to
   "Benchmarking Terminology for Large Language Model Serving" and
   SHOULD be consulted alongside that terminology document.

full_text:
Network Working Group                                         M. Gaikwad
Internet-Draft                                    Independent Researcher
Intended status: Informational                              January 2026
Expires: 24 July 2026


       Benchmarking Methodology for Large Language Model Serving
             draft-gaikwad-llm-benchmarking-methodology-00

Abstract

   This document defines benchmarking methodologies for Large Language
   Model (LLM) inference serving systems.  It provides test procedures,
   setup parameters, measurement specifications, and reporting formats
   for evaluating latency, throughput, scheduling, and resource
   management characteristics.  This document is a companion to
   "Benchmarking Terminology for Large Language Model Serving" and
   SHOULD be consulted alongside that terminology document.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 5 July 2026.

Copyright Notice

   Copyright (c) 2026 IETF Trust and the persons identified as the
   document authors.  All rights reserved.











Gaikwad                   Expires 24 July 2026                  [Page 1]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   5
   2.  Requirements Language . . . . . . . . . . . . . . . . . . . .   6
   3.  Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . .   6
   4.  Test Setup  . . . . . . . . . . . . . . . . . . . . . . . . .   7
     4.1.  System Under Test Configurations  . . . . . . . . . . . .   7
       4.1.1.  Model Engine Configuration  . . . . . . . . . . . . .   7
       4.1.2.  Application Gateway Configuration . . . . . . . . . .   8
       4.1.3.  Compound System Configuration . . . . . . . . . . . .   9
     4.2.  Load Generator Requirements . . . . . . . . . . . . . . .   9
       4.2.1.  Timing Resolution . . . . . . . . . . . . . . . . . .   9
       4.2.2.  Streaming Support . . . . . . . . . . . . . . . . . .  10
       4.2.3.  Open-Loop Load Generation . . . . . . . . . . . . . .  10
       4.2.4.  Closed-Loop Load Generation . . . . . . . . . . . . .  10
       4.2.5.  Request Isolation . . . . . . . . . . . . . . . . . .  10
       4.2.6.  Output Recording  . . . . . . . . . . . . . . . . . .  10
     4.3.  Reference Workloads . . . . . . . . . . . . . . . . . . .  10
       4.3.1.  Workload Parameters . . . . . . . . . . . . . . . . .  11
       4.3.2.  Standard Workloads  . . . . . . . . . . . . . . . . .  11
       4.3.3.  Workload Reproducibility  . . . . . . . . . . . . . .  13
     4.4.  Tokenization  . . . . . . . . . . . . . . . . . . . . . .  13
       4.4.1.  Tokenizer Specification . . . . . . . . . . . . . . .  13
       4.4.2.  Token Counting Method . . . . . . . . . . . . . . . .  13
       4.4.3.  Special Token Handling  . . . . . . . . . . . . . . .  14
     4.5.  Warm-up Procedures  . . . . . . . . . . . . . . . . . . .  14
       4.5.1.  Warm-up Requirements  . . . . . . . . . . . . . . . .  14
       4.5.2.  Warm-up Verification  . . . . . . . . . . . . . . . .  14
       4.5.3.  Cold Start Measurement  . . . . . . . . . . . . . . .  14
     4.6.  Streaming Protocol  . . . . . . . . . . . . . . . . . . .  15
       4.6.1.  Supported Protocols . . . . . . . . . . . . . . . . .  15
       4.6.2.  Token Chunking  . . . . . . . . . . . . . . . . . . .  15
       4.6.3.  ITL Calculation with Chunked Delivery . . . . . . . .  15
     4.7.  Clock Synchronization . . . . . . . . . . . . . . . . . .  16
       4.7.1.  Single-Machine Testing  . . . . . . . . . . . . . . .  16
       4.7.2.  Distributed Testing . . . . . . . . . . . . . . . . .  16
       4.7.3.  Network Latency Measurement . . . . . . . . . . . . .  16
       4.7.4.  Timestamp Format  . . . . . . . . . . . . . . . . . .  16
     4.8.  Safety and Guardrail Configuration  . . . . . . . . . . .  16



Gaikwad                   Expires 24 July 2026                  [Page 2]

Internet-Draft        LLM Benchmarking Methodology          January 2026


       4.8.1.  Guardrail Disclosure  . . . . . . . . . . . . . . . .  17
       4.8.2.  Production-Representative Testing . . . . . . . . . .  17
   5.  Benchmarking Tests  . . . . . . . . . . . . . . . . . . . . .  17
     5.1.  Time to First Token . . . . . . . . . . . . . . . . . . .  17
       5.1.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  17
       5.1.2.  Setup Parameters  . . . . . . . . . . . . . . . . . .  17
       5.1.3.  Procedure . . . . . . . . . . . . . . . . . . . . . .  18
       5.1.4.  Measurements  . . . . . . . . . . . . . . . . . . . .  19
       5.1.5.  Reporting Format  . . . . . . . . . . . . . . . . . .  20
     5.2.  Output Token Throughput . . . . . . . . . . . . . . . . .  22
       5.2.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  22
       5.2.2.  Setup Parameters  . . . . . . . . . . . . . . . . . .  22
       5.2.3.  Procedure . . . . . . . . . . . . . . . . . . . . . .  23
       5.2.4.  Measurements  . . . . . . . . . . . . . . . . . . . .  24
       5.2.5.  Reporting Format  . . . . . . . . . . . . . . . . . .  24
     5.3.  Throughput-Latency Tradeoff . . . . . . . . . . . . . . .  25
       5.3.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  25
       5.3.2.  Setup Parameters  . . . . . . . . . . . . . . . . . .  25
       5.3.3.  Procedure . . . . . . . . . . . . . . . . . . . . . .  25
       5.3.4.  Measurements  . . . . . . . . . . . . . . . . . . . .  26
       5.3.5.  Reporting Format  . . . . . . . . . . . . . . . . . .  27
     5.4.  Inter-Token Latency Distribution  . . . . . . . . . . . .  27
       5.4.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  27
       5.4.2.  Setup Parameters  . . . . . . . . . . . . . . . . . .  27
       5.4.3.  Procedure . . . . . . . . . . . . . . . . . . . . . .  28
       5.4.4.  Measurements  . . . . . . . . . . . . . . . . . . . .  28
       5.4.5.  Reporting Format  . . . . . . . . . . . . . . . . . .  29
     5.5.  Concurrent Request Capacity . . . . . . . . . . . . . . .  29
       5.5.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  29
       5.5.2.  Setup Parameters  . . . . . . . . . . . . . . . . . .  29
       5.5.3.  Procedure . . . . . . . . . . . . . . . . . . . . . .  30
       5.5.4.  Measurements  . . . . . . . . . . . . . . . . . . . .  30
       5.5.5.  Reporting Format  . . . . . . . . . . . . . . . . . .  31
     5.6.  Scheduling Fairness . . . . . . . . . . . . . . . . . . .  31
       5.6.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  31
       5.6.2.  Setup Parameters  . . . . . . . . . . . . . . . . . .  31
       5.6.3.  Procedure . . . . . . . . . . . . . . . . . . . . . .  32
       5.6.4.  Measurements  . . . . . . . . . . . . . . . . . . . .  32
       5.6.5.  Reporting Format  . . . . . . . . . . . . . . . . . .  32
     5.7.  Prefix Cache Effectiveness  . . . . . . . . . . . . . . .  33
       5.7.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  33
       5.7.2.  Setup Parameters  . . . . . . . . . . . . . . . . . .  33
       5.7.3.  Procedure . . . . . . . . . . . . . . . . . . . . . .  33
       5.7.4.  Measurements  . . . . . . . . . . . . . . . . . . . .  34
       5.7.5.  Reporting Format  . . . . . . . . . . . . . . . . . .  34
     5.8.  Memory Pressure Behavior  . . . . . . . . . . . . . . . .  34
       5.8.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  34
       5.8.2.  Setup Parameters  . . . . . . . . . . . . . . . . . .  34



Gaikwad                   Expires 24 July 2026                  [Page 3]

Internet-Draft        LLM Benchmarking Methodology          January 2026


       5.8.3.  Procedure . . . . . . . . . . . . . . . . . . . . . .  34
       5.8.4.  Measurements  . . . . . . . . . . . . . . . . . . . .  35
       5.8.5.  Reporting Format  . . . . . . . . . . . . . . . . . .  35
     5.9.  Long Context Scaling  . . . . . . . . . . . . . . . . . .  35
       5.9.1.  Objective . . . . . . . . . . . . . . . . . . . . . .  35
       5.9.2.  Setup Parameters  . . . . . . . . . . . . . . . . . .  36
       5.9.3.  Procedure . . . . . . . . . . . . . . . . . . . . . .  36
       5.9.4.  Measurements  . . . . . . . . . . . . . . . . . . . .  36
       5.9.5.  Reporting Format  . . . . . . . . . . . . . . . . . .  36
     5.10. Guardrail Overhead  . . . . . . . . . . . . . . . . . . .  37
       5.10.1.  Objective  . . . . . . . . . . . . . . . . . . . . .  37
       5.10.2.  Setup Parameters . . . . . . . . . . . . . . . . . .  37
       5.10.3.  Procedure  . . . . . . . . . . . . . . . . . . . . .  37
       5.10.4.  Measurements . . . . . . . . . . . . . . . . . . . .  38
       5.10.5.  Reporting Format . . . . . . . . . . . . . . . . . .  38
   6.  Multi-System Comparison Guidelines  . . . . . . . . . . . . .  38
     6.1.  Equivalence Requirements  . . . . . . . . . . . . . . . .  39
     6.2.  Normalization . . . . . . . . . . . . . . . . . . . . . .  39
     6.3.  Statistical Significance  . . . . . . . . . . . . . . . .  39
     6.4.  Fair Comparison Checklist . . . . . . . . . . . . . . . .  39
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .  40
     7.1.  Side-Channel Risks  . . . . . . . . . . . . . . . . . . .  40
     7.2.  Benchmark Gaming  . . . . . . . . . . . . . . . . . . . .  40
     7.3.  Adversarial Workloads . . . . . . . . . . . . . . . . . .  40
     7.4.  Resource Exhaustion . . . . . . . . . . . . . . . . . . .  40
   8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  41
     8.1.  Normative References  . . . . . . . . . . . . . . . . . .  41
     8.2.  Informative References  . . . . . . . . . . . . . . . . .  41
   Appendix A.  Reference Workload Specifications  . . . . . . . . .  42
     A.1.  Synthetic-Uniform Workload  . . . . . . . . . . . . . . .  42
       A.1.1.  Input Specification . . . . . . . . . . . . . . . . .  42
       A.1.2.  Output Specification  . . . . . . . . . . . . . . . .  42
       A.1.3.  Other Parameters  . . . . . . . . . . . . . . . . . .  42
       A.1.4.  Generation Method . . . . . . . . . . . . . . . . . .  42
     A.2.  Synthetic-Skewed Workload . . . . . . . . . . . . . . . .  43
       A.2.1.  Input Specification . . . . . . . . . . . . . . . . .  43
       A.2.2.  Output Specification  . . . . . . . . . . . . . . . .  43
     A.3.  Conversation Workload . . . . . . . . . . . . . . . . . .  44
       A.3.1.  Data Source . . . . . . . . . . . . . . . . . . . . .  44
       A.3.2.  Length Statistics (Reference) . . . . . . . . . . . .  44
     A.4.  Code Completion Workload  . . . . . . . . . . . . . . . .  44
       A.4.1.  Data Source . . . . . . . . . . . . . . . . . . . . .  44
       A.4.2.  Prefix Sharing Pattern  . . . . . . . . . . . . . . .  44
     A.5.  Long Context Workload . . . . . . . . . . . . . . . . . .  45
       A.5.1.  Input Specification . . . . . . . . . . . . . . . . .  45
       A.5.2.  Output Specification  . . . . . . . . . . . . . . . .  45
   Appendix B.  Timing Measurement Reference . . . . . . . . . . . .  45
     B.1.  TTFT Measurement Points . . . . . . . . . . . . . . . . .  45



Gaikwad                   Expires 24 July 2026                  [Page 4]

Internet-Draft        LLM Benchmarking Methodology          January 2026


       B.1.1.  HTTP/SSE Measurement  . . . . . . . . . . . . . . . .  45
       B.1.2.  gRPC Streaming Measurement  . . . . . . . . . . . . .  45
       B.1.3.  Server-Side Measurement . . . . . . . . . . . . . . .  46
     B.2.  ITL Measurement with SSE  . . . . . . . . . . . . . . . .  46
       B.2.1.  Recommended Approach  . . . . . . . . . . . . . . . .  46
     B.3.  Clock Synchronization Methods . . . . . . . . . . . . . .  46
       B.3.1.  NTP Synchronization . . . . . . . . . . . . . . . . .  46
       B.3.2.  PTP Synchronization . . . . . . . . . . . . . . . . .  46
       B.3.3.  Single-Machine Alternative  . . . . . . . . . . . . .  47
   Appendix C.  Reporting Templates  . . . . . . . . . . . . . . . .  47
     C.1.  Minimum Viable Report . . . . . . . . . . . . . . . . . .  47
     C.2.  Full Report Template  . . . . . . . . . . . . . . . . . .  48
   Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  48
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  48

1.  Introduction

   This document provides benchmarking methodologies for Large Language
   Model inference serving systems.  It defines test procedures,
   measurement specifications, and reporting formats that enable
   meaningful performance comparison.

   A companion document, "Benchmarking Terminology for Large Language
   Model Serving" [LLM-TERMS], defines the metrics referenced in this
   methodology.  That terminology document SHOULD be consulted before
   attempting to make use of this document.

   LLM serving systems present unique benchmarking challenges:

   Streaming responses:  Output tokens arrive incrementally over seconds
      or minutes, requiring timing measurements at multiple points
      within a single request.

   Phase separation:  The prefill phase (processing input) and decode
      phase (generating output) have distinct computational profiles and
      optimization targets.

   Memory-bound decoding:  The decode phase is limited by memory
      bandwidth rather than compute, creating different bottlenecks than
      traditional neural network inference.

   Dynamic batching:  Continuous batching systems interleave requests,
      causing per-request performance to depend on concurrent load.

   Context-dependent performance:  Request latency varies with input
      length, output length, and cache state, making workload
      specification critical.




Gaikwad                   Expires 24 July 2026                  [Page 5]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   These characteristics require methodology beyond traditional
   throughput and latency measurement.  This document addresses these
   challenges by specifying:

   *  Test configurations for different system boundaries

   *  Reference workloads with defined characteristics

   *  Measurement procedures for streaming responses

   *  Statistical requirements for reliable percentile estimation

   *  Reporting formats enabling meaningful comparison

   This document does not specify acceptance thresholds or recommend
   particular systems.  It provides methodology for fair comparison.

2.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

   An implementation is not compliant if it fails to satisfy one or more
   of the MUST requirements for a given test.  An implementation that
   satisfies all the MUST and all the SHOULD requirements for a test is
   said to be "unconditionally compliant" for that test; one that
   satisfies all the MUST requirements but not all the SHOULD
   requirements is said to be "conditionally compliant."

3.  Scope

   This document covers benchmarking methodology for transformer-based
   autoregressive language models deployed as network services.  The
   methodology applies to:

   *  Inference engines executing model forward passes

   *  Application gateways providing API endpoints

   *  Compound systems with retrieval or tool execution

   The following are out of scope:

   *  Model training or fine-tuning performance




Gaikwad                   Expires 24 July 2026                  [Page 6]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   *  Model quality or accuracy evaluation

   *  Non-autoregressive models (diffusion, encoder-only)

   *  Edge deployment or on-device inference

   *  Specific vendor implementations or products

4.  Test Setup

4.1.  System Under Test Configurations

   The System Under Test (SUT) boundary MUST be declared before
   benchmarking.  This document defines three standard configurations.

4.1.1.  Model Engine Configuration

   The Model Engine configuration measures raw inference capability.

                   +------------------+
                   |   Load Generator |
                   +--------+---------+
                            |
                 Internal API (gRPC/HTTP)
                            |
                   +--------v---------+
                   |   Model Engine   |
                   |  (SUT Boundary)  |
                   +------------------+

                    Figure 1: Model Engine Configuration

   Included components:

   *  Model weights and inference runtime

   *  Batching and scheduling logic

   *  KV cache management

   *  Tensor operations and kernels

   Excluded components:

   *  External network transport

   *  Authentication and authorization




Gaikwad                   Expires 24 July 2026                  [Page 7]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   *  Rate limiting

   *  Input/output safety filtering

   *  Load balancing

   This configuration is appropriate for comparing inference engines
   (vLLM, TensorRT-LLM, SGLang) independent of deployment stack.

4.1.2.  Application Gateway Configuration

   The Application Gateway configuration measures user-observable API
   performance.

                   +------------------+
                   |   Load Generator |
                   +--------+---------+
                            |
                 External API (HTTPS)
                            |
                   +--------v---------+
                   | Application GW   |
                   |  (SUT Boundary)  |
                   |  +------------+  |
                   |  |   Engine   |  |
                   |  +------------+  |
                   +------------------+

                Figure 2: Application Gateway Configuration

   Included components (in addition to Model Engine):

   *  TLS termination

   *  Authentication and session management

   *  Rate limiting and quota enforcement

   *  Input validation and output filtering

   *  Safety guardrails

   This configuration is appropriate for comparing API providers or
   evaluating production deployment performance.







Gaikwad                   Expires 24 July 2026                  [Page 8]

Internet-Draft        LLM Benchmarking Methodology          January 2026


4.1.3.  Compound System Configuration

   The Compound System configuration measures end-to-end task completion
   for agentic or retrieval-augmented workloads.

                   +------------------+
                   |   Task Driver    |
                   +--------+---------+
                            |
                   +--------v---------+
                   |  Compound System |
                   |  (SUT Boundary)  |
                   |  +------------+  |
                   |  | Retrieval  |  |
                   |  +------------+  |
                   |  +------------+  |
                   |  |   Tools    |  |
                   |  +------------+  |
                   |  +------------+  |
                   |  |  Gateway   |  |
                   |  +------------+  |
                   +------------------+

                  Figure 3: Compound System Configuration

   Included components (in addition to Application Gateway):

   *  Retrieval pipeline (embedding, vector search, reranking)

   *  Tool execution environment

   *  Orchestration logic

   *  Multi-turn conversation state

   This configuration is appropriate for evaluating RAG systems or
   agentic applications.

4.2.  Load Generator Requirements

   The load generator produces requests and measures responses.  It MUST
   satisfy the following requirements.

4.2.1.  Timing Resolution

   The load generator MUST measure time with resolution of 1 millisecond
   or better.  Microsecond resolution is RECOMMENDED for ITL
   measurement.



Gaikwad                   Expires 24 July 2026                  [Page 9]

Internet-Draft        LLM Benchmarking Methodology          January 2026


4.2.2.  Streaming Support

   The load generator MUST support streaming response protocols (SSE,
   WebSocket, or gRPC streaming).  It MUST record the arrival time of
   each token or chunk, not only the complete response.

4.2.3.  Open-Loop Load Generation

   The load generator MUST support open-loop load generation where
   request arrival times are determined by a specified distribution
   independent of response times.  Poisson arrivals MUST be supported.
   Uniform and bursty arrival patterns are RECOMMENDED.

4.2.4.  Closed-Loop Load Generation

   The load generator MUST support closed-loop load generation where a
   fixed number of concurrent requests are maintained.  When a request
   completes, a new request is immediately submitted.

4.2.5.  Request Isolation

   The load generator MUST NOT allow slow responses to delay the
   submission of subsequent requests in open-loop mode.  Asynchronous or
   multi-threaded implementation is REQUIRED.

4.2.6.  Output Recording

   The load generator MUST record for each request:

   *  Request submission timestamp

   *  First token arrival timestamp

   *  Each subsequent token arrival timestamp

   *  Final token arrival timestamp

   *  Total input token count

   *  Total output token count

   *  Request success/failure status

4.3.  Reference Workloads

   Workload specification is critical for reproducible benchmarking.
   This document defines reference workloads with fixed characteristics.
   Testers MAY use custom workloads but MUST fully specify them.



Gaikwad                   Expires 24 July 2026                 [Page 10]

Internet-Draft        LLM Benchmarking Methodology          January 2026


4.3.1.  Workload Parameters

   Each workload MUST specify:

   Input length distribution:  Distribution type (fixed, uniform,
      normal, empirical), parameters (mean, std, min, max, or
      histogram), and unit (tokens using specified tokenizer).

   Output length distribution:  Distribution type (fixed, uniform,
      normal, empirical), parameters (mean, std, min, max, or
      histogram), control method (max_tokens parameter, stop sequence,
      or both), and unit (tokens using specified tokenizer).

   Content characteristics:  Domain (general, code, conversation,
      instruction), language (English, multilingual, code languages),
      and system prompt presence and typical length.

   Prefix sharing:  Fraction of requests sharing common prefix and
      shared prefix length distribution.

4.3.2.  Standard Workloads

   This document defines five standard workloads.  Full specifications
   appear in Appendix A.

4.3.2.1.  Synthetic-Uniform

   Purpose: Baseline comparison with controlled variability

   *  Input length: Uniform(128, 512) tokens

   *  Output length: Uniform(64, 256) tokens

   *  Content: Random token sequences (no semantic meaning)

   *  Prefix sharing: None

   This workload isolates inference performance from content effects.
   It is REQUIRED for Model Engine benchmarking.

4.3.2.2.  Synthetic-Skewed

   Purpose: Test behavior under realistic length variation

   *  Input length: Log-normal(mu=5.5, sigma=1.0) tokens, capped at 4096

   *  Output length: Log-normal(mu=4.5, sigma=1.2) tokens, capped at
      2048



Gaikwad                   Expires 24 July 2026                 [Page 11]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   *  Content: Random token sequences

   *  Prefix sharing: None

   This workload tests scheduling fairness with high length variance.

4.3.2.3.  Conversation

   Purpose: Simulate interactive chat workloads

   *  Input length: Empirical distribution from ShareGPT dataset

   *  Output length: Empirical distribution from ShareGPT dataset

   *  Content: Natural language conversation

   *  Prefix sharing: 50% share 200-token system prompt

   This workload is RECOMMENDED for Application Gateway benchmarking.

4.3.2.4.  Code Completion

   Purpose: Simulate coding assistant workloads

   *  Input length: Empirical from code completion datasets

   *  Output length: Log-normal(mu=4.0, sigma=1.5) tokens

   *  Content: Source code in Python, JavaScript, TypeScript

   *  Prefix sharing: 80% share repository context prefix

   This workload tests prefix caching effectiveness.

4.3.2.5.  Long Context

   Purpose: Test long-context behavior

   *  Input length: Uniform(8192, 32768) tokens

   *  Output length: Fixed at 256 tokens

   *  Content: Document + question format

   *  Prefix sharing: None

   This workload is REQUIRED for Long Context Scaling tests.




Gaikwad                   Expires 24 July 2026                 [Page 12]

Internet-Draft        LLM Benchmarking Methodology          January 2026


4.3.3.  Workload Reproducibility

   For reproducible benchmarking:

   *  Testers MUST use deterministic random seeds for workload
      generation.  The seed MUST be reported.

   *  Testers SHOULD publish the exact request sequences used, or
      provide generation code with fixed seeds.

   *  When using dataset-derived workloads (ShareGPT, HumanEval),
      testers MUST specify the dataset version, subset selection method,
      and any preprocessing applied.

4.4.  Tokenization

   Token counts depend on the tokenizer.  Different tokenizers produce
   different counts for identical text, making cross-system comparison
   challenging.

4.4.1.  Tokenizer Specification

   The test report MUST specify:

   *  Tokenizer name and version (e.g., "cl100k_base", "Llama-3
      tokenizer")

   *  Vocabulary size

   *  Source (Hugging Face model ID, tiktoken name, or custom)

4.4.2.  Token Counting Method

   For cross-system comparison where systems use different tokenizers:

   Option A - Native tokenizer:  Count tokens using each system's native
      tokenizer.  Report results separately with tokenizer identified.
      This method reflects actual system behavior but complicates
      comparison.

   Option B - Reference tokenizer:  Count tokens using a declared
      reference tokenizer for all systems.  This enables direct
      comparison but may not reflect actual system token counts.

   The test report MUST declare which option is used.  Option B with
   cl100k_base (GPT-4 tokenizer) as reference is RECOMMENDED for cross-
   system comparison.




Gaikwad                   Expires 24 July 2026                 [Page 13]

Internet-Draft        LLM Benchmarking Methodology          January 2026


4.4.3.  Special Token Handling

   The test report MUST specify handling of:

   *  BOS/EOS tokens (included or excluded from counts)

   *  System prompt tokens (counted separately or included)

   *  Tool/function call formatting tokens

4.5.  Warm-up Procedures

   LLM serving systems require warm-up before reaching steady-state
   performance.  Warm-up effects include JIT compilation, memory
   allocator initialization, prefix cache population, and batch size
   ramp-up.

4.5.1.  Warm-up Requirements

   Before measurement begins, testers MUST:

   1.  Load the model fully into accelerator memory

   2.  Process at least 100 requests or 10,000 output tokens, whichever
       is greater

   3.  Wait for request queue to drain completely

   4.  If prefix caching is enabled and being tested, populate the cache
       with representative prefixes

4.5.2.  Warm-up Verification

   Testers SHOULD verify warm-up completion by:

   1.  Measuring latency for a probe request before and after warm-up

   2.  Confirming latency stabilization (less than 10% variation across
       consecutive probe requests)

4.5.3.  Cold Start Measurement

   When cold start performance is being measured (Model Load Time, Cold
   Start Latency), warm-up MUST be skipped.  The test report MUST
   clearly indicate cold start measurement.






Gaikwad                   Expires 24 July 2026                 [Page 14]

Internet-Draft        LLM Benchmarking Methodology          January 2026


4.6.  Streaming Protocol

   LLM serving systems deliver tokens via streaming protocols.  The
   choice of protocol affects timing measurement.

4.6.1.  Supported Protocols

   This methodology supports:

   Server-Sent Events (SSE):  HTTP-based streaming.  Each event contains
      one or more tokens.  RECOMMENDED for Application Gateway testing.

   WebSocket:  Bidirectional streaming.  Each message contains one or
      more tokens.

   gRPC streaming:  Binary streaming protocol.  Each message contains
      one or more tokens.  RECOMMENDED for Model Engine testing.

4.6.2.  Token Chunking

   Streaming protocols may deliver multiple tokens per chunk due to
   batching or network buffering.  The test report MUST specify:

   *  Protocol used

   *  Whether each chunk contains exactly one token or potentially
      multiple tokens

   *  How multi-token chunks are handled for ITL calculation

4.6.3.  ITL Calculation with Chunked Delivery

   When chunks contain multiple tokens:

   Option A - Chunk timing:  Measure inter-chunk latency.  Report as
      "Time Between Chunks" rather than ITL.  Note chunk size
      distribution.

   Option B - Distributed timing:  Distribute chunk arrival time across
      tokens.  If a chunk with N tokens arrives at time T, assign
      arrival time T to all N tokens.  This understates ITL variance.

   Option C - Server-side timing:  Use server-reported per-token
      timestamps if available.  This measures ITL independent of network
      effects.

   The test report MUST declare which option is used.  Option C is
   RECOMMENDED when available.



Gaikwad                   Expires 24 July 2026                 [Page 15]

Internet-Draft        LLM Benchmarking Methodology          January 2026


4.7.  Clock Synchronization

   Accurate timing requires synchronized clocks between load generator
   and SUT, and between distributed SUT components.

4.7.1.  Single-Machine Testing

   When load generator and SUT run on the same machine, clock
   synchronization is inherent.  This configuration is RECOMMENDED for
   Model Engine testing.

4.7.2.  Distributed Testing

   When load generator and SUT are on different machines:

   *  NTP synchronization MUST achieve accuracy of 10ms or better

   *  PTP synchronization SHOULD be used when sub-millisecond accuracy
      is required

   *  The test report MUST state the synchronization method and
      estimated accuracy

4.7.3.  Network Latency Measurement

   For Application Gateway testing where network latency is significant:

   *  Testers SHOULD measure and report network RTT separately

   *  Testers MAY subtract estimated network latency from TTFT to
      isolate server-side processing time

   *  Any latency adjustment MUST be documented in the test report

4.7.4.  Timestamp Format

   All timestamps MUST be recorded in a format with at least millisecond
   precision.  ISO 8601 with milliseconds (YYYY-MM-DDTHH:MM:SS.sssZ) or
   Unix epoch with milliseconds is RECOMMENDED.

4.8.  Safety and Guardrail Configuration

   Production LLM deployments include safety systems that affect
   performance.  Benchmarking MUST account for these systems.







Gaikwad                   Expires 24 July 2026                 [Page 16]

Internet-Draft        LLM Benchmarking Methodology          January 2026


4.8.1.  Guardrail Disclosure

   The test report MUST disclose:

   *  Whether input content filtering is enabled

   *  Whether output content filtering is enabled

   *  Names of safety systems if known (e.g., "Llama Guard")

   *  Whether any requests were refused during testing

4.8.2.  Production-Representative Testing

   For Application Gateway benchmarking intended to represent production
   performance:

   *  Safety systems SHOULD be enabled in their default configuration

   *  The test report MUST note if safety systems are disabled

   *  Testers SHOULD run comparative tests with safety enabled and
      disabled to quantify overhead

5.  Benchmarking Tests

   This section defines benchmarking tests.  Each test includes:
   objective, setup parameters, procedure, measurements, and reporting
   format.

5.1.  Time to First Token

5.1.1.  Objective

   To determine the latency from request submission to first token
   receipt under varying load conditions.  TTFT measures perceived
   responsiveness for interactive applications.

5.1.2.  Setup Parameters

   The following parameters MUST be defined:

5.1.2.1.  Workload Parameters

   Workload:  One of the standard workloads (Section 4.3.2) or a fully
      specified custom workload.

   Request count:  Total number of requests to execute.  MUST be at



Gaikwad                   Expires 24 July 2026                 [Page 17]

Internet-Draft        LLM Benchmarking Methodology          January 2026


      least 1000 for P99 measurement, 10000 for P99.9.

5.1.2.2.  Load Parameters

   Load model:  Open-loop or closed-loop.

   For open-loop:

   Arrival rate:  Requests per second.

   Arrival distribution:  Poisson (REQUIRED), uniform, or bursty.

   For closed-loop:

   Concurrency:  Number of concurrent requests maintained.

5.1.2.3.  System Parameters

   SUT configuration:  Model Engine, Application Gateway, or Compound
      System.

   Model identifier:  Model name, version, and quantization if
      applicable.

   Hardware:  Accelerator type, count, and memory.

   Prefix caching:  Enabled or disabled.

5.1.3.  Procedure

   1.  Configure the SUT with specified parameters.

   2.  Complete warm-up procedure (Section 4.5).

   3.  Begin load generation at the specified arrival rate or
       concurrency.

   4.  For each request:

       a.  Record submission timestamp (T_submit)

       b.  Record first token arrival timestamp (T_first)

       c.  Calculate TTFT = T_first - T_submit

       d.  Record input token count

   5.  Continue until request count is reached.



Gaikwad                   Expires 24 July 2026                 [Page 18]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   6.  Compute distribution statistics.

5.1.3.1.  First Token Definition

   The first token is defined as the first content token received,
   excluding:

   *  Empty tokens or whitespace-only tokens

   *  Protocol overhead (SSE event markers, JSON framing)

   *  Metadata tokens (token IDs, logprobs if requested separately)

   If the system emits non-content tokens before content, the test
   report MUST note this and specify whether TTFT measures time to any
   token or time to first content token.

5.1.4.  Measurements

5.1.4.1.  Primary Measurements

   TTFT Percentiles:  P50, P90, P95, P99, and P99.9 of TTFT
      distribution.  All percentiles MUST be reported.

   TTFT Mean:  Arithmetic mean of TTFT values.

   TTFT Minimum:  Smallest TTFT observed.

   TTFT Maximum:  Largest TTFT observed.

5.1.4.2.  Conditional Measurements

   TTFT by input length:  When workload has variable input length,
      report TTFT percentiles bucketed by input length ranges.
      RECOMMENDED buckets: [0-256), [256-512), [512-1024), [1024-2048),
      [2048-4096), [4096+) tokens.

   Queue wait time:  If measurable (server instrumentation), report the
      queue wait component of TTFT separately.

   Prefill latency:  If measurable, report the prefill computation
      component of TTFT separately.

5.1.4.3.  Statistical Requirements

   For P99 accuracy within 10% relative error at 95% confidence, at
   least 1000 samples are required.  For P99.9, at least 10000 samples.
   The test report MUST state the sample count.



Gaikwad                   Expires 24 July 2026                 [Page 19]

Internet-Draft        LLM Benchmarking Methodology          January 2026


5.1.5.  Reporting Format

   The test report MUST include:

5.1.5.1.  Configuration Summary

   *  SUT configuration and boundary

   *  Model identifier and hardware

   *  Workload name or full specification

   *  Load model and parameters

   *  Request count and test duration

   *  Warm-up procedure followed

   *  Prefix caching state

   *  Guardrail configuration

5.1.5.2.  Results Table

   The results SHOULD be reported in tabular format:


























Gaikwad                   Expires 24 July 2026                 [Page 20]

Internet-Draft        LLM Benchmarking Methodology          January 2026


                          +============+=========+
                          | Metric     | Value   |
                          +============+=========+
                          | Requests   | 10000   |
                          +------------+---------+
                          | TTFT P50   | 127 ms  |
                          +------------+---------+
                          | TTFT P90   | 245 ms  |
                          +------------+---------+
                          | TTFT P95   | 312 ms  |
                          +------------+---------+
                          | TTFT P99   | 524 ms  |
                          +------------+---------+
                          | TTFT P99.9 | 891 ms  |
                          +------------+---------+
                          | TTFT Mean  | 156 ms  |
                          +------------+---------+
                          | TTFT Min   | 89 ms   |
                          +------------+---------+
                          | TTFT Max   | 1243 ms |
                          +------------+---------+

                           Table 1: TTFT Results
                                  Example

5.1.5.3.  TTFT by Input Length

   If applicable:

             +==============+==========+==========+==========+
             | Input Tokens | P50 (ms) | P95 (ms) | P99 (ms) |
             +==============+==========+==========+==========+
             | 0-256        | 95       | 198      | 312      |
             +--------------+----------+----------+----------+
             | 256-512      | 142      | 287      | 445      |
             +--------------+----------+----------+----------+
             | 512-1024     | 198      | 412      | 623      |
             +--------------+----------+----------+----------+
             | 1024-2048    | 312      | 587      | 891      |
             +--------------+----------+----------+----------+
             | 2048+        | 523      | 912      | 1243     |
             +--------------+----------+----------+----------+

                   Table 2: TTFT by Input Length Example







Gaikwad                   Expires 24 July 2026                 [Page 21]

Internet-Draft        LLM Benchmarking Methodology          January 2026


5.1.5.4.  Distribution Visualization

   Testers SHOULD include a histogram or CDF plot of the TTFT
   distribution.

5.2.  Output Token Throughput

5.2.1.  Objective

   To determine the maximum rate at which the SUT can generate output
   tokens while maintaining acceptable latency.  This test measures
   system capacity under load.

5.2.2.  Setup Parameters

   The following parameters MUST be defined:

5.2.2.1.  Workload Parameters

   Workload:  One of the standard workloads or fully specified custom
      workload.

   Test duration:  Minimum 60 seconds.  RECOMMENDED 300 seconds for
      stable measurement.

5.2.2.2.  Load Parameters

   Load model:  Open-loop or closed-loop.

   For open-loop:

   Arrival rate range:  Minimum and maximum request rates to test.

   Rate increment:  Step size for iterative search.

   For closed-loop:

   Concurrency range:  Minimum and maximum concurrent requests.

   Concurrency increment:  Step size for iterative search.

5.2.2.3.  Latency Constraint (Optional)

   TTFT SLO:  Maximum acceptable P99 TTFT.

   TPOT SLO:  Maximum acceptable P99 TPOT.





Gaikwad                   Expires 24 July 2026                 [Page 22]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   When specified, throughput is measured as the maximum rate achieving
   these SLOs.

5.2.3.  Procedure

   This test employs an iterative search to find maximum throughput.

   1.  Configure the SUT with specified parameters.

   2.  Complete warm-up procedure.

   3.  For each load level (arrival rate or concurrency):

       a.  Run load for the specified test duration.

       b.  Record all request timings.

       c.  Compute throughput as total output tokens divided by test
           duration.

       d.  Compute TTFT and TPOT percentiles.

       e.  If latency constraint specified, check SLO compliance.

   4.  Use binary search to find maximum throughput:

       a.  If no latency constraint: find load level where queue grows
           unboundedly (system saturation).

       b.  If latency constraint: find highest load level meeting SLO.

   5.  Report throughput at the maximum sustainable load level.

5.2.3.1.  Saturation Detection

   System saturation is detected when:

   *  Queue depth grows continuously during test duration, OR

   *  Request completion rate is less than 90% of arrival rate, OR

   *  P99 latency exceeds 10x the P50 latency at lower load

5.2.3.2.  Steady State Verification

   At each load level, verify steady state by:

   *  Confirming queue depth is stable (not growing)



Gaikwad                   Expires 24 July 2026                 [Page 23]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   *  Confirming throughput is stable across test duration

   *  Excluding initial ramp-up period (first 10% of duration)

5.2.4.  Measurements

5.2.4.1.  Primary Measurements

   Maximum output token throughput:  Output tokens per second at maximum
      sustainable load.  Report with or without latency constraint as
      specified.

   Request throughput:  Requests completed per second at maximum load.

   Input token throughput:  Input tokens processed per second (measures
      prefill capacity).

5.2.4.2.  Efficiency Measurements

   Tokens per GPU-second:  Output tokens per second divided by GPU
      count.  Enables comparison across different hardware
      configurations.

   Batch utilization:  If measurable, report average batch size divided
      by maximum batch size.

5.2.4.3.  Latency at Maximum Throughput

   At the maximum sustainable load level, report:

   *  TTFT P50, P95, P99

   *  TPOT P50, P95, P99

   *  End-to-end latency P50, P95, P99

5.2.5.  Reporting Format

5.2.5.1.  Summary Results

                +========================+===============+
                | Metric                 | Value         |
                +========================+===============+
                | Max Output Throughput  | 2847 tok/s    |
                +------------------------+---------------+
                | Max Request Throughput | 18.2 req/s    |
                +------------------------+---------------+
                | Max Input Throughput   | 5123 tok/s    |



Gaikwad                   Expires 24 July 2026                 [Page 24]

Internet-Draft        LLM Benchmarking Methodology          January 2026


                +------------------------+---------------+
                | Sustainable Load       | 20 req/s      |
                +------------------------+---------------+
                | Tokens per GPU-second  | 356 tok/s/GPU |
                +------------------------+---------------+

                   Table 3: Throughput Summary Example

5.2.5.2.  Latency at Maximum Throughput

                +============+========+========+=========+
                | Metric     | P50    | P95    | P99     |
                +============+========+========+=========+
                | TTFT       | 312 ms | 687 ms | 1124 ms |
                +------------+--------+--------+---------+
                | TPOT       | 42 ms  | 78 ms  | 134 ms  |
                +------------+--------+--------+---------+
                | End-to-End | 6.2 s  | 11.4 s | 18.7 s  |
                +------------+--------+--------+---------+

                  Table 4: Latency at Maximum Throughput
                                 Example

5.3.  Throughput-Latency Tradeoff

5.3.1.  Objective

   To characterize the relationship between throughput and latency
   across the operating range of the SUT.  This test produces a
   throughput-latency curve revealing system behavior better than point
   measurements.

5.3.2.  Setup Parameters

   Workload:  One of the standard workloads or fully specified custom
      workload.

   Test duration per point:  Minimum 60 seconds per load level.

   Load levels:  At least 10 load levels spanning from low load (10% of
      estimated capacity) to saturation.

   Load model:  Open-loop is REQUIRED for this test.  Closed-loop cannot
      reveal behavior beyond capacity.

5.3.3.  Procedure





Gaikwad                   Expires 24 July 2026                 [Page 25]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   1.  Estimate system capacity using a preliminary throughput test or
       published specifications.

   2.  Define load levels: 10%, 20%, 30%, ..., 100%, 110%, 120% of
       estimated capacity.

   3.  For each load level in ascending order:

       a.  Run load for specified duration.

       b.  Record all request timings.

       c.  Compute achieved throughput (may differ from offered load at
           saturation).

       d.  Compute latency percentiles.

   4.  Plot throughput vs latency curves.

5.3.4.  Measurements

   For each load level, record:

   *  Offered load (request rate)

   *  Achieved throughput (output tokens per second)

   *  TTFT: P50, P95, P99

   *  TPOT: P50, P95, P99

   *  End-to-end latency: P50, P95, P99

   *  Request success rate

   *  Queue growth indicator (stable/growing)

   Derived metrics:

   Optimal operating point:  Load level achieving highest throughput
      while meeting specified SLO.

   Knee point:  Load level where P99 latency exceeds 2x the minimum P99
      latency observed.

   Saturation point:  Load level where achieved throughput first
      decreases from previous level.




Gaikwad                   Expires 24 July 2026                 [Page 26]

Internet-Draft        LLM Benchmarking Methodology          January 2026


5.3.5.  Reporting Format

    +===============+==========+======+======+======+======+=========+
    | Offered (r/s) | Achieved | TTFT | TTFT | TPOT | TPOT | Success |
    |               | (tok/s)  | P50  | P99  | P50  | P99  |         |
    +===============+==========+======+======+======+======+=========+
    | 2             | 284      | 95   | 142  | 32   | 41   | 100%    |
    +---------------+----------+------+------+------+------+---------+
    | 6             | 852      | 102  | 178  | 34   | 48   | 100%    |
    +---------------+----------+------+------+------+------+---------+
    | 10            | 1420     | 128  | 267  | 38   | 62   | 100%    |
    +---------------+----------+------+------+------+------+---------+
    | 14            | 1988     | 198  | 512  | 48   | 98   | 100%    |
    +---------------+----------+------+------+------+------+---------+
    | 18            | 2534     | 378  | 1234 | 72   | 198  | 99.8%   |
    +---------------+----------+------+------+------+------+---------+
    | 22            | 2712     | 823  | 3456 | 142  | 523  | 94.1%   |
    +---------------+----------+------+------+------+------+---------+

                Table 5: Throughput-Latency Table Example

   Knee point: 14 req/s (TTFT P99 exceeds 2x minimum)

   Saturation point: 22 req/s (throughput peaks)

5.4.  Inter-Token Latency Distribution

5.4.1.  Objective

   To characterize the variability of token delivery during the decode
   phase.  ITL distribution determines streaming smoothness experienced
   by users.

5.4.2.  Setup Parameters

   Workload:  Synthetic-Uniform or Conversation workload RECOMMENDED.

   Minimum output length:  Requests MUST generate at least 50 output
      tokens to provide meaningful ITL samples.

   Request count:  At least 100 requests for per-request statistics,
      yielding 5000+ ITL samples.

   Load level:  Specify as percentage of maximum throughput.  Multiple
      load levels RECOMMENDED: 25%, 50%, 75%, 90% of saturation.

   Measurement method:  Specify per Section 4.6.3 (chunk timing,
      distributed timing, or server-side timing).



Gaikwad                   Expires 24 July 2026                 [Page 27]

Internet-Draft        LLM Benchmarking Methodology          January 2026


5.4.3.  Procedure

   1.  Configure SUT and complete warm-up.

   2.  For each load level:

       a.  Generate requests at specified load.

       b.  For each request, record arrival time of each token after the
           first.

       c.  Calculate ITL_i = T(token_i) - T(token_{i-1}) for each
           consecutive token pair.

       d.  Aggregate ITL samples across all requests.

       e.  Calculate per-request jitter (standard deviation of ITL
           within each request).

       f.  Record maximum pause duration per request.

   The interval between request submission and first token (TTFT) MUST
   NOT be included in ITL calculation.

5.4.4.  Measurements

5.4.4.1.  Aggregate ITL Statistics

   ITL Percentiles:  P50, P90, P95, P99, P99.9 across all ITL samples.

   ITL Mean:  Arithmetic mean of all ITL samples.

   ITL Standard Deviation:  Standard deviation across all samples.

5.4.4.2.  Per-Request Statistics

   Jitter Distribution:  P50, P95, P99 of per-request standard
      deviation.

   Maximum Pause Distribution:  P50, P95, P99 of per-request maximum
      ITL.

5.4.4.3.  Distribution Shape

   Modality:  Whether ITL distribution is unimodal or multimodal.
      Multimodal distributions indicate distinct operating regimes
      (e.g., batching effects).




Gaikwad                   Expires 24 July 2026                 [Page 28]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   Tail behavior:  Characterize tail (exponential, heavy-tailed).
      Report the ratio P99/P50 as a tail heaviness indicator.

5.4.5.  Reporting Format

                        +===============+========+
                        | Metric        | Value  |
                        +===============+========+
                        | ITL Samples   | 15234  |
                        +---------------+--------+
                        | ITL P50       | 38 ms  |
                        +---------------+--------+
                        | ITL P90       | 52 ms  |
                        +---------------+--------+
                        | ITL P95       | 67 ms  |
                        +---------------+--------+
                        | ITL P99       | 124 ms |
                        +---------------+--------+
                        | ITL P99.9     | 312 ms |
                        +---------------+--------+
                        | ITL Mean      | 42 ms  |
                        +---------------+--------+
                        | ITL Std Dev   | 28 ms  |
                        +---------------+--------+
                        | P99/P50 Ratio | 3.26   |
                        +---------------+--------+

                           Table 6: ITL Results
                                 Example

5.5.  Concurrent Request Capacity

5.5.1.  Objective

   To determine the maximum number of concurrent requests the SUT can
   maintain while meeting latency objectives.  This test measures memory
   capacity and scheduling limits.

5.5.2.  Setup Parameters

   Workload:  Synthetic-Uniform RECOMMENDED for controlled testing.

   Fixed output length:  Use fixed output length (e.g., 256 tokens) to
      ensure all requests have similar duration.

   Initial concurrency:  Starting number of concurrent requests (e.g.,
      8).




Gaikwad                   Expires 24 July 2026                 [Page 29]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   Maximum concurrency:  Upper bound for search (e.g., 512).

   Success criteria:  Request completion rate >= 99%, TTFT P99 <=
      specified threshold, and no out-of-memory errors.

5.5.3.  Procedure

   This test employs binary search to find maximum concurrent capacity.

   1.  Configure SUT and complete warm-up.

   2.  Set concurrency = initial concurrency.

   3.  For each concurrency level:

       a.  Submit [concurrency] requests simultaneously.

       b.  Maintain concurrency: when a request completes, immediately
           submit a replacement.

       c.  Run for at least 60 seconds or 100 request completions per
           slot, whichever is longer.

       d.  Record completion rate, latency percentiles, and any errors.

       e.  Check success criteria.

   4.  Binary search:

       a.  If success criteria met: increase concurrency toward maximum.

       b.  If success criteria not met: decrease concurrency.

       c.  Continue until convergence.

   5.  Report maximum concurrency meeting success criteria.

5.5.4.  Measurements

   Maximum concurrent requests:  Highest concurrency meeting success
      criteria.

   Achieved throughput at maximum:  Output tokens per second at maximum
      concurrency.

   Tokens in flight at maximum:  Approximate total tokens (input +
      output so far) across all concurrent requests.




Gaikwad                   Expires 24 July 2026                 [Page 30]

Internet-Draft        LLM Benchmarking Methodology          January 2026


5.5.5.  Reporting Format

   +=============+============+==========+==========+========+========+
   | Concurrency | Completion | TTFT P99 | TPOT P99 | Errors | Status |
   +=============+============+==========+==========+========+========+
   | 8           | 100%       | 142 ms   | 38 ms    | 0      | Pass   |
   +-------------+------------+----------+----------+--------+--------+
   | 16          | 100%       | 178 ms   | 42 ms    | 0      | Pass   |
   +-------------+------------+----------+----------+--------+--------+
   | 32          | 100%       | 267 ms   | 52 ms    | 0      | Pass   |
   +-------------+------------+----------+----------+--------+--------+
   | 64          | 99.7%      | 523 ms   | 78 ms    | 0      | Pass   |
   +-------------+------------+----------+----------+--------+--------+
   | 128         | 97.2%      | 1234 ms  | 156 ms   | 3      | Fail   |
   +-------------+------------+----------+----------+--------+--------+

                 Table 7: Capacity Search Results Example

   Maximum concurrent requests meeting criteria: 64

5.6.  Scheduling Fairness

5.6.1.  Objective

   To evaluate how equitably the SUT allocates resources across
   concurrent requests with different characteristics.  This test
   reveals head-of-line blocking, starvation, and priority effects.

5.6.2.  Setup Parameters

   Workload:  Synthetic-Skewed REQUIRED.  The high length variance
      creates fairness-sensitive conditions.

   Request classes:  Define two or more request classes:

      *  Short requests: Input [64, 256] tokens, output [32, 128] tokens

      *  Long requests: Input [1024, 4096] tokens, output [256, 1024]
         tokens

   Class mix:  Ratio of request classes (e.g., 80% short, 20% long).

   Load level:  70-90% of saturation throughput RECOMMENDED to create
      contention.

   Request count:  At least 500 requests per class.





Gaikwad                   Expires 24 July 2026                 [Page 31]

Internet-Draft        LLM Benchmarking Methodology          January 2026


5.6.3.  Procedure

   1.  Configure SUT and complete warm-up.

   2.  Measure baseline: performance of each class in isolation at same
       total load.

   3.  Generate mixed workload with specified class ratio.

   4.  Run at specified load level for at least 300 seconds.

   5.  For each request, record class membership, submission time, first
       token time, completion time.

   6.  Compute per-class statistics and fairness metrics.

5.6.4.  Measurements

   Per-class latency:  TTFT P50, P95, P99 for each request class.

   Latency inflation:  (Mixed workload TTFT) / (Isolated TTFT) per
      class.

   Jain's Fairness Index:  J = (sum(x_i))^2 / (n * sum(x_i^2)) where x_i
      is normalized latency.  J = 1.0 indicates perfect fairness.  J <
      0.9 indicates significant unfairness.

   Starvation rate:  Fraction of requests waiting longer than 5x the
      median wait time for their class.

5.6.5.  Reporting Format

       +=======+=======+==========+==========+==========+==========+
       | Class | Count | TTFT P50 | TTFT P99 | TPOT P50 | TPOT P99 |
       +=======+=======+==========+==========+==========+==========+
       | Short | 4012  | 89 ms    | 234 ms   | 35 ms    | 67 ms    |
       +-------+-------+----------+----------+----------+----------+
       | Long  | 988   | 312 ms   | 1234 ms  | 42 ms    | 89 ms    |
       +-------+-------+----------+----------+----------+----------+

                     Table 8: Per-Class Results Example










Gaikwad                   Expires 24 July 2026                 [Page 32]

Internet-Draft        LLM Benchmarking Methodology          January 2026


                    +========================+=======+
                    | Metric                 | Value |
                    +========================+=======+
                    | Jain's Fairness Index  | 0.87  |
                    +------------------------+-------+
                    | Short Class Starvation | 0.3%  |
                    +------------------------+-------+
                    | Long Class Starvation  | 2.1%  |
                    +------------------------+-------+

                    Table 9: Fairness Metrics Example

5.7.  Prefix Cache Effectiveness

5.7.1.  Objective

   To evaluate the performance benefit of prefix caching under workloads
   with shared prefixes.  This test quantifies TTFT reduction from cache
   hits.

5.7.2.  Setup Parameters

   Workload:  Code Completion workload RECOMMENDED (high prefix
      sharing).

   Shared prefix:  Define a prefix shared across requests.

   Prefix length:  Length in tokens of shared prefix.

   Sharing fraction:  Percentage of requests sharing the prefix.

   Comparison mode:  Test MUST run in two configurations: cache disabled
      (baseline) and cache enabled.

5.7.3.  Procedure

   1.  Configure SUT with cache disabled.

   2.  Complete warm-up (without populating prefix cache).

   3.  Run workload, record TTFT for all requests.

   4.  Enable prefix cache.

   5.  Optionally pre-populate cache with shared prefix.

   6.  Run identical workload, record TTFT for all requests.




Gaikwad                   Expires 24 July 2026                 [Page 33]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   7.  Compare results.

5.7.4.  Measurements

   TTFT without cache:  P50, P95, P99 with caching disabled.

   TTFT with cache:  P50, P95, P99 with caching enabled.

   TTFT reduction:  (TTFT_no_cache - TTFT_cache) / TTFT_no_cache as
      percentage.

   Cache hit rate:  Fraction of prefix tokens served from cache.

   Throughput improvement:  Percentage increase from caching.

5.7.5.  Reporting Format

            +================+==========+==========+==========+
            | Configuration  | TTFT P50 | TTFT P95 | TTFT P99 |
            +================+==========+==========+==========+
            | Cache Disabled | 312 ms   | 423 ms   | 534 ms   |
            +----------------+----------+----------+----------+
            | Cache (Cold)   | 134 ms   | 198 ms   | 267 ms   |
            +----------------+----------+----------+----------+
            | Cache (Warm)   | 98 ms    | 156 ms   | 212 ms   |
            +----------------+----------+----------+----------+

                   Table 10: Cache Effectiveness Example

5.8.  Memory Pressure Behavior

5.8.1.  Objective

   To characterize SUT behavior when memory resources are constrained,
   including preemption, swapping, and degradation patterns.

5.8.2.  Setup Parameters

   Workload:  Long Context workload RECOMMENDED to create memory
      pressure.

   Oversubscription level:  Percentage above maximum capacity (e.g.,
      110%, 125%, 150%).

5.8.3.  Procedure

   1.  Determine maximum concurrent capacity from Section 5.5.




Gaikwad                   Expires 24 July 2026                 [Page 34]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   2.  Configure SUT and complete warm-up.

   3.  For each oversubscription level:

       a.  Submit requests at concurrency exceeding capacity.

       b.  Run for at least 120 seconds.

       c.  Monitor request completions, preemption events, latency.

       d.  Record any OOM errors or system failures.

   4.  Analyze degradation patterns.

5.8.4.  Measurements

   Completion rate:  Percentage of requests completing successfully at
      each level.

   Preemption rate:  Fraction of requests preempted at least once.

   Preemption recovery rate:  Fraction of preempted requests that
      eventually complete.

   Preemption loss:  Average tokens discarded per preemption event.

5.8.5.  Reporting Format

       +===============+==========+=========+===========+==========+
       | Oversub Level | Complete | Preempt | Fail Rate | TTFT P99 |
       +===============+==========+=========+===========+==========+
       | 100% (base)   | 99.7%    | 0%      | 0.3%      | 523 ms   |
       +---------------+----------+---------+-----------+----------+
       | 110%          | 98.2%    | 5.2%    | 1.8%      | 789 ms   |
       +---------------+----------+---------+-----------+----------+
       | 125%          | 94.5%    | 18.7%   | 5.5%      | 1456 ms  |
       +---------------+----------+---------+-----------+----------+
       | 150%          | 82.3%    | 42.1%   | 17.7%     | 3234 ms  |
       +---------------+----------+---------+-----------+----------+

               Table 11: Memory Pressure Degradation Example

5.9.  Long Context Scaling

5.9.1.  Objective

   To characterize how latency and throughput scale with context length.




Gaikwad                   Expires 24 July 2026                 [Page 35]

Internet-Draft        LLM Benchmarking Methodology          January 2026


5.9.2.  Setup Parameters

   Workload:  Long Context workload REQUIRED.

   Context length range:  Sequence of lengths to test (e.g., 1K, 2K, 4K,
      8K, 16K, 32K, 64K, 128K tokens).

   Fixed output length:  Use consistent short output (256 tokens) to
      isolate prefill impact.

   Load model:  Closed-loop with low concurrency (1-4).

   Requests per length:  At least 20 requests per context length.

5.9.3.  Procedure

   1.  Configure SUT and complete warm-up with short-context requests.

   2.  For each context length in ascending order:

       a.  Generate requests with specified input length.

       b.  Submit requests at low concurrency.

       c.  Record TTFT and total latency for each request.

   3.  Analyze scaling behavior and fit to scaling models.

5.9.4.  Measurements

   Per-length latency:  TTFT Mean, P50, P95 for each context length.

   Prefill scaling:  Time per input token (TTFT / input_length).

   Scaling exponent:  Fit exponent k where TTFT proportional to
      context_length^k.

   Throughput at length:  Maximum throughput achievable at each context
      length.

5.9.5.  Reporting Format

        +==================+===========+==========+==============+
        | Context (tokens) | TTFT Mean | TTFT P95 | ms/1K tokens |
        +==================+===========+==========+==============+
        | 1024             | 89 ms     | 112 ms   | 76           |
        +------------------+-----------+----------+--------------+
        | 4096             | 289 ms    | 367 ms   | 63           |



Gaikwad                   Expires 24 July 2026                 [Page 36]

Internet-Draft        LLM Benchmarking Methodology          January 2026


        +------------------+-----------+----------+--------------+
        | 16384            | 1023 ms   | 1287 ms  | 59           |
        +------------------+-----------+----------+--------------+
        | 65536            | 4234 ms   | 5123 ms  | 62           |
        +------------------+-----------+----------+--------------+
        | 131072           | 9123 ms   | 11234 ms | 68           |
        +------------------+-----------+----------+--------------+

                  Table 12: Long Context Scaling Example

   Best fit: Linear (R^2 = 0.9987), ~68 microseconds per input token

5.10.  Guardrail Overhead

5.10.1.  Objective

   To quantify the latency impact of safety systems and content
   filtering.

5.10.2.  Setup Parameters

   Workload:  Conversation workload RECOMMENDED.

   Content mix:  Use benign content to measure processing overhead.

   Configurations to compare:  The following configurations should be
      tested:

      *  Baseline: All guardrails disabled (if possible)

      *  Input filtering only

      *  Output filtering only

      *  Full filtering: All production guardrails enabled

   Load levels:  Test at 25%, 50%, 75% of capacity.

5.10.3.  Procedure

   1.  Configure SUT with baseline (no guardrails).

   2.  Complete warm-up and run workload at each load level.

   3.  Enable each guardrail configuration and repeat.

   4.  Compare results across configurations.




Gaikwad                   Expires 24 July 2026                 [Page 37]

Internet-Draft        LLM Benchmarking Methodology          January 2026


5.10.4.  Measurements

   Per-configuration latency:  TTFT P50, P95, P99 and End-to-end latency
      for each configuration.

   Input filter overhead:  TTFT(input_filter) - TTFT(baseline)

   Total guardrail overhead:  End-to-end(full) - End-to-end(baseline)

   Throughput reduction:  Percentage reduction from guardrails.

5.10.5.  Reporting Format

        +===============+==========+==========+=========+=========+
        | Configuration | TTFT P50 | TTFT P99 | E2E P50 | E2E P99 |
        +===============+==========+==========+=========+=========+
        | Baseline      | 98 ms    | 234 ms   | 4.2 s   | 8.7 s   |
        +---------------+----------+----------+---------+---------+
        | Input Filter  | 112 ms   | 267 ms   | 4.3 s   | 8.9 s   |
        +---------------+----------+----------+---------+---------+
        | Output Filter | 101 ms   | 242 ms   | 4.8 s   | 9.8 s   |
        +---------------+----------+----------+---------+---------+
        | Full Filter   | 118 ms   | 289 ms   | 5.0 s   | 10.2 s  |
        +---------------+----------+----------+---------+---------+

                    Table 13: Guardrail Overhead Example

              +===============+================+===========+
              | Configuration | Max Throughput | Reduction |
              +===============+================+===========+
              | Baseline      | 2867 tok/s     | -         |
              +---------------+----------------+-----------+
              | Input Filter  | 2756 tok/s     | -3.9%     |
              +---------------+----------------+-----------+
              | Output Filter | 2412 tok/s     | -15.9%    |
              +---------------+----------------+-----------+
              | Full Filter   | 2289 tok/s     | -20.2%    |
              +---------------+----------------+-----------+

                   Table 14: Throughput Impact Example

6.  Multi-System Comparison Guidelines

   When comparing multiple SUTs:







Gaikwad                   Expires 24 July 2026                 [Page 38]

Internet-Draft        LLM Benchmarking Methodology          January 2026


6.1.  Equivalence Requirements

   Testers MUST ensure:

   *  Identical workload (same requests in same order with same seeds)

   *  Equivalent SUT boundary (all systems at same boundary)

   *  Comparable hardware (or normalize by hardware capability)

   *  Same load model and parameters

6.2.  Normalization

   When hardware differs:

   *  Report tokens per GPU-second (normalized by GPU count)

   *  Report cost-normalized throughput (tokens per dollar-hour)

   *  Clearly state normalization method

6.3.  Statistical Significance

   For comparative claims:

   *  Report confidence intervals for key metrics

   *  Conduct multiple independent runs (at least 3)

   *  Use appropriate statistical tests for comparison

6.4.  Fair Comparison Checklist

   Before publishing comparative results, verify:

   *  Same workload specification

   *  Same test duration

   *  Same warm-up procedure

   *  Same success criteria

   *  Both systems tested at same time (if using shared resources)

   *  Both systems in production-representative configuration




Gaikwad                   Expires 24 July 2026                 [Page 39]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   *  Differences in configuration explicitly noted

7.  Security Considerations

   Benchmarking methodology intersects with security in several ways.

7.1.  Side-Channel Risks

   Benchmark results may reveal:

   *  System capacity limits useful for DoS planning

   *  Timing patterns enabling cache probing attacks

   *  Memory pressure thresholds for resource exhaustion

   Operators SHOULD consider whether to publish detailed capacity
   information publicly.

7.2.  Benchmark Gaming

   Systems may be optimized specifically for benchmark workloads in ways
   that do not generalize:

   *  Detecting benchmark patterns and applying special handling

   *  Caching benchmark-specific prefixes

   *  Prioritizing benchmark-like requests

   Testers SHOULD vary workloads and verify results with production
   traffic samples.

7.3.  Adversarial Workloads

   This methodology uses benign workloads.  Adversarial inputs
   (jailbreak attempts, prompt injections) may have different
   performance characteristics due to guardrail processing.

   Testing with adversarial workloads requires additional ethical and
   safety considerations not covered here.

7.4.  Resource Exhaustion

   Memory pressure tests (Section 5.8) intentionally push systems beyond
   capacity.  Testers SHOULD:

   *  Conduct such tests on isolated systems



Gaikwad                   Expires 24 July 2026                 [Page 40]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   *  Have recovery procedures ready

   *  Monitor for cascading failures

8.  References

8.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [LLM-TERMS]
              Gaikwad, M., "Benchmarking Terminology for Large Language
              Model Serving", Work in Progress, Internet-Draft, draft-
              gaikwad-llm-benchmarking-terminology-00, January 2026,
              <https://datatracker.ietf.org/doc/draft-gaikwad-llm-
              benchmarking-terminology/>.

8.2.  Informative References

   [RFC1242]  Bradner, S., "Benchmarking Terminology for Network
              Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242,
              July 1991, <https://www.rfc-editor.org/info/rfc1242>.

   [RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
              Network Interconnect Devices", RFC 2544,
              DOI 10.17487/RFC2544, March 1999,
              <https://www.rfc-editor.org/info/rfc2544>.

   [RFC3511]  Hickman, B., Newman, D., Tadjudin, S., and T. Martin,
              "Benchmarking Methodology for Firewall Performance",
              RFC 3511, DOI 10.17487/RFC3511, April 2003,
              <https://www.rfc-editor.org/info/rfc3511>.

   [VLLM]     Kwon, W., "Efficient Memory Management for Large Language
              Model Serving with PagedAttention", Proceedings of SOSP
              2023, DOI 10.1145/3600006.3613165, 2023,
              <https://doi.org/10.1145/3600006.3613165>.

   [SARATHI]  Agrawal, A., "Taming Throughput-Latency Tradeoff in LLM
              Inference with Sarathi-Serve", Proceedings of OSDI 2024,
              2024.



Gaikwad                   Expires 24 July 2026                 [Page 41]

Internet-Draft        LLM Benchmarking Methodology          January 2026


Appendix A.  Reference Workload Specifications

   This appendix provides complete specifications for standard
   workloads.

A.1.  Synthetic-Uniform Workload

   Purpose: Controlled baseline with minimal variance

A.1.1.  Input Specification

   Distribution:  Uniform

   Minimum:  128 tokens

   Maximum:  512 tokens

   Mean:  320 tokens

   Content:  Random token IDs from vocabulary

A.1.2.  Output Specification

   Distribution:  Uniform

   Minimum:  64 tokens

   Maximum:  256 tokens

   Mean:  160 tokens

   Control:  max_tokens parameter

A.1.3.  Other Parameters

   System prompt:  None

   Prefix sharing:  None

   Temperature:  0.0 (deterministic)

   Stop sequences:  None

A.1.4.  Generation Method

   Python pseudocode:





Gaikwad                   Expires 24 July 2026                 [Page 42]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   def generate_synthetic_uniform(n_requests, seed=42):
       rng = random.Random(seed)
       requests = []
       for i in range(n_requests):
           input_len = rng.randint(128, 512)
           output_len = rng.randint(64, 256)
           input_tokens = [rng.randint(0, 100255)
                           for _ in range(input_len)]
           requests.append({
               'input_tokens': input_tokens,
               'max_tokens': output_len,
               'temperature': 0.0
           })
       return requests

A.2.  Synthetic-Skewed Workload

   Purpose: Test scheduling with high length variance

A.2.1.  Input Specification

   Distribution:  Log-normal

   mu:  5.5 (in log space)

   sigma:  1.0 (in log space)

   Minimum:  32 tokens (floor)

   Maximum:  4096 tokens (cap)

   Median:  ~245 tokens

   Mean:  ~405 tokens

A.2.2.  Output Specification

   Distribution:  Log-normal

   mu:  4.5 (in log space)

   sigma:  1.2 (in log space)

   Minimum:  16 tokens (floor)

   Maximum:  2048 tokens (cap)





Gaikwad                   Expires 24 July 2026                 [Page 43]

Internet-Draft        LLM Benchmarking Methodology          January 2026


A.3.  Conversation Workload

   Purpose: Realistic interactive chat patterns

A.3.1.  Data Source

   Dataset:  ShareGPT (vicuna_cleaned subset)

   Version:  2023-04-12

   Preprocessing:  Filter conversations with 1+ assistant turns

A.3.2.  Length Statistics (Reference)

   Input tokens:

   *  P50: 156

   *  P95: 892

   *  P99: 2134

   Output tokens:

   *  P50: 234

   *  P95: 789

   *  P99: 1567

A.4.  Code Completion Workload

   Purpose: Test prefix caching with code context

A.4.1.  Data Source

   Dataset:  The Stack (Python, JavaScript, TypeScript subset)

   Preprocessing:  Extract function-level completions

A.4.2.  Prefix Sharing Pattern

   *  10 unique repository contexts

   *  Each 512-1024 tokens

   *  80% of requests share one of these prefixes




Gaikwad                   Expires 24 July 2026                 [Page 44]

Internet-Draft        LLM Benchmarking Methodology          January 2026


   *  Distribution: Zipf with s=1.5

A.5.  Long Context Workload

   Purpose: Test long-context handling

A.5.1.  Input Specification

   Distribution:  Uniform over target lengths

   Target lengths:  [8192, 16384, 32768, 65536, 131072] tokens

   Structure:  [document][question]

   Document:  Fills target length minus 100 tokens

   Question:  Fixed ~100 token question about document

A.5.2.  Output Specification

   Distribution:  Fixed

   Length:  256 tokens

   Control:  max_tokens = 256

Appendix B.  Timing Measurement Reference

   This appendix provides detailed guidance for timing measurements.

B.1.  TTFT Measurement Points

B.1.1.  HTTP/SSE Measurement

   Client-side TTFT:

   T_submit:  time of sending final byte of HTTP request

   T_first:  time of receiving first data event with content token

   T_first is when the complete "data:" line is received and parsed, not
   when the first byte of the response arrives.

B.1.2.  gRPC Streaming Measurement

   T_submit:  time of sending request message

   T_first:  time of receiving first response message with token



Gaikwad                   Expires 24 July 2026                 [Page 45]

Internet-Draft        LLM Benchmarking Methodology          January 2026


B.1.3.  Server-Side Measurement

   If server instrumentation available:

   T_submit:  time request enters inference queue

   T_first:  time first token exits model forward pass

   Server-side excludes network latency but may include internal queue
   time.

B.2.  ITL Measurement with SSE

   SSE delivery may batch multiple tokens per event due to server-side
   batching, TCP buffering, or client-side buffering.

B.2.1.  Recommended Approach

   1.  First, characterize delivery pattern (tokens per chunk)

   2.  If single-token chunks dominate (>90%): use direct measurement

   3.  If multi-token chunks common: prefer server timestamps

   4.  If server timestamps unavailable: use chunk timing and document

B.3.  Clock Synchronization Methods

B.3.1.  NTP Synchronization

   1.  Both machines sync to same NTP server

   2.  Verify offset: ntpq -p (check offset column)

   3.  Acceptable offset: < 10ms for most LLM benchmarking

   4.  Document NTP server and measured offset

B.3.2.  PTP Synchronization

   For sub-millisecond accuracy:

   1.  Use PTP-capable network hardware

   2.  Configure ptp4l on Linux systems

   3.  Acceptable offset: < 1 microsecond




Gaikwad                   Expires 24 July 2026                 [Page 46]

Internet-Draft        LLM Benchmarking Methodology          January 2026


B.3.3.  Single-Machine Alternative

   Recommended for Model Engine testing:

   1.  Run load generator on same machine as SUT

   2.  Use loopback network interface

   3.  Clock synchronization inherent

   4.  Eliminates network latency from measurement

Appendix C.  Reporting Templates

C.1.  Minimum Viable Report

   For quick comparisons, include at minimum:

   === LLM Benchmark Report (Minimum) ===

   System Identification:
   - Model: [model name and version]
   - Hardware: [GPU type] x [count]
   - Software: [inference engine and version]
   - SUT Boundary: [Model Engine | Gateway | Compound]

   Test Configuration:
   - Workload: [workload name]
   - Load Model: [open-loop rate | closed-loop concurrency]
   - Request Count: [N]
   - Test Duration: [seconds]

   Key Results:
   - TTFT P50: [value] ms
   - TTFT P99: [value] ms
   - TPOT P50: [value] ms
   - TPOT P99: [value] ms
   - Max Throughput: [value] tok/s
   - Throughput at P99 TTFT < 500ms: [value] tok/s

   Notes:
   - [Any deviations from methodology]
   - [Guardrail configuration]

   === End Report ===






Gaikwad                   Expires 24 July 2026                 [Page 47]

Internet-Draft        LLM Benchmarking Methodology          January 2026


C.2.  Full Report Template

   A complete benchmark report should include the following sections:

   1.  System Identification (model, hardware, software)

   2.  Test Configuration (workload, load, execution parameters)

   3.  Results (latency summary, throughput summary, success metrics)

   4.  Detailed Results (per-test tables and visualizations)

   5.  Methodology Compliance (tests performed, deviations, limitations)

   6.  Reproduction Information (test harness, configuration, data)

Acknowledgements

   This document draws on the structure and approach established by RFC
   3511 for firewall benchmarking methodology.  The author thanks the
   Benchmarking Methodology Working Group for their foundational work in
   network device benchmarking.

Author's Address

   Madhava Gaikwad
   Independent Researcher
   Email: gaikwad.madhav@gmail.com























Gaikwad                   Expires 24 July 2026                 [Page 48]


name: draft-gaikwad-llm-benchmarking-profiles
rev: 00
title: Performance Benchmarking Profiles for Large Language Model Serving Systems
time: 2026-01-20T18:16:53Z
group_uri: /api/v1/group/group/1027/
expires: 2026-07-24T18:16:53Z
states: ["/api/v1/doc/state/1/", "/api/v1/doc/state/150/"]
categories: ["Model serving/inference", "Data formats/interop"]
tags: []
fetched_at: 2026-02-27T22:56:07.990772+00:00
abstract:
This document defines performance benchmarking profiles for Large
   Language Model (LLM) serving systems.  Profiles bind the terminology
   defined in draft-gaikwad-llm-benchmarking-terminology and the
   procedures described in draft-gaikwad-llm-benchmarking-methodology to
   concrete architectural roles and workload patterns.  Each profile
   clarifies the System Under Test (SUT) boundary, measurement points,
   and interpretation constraints required for reproducible and
   comparable benchmarking.

   This document specifies profiles only.  It does not define new
   metrics, benchmark workloads, or acceptance thresholds.

full_text:
Network Working Group                                         M. Gaikwad
Internet-Draft                                    Independent Researcher
Intended status: Informational                              January 2026
Expires: 24 July 2026


   Performance Benchmarking Profiles for Large Language Model Serving
                                Systems
               draft-gaikwad-llm-benchmarking-profiles-00

Abstract

   This document defines performance benchmarking profiles for Large
   Language Model (LLM) serving systems.  Profiles bind the terminology
   defined in draft-gaikwad-llm-benchmarking-terminology and the
   procedures described in draft-gaikwad-llm-benchmarking-methodology to
   concrete architectural roles and workload patterns.  Each profile
   clarifies the System Under Test (SUT) boundary, measurement points,
   and interpretation constraints required for reproducible and
   comparable benchmarking.

   This document specifies profiles only.  It does not define new
   metrics, benchmark workloads, or acceptance thresholds.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 5 July 2026.

Copyright Notice

   Copyright (c) 2026 IETF Trust and the persons identified as the
   document authors.  All rights reserved.






Gaikwad                   Expires 24 July 2026                  [Page 1]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   4
     1.1.  Terminology Alignment . . . . . . . . . . . . . . . . . .   5
   2.  Requirements Language . . . . . . . . . . . . . . . . . . . .   6
   3.  Profile Taxonomy  . . . . . . . . . . . . . . . . . . . . . .   6
     3.1.  Infrastructure Profiles . . . . . . . . . . . . . . . . .   6
     3.2.  Workload Profiles . . . . . . . . . . . . . . . . . . . .   7
     3.3.  Profile Selection Guidance  . . . . . . . . . . . . . . .   8
   4.  Infrastructure Profiles . . . . . . . . . . . . . . . . . . .   8
     4.1.  Model Engine Profile  . . . . . . . . . . . . . . . . . .   9
       4.1.1.  Definition and Concepts . . . . . . . . . . . . . . .   9
       4.1.2.  Boundary Specification  . . . . . . . . . . . . . . .   9
       4.1.3.  Architecture Variants . . . . . . . . . . . . . . . .  10
       4.1.4.  Configuration Disclosure  . . . . . . . . . . . . . .  14
       4.1.5.  Primary Metrics . . . . . . . . . . . . . . . . . . .  16
       4.1.6.  Secondary Metrics . . . . . . . . . . . . . . . . . .  16
       4.1.7.  Benchmarking Constraints  . . . . . . . . . . . . . .  17
     4.2.  AI Gateway Profile  . . . . . . . . . . . . . . . . . . .  17
       4.2.1.  Definition and Concepts . . . . . . . . . . . . . . .  17
       4.2.2.  Boundary Specification  . . . . . . . . . . . . . . .  17
       4.2.3.  Baseline Requirement  . . . . . . . . . . . . . . . .  18
       4.2.4.  Load Balancing Disclosure . . . . . . . . . . . . . .  19
       4.2.5.  Multi-Model Gateway . . . . . . . . . . . . . . . . .  19
       4.2.6.  Semantic Cache  . . . . . . . . . . . . . . . . . . .  20
     4.3.  AI Firewall Profile . . . . . . . . . . . . . . . . . . .  22
       4.3.1.  Definition and Concepts . . . . . . . . . . . . . . .  22
       4.3.2.  Boundary Specification  . . . . . . . . . . . . . . .  22
       4.3.3.  Enforcement Directions  . . . . . . . . . . . . . . .  23
       4.3.4.  Inspection Architecture . . . . . . . . . . . . . . .  24
       4.3.5.  Required Metrics  . . . . . . . . . . . . . . . . . .  25
       4.3.6.  Workload Specification  . . . . . . . . . . . . . . .  27
       4.3.7.  Multi-Layer Firewall  . . . . . . . . . . . . . . . .  28
       4.3.8.  Benchmarking Constraints  . . . . . . . . . . . . . .  28
     4.4.  Compound System Profile . . . . . . . . . . . . . . . . .  29
       4.4.1.  Definition and Concepts . . . . . . . . . . . . . . .  29
       4.4.2.  Boundary Specification  . . . . . . . . . . . . . . .  29
       4.4.3.  Primary Metrics . . . . . . . . . . . . . . . . . . .  30
       4.4.4.  Evaluation Oracle . . . . . . . . . . . . . . . . . .  31



Gaikwad                   Expires 24 July 2026                  [Page 2]

Internet-Draft          LLM Benchmarking Profiles           January 2026


       4.4.5.  Secondary Metrics . . . . . . . . . . . . . . . . . .  32
       4.4.6.  RAG Sub-Profile . . . . . . . . . . . . . . . . . . .  33
       4.4.7.  Agentic System Boundaries . . . . . . . . . . . . . .  34
       4.4.8.  Exclusions  . . . . . . . . . . . . . . . . . . . . .  36
   5.  Workload Profiles . . . . . . . . . . . . . . . . . . . . . .  36
     5.1.  Chatbot Workload Profile  . . . . . . . . . . . . . . . .  36
       5.1.1.  Characteristics . . . . . . . . . . . . . . . . . . .  36
       5.1.2.  Required Parameters . . . . . . . . . . . . . . . . .  37
       5.1.3.  Example . . . . . . . . . . . . . . . . . . . . . . .  37
     5.2.  Compound Workflow Workload Profile  . . . . . . . . . . .  37
       5.2.1.  Characteristics . . . . . . . . . . . . . . . . . . .  37
       5.2.2.  Required Parameters . . . . . . . . . . . . . . . . .  38
       5.2.3.  External Dependency Handling  . . . . . . . . . . . .  38
   6.  Delta Measurement Model . . . . . . . . . . . . . . . . . . .  39
     6.1.  Timestamp Reference . . . . . . . . . . . . . . . . . . .  39
     6.2.  Timestamp Definitions . . . . . . . . . . . . . . . . . .  40
     6.3.  Component Deltas  . . . . . . . . . . . . . . . . . . . .  41
     6.4.  End-to-End Metrics  . . . . . . . . . . . . . . . . . . .  42
     6.5.  Clock Synchronization . . . . . . . . . . . . . . . . . .  42
   7.  Profile Composition . . . . . . . . . . . . . . . . . . . . .  42
     7.1.  Composite SUT Declaration . . . . . . . . . . . . . . . .  43
     7.2.  Composition Validation  . . . . . . . . . . . . . . . . .  43
     7.3.  Interaction Effects . . . . . . . . . . . . . . . . . . .  44
   8.  Access Logging Requirements . . . . . . . . . . . . . . . . .  45
     8.1.  Minimum Fields  . . . . . . . . . . . . . . . . . . . . .  45
     8.2.  Model Engine Fields . . . . . . . . . . . . . . . . . . .  46
     8.3.  AI Firewall Fields  . . . . . . . . . . . . . . . . . . .  46
     8.4.  Compound System Fields  . . . . . . . . . . . . . . . . .  47
     8.5.  AI Gateway Fields . . . . . . . . . . . . . . . . . . . .  47
     8.6.  OpenTelemetry Integration . . . . . . . . . . . . . . . .  47
   9.  Measurement Considerations  . . . . . . . . . . . . . . . . .  48
     9.1.  Baseline and Delta Reporting  . . . . . . . . . . . . . .  48
     9.2.  Warm-up and Steady State  . . . . . . . . . . . . . . . .  48
     9.3.  Clock Synchronization . . . . . . . . . . . . . . . . . .  48
     9.4.  Streaming Protocol Considerations . . . . . . . . . . . .  49
   10. Security Considerations . . . . . . . . . . . . . . . . . . .  49
     10.1.  Bidirectional Enforcement Gaps . . . . . . . . . . . . .  49
     10.2.  Adversarial Workload Handling  . . . . . . . . . . . . .  50
     10.3.  Side-Channel Considerations  . . . . . . . . . . . . . .  50
   11. References  . . . . . . . . . . . . . . . . . . . . . . . . .  50
     11.1.  Normative References . . . . . . . . . . . . . . . . . .  50
     11.2.  Informative References . . . . . . . . . . . . . . . . .  51
   Appendix A.  Example Benchmark Report Structure . . . . . . . . .  52
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  52







Gaikwad                   Expires 24 July 2026                  [Page 3]

Internet-Draft          LLM Benchmarking Profiles           January 2026


1.  Introduction

   LLM serving systems are rarely monolithic.  Production deployments
   typically compose multiple infrastructural intermediaries before a
   request reaches a Model Engine.  A request may pass through an API
   gateway for authentication, an AI firewall for prompt inspection, a
   load balancer for routing, and finally arrive at an inference engine.
   Each component adds latency and affects throughput.

   Performance metrics such as Time to First Token (TTFT) or throughput
   are boundary dependent.  A TTFT measurement taken at the client
   includes network latency, gateway processing, firewall inspection,
   queue wait time, and prefill computation.  The same measurement taken
   at the engine boundary includes only queue wait and prefill.  Without
   explicit boundary declaration, reported results cannot be compared.

   This document addresses this ambiguity by defining benchmarking
   profiles: standardized descriptions of SUT boundaries and their
   associated performance interpretation rules.  Section 4 defines four
   infrastructure profiles that specify what component is being
   measured.  Section 5 defines workload profiles that specify how that
   component is tested.  Section 6 then shows how to attribute latency
   across composed systems using delta measurement.




























Gaikwad                   Expires 24 July 2026                  [Page 4]

Internet-Draft          LLM Benchmarking Profiles           January 2026


                Client
                  |
                  v
        +------------------+
        |   AI Gateway     |<-- Auth, routing, caching
        +--------+---------+
                 |
                 v
        +------------------+
        |   AI Firewall    |<-- Prompt inspection
        +--------+---------+
                 |
                 v
        +------------------+
        |  Model Engine    |<-- Inference
        +--------+---------+
                 |
                 v
        +------------------+
        |   AI Firewall    |<-- Output inspection
        +--------+---------+
                 |
                 v
        +------------------+
        |   AI Gateway     |<-- Response normalization
        +--------+---------+
                 |
                 v
               Client

Each layer adds latency. Benchmarks must declare which layers are included.

                 Figure 1: Typical LLM Serving Stack

1.1.  Terminology Alignment

   This document uses metrics defined in
   [I-D.gaikwad-llm-benchmarking-terminology].  The following table maps
   profile-specific terms to their normative definitions.












Gaikwad                   Expires 24 July 2026                  [Page 5]

Internet-Draft          LLM Benchmarking Profiles           January 2026


        +=========================+===============================+
        | Term Used in Profiles   | Terminology Draft Reference   |
        +=========================+===============================+
        | TTFT                    | Time to First Token           |
        +-------------------------+-------------------------------+
        | ITL                     | Inter-Token Latency           |
        +-------------------------+-------------------------------+
        | TPOT                    | Time per Output Token         |
        +-------------------------+-------------------------------+
        | Queue Residence Time    | Queue Wait Time               |
        +-------------------------+-------------------------------+
        | FRR                     | False Refusal Rate            |
        +-------------------------+-------------------------------+
        | Guardrail Overhead      | Guardrail Processing Overhead |
        +-------------------------+-------------------------------+
        | Task Completion Latency | Task Completion Latency       |
        +-------------------------+-------------------------------+
        | Goodput                 | Goodput                       |
        +-------------------------+-------------------------------+

                        Table 1: Terminology Mapping

2.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

3.  Profile Taxonomy

   Profiles divide into two categories that serve orthogonal purposes.
   Conflating them produces misleading benchmarks.

3.1.  Infrastructure Profiles

   Infrastructure Profiles define what is being tested.  They specify
   the SUT boundary: where measurements start and end, what components
   are included, and what is excluded.











Gaikwad                   Expires 24 July 2026                  [Page 6]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   +==============+========================+===========================+
   | Profile      | SUT Boundary           | Primary Question Answered |
   +==============+========================+===========================+
   | Model Engine | Inference              | How fast can this engine  |
   |              | runtime only           | generate tokens?          |
   +--------------+------------------------+---------------------------+
   | AI Gateway   | API intermediary       | What overhead does the    |
   |              | layer                  | gateway add?              |
   +--------------+------------------------+---------------------------+
   | AI Firewall  | Security               | What latency and accuracy |
   |              | inspection layer       | does inspection cost?     |
   +--------------+------------------------+---------------------------+
   | Compound     | End-to-end             | How long does it take to  |
   | System       | orchestration          | complete a task?          |
   +--------------+------------------------+---------------------------+

                      Table 2: Infrastructure Profiles

   The choice of infrastructure profile determines which metrics are
   meaningful.  Measuring "AI Firewall throughput" in tokens per second
   conflates firewall performance with downstream engine performance.
   The firewall does not generate tokens; it inspects them.  Appropriate
   firewall metrics include inspection latency, detection rate, and
   false positive rate.

3.2.  Workload Profiles

   Workload Profiles define how the SUT is tested.  They specify traffic
   patterns, request characteristics, and arrival models.  Workload
   profiles are independent of infrastructure profiles.

    +==================+========================+====================+
    | Profile          | Traffic Pattern        | Applicable To      |
    +==================+========================+====================+
    | Chatbot Workload | Multi-turn, streaming, | Engine, Gateway,   |
    |                  | human-paced            | Firewall, Compound |
    +------------------+------------------------+--------------------+
    | Compound         | Multi-step, tool-      | Compound System    |
    | Workflow         | using, machine-paced   | primarily          |
    +------------------+------------------------+--------------------+

                        Table 3: Workload Profiles









Gaikwad                   Expires 24 July 2026                  [Page 7]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   A Chatbot Workload can be applied to a Model Engine (measuring raw
   inference speed), an AI Gateway (measuring gateway overhead under
   conversational traffic), or a Compound System (measuring end-to-end
   chat latency including retrieval).  The infrastructure profile
   determines the measurement boundary; the workload profile determines
   the traffic shape.

   Conflating infrastructure and workload profiles produces non-
   comparable results.  "Chatbot benchmark on Gateway A" versus "Chatbot
   benchmark on Engine B" compares different things.  The former
   includes gateway overhead; the latter does not.  Valid comparison
   requires either:

   *  Same infrastructure profile, different implementations (Gateway A
      vs Gateway B)

   *  Same implementation, different workload profiles (Chatbot vs
      Compound Workflow on Engine A)

   Cross-profile comparisons require explicit delta decomposition
   (Section 6).

3.3.  Profile Selection Guidance

   +===========================+====================+==================+
   | If you want to            | Use Infrastructure | Apply Workload   |
   | measure...                | Profile            | Profile          |
   +===========================+====================+==================+
   | Raw model inference       | Model Engine       | Chatbot or       |
   | speed                     |                    | synthetic        |
   +---------------------------+--------------------+------------------+
   | Gateway routing           | AI Gateway         | Match production |
   | overhead                  |                    | traffic          |
   +---------------------------+--------------------+------------------+
   | Security inspection       | AI Firewall        | Mixed benign/    |
   | cost                      |                    | adversarial      |
   +---------------------------+--------------------+------------------+
   | End-to-end agent          | Compound System    | Compound         |
   | latency                   |                    | Workflow         |
   +---------------------------+--------------------+------------------+
   | Full-stack production     | Composite (see     | Match production |
   | performance               | Section 7)         | traffic          |
   +---------------------------+--------------------+------------------+

                      Table 4: Profile Selection Guide

4.  Infrastructure Profiles




Gaikwad                   Expires 24 July 2026                  [Page 8]

Internet-Draft          LLM Benchmarking Profiles           January 2026


4.1.  Model Engine Profile

4.1.1.  Definition and Concepts

   A Model Engine is the runtime responsible for executing LLM
   inference.  Before specifying the benchmark boundary, understanding
   three core operations is necessary:

   *Prefill* (also called prompt processing): The engine processes all
   input tokens in parallel to build initial hidden states.  Prefill is
   compute-bound and benefits from parallelism.  Prefill latency scales
   with input length but can be reduced by adding more compute.

   *Decode* (also called autoregressive generation): The engine
   generates output tokens one at a time, each depending on all previous
   tokens.  Decode is memory-bandwidth-bound because each token requires
   reading the full model weights.  Decode latency per token is
   relatively constant regardless of batch size, but throughput
   increases with batching.

   *KV Cache*: To avoid recomputing attention over previous tokens, the
   engine stores key-value pairs from prior tokens.  The KV cache grows
   with sequence length and consumes GPU memory.  Cache management
   (allocation, eviction, swapping to CPU) directly affects how many
   concurrent sequences the engine can handle.

   These three operations determine the fundamental performance
   characteristics:

   *  TTFT depends primarily on prefill time plus any queue wait

   *  ITL depends on decode time per token

   *  Maximum concurrency depends on KV cache capacity

   *  Throughput depends on batching efficiency during decode

4.1.2.  Boundary Specification

   Included in SUT:

   *  Model weights and inference kernels

   *  Prefill and decode computation

   *  Batch formation and scheduling logic

   *  KV cache allocation, eviction, and swapping



Gaikwad                   Expires 24 July 2026                  [Page 9]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   *  Speculative decoding (if enabled)

   *  Quantization and precision handling

   Excluded from SUT:

   *  Network transport beyond local interface

   *  Authentication and authorization

   *  Policy enforcement and content inspection

   *  Request routing between multiple engines

   *  Protocol translation (handled by gateway)

4.1.3.  Architecture Variants

   Model Engines exist in several architectural configurations that
   affect measurement interpretation.

4.1.3.1.  Monolithic Architecture

   Prefill and decode execute on the same hardware.  This is the
   simplest configuration and the most common in single-GPU deployments.

               MONOLITHIC ENGINE

   +------------------------------------------+
   |               GPU / Accelerator          |
   |                                          |
   |  Request -> [Prefill] -> [Decode] -> Out |
   |               |            |             |
   |            KV Cache <------+             |
   |                                          |
   +------------------------------------------+

   Timeline for single request:
   |---- Queue ----|---- Prefill ----|---- Decode (N tokens) ----|
         t6             t6a -> t7            t7 -> t8

                  Figure 2: Monolithic Engine Architecture

   Timestamp mapping:







Gaikwad                   Expires 24 July 2026                 [Page 10]

Internet-Draft          LLM Benchmarking Profiles           January 2026


       +========+==================================================+
       | Symbol | Event                                            |
       +========+==================================================+
       | t6     | Request enters engine queue                      |
       +--------+--------------------------------------------------+
       | t6a    | Prefill computation begins (batch slot acquired) |
       +--------+--------------------------------------------------+
       | t7     | First output token generated                     |
       +--------+--------------------------------------------------+
       | t8     | Last output token generated                      |
       +--------+--------------------------------------------------+

                                  Table 5

   Derived metrics:

   Queue residence = t6a - t6
   Prefill latency = t7 - t6a
   Engine TTFT = t7 - t6
   Generation time = t8 - t7

4.1.3.2.  Disaggregated Architecture

   Prefill and decode execute on separate hardware pools.  Prefill nodes
   are optimized for compute throughput; decode nodes are optimized for
   memory bandwidth.  After prefill completes, the KV cache must
   transfer across the network to the decode pool.

   This architecture appears in published systems including DistServe
   [DISTSERVE] and Mooncake [MOONCAKE], and in open-source projects such
   as llm-d.




















Gaikwad                   Expires 24 July 2026                 [Page 11]

Internet-Draft          LLM Benchmarking Profiles           January 2026


                       DISAGGREGATED SERVING

   +------------------+                      +------------------+
   |  PREFILL POOL    |                      |  DECODE POOL     |
   |                  |                      |                  |
   |  High compute    |    KV Cache          |  High memory BW  |
   |  utilization     |    Transfer          |  utilization     |
   |                  |  ================>   |                  |
   |  +-----------+   |   Network link       |  +-----------+   |
   |  | GPU 0     |   |   (RDMA or TCP)      |  | GPU 0     |   |
   |  +-----------+   |                      |  +-----------+   |
   |  | GPU 1     |   |   Bottleneck at      |  | GPU 1     |   |
   |  +-----------+   |   high context       |  +-----------+   |
   |  | ...       |   |   lengths            |  | ...       |   |
   |  +-----------+   |                      |  +-----------+   |
   +------------------+                      +------------------+

   Timeline:
   |-- Queue --|-- Prefill --|-- KV Transfer --|-- Decode --|
        t6          t6a            t7a              t7 -> t8

                Figure 3: Disaggregated Serving Architecture

   The KV transfer phase (t7a) does not exist in monolithic deployments.
   This phase can become the bottleneck for long contexts.

   KV Transfer Constraint:

   Transfer time depends on context length and network bandwidth:

KV_transfer_time = (context_length * kv_bytes_per_token) / effective_bandwidth

   Where:

   *  context_length = input tokens processed

   *  kv_bytes_per_token = 2 * num_layers * head_dim * num_heads *
      bytes_per_element

   *  effective_bandwidth = min(network_bandwidth, memory_bandwidth) *
      efficiency

   Bandwidth Saturation Threshold: The context length at which KV
   transfer time exceeds prefill compute time.  Beyond this threshold,
   adding more prefill compute does not reduce TTFT.






Gaikwad                   Expires 24 July 2026                 [Page 12]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   Configuration:
     Model: 70B parameters
     KV cache: 80 layers, 128 heads, 128 dim, BF16
     KV bytes per token: 2 * 80 * 128 * 128 * 2 = 5.24 MB
     Inter-pool bandwidth: 400 Gbps = 50 GB/s effective

   At 4K context:
     KV transfer = 4096 * 5.24 MB / 50 GB/s = 430 ms

   At 32K context:
     KV transfer = 32768 * 5.24 MB / 50 GB/s = 3.44 s

   If prefill compute takes 500ms for 32K tokens:
     Bottleneck shifts to KV transfer at ~4.8K tokens

                 Figure 4: KV Transfer Example Calculation

   Testers benchmarking disaggregated architectures MUST report:

          +======================+=============================+
          | Parameter            | Description                 |
          +======================+=============================+
          | Pool configuration   | Number and type of prefill  |
          |                      | vs decode accelerators      |
          +----------------------+-----------------------------+
          | KV transfer          | RDMA, TCP, or other;        |
          | mechanism            | theoretical bandwidth       |
          +----------------------+-----------------------------+
          | KV bytes per token   | Calculated from model       |
          |                      | architecture                |
          +----------------------+-----------------------------+
          | Observed transfer    | Measured, not calculated    |
          | latency              |                             |
          +----------------------+-----------------------------+
          | Bandwidth saturation | Context length where        |
          | threshold            | transfer becomes bottleneck |
          +----------------------+-----------------------------+
          | TTFT boundary        | Whether reported TTFT       |
          |                      | includes KV transfer        |
          +----------------------+-----------------------------+

                                 Table 6

   Results from disaggregated and monolithic deployments MUST NOT be
   directly compared without explicit architectural notation.






Gaikwad                   Expires 24 July 2026                 [Page 13]

Internet-Draft          LLM Benchmarking Profiles           January 2026


           MONOLITHIC                         DISAGGREGATED

   +----------------------+      +-----------+    +-----------+
   |     Single Pool      |      |  Prefill  |    |  Decode   |
   |                      |      |   Pool    |    |   Pool    |
   |  Prefill --> Decode  |      |           |    |           |
   |      |         |     |      | +-------+ |    | +-------+ |
   |      +> KV Cache <---+      | | GPU 0 | |    | | GPU 0 | |
   |                      |      | +-------+ |    | +-------+ |
   |  Same memory space   |      | +-------+ |    | +-------+ |
   |  No transfer needed  |      | | GPU 1 |======>| GPU 1 | |
   |                      |      | +-------+ |    | +-------+ |
   +----------------------+      |   KV transfer  |           |
                                 +-----------+    +-----------+

   TTFT = Queue + Prefill        TTFT = Queue + Prefill + KV_Transfer

   Best for:                     Best for:
   - Smaller models              - Large models (70B+)
   - Lower latency               - Higher throughput
   - Simpler deployment          - Independent scaling

              Figure 5: Monolithic vs Disaggregated Comparison

4.1.3.3.  Distributed Architecture

   Model sharded across multiple accelerators using tensor parallelism
   (TP), pipeline parallelism (PP), or expert parallelism (EP for
   mixture-of-experts models).

   Testers MUST report:

   *  Parallelism strategy and degree (e.g., TP=8, PP=2)

   *  Interconnect type (NVLink, PCIe, InfiniBand)

   *  Collective communication overhead if measurable

4.1.4.  Configuration Disclosure

   Testers MUST disclose:










Gaikwad                   Expires 24 July 2026                 [Page 14]

Internet-Draft          LLM Benchmarking Profiles           January 2026


     +=================+=======================+=====================+
     | Configuration   | Example Values        | Why It Matters      |
     +=================+=======================+=====================+
     | Model precision | FP16, BF16, INT8, FP8 | Affects throughput, |
     |                 |                       | memory, and quality |
     +-----------------+-----------------------+---------------------+
     | Quantization    | GPTQ, AWQ,            | Different speed/    |
     | method          | SmoothQuant           | quality tradeoffs   |
     +-----------------+-----------------------+---------------------+
     | Batch strategy  | Static, continuous,   | Affects latency     |
     |                 | chunked prefill       | distribution        |
     +-----------------+-----------------------+---------------------+
     | Max batch size  | 64 requests           | Limits concurrency  |
     +-----------------+-----------------------+---------------------+
     | Max sequence    | 8192 tokens           | Limits context      |
     | length          |                       | window              |
     +-----------------+-----------------------+---------------------+
     | KV cache memory | 24 GB                 | Limits concurrent   |
     |                 |                       | sequences           |
     +-----------------+-----------------------+---------------------+

                                  Table 7

4.1.4.1.  Speculative Decoding

   Speculative decoding uses a smaller draft model to propose multiple
   tokens, then verifies them in parallel with the target model.  When
   draft tokens are accepted, generation is faster.  When rejected,
   compute is wasted.

   If speculative decoding is enabled, testers MUST report:

    +========================+========================================+
    | Parameter              | Description                            |
    +========================+========================================+
    | Draft model            | Identifier and parameter count         |
    +------------------------+----------------------------------------+
    | Speculation window (k) | Tokens proposed per verification step  |
    +------------------------+----------------------------------------+
    | Acceptance rate        | Fraction of draft tokens accepted      |
    +------------------------+----------------------------------------+
    | Verification overhead  | Latency when draft tokens are rejected |
    +------------------------+----------------------------------------+

                                  Table 8

   Acceptance rate directly affects efficiency:




Gaikwad                   Expires 24 July 2026                 [Page 15]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   High acceptance (80%), k=5:
     Expected accepted per step = 4 tokens
     Verification passes per output token = 0.25

   Low acceptance (30%), k=5:
     Expected accepted per step = 1.5 tokens
     Verification passes per output token = 0.67

   Result: 2.7x more verification overhead at low acceptance

   Results with speculative decoding MUST be labeled separately and
   include observed acceptance rate.

4.1.4.2.  Chunked Prefill

   Chunked prefill splits long prompts into smaller pieces, processing
   each chunk and potentially interleaving with decode iterations from
   other requests.  This reduces head-of-line blocking but increases
   total prefill time for the chunked request.

   If chunked prefill is enabled, testers MUST report:

   *  Chunk size in tokens

   *  Whether chunks interleave with other requests

   *  Impact on TTFT for long prompts

4.1.5.  Primary Metrics

   From [I-D.gaikwad-llm-benchmarking-terminology]:

   *  Time to First Token (TTFT)

   *  Inter-Token Latency (ITL)

   *  Time per Output Token (TPOT)

   *  Output Token Throughput

4.1.6.  Secondary Metrics

   *  Request Throughput

   *  Queue Depth over time

   *  Queue Residence Time




Gaikwad                   Expires 24 July 2026                 [Page 16]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   *  Prefill Latency (TTFT minus queue residence)

   *  Batch Utilization

4.1.7.  Benchmarking Constraints

   Request rate saturation differs from token saturation.  A system
   might handle 2000 output tokens per second but only 50 requests per
   second if scheduling overhead dominates.  Testers SHOULD measure both
   dimensions.

   Mixed-length workloads increase tail latency under continuous
   batching.  Short requests arriving behind long prefills experience
   head-of-line blocking.  When workload includes high length variance,
   measure fairness: the ratio of actual latency to expected latency
   based on request size.

4.2.  AI Gateway Profile

4.2.1.  Definition and Concepts

   An AI Gateway is a network-facing intermediary that virtualizes
   access to one or more Model Engines.  Gateways handle cross-cutting
   concerns that do not belong in the inference engine itself.

   Gateways perform several functions that affect latency:

   *Request Processing:* TLS termination, authentication, schema
   validation, and protocol translation.  These operations add fixed
   overhead per request.

   *Routing:* Selection of backend engine based on load, capability, or
   policy.  Intelligent routing (e.g., KV-cache-aware) adds decision
   latency but may reduce overall latency by improving cache hit rates.

   *Caching:* Gateways may implement response caching.  Traditional
   exact-match caching has limited utility for LLM traffic due to low
   query repetition.  Semantic caching (matching similar queries)
   improves hit rates but introduces quality risk from approximate
   matches.

   *Admission Control:* Rate limiting and quota enforcement.  Under
   load, admission control adds queuing delay or rejects requests.

4.2.2.  Boundary Specification

   Included in SUT:




Gaikwad                   Expires 24 July 2026                 [Page 17]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   *  TLS termination

   *  Authentication and authorization

   *  Schema validation and protocol translation

   *  Load balancing across engines or model replicas

   *  Semantic cache lookup and population

   *  Admission control and rate limiting

   *  Retry and fallback logic

   *  Response normalization

   Excluded from SUT:

   *  Model inference computation (handled by downstream engine)

   *  Content inspection for safety (handled by AI Firewall)

4.2.3.  Baseline Requirement

   Gateway overhead is meaningful only relative to direct engine access.
   Gateway benchmarks MUST declare measurement type:

          +==================+=================================+
          | Measurement Type | What It Includes                |
          +==================+=================================+
          | Aggregate        | Gateway processing plus         |
          |                  | downstream engine latency       |
          +------------------+---------------------------------+
          | Differential     | Gateway overhead only, relative |
          |                  | to direct engine access         |
          +------------------+---------------------------------+

                                 Table 9

   To measure differential latency:

   1.  Benchmark the Model Engine directly (baseline)

   2.  Benchmark through the Gateway to the same engine (same workload,
       same conditions)

   3.  Compute delta: Gateway_overhead = Gateway_TTFT - Engine_TTFT




Gaikwad                   Expires 24 July 2026                 [Page 18]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   Report both absolute values and delta.

4.2.4.  Load Balancing Disclosure

   Load balancing strategy affects tail latency.  Testers MUST report:

    +===============+=================================+==============+
    | Configuration | Options                         | Impact       |
    +===============+=================================+==============+
    | Algorithm     | Round-robin, least-connections, | Tail latency |
    |               | weighted, adaptive              | variance     |
    +---------------+---------------------------------+--------------+
    | Health checks | Interval, timeout, failure      | Failover     |
    |               | threshold                       | speed        |
    +---------------+---------------------------------+--------------+
    | Sticky        | Enabled/disabled, key type      | Cache        |
    | sessions      |                                 | locality     |
    +---------------+---------------------------------+--------------+
    | Retry policy  | Max retries, backoff strategy   | Failure      |
    |               |                                 | handling     |
    +---------------+---------------------------------+--------------+

                                 Table 10

   For intelligent routing (KV-cache-aware, cost-optimized, latency-
   optimized):

   *  Routing signals used (queue depth, cache locality, model cost)

   *  Decision latency overhead

   *  Routing effectiveness (e.g., cache hit improvement from routing)

4.2.5.  Multi-Model Gateway

   Modern gateways route to multiple backend models based on capability,
   cost, or latency.

   When gateway routes to heterogeneous backends, testers MUST report:

   *  Model selection logic: Rule-based, cost-optimized, capability-
      based

   *  Backend composition: List of models and their roles

   *  Fallback behavior: Conditions triggering model switching

   Per-model metrics SHOULD be reported separately.



Gaikwad                   Expires 24 July 2026                 [Page 19]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   Cross-gateway comparison requires backend normalization.  Comparing
   Gateway A (routing to GPT-4) against Gateway B (routing to Llama-70B)
   conflates gateway performance with model performance.

4.2.6.  Semantic Cache

   Semantic caching matches queries by meaning rather than exact text.
   A cache hit on "What is the capital of France?" might serve a
   response cached from "France's capital city?"  This improves hit
   rates but risks serving inappropriate responses for queries that are
   similar but not equivalent.

   Configuration Disclosure:

      +============+========================+=======================+
      | Parameter  | Example                | Why It Matters        |
      +============+========================+=======================+
      | Similarity | Cosine >= 0.92         | Lower threshold: more |
      | threshold  |                        | hits, more mismatches |
      +------------+------------------------+-----------------------+
      | Embedding  | text-embedding-3-small | Affects similarity    |
      | model      |                        | quality               |
      +------------+------------------------+-----------------------+
      | Cache      | 100,000 entries        | Hit rate ceiling      |
      | capacity   |                        |                       |
      +------------+------------------------+-----------------------+
      | Eviction   | LRU, frequency-based   | Long-term hit rate    |
      | policy     |                        |                       |
      +------------+------------------------+-----------------------+
      | Cache      | Global, per-tenant,    | Security and hit rate |
      | scope      | per-user               | tradeoff              |
      +------------+------------------------+-----------------------+
      | TTL        | 1 hour                 | Staleness vs hit rate |
      +------------+------------------------+-----------------------+

                                  Table 11

   Required Metrics:













Gaikwad                   Expires 24 July 2026                 [Page 20]

Internet-Draft          LLM Benchmarking Profiles           January 2026


    +=======================+========================================+
    | Metric                | Definition                             |
    +=======================+========================================+
    | Hit rate              | Fraction of requests served from cache |
    +-----------------------+----------------------------------------+
    | Hit rate distribution | P50, P95, P99 of per-session hit rates |
    +-----------------------+----------------------------------------+
    | Latency on hit        | TTFT when cache serves response        |
    +-----------------------+----------------------------------------+
    | Latency on miss       | TTFT when engine generates             |
    +-----------------------+----------------------------------------+
    | Cache delta           | Latency_miss minus Latency_hit         |
    +-----------------------+----------------------------------------+
    | Mismatch rate         | Fraction of hits where cached response |
    |                       | was inappropriate                      |
    +-----------------------+----------------------------------------+

                                 Table 12

   Mismatch rate requires evaluation.  Testers SHOULD disclose
   evaluation methodology (human review, automated comparison, or LLM-
   as-judge).

   Session Definition: For per-session metrics, define what constitutes
   a session: requests sharing a session identifier, requests from the
   same user within a time window, or another definition.  Testers MUST
   disclose session definition.

   Staleness in RAG Systems: When semantic cache operates with a RAG
   system, cached responses may reference documents that have since been
   updated.

    +========================+========================================+
    | Parameter              | Description                            |
    +========================+========================================+
    | Index update frequency | How often RAG index refreshes          |
    +------------------------+----------------------------------------+
    | Cache TTL              | Maximum age of cached entries          |
    +------------------------+----------------------------------------+
    | Staleness risk         | Estimated fraction of stale cache hits |
    +------------------------+----------------------------------------+

                                  Table 13

   Staleness risk estimate:

staleness_risk = (average_cache_age / index_update_interval) * corpus_change_rate




Gaikwad                   Expires 24 July 2026                 [Page 21]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   Benchmarking Constraints: Workload diversity determines hit rate.
   Testers MUST report:

   *  Number of distinct query clusters in workload

   *  Cache state at test start (cold or warm)

   *  Time until hit rate stabilizes

4.3.  AI Firewall Profile

4.3.1.  Definition and Concepts

   An AI Firewall is a bidirectional security intermediary that inspects
   LLM inputs and outputs to detect and prevent policy violations.

   Unlike traditional firewalls that examine packet headers or match
   byte patterns, AI Firewalls analyze semantic content.  They must
   understand what a prompt is asking and what a response is saying.
   This requires ML models, making firewall latency fundamentally
   different from network firewall latency.

   The firewall sits on the request path and adds latency to every
   request.  The core tradeoff: more thorough inspection catches more
   threats but costs more time.

4.3.2.  Boundary Specification

   Included in SUT:

   *  Prompt analysis and classification

   *  Output content inspection

   *  Policy decision engine

   *  Block, allow, or modify actions

   Excluded from SUT:

   *  Model inference (upstream or downstream)

   *  Network-layer firewalling (traditional WAF)

   *  Authentication (handled by gateway)






Gaikwad                   Expires 24 July 2026                 [Page 22]

Internet-Draft          LLM Benchmarking Profiles           January 2026


4.3.3.  Enforcement Directions

   AI Firewalls operate bidirectionally.  Each direction addresses
   different threats.

   Inbound Enforcement inspects user prompts before they reach the
   model:

          +=========================+===========================+
          | Threat                  | Description               |
          +=========================+===========================+
          | Direct prompt injection | User attempts to override |
          |                         | system instructions       |
          +-------------------------+---------------------------+
          | Indirect prompt         | Malicious content in      |
          | injection               | retrieved documents       |
          +-------------------------+---------------------------+
          | Jailbreak attempts      | Techniques to bypass      |
          |                         | model safety training     |
          +-------------------------+---------------------------+
          | Context poisoning       | Adversarial content to    |
          |                         | manipulate model behavior |
          +-------------------------+---------------------------+

                                  Table 14

   Outbound Enforcement inspects model outputs before they reach the
   user:

      +===================+=========================================+
      | Threat            | Description                             |
      +===================+=========================================+
      | PII leakage       | Model outputs personal information      |
      +-------------------+-----------------------------------------+
      | Policy violation  | Output violates content policies        |
      +-------------------+-----------------------------------------+
      | Tool misuse       | Model attempts unauthorized actions     |
      +-------------------+-----------------------------------------+
      | Data exfiltration | Sensitive information encoded in output |
      +-------------------+-----------------------------------------+

                                  Table 15

   Testers MUST declare which directions are enforced.  A benchmark
   testing inbound-only enforcement MUST NOT claim protection against
   outbound threats.





Gaikwad                   Expires 24 July 2026                 [Page 23]

Internet-Draft          LLM Benchmarking Profiles           January 2026


4.3.4.  Inspection Architecture

   Firewalls use different inspection strategies with distinct latency
   characteristics.

   INSPECTION ARCHITECTURE COMPARISON

   BUFFERED (adds to TTFT, no ITL impact):

     Input:   ================........................
              |-- collect --|-- analyze --|-- forward -->

     Output:  ..................========........
              |- engine generates -|- buffer -|- analyze -|-->
                                   t7                     t12
                                   |-- inspection delay --|

   STREAMING (no TTFT impact, adds to ITL):

     Output:  ..o..o..o..o..o..o..o..o..o..o..o..o..o..o..o.
                 |     |     |     |     |     |     |
              inspect  |  inspect  |  inspect  |  inspect
                 |     |     |     |     |     |     |
              --o-----o-----o-----o-----o-----o-----o----->
              Variable delays, increased jitter

                Figure 6: Inspection Architecture Comparison

   Buffered Inspection: The firewall collects complete input (or output)
   before analysis.

   Characteristics:

   *  Adds to TTFT (inbound) or delays token delivery (outbound)

   *  No impact on ITL once streaming starts

   *  Enables deep analysis requiring full context

   For outbound buffered inspection, the client receives the first token
   later than the engine generates it.  This distinction matters:

   Engine TTFT (t7 - t6): 200ms
   Outbound inspection: 50ms
   Client-observed TTFT (t12 - t0): 250ms + network

   Streaming Inspection: The firewall analyzes content as tokens flow
   through.



Gaikwad                   Expires 24 July 2026                 [Page 24]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   Characteristics:

   *  Adds per-token overhead to ITL

   *  May batch or pause tokens during analysis

   *  Introduces jitter in token delivery

   Required measurements:

   +============================+======================================+
   | Metric                     | Definition                           |
   +============================+======================================+
   | Per-token inspection delay | Average latency added per token      |
   +----------------------------+--------------------------------------+
   | Maximum pause duration     | Longest delay during streaming       |
   +----------------------------+--------------------------------------+
   | Pause frequency            | How often inspection causes          |
   |                            | batching                             |
   +----------------------------+--------------------------------------+
   | Jitter contribution        | Standard deviation of delays         |
   +----------------------------+--------------------------------------+

                                  Table 16

   Hybrid Inspection: Initial buffering followed by streaming.  Common
   pattern: buffer first N tokens for context, then stream with spot-
   checks.

   Configuration to disclose:

   *  Buffer threshold (tokens before streaming starts)

   *  Spot-check frequency

   *  Escalation triggers (patterns that switch to full buffering)

4.3.5.  Required Metrics

   Accuracy Metrics:











Gaikwad                   Expires 24 July 2026                 [Page 25]

Internet-Draft          LLM Benchmarking Profiles           January 2026


    +================+================================================+
    | Metric         | Definition                                     |
    +================+================================================+
    | Detection Rate | Fraction of malicious inputs correctly blocked |
    +----------------+------------------------------------------------+
    | False Positive | Fraction of benign inputs blocked by firewall  |
    | Rate (FPR)     |                                                |
    +----------------+------------------------------------------------+
    | False Refusal  | Fraction of policy-compliant requests refused  |
    | Rate (FRR)     | at system boundary                             |
    +----------------+------------------------------------------------+
    | Over-Defense   | FPR conditional on trigger-word presence in    |
    | Rate           | benign inputs                                  |
    +----------------+------------------------------------------------+

                                  Table 17

   FPR vs FRR: FPR measures firewall classifier errors on a benign test
   set.  FRR measures all refusals observed at the system boundary,
   which may include:

   *  Firewall blocks (captured in FPR)

   *  Model refusals (model's own safety behavior)

   *  Policy engine blocks (business rules)

   *  Rate limiting (capacity rejection)

   Therefore: FRR >= FPR when other refusal sources exist.

   When reporting both, attribute refusals by source when possible.

   Over-Defense Rate: Measures false positives on benign inputs that
   contain words commonly associated with attacks.

   Over-Defense Rate = P(Block | Benign AND Contains_Trigger_Words)

   Examples of benign inputs that may trigger over-defense:

   *  "Explain how prompt injection attacks work" (security education)

   *  "What does 'ignore previous instructions' mean?" (linguistic
      question)

   *  "How do I kill a process in Linux?" (technical query)





Gaikwad                   Expires 24 July 2026                 [Page 26]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   The test corpus for over-defense MUST contain semantically benign
   inputs that happen to include trigger words.  Testing with trivially
   benign inputs does not measure real over-defense risk.

   Latency Metrics:

    +========================+=======================================+
    | Metric                 | Definition                            |
    +========================+=======================================+
    | Passing latency        | Overhead when firewall allows request |
    +------------------------+---------------------------------------+
    | Blocking latency       | Time to reach block decision          |
    +------------------------+---------------------------------------+
    | Throughput degradation | Reduction in requests per second      |
    +------------------------+---------------------------------------+

                                 Table 18

   Latency may vary by decision path:

   Example:
     Allow (no flags): 8ms
     Allow (flagged, deep analysis): 45ms
     Block (pattern match): 3ms
     Block (semantic analysis): 67ms

   Report latency distribution by decision type.

4.3.6.  Workload Specification

   AI Firewall benchmarks require careful workload design.

   Benign Workload: Normal traffic with no policy violations.  Measures
   passing latency, FRR, and throughput impact on legitimate use.
   Source: Sanitized production samples or standard datasets.

   Adversarial Workload: Known attack patterns.  Measures detection
   rate, blocking latency, and FPR.  Source: Published datasets (BIPIA
   [BIPIA], JailbreakBench, PromptInject) or red team generated.  Do not
   publish working exploits.

   Mixed Workload (recommended): Combines benign and adversarial at
   declared ratio.








Gaikwad                   Expires 24 July 2026                 [Page 27]

Internet-Draft          LLM Benchmarking Profiles           January 2026


    +========================+=======================================+
    | Parameter              | Example                               |
    +========================+=======================================+
    | Mix ratio              | 95% benign, 5% adversarial            |
    +------------------------+---------------------------------------+
    | Adversarial categories | 40% injection, 30% jailbreak, 30% PII |
    +------------------------+---------------------------------------+
    | Arrival pattern        | Uniform or bursty                     |
    +------------------------+---------------------------------------+

                                 Table 19

4.3.7.  Multi-Layer Firewall

   Production deployments often stack multiple inspection layers.

Request -> Quick Filter -> ML Classifier -> Model -> Semantic Check -> PII Scan -> Response
              |               |                        |              |
           regex           embedding               output           entity
           + rules         classifier             analysis         detection

   When multiple layers exist, report:

   *  Number and position of layers

   *  Per-layer latency

   *  Execution model: Series (latencies add) or parallel

   *  Short-circuit behavior: Does blocking at layer N skip later
      layers?

   Delta decomposition:

Total overhead = Quick(2ms) + ML(12ms) + Semantic(34ms) + PII(8ms) = 56ms

With short-circuit on input block:
  Overhead = Quick(2ms) + ML(12ms) = 14ms

4.3.8.  Benchmarking Constraints

   Blocking speed alone is meaningless.  A firewall blocking all
   requests in 1ms is useless.  Always measure impact on benign traffic
   alongside detection effectiveness.

   Disclose integration with WAF, rate limiting, or DDoS protection.
   These add latency.




Gaikwad                   Expires 24 July 2026                 [Page 28]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   Different attack categories have different detection latencies.
   Pattern-based detection is faster than semantic analysis.  Report
   detection latency by category.

4.4.  Compound System Profile

4.4.1.  Definition and Concepts

   A Compound System executes multiple inference, retrieval, and tool-
   use steps to satisfy a user intent.  The system orchestrates these
   steps, manages state across them, and produces a final response.

   Examples: RAG pipelines, multi-agent systems, tool-using assistants,
   coding agents.

   Unlike single-inference benchmarks, compound system benchmarks
   measure task completion, not token generation.  The primary question
   is "Did it accomplish the goal?" not "How fast did it generate
   tokens?"

4.4.2.  Boundary Specification

   Included in SUT:

   *  Orchestration and planning logic

   *  Multiple LLM inference calls

   *  Retrieval pipeline (embedding, search, reranking)

   *  Tool execution environment

   *  Conversation state management

   *  Agent-to-agent communication

   Excluded from SUT:

   *  External APIs outside the system boundary (latency measured but
      not controlled)

   *  User interface rendering

   *  Arbitrary user-supplied code







Gaikwad                   Expires 24 July 2026                 [Page 29]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   Boundary Rule: The Compound System boundary includes only components
   deployed and controlled as part of the serving system.  User-provided
   plugins or custom code at runtime are excluded.  This prevents
   ambiguity when comparing systems with different extensibility models.

      +========================+===========+========================+
      | Component              | Included? | Rationale              |
      +========================+===========+========================+
      | Built-in retrieval     | Yes       | Part of serving system |
      +------------------------+-----------+------------------------+
      | Standard tool library  | Yes       | Shipped with system    |
      +------------------------+-----------+------------------------+
      | User-uploaded plugin   | No        | User-supplied          |
      +------------------------+-----------+------------------------+
      | External API (weather) | Latency   | Outside boundary       |
      |                        | measured  |                        |
      +------------------------+-----------+------------------------+

                                  Table 20

4.4.3.  Primary Metrics

           +=========================+========================+
           | Metric                  | Definition             |
           +=========================+========================+
           | Task Completion Latency | Time from user request |
           |                         | to final response      |
           +-------------------------+------------------------+
           | Task Success Rate       | Fraction of tasks      |
           |                         | completed correctly    |
           +-------------------------+------------------------+

                                 Table 21

   Task Success has two dimensions:

       +==============+========================+==================+
       | Type         | Definition             | Evaluation       |
       +==============+========================+==================+
       | Hard Success | Structural correctness | Automated (valid |
       |              |                        | JSON, no errors) |
       +--------------+------------------------+------------------+
       | Soft Success | Semantic correctness   | Requires         |
       |              |                        | evaluation       |
       +--------------+------------------------+------------------+

                                 Table 22




Gaikwad                   Expires 24 July 2026                 [Page 30]

Internet-Draft          LLM Benchmarking Profiles           January 2026


4.4.4.  Evaluation Oracle

   When using automated evaluation for Task Success Rate, disclose
   oracle methodology.

   LLM-as-Judge:

    +=====================+===========================================+
    | Parameter           | Report                                    |
    +=====================+===========================================+
    | Judge model         | Identifier and version                    |
    +---------------------+-------------------------------------------+
    | Judge prompt        | Full prompt or published rubric reference |
    +---------------------+-------------------------------------------+
    | Ground truth access | Whether judge sees reference answers      |
    +---------------------+-------------------------------------------+
    | Sampling            | Temperature, judgments per task           |
    +---------------------+-------------------------------------------+

                                  Table 23

   Report inter-rater agreement if using multiple judges.

   Rule-Based Evaluation:

   +====================+==============================================+
   | Parameter          | Report                                       |
   +====================+==============================================+
   | Rule specification | Formal definition                            |
   +--------------------+----------------------------------------------+
   | Coverage           | Fraction of criteria                         |
   |                    | that are rule-checkable                      |
   +--------------------+----------------------------------------------+
   | Edge case handling | How ambiguous cases                          |
   |                    | resolve                                      |
   +--------------------+----------------------------------------------+

                                  Table 24

   Human Evaluation:











Gaikwad                   Expires 24 July 2026                 [Page 31]

Internet-Draft          LLM Benchmarking Profiles           January 2026


    +=================+===============================================+
    | Parameter       | Report                                        |
    +=================+===============================================+
    | Evaluator count | Number of humans                              |
    +-----------------+-----------------------------------------------+
    | Rubric          | Criteria and scoring                          |
    +-----------------+-----------------------------------------------+
    | Agreement       | Inter-rater reliability (e.g., Cohen's Kappa) |
    +-----------------+-----------------------------------------------+
    | Blinding        | Whether evaluators knew system identity       |
    +-----------------+-----------------------------------------------+

                                  Table 25

4.4.5.  Secondary Metrics

        +=====================+===================================+
        | Metric              | Definition                        |
        +=====================+===================================+
        | Trace Depth         | Sequential steps in execution     |
        +---------------------+-----------------------------------+
        | Fan-out Factor      | Maximum parallel sub-requests     |
        +---------------------+-----------------------------------+
        | Sub-Request Count   | Total LLM calls per user request  |
        +---------------------+-----------------------------------+
        | Loop Incidence Rate | Fraction of tasks with repetitive |
        |                     | non-progressing actions           |
        +---------------------+-----------------------------------+
        | Stalled Task Rate   | Fraction of tasks hitting step    |
        |                     | limit without resolution          |
        +---------------------+-----------------------------------+
        | State Management    | Latency and memory for multi-turn |
        | Overhead            | context                           |
        +---------------------+-----------------------------------+

                                  Table 26

   Stalled Task Rate:

   Stalled Task Rate = Tasks_reaching_max_steps / Total_tasks

   Stalled tasks differ from loops.  A loop repeats similar actions.  A
   stalled task may try diverse actions but fail to converge.  Both
   indicate problems but different ones.







Gaikwad                   Expires 24 July 2026                 [Page 32]

Internet-Draft          LLM Benchmarking Profiles           January 2026


4.4.6.  RAG Sub-Profile

   When Compound System includes Retrieval-Augmented Generation:

                           RAG PIPELINE LATENCY

       Query --> Embed --> Search --> Rerank --> Inject --> Generate
         |         |          |          |          |          |
         Q         E          S          R          I          G
         |         |          |          |          |          |
       ----------------------------------------------------------->
       0ms      15ms       60ms       180ms      185ms      385ms
                 |          |          |          |          |
                 +--15ms----+          |          |          |
                      +-----45ms-------+          |          |
                                 +-----120ms------+          |
                                           +--5ms-+          |
                                                 +--200ms----+

       TTFT = E + S + R + I + Prefill + Queue = 385ms

                       Figure 7: RAG Pipeline Latency

   Configuration Disclosure:

          +==============+=====================================+
          | Component    | Parameters                          |
          +==============+=====================================+
          | Embedding    | Model, dimensions, batch size       |
          +--------------+-------------------------------------+
          | Vector store | Type, index configuration           |
          +--------------+-------------------------------------+
          | Search       | Top-k, similarity metric, filters   |
          +--------------+-------------------------------------+
          | Reranking    | Model (if used), top-n after rerank |
          +--------------+-------------------------------------+
          | Context      | Max tokens, formatting template     |
          +--------------+-------------------------------------+

                                 Table 27

   RAG-Specific Metrics:









Gaikwad                   Expires 24 July 2026                 [Page 33]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   +============================+=====================================+
   | Metric                     | Definition                          |
   +============================+=====================================+
   | Embedding Latency          | Query to vector conversion          |
   +----------------------------+-------------------------------------+
   | Retrieval Latency          | Search and fetch time               |
   +----------------------------+-------------------------------------+
   | Retrieval Recall           | Fraction of relevant docs retrieved |
   +----------------------------+-------------------------------------+
   | Context Injection Overhead | Additional prefill from retrieved   |
   |                            | content                             |
   +----------------------------+-------------------------------------+

                                 Table 28

   Corpus Constraints:

        +====================+====================================+
        | Characteristic     | Impact                             |
        +====================+====================================+
        | Corpus size        | Larger means longer search         |
        +--------------------+------------------------------------+
        | Document length    | Longer means more context overhead |
        +--------------------+------------------------------------+
        | Semantic diversity | More diverse reduces precision     |
        +--------------------+------------------------------------+

                                  Table 29

   Report corpus statistics: document count, average length, domain.

   Vector index must be fully built before measurement.

4.4.7.  Agentic System Boundaries

   For multi-agent or tool-using systems:















Gaikwad                   Expires 24 July 2026                 [Page 34]

Internet-Draft          LLM Benchmarking Profiles           January 2026


                       AGENTIC EXECUTION TRACE

   User Request
        |
        v
   +---------+     +---------+     +---------+
   | Planner |---->| Agent A |---->| Agent B |
   |  (LLM)  |     |  (LLM)  |     |  (LLM)  |
   +----+----+     +----+----+     +----+----+
        |               |               |
        |          +----+----+          |
        |          v         v          |
        |     +-------+ +-------+       |
        |     | Tool  | | Tool  |       |
        |     |  API  | |  DB   |       |
        |     +-------+ +-------+       |
        |          |         |          |
        |          +----+----+          |
        |               v               |
        |          +---------+          |
        +--------->|  Final  |<---------+
                   | Response|
                   +---------+

   Trace depth: 4 (Planner -> A -> Tools -> B)
   Fan-out: 2 (parallel tool calls)
   Sub-requests: 3 LLM calls

                     Figure 8: Agentic Execution Trace

   Definitions:

      +====================+========================================+
      | Term               | Definition                             |
      +====================+========================================+
      | Agent invocation   | Single LLM call with specific role     |
      +--------------------+----------------------------------------+
      | Tool call          | External capability invocation         |
      +--------------------+----------------------------------------+
      | Orchestration step | Planning/routing decision              |
      +--------------------+----------------------------------------+
      | Trace              | Complete sequence for one user request |
      +--------------------+----------------------------------------+

                                  Table 30

   Measurement Points:




Gaikwad                   Expires 24 July 2026                 [Page 35]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   +===================+======================+=======================+
   | Metric            | Start                | End                   |
   +===================+======================+=======================+
   | Per-agent latency | Agent receives input | Agent produces output |
   +-------------------+----------------------+-----------------------+
   | Per-tool latency  | Tool call initiated  | Response received     |
   +-------------------+----------------------+-----------------------+
   | Orchestration     | Previous step        | Next step starts      |
   | overhead          | complete             |                       |
   +-------------------+----------------------+-----------------------+
   | Task completion   | User request         | Final response        |
   |                   | received             | delivered             |
   +-------------------+----------------------+-----------------------+

                                 Table 31

4.4.8.  Exclusions

   Custom user application logic and bespoke agent frameworks are out of
   scope.  This profile covers general patterns, not specific
   implementations.

5.  Workload Profiles

   Workload profiles specify traffic patterns applied to infrastructure
   profiles.  They do not define measurement boundaries.

5.1.  Chatbot Workload Profile

5.1.1.  Characteristics

       +================+=========================================+
       | Characteristic | Description                             |
       +================+=========================================+
       | Interaction    | Stateful, multi-turn                    |
       +----------------+-----------------------------------------+
       | Delivery       | Streaming                               |
       +----------------+-----------------------------------------+
       | Arrival        | Closed-loop (user thinks between turns) |
       +----------------+-----------------------------------------+
       | Session length | Variable, typically 3-20 turns          |
       +----------------+-----------------------------------------+

                                 Table 32







Gaikwad                   Expires 24 July 2026                 [Page 36]

Internet-Draft          LLM Benchmarking Profiles           January 2026


5.1.2.  Required Parameters

   +===============+==========================+========================+
   | Parameter     | Description              | Example                |
   +===============+==========================+========================+
   | Arrival model | Open or closed loop      | Closed-loop            |
   +---------------+--------------------------+------------------------+
   | Think-time    | User delay between turns | Exponential, mean=5s   |
   +---------------+--------------------------+------------------------+
   | Input length  | Tokens per user message  | Log-normal, median=50  |
   +---------------+--------------------------+------------------------+
   | Output length | Tokens per response      | Log-normal,            |
   |               |                          | median=150             |
   +---------------+--------------------------+------------------------+
   | Context       | History handling         | Sliding window, 4K     |
   | retention     |                          | tokens                 |
   +---------------+--------------------------+------------------------+
   | Session       | Turns per conversation   | Geometric, mean=8      |
   | length        |                          |                        |
   +---------------+--------------------------+------------------------+

                                  Table 33

5.1.3.  Example

   Chatbot Workload: Customer Support

   Arrival: Closed-loop, 100 concurrent sessions
   Think-time: Exponential(mean=8s)
   Input: Log-normal(mu=4.0, sigma=0.8), range [10, 500]
   Output: Log-normal(mu=5.0, sigma=1.0), range [20, 1000]
   Context: Sliding window, last 4000 tokens
   Session: Geometric(p=0.12), mean ~8 turns
   System prompt: 200 tokens, shared

5.2.  Compound Workflow Workload Profile

5.2.1.  Characteristics

   +================+=================================================+
   | Characteristic | Description                                     |
   +================+=================================================+
   | Execution      | Multi-step, may include parallel branches       |
   +----------------+-------------------------------------------------+
   | Tool usage     | API calls, code execution, database queries     |
   +----------------+-------------------------------------------------+
   | Dependencies   | Steps may depend on previous outputs            |
   +----------------+-------------------------------------------------+



Gaikwad                   Expires 24 July 2026                 [Page 37]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   | Failure modes  | Steps may fail, requiring retry or alternatives |
   +----------------+-------------------------------------------------+

                                 Table 34

5.2.2.  Required Parameters

    +=================+=====================+=========================+
    | Parameter       | Description         | Example                 |
    +=================+=====================+=========================+
    | Task complexity | Steps per task      | Fixed=5 or distribution |
    +-----------------+---------------------+-------------------------+
    | Fan-out pattern | Parallel vs         | Max parallel=3          |
    |                 | sequential          |                         |
    +-----------------+---------------------+-------------------------+
    | Tool latency    | External dependency | Real, mocked, simulated |
    |                 | behavior            |                         |
    +-----------------+---------------------+-------------------------+
    | Failure         | Simulated failures  | 5% tool failure rate    |
    | injection       |                     |                         |
    +-----------------+---------------------+-------------------------+
    | Retry behavior  | Failure handling    | Max 2 retries,          |
    |                 |                     | exponential backoff     |
    +-----------------+---------------------+-------------------------+

                                  Table 35

5.2.3.  External Dependency Handling

   Compound workflows depend on external systems.  Disclose handling:

       +===========+===================+===========================+
       | Approach  | Description       | When                      |
       +===========+===================+===========================+
       | Real      | Actual API calls  | Production-representative |
       +-----------+-------------------+---------------------------+
       | Mocked    | Fixed responses   | Controlled experiments    |
       +-----------+-------------------+---------------------------+
       | Simulated | Statistical model | Reproducible benchmarks   |
       +-----------+-------------------+---------------------------+

                                  Table 36

   Report observed latency and failure rate for real dependencies.
   Report configured values for mocked dependencies.






Gaikwad                   Expires 24 July 2026                 [Page 38]

Internet-Draft          LLM Benchmarking Profiles           January 2026


6.  Delta Measurement Model

   Section 4 and Section 5 defined individual profiles.  Production
   systems compose multiple profiles.  A request may pass through
   Gateway, Firewall, and Engine before response generation.

   Meaningful comparison across composed systems requires attributing
   latency to each component.  This section defines the delta
   measurement model.

6.1.  Timestamp Reference

   Consider a request flowing through a full stack:






































Gaikwad                   Expires 24 July 2026                 [Page 39]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   Client
     |
     |  t0: request sent
     v
   +-------------------+
   |    AI Gateway     |
   |  t1: arrives      |
   |  t2: exits        |
   +-------------------+
     |
     v
   +-------------------+
   |   AI Firewall     |
   |  t3: arrives      |
   |  t4: decision     |
   |  t5: exits        |
   +-------------------+
     |
     v
   +-------------------+
   |   Model Engine    |
   |  t6: queue entry  |
   |  t6a: exec start  |
   |  t7: first token  |
   |  t8: last token   |
   +-------------------+
     |
     v
   +-------------------+
   |   Output Path     |
   |  t9: fw receives  |
   |  t10: fw releases |
   |  t11: gw releases |
   +-------------------+
     |
     v
     t12: client receives first token

                     Figure 9: Request Flow Timestamps

6.2.  Timestamp Definitions

         +===========+==========+===============================+
         | Timestamp | Location | Event                         |
         +===========+==========+===============================+
         | t0        | Client   | Request transmission begins   |
         +-----------+----------+-------------------------------+
         | t1        | Gateway  | Request arrives               |



Gaikwad                   Expires 24 July 2026                 [Page 40]

Internet-Draft          LLM Benchmarking Profiles           January 2026


         +-----------+----------+-------------------------------+
         | t2        | Gateway  | Request exits toward firewall |
         +-----------+----------+-------------------------------+
         | t3        | Firewall | Request arrives               |
         +-----------+----------+-------------------------------+
         | t4        | Firewall | Inbound decision reached      |
         +-----------+----------+-------------------------------+
         | t5        | Firewall | Request exits toward engine   |
         +-----------+----------+-------------------------------+
         | t6        | Engine   | Request enters queue          |
         +-----------+----------+-------------------------------+
         | t6a       | Engine   | Prefill computation begins    |
         +-----------+----------+-------------------------------+
         | t7        | Engine   | First output token generated  |
         +-----------+----------+-------------------------------+
         | t8        | Engine   | Last output token generated   |
         +-----------+----------+-------------------------------+
         | t9        | Firewall | First token arrives for       |
         |           |          | outbound inspection           |
         +-----------+----------+-------------------------------+
         | t10       | Firewall | First token released after    |
         |           |          | inspection                    |
         +-----------+----------+-------------------------------+
         | t11       | Gateway  | First token exits toward      |
         |           |          | client                        |
         +-----------+----------+-------------------------------+
         | t12       | Client   | Client receives first token   |
         +-----------+----------+-------------------------------+

                                 Table 37

6.3.  Component Deltas

   +==========================+===========+===========================+
   | Component                | Formula   | Measures                  |
   +==========================+===========+===========================+
   | Gateway inbound          | t2 - t1   | Auth, validation, routing |
   +--------------------------+-----------+---------------------------+
   | Firewall inbound (pass)  | t5 - t3   | Prompt inspection         |
   +--------------------------+-----------+---------------------------+
   | Firewall inbound (block) | t4 - t3   | Time to block             |
   +--------------------------+-----------+---------------------------+
   | Engine queue             | t6a - t6  | Wait before execution     |
   +--------------------------+-----------+---------------------------+
   | Engine prefill           | t7 - t6a  | Prefill computation       |
   +--------------------------+-----------+---------------------------+
   | Engine TTFT              | t7 - t6   | Queue plus prefill        |
   +--------------------------+-----------+---------------------------+



Gaikwad                   Expires 24 July 2026                 [Page 41]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   | Firewall outbound        | t10 - t9  | Output inspection         |
   +--------------------------+-----------+---------------------------+
   | Gateway outbound         | t11 - t10 | Response processing       |
   +--------------------------+-----------+---------------------------+

                                 Table 38

6.4.  End-to-End Metrics

       +======================+==========+========================+
       | Metric               | Formula  | Notes                  |
       +======================+==========+========================+
       | Engine TTFT          | t7 - t6  | At engine boundary     |
       +----------------------+----------+------------------------+
       | System TTFT          | t12 - t0 | Client-observed        |
       +----------------------+----------+------------------------+
       | Output path overhead | t12 - t7 | Delay from engine emit |
       |                      |          | to client receive      |
       +----------------------+----------+------------------------+

                                 Table 39

6.5.  Clock Synchronization

   Delta metrics within a single component (t2 - t1, both from gateway
   clock) are reliable.  Cross-component deltas (t6 - t5) require clock
   synchronization.

   For end-to-end metrics involving client timestamps (t0, t12), clock
   skew introduces error.

   Options:

   1.  Single-machine measurement (client and server share clock)

   2.  Measure and report skew bounds

   3.  Report server-side metrics only when skew is too large

   Recommended practice: Calculate deltas within components rather than
   across boundaries when possible.

   See Section 9.3 for synchronization requirements.

7.  Profile Composition






Gaikwad                   Expires 24 July 2026                 [Page 42]

Internet-Draft          LLM Benchmarking Profiles           January 2026


7.1.  Composite SUT Declaration

   When SUT includes multiple profiles, testers MUST:

   1.  Enumerate all components in request path:

Client -> AI Gateway -> AI Firewall -> Model Engine -> AI Firewall -> Client

   2.  Declare measurement boundary:

       +===============+==========================================+
       | Type          | Description                              |
       +===============+==========================================+
       | Full-stack    | Client to response, all components       |
       +---------------+------------------------------------------+
       | Per-component | Separate measurement at each boundary    |
       +---------------+------------------------------------------+
       | Partial       | Specific subset (e.g., Gateway + Engine) |
       +---------------+------------------------------------------+

                                 Table 40

   3.  Provide delta decomposition:

   Component          | TTFT Contribution | Throughput Impact
   -------------------|-------------------|------------------
   AI Gateway         | +15ms             | -3%
   AI Firewall (in)   | +45ms             | -8%
   Model Engine       | 180ms (baseline)  | baseline
   AI Firewall (out)  | +12ms*            | -12%
   -------------------|-------------------|------------------
   Total              | 252ms             | -22%

   *Outbound adds to client-observed TTFT, not engine TTFT

7.2.  Composition Validation

   Measure components independently before measuring composite:

   1.  Engine alone: TTFT_engine = 180ms

   2.  Gateway + Engine: TTFT_gw = 195ms, Gateway_delta = 15ms

   3.  Firewall + Engine: TTFT_fw = 225ms, Firewall_delta = 45ms

   4.  Full stack: TTFT_full = 252ms

   5.  Validate: TTFT_engine + deltas approximately equals TTFT_full



Gaikwad                   Expires 24 July 2026                 [Page 43]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   If validation fails, interaction effects exist.  Document them.

7.3.  Interaction Effects

   Components may interact beyond simple addition:

   +==============+==============================+=====================+
   | Effect       | Description                  | Example             |
   +==============+==============================+=====================+
   | Batching     | Gateway batching conflicts   | Gateway batches 8,  |
   | interference | with engine                  | engine max is 4     |
   +--------------+------------------------------+---------------------+
   | Cache        | High gateway cache hit means | Biased difficulty   |
   | interaction  | engine sees hard queries     |                     |
   +--------------+------------------------------+---------------------+
   | Backpressure | Slow component causes        | Firewall slowdown   |
   |              | upstream queuing             | grows gateway       |
   |              |                              | queue               |
   +--------------+------------------------------+---------------------+
   | Timeout      | Mismatched timeouts waste    | See below           |
   | cascades     | resources                    |                     |
   +--------------+------------------------------+---------------------+

                                  Table 41

   Timeout Cascades:

























Gaikwad                   Expires 24 July 2026                 [Page 44]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   TIMEOUT CASCADE (mismatched configurations)

   Gateway timeout:  10s  -------------+
   Firewall timeout: 15s  ---------------------+
   Engine timeout:   30s  -------------------------------------+
                          |            |       |               |
   Time: 0s              10s          15s     30s
         |                |            |       |
         +-- Request ---->|            |       |
         |                |            |       |
         |   Gateway -----X timeout    |       |
         |   (returns error to client) |       |
         |                |            |       |
         |   Firewall -----------------+       |
         |   (still waiting)           |       |
         |                             |       |
         |   Engine ---------------------------+
         |   (completes at 12s, result discarded)
         |                             |       |
         +-------------------------------------+

   Result: Client gets error at 10s. Engine wastes 12s of compute.

                     Figure 10: Timeout Cascade Example

   Report timeout configurations and note mismatches.

8.  Access Logging Requirements

8.1.  Minimum Fields

   All profiles MUST log:



















Gaikwad                   Expires 24 July 2026                 [Page 45]

Internet-Draft          LLM Benchmarking Profiles           January 2026


            +============+===================================+
            | Field      | Description                       |
            +============+===================================+
            | timestamp  | Request start time                |
            +------------+-----------------------------------+
            | request_id | Unique identifier                 |
            +------------+-----------------------------------+
            | profile    | Infrastructure profile under test |
            +------------+-----------------------------------+
            | workload   | Workload profile applied          |
            +------------+-----------------------------------+
            | latency_ms | Total request latency             |
            +------------+-----------------------------------+
            | status     | Success, error, timeout           |
            +------------+-----------------------------------+

                                 Table 42

8.2.  Model Engine Fields

            +=================+==============================+
            | Field           | Description                  |
            +=================+==============================+
            | queue_time_ms   | Time in queue                |
            +-----------------+------------------------------+
            | prefill_time_ms | Prefill latency              |
            +-----------------+------------------------------+
            | decode_time_ms  | Generation time              |
            +-----------------+------------------------------+
            | batch_size      | Concurrent requests in batch |
            +-----------------+------------------------------+
            | token_count_in  | Input tokens                 |
            +-----------------+------------------------------+
            | token_count_out | Output tokens                |
            +-----------------+------------------------------+

                                 Table 43

8.3.  AI Firewall Fields

               +====================+======================+
               | Field              | Description          |
               +====================+======================+
               | direction          | Inbound or outbound  |
               +--------------------+----------------------+
               | decision           | Allow, block, modify |
               +--------------------+----------------------+
               | policy_triggered   | Which policy matched |



Gaikwad                   Expires 24 July 2026                 [Page 46]

Internet-Draft          LLM Benchmarking Profiles           January 2026


               +--------------------+----------------------+
               | confidence         | Detection confidence |
               +--------------------+----------------------+
               | inspection_time_ms | Analysis time        |
               +--------------------+----------------------+

                                  Table 44

8.4.  Compound System Fields

              +==============+==============================+
              | Field        | Description                  |
              +==============+==============================+
              | trace_id     | Identifier linking all steps |
              +--------------+------------------------------+
              | step_count   | Total orchestration steps    |
              +--------------+------------------------------+
              | tool_calls   | List of tools invoked        |
              +--------------+------------------------------+
              | success_type | Hard, soft, or failure       |
              +--------------+------------------------------+

                                  Table 45

8.5.  AI Gateway Fields

                +=================+======================+
                | Field           | Description          |
                +=================+======================+
                | cache_status    | Hit, miss, or bypass |
                +-----------------+----------------------+
                | route_target    | Selected backend     |
                +-----------------+----------------------+
                | token_count_in  | Input tokens         |
                +-----------------+----------------------+
                | token_count_out | Output tokens        |
                +-----------------+----------------------+

                                 Table 46

8.6.  OpenTelemetry Integration

   OpenTelemetry integration SHOULD be supported.  Reference GenAI
   semantic conventions when available.







Gaikwad                   Expires 24 July 2026                 [Page 47]

Internet-Draft          LLM Benchmarking Profiles           January 2026


9.  Measurement Considerations

9.1.  Baseline and Delta Reporting

   For intermediary components (Gateway, Firewall), provide differential
   measurements:

   1.  Measure downstream directly (baseline)

   2.  Measure through intermediary

   3.  Compute delta

   4.  Report both absolute and delta

9.2.  Warm-up and Steady State

   Declare whether results include cold start.

               +==============+===========================+
               | Profile      | Cold Start Factors        |
               +==============+===========================+
               | Model Engine | JIT compilation, KV cache |
               |              | allocation, batch ramp-up |
               +--------------+---------------------------+
               | AI Gateway   | Connection pool, cache    |
               |              | population                |
               +--------------+---------------------------+
               | AI Firewall  | Model loading, rule       |
               |              | compilation               |
               +--------------+---------------------------+
               | Compound     | All above plus retrieval  |
               | System       | index loading             |
               +--------------+---------------------------+

                                 Table 47

   If excluding cold start, report warm-up procedure and duration.

9.3.  Clock Synchronization

    +==========================+==================+===================+
    | Configuration            | Minimum Accuracy | Method            |
    +==========================+==================+===================+
    | Single-machine           | Inherent         | N/A               |
    +--------------------------+------------------+-------------------+
    | Same rack                | 1ms              | NTP               |
    +--------------------------+------------------+-------------------+



Gaikwad                   Expires 24 July 2026                 [Page 48]

Internet-Draft          LLM Benchmarking Profiles           January 2026


    | Distributed              | 100us            | PTP               |
    +--------------------------+------------------+-------------------+
    | Sub-millisecond analysis | 10us             | PTP with hardware |
    |                          |                  | timestamps        |
    +--------------------------+------------------+-------------------+

                                  Table 48

   Reports MUST declare:

   *  Synchronization method

   *  Estimated maximum skew

   *  Single-point or distributed measurement

9.4.  Streaming Protocol Considerations

    +==============+===========================+======================+
    | Profile      | Recommended Protocol      | Notes                |
    +==============+===========================+======================+
    | Model Engine | gRPC streaming            | Lower overhead       |
    +--------------+---------------------------+----------------------+
    | AI Gateway   | SSE over HTTP             | Broad compatibility  |
    +--------------+---------------------------+----------------------+
    | AI Firewall  | Match upstream/downstream | Minimize translation |
    +--------------+---------------------------+----------------------+
    | Compound     | SSE or WebSocket          | Client dependent     |
    | System       |                           |                      |
    +--------------+---------------------------+----------------------+

                                  Table 49

   Report chunk size distribution when measuring ITL.

10.  Security Considerations

10.1.  Bidirectional Enforcement Gaps

   AI Firewalls enforcing only one direction leave systems exposed.

   Inbound-only gaps:

   *  Cannot prevent PII leakage

   *  Cannot catch policy violations from model

   *  Cannot stop tool misuse



Gaikwad                   Expires 24 July 2026                 [Page 49]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   Outbound-only gaps:

   *  Cannot prevent prompt injection

   *  Cannot stop jailbreak attempts

   *  Malicious content reaches model

   Declare which directions are enforced.  "AI Firewall protection"
   without direction is incomplete.

10.2.  Adversarial Workload Handling

   Security requirements for adversarial benchmarks:

   *  Samples MUST NOT contain working exploits

   *  Use sanitized patterns or synthetic constructs

   *  Reference published taxonomies (OWASP LLM Top 10, MITRE ATLAS)

   *  Do not publish novel attacks discovered during testing

10.3.  Side-Channel Considerations

   Performance characteristics may leak information:

      +=========+======================================+============+
      | Channel | Risk                                 | Mitigation |
      +=========+======================================+============+
      | Timing  | Decision time reveals classification | Add noise  |
      +---------+--------------------------------------+------------+
      | Cache   | Hit patterns reveal similarity       | Per-tenant |
      |         |                                      | isolation  |
      +---------+--------------------------------------+------------+
      | Routing | Balancing reveals backend state      | Randomize  |
      +---------+--------------------------------------+------------+

                                  Table 50

   Multi-tenant benchmarks SHOULD measure side-channel exposure.

11.  References

11.1.  Normative References






Gaikwad                   Expires 24 July 2026                 [Page 50]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   [I-D.gaikwad-llm-benchmarking-terminology]
              Gaikwad, M., "Benchmarking Terminology for Large Language
              Model Serving", Work in Progress, Internet-Draft, draft-
              gaikwad-llm-benchmarking-terminology, 2026,
              <https://datatracker.ietf.org/doc/html/draft-gaikwad-llm-
              benchmarking-terminology>.

   [I-D.gaikwad-llm-benchmarking-methodology]
              Gaikwad, M., "Benchmarking Methodology for Large Language
              Model Serving", Work in Progress, Internet-Draft, draft-
              gaikwad-llm-benchmarking-methodology, 2026,
              <https://datatracker.ietf.org/doc/html/draft-gaikwad-llm-
              benchmarking-methodology>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

11.2.  Informative References

   [RFC2647]  Newman, D., "Benchmarking Terminology for Firewall
              Performance", RFC 2647, August 1999,
              <https://www.rfc-editor.org/info/rfc2647>.

   [RFC3511]  Hickman, B., Newman, D., Tadjudin, S., and T. Martin,
              "Benchmarking Methodology for Firewall Performance",
              RFC 3511, April 2003,
              <https://www.rfc-editor.org/info/rfc3511>.

   [OWASP-LLM]
              OWASP Foundation, "OWASP Top 10 for Large Language Model
              Applications", 2023, <https://owasp.org/www-project-top-
              10-for-large-language-model-applications/>.

   [BIPIA]    Yi, J., Xie, Y., Zhu, B., Hines, K., Kiciman, E., Sun, G.,
              and X. Xie, "Benchmarking and Defending Against Indirect
              Prompt Injection Attacks on Large Language Models", 2023,
              <https://arxiv.org/abs/2312.14197>.

   [DISTSERVE]
              Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X.,
              Jin, X., and H. Zhang, "DistServe: Disaggregating Prefill
              and Decoding for Goodput-optimized Large Language Model



Gaikwad                   Expires 24 July 2026                 [Page 51]

Internet-Draft          LLM Benchmarking Profiles           January 2026


              Serving", OSDI 2024, 2024,
              <https://www.usenix.org/conference/osdi24/presentation/
              zhong-yinmin>.

   [MOONCAKE] Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., and
              L. Zhou, "Mooncake: A KVCache-centric Disaggregated
              Architecture for LLM Serving", 2024,
              <https://arxiv.org/abs/2407.00079>.

Appendix A.  Example Benchmark Report Structure

   1. Executive Summary
      - SUT and profile(s) used
      - Key results

   2. System Configuration
      - Hardware
      - Software versions
      - Profile-specific config (per Section 4)

   3. Workload Specification
      - Workload profile
      - Parameters (per Section 5)
      - Dataset sources

   4. Methodology
      - Measurement boundary
      - Clock synchronization
      - Warm-up procedure
      - Duration and request counts

   5. Results
      - Primary metrics with percentiles
      - Secondary metrics
      - Delta decomposition (if composite)

   6. Analysis
      - Observations
      - Interaction effects
      - Limitations

   7. Reproduction
      - Config files
      - Scripts
      - Random seeds

Author's Address




Gaikwad                   Expires 24 July 2026                 [Page 52]

Internet-Draft          LLM Benchmarking Profiles           January 2026


   Madhava Gaikwad
   Independent Researcher
   Email: gaikwad.madhav@gmail.com
















































Gaikwad                   Expires 24 July 2026                 [Page 53]


name: draft-levy-llm-uri-scheme
rev: 00
title: The "llm" URI Scheme
time: 2026-02-06T17:55:33Z
group_uri: /api/v1/group/group/1027/
expires: 2026-08-10T17:55:32Z
states: ["/api/v1/doc/state/1/", "/api/v1/doc/state/150/"]
categories: ["Model serving/inference", "Agent discovery/reg", "Data formats/interop"]
tags: []
fetched_at: 2026-02-27T22:56:07.990775+00:00
abstract:
This document defines the "llm" Uniform Resource Identifier (URI)
   scheme for identifying Large Language Model (LLM) endpoints and their
   configuration.  The scheme encodes provider host, model identifier,
   authentication credentials, and inference parameters into a single,
   portable connection string, analogous to database connection URIs
   such as those used by PostgreSQL and MongoDB.

   This document registers the scheme name in the IANA "Uniform Resource
   Identifier (URI) Schemes" registry as a provisional registration per
   RFC 7595.

full_text:
Individual Submission                                            D. Levy
Internet-Draft                                        DanLevy Consulting
Intended status: Informational                           6 February 2026
Expires: 10 August 2026


                          The "llm" URI Scheme
                      draft-levy-llm-uri-scheme-00

Abstract

   This document defines the "llm" Uniform Resource Identifier (URI)
   scheme for identifying Large Language Model (LLM) endpoints and their
   configuration.  The scheme encodes provider host, model identifier,
   authentication credentials, and inference parameters into a single,
   portable connection string, analogous to database connection URIs
   such as those used by PostgreSQL and MongoDB.

   This document registers the scheme name in the IANA "Uniform Resource
   Identifier (URI) Schemes" registry as a provisional registration per
   RFC 7595.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 10 August 2026.

Copyright Notice

   Copyright (c) 2026 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights



Levy                     Expires 10 August 2026                 [Page 1]

Internet-Draft               LLM URI Scheme                February 2026


   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Motivation  . . . . . . . . . . . . . . . . . . . . . . .   3
     1.2.  Relationship to Other Work  . . . . . . . . . . . . . . .   4
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   4
   3.  URI Syntax  . . . . . . . . . . . . . . . . . . . . . . . . .   4
     3.1.  The "llm" Scheme  . . . . . . . . . . . . . . . . . . . .   4
     3.2.  ABNF  . . . . . . . . . . . . . . . . . . . . . . . . . .   5
   4.  URI Components  . . . . . . . . . . . . . . . . . . . . . . .   5
     4.1.  Scheme  . . . . . . . . . . . . . . . . . . . . . . . . .   5
     4.2.  Authority . . . . . . . . . . . . . . . . . . . . . . . .   5
       4.2.1.  Userinfo  . . . . . . . . . . . . . . . . . . . . . .   5
       4.2.2.  Host  . . . . . . . . . . . . . . . . . . . . . . . .   6
       4.2.3.  Port  . . . . . . . . . . . . . . . . . . . . . . . .   6
     4.3.  Path (Model Identifier) . . . . . . . . . . . . . . . . .   6
     4.4.  Query Parameters  . . . . . . . . . . . . . . . . . . . .   7
     4.5.  Fragment Identifier . . . . . . . . . . . . . . . . . . .   8
   5.  Semantics and Operations  . . . . . . . . . . . . . . . . . .   8
     5.1.  Resolution  . . . . . . . . . . . . . . . . . . . . . . .   8
     5.2.  Operations  . . . . . . . . . . . . . . . . . . . . . . .   9
   6.  Well-Known Query Parameters . . . . . . . . . . . . . . . . .   9
   7.  Examples  . . . . . . . . . . . . . . . . . . . . . . . . . .  10
   8.  Encoding Considerations . . . . . . . . . . . . . . . . . . .  11
   9.  Interoperability Considerations . . . . . . . . . . . . . . .  12
   10. Security Considerations . . . . . . . . . . . . . . . . . . .  12
     10.1.  Credential Exposure  . . . . . . . . . . . . . . . . . .  12
     10.2.  Transport Security . . . . . . . . . . . . . . . . . . .  13
     10.3.  Injection Attacks  . . . . . . . . . . . . . . . . . . .  13
     10.4.  Prompt Injection via URI . . . . . . . . . . . . . . . .  13
     10.5.  Server-Side Request Forgery (SSRF) . . . . . . . . . . .  14
   11. Privacy Considerations  . . . . . . . . . . . . . . . . . . .  14
   12. IANA Considerations . . . . . . . . . . . . . . . . . . . . .  15
     12.1.  URI Scheme Registration: llm . . . . . . . . . . . . . .  15
     12.2.  Well-Known Query Parameters Registry . . . . . . . . . .  15
   13. References  . . . . . . . . . . . . . . . . . . . . . . . . .  15
     13.1.  Normative References . . . . . . . . . . . . . . . . . .  15
     13.2.  Informative References . . . . . . . . . . . . . . . . .  16
   Appendix A.  Comparison with Database Connection URIs . . . . . .  16
   Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  17
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  17





Levy                     Expires 10 August 2026                 [Page 2]

Internet-Draft               LLM URI Scheme                February 2026


1.  Introduction

   Large Language Model (LLM) APIs have become a foundational component
   of modern software systems.  As of 2025, applications commonly
   integrate with multiple LLM providers including OpenAI, Anthropic,
   Google, Mistral, Cohere, and self-hosted models via frameworks such
   as Ollama and vLLM.

   Currently, there is no standardized way to represent an LLM endpoint
   configuration as a single, portable identifier.  Developers typically
   manage provider connections through a proliferation of environment
   variables (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY, MODEL_NAME,
   API_BASE_URL, TEMPERATURE, MAX_TOKENS) that vary by provider,
   framework, and deployment context.

   This document defines the "llm" URI scheme to address this
   fragmentation by encoding all connection parameters into a single URI
   string, following the well-established precedent of database
   connection URIs.

1.1.  Motivation

   The database ecosystem underwent a similar standardization journey.
   Early database clients required separate configuration for host,
   port, username, password, database name, and various driver options.
   The adoption of connection URIs (e.g.,
   postgres://user:pass@host:5432/dbname) dramatically simplified
   configuration, improved portability, and enabled tooling
   standardization.

   The LLM ecosystem is at an analogous inflection point.  A single
   connection string encoding provider, model, authentication, and
   inference parameters would provide:

   *  *Portability:* A single string that moves between environments,
      scripts, and deployment targets.

   *  *CLI ergonomics:* One argument instead of many flags.

   *  *Language agnosticism:* Every programming language includes a
      robust URI parser per RFC 3986, providing parsing, validation, and
      sanitization without additional dependencies.

   *  *Framework interoperability:* A common configuration format that
      middleware, routers, and observability tools can consume.






Levy                     Expires 10 August 2026                 [Page 3]

Internet-Draft               LLM URI Scheme                February 2026


1.2.  Relationship to Other Work

   The "ai" URI scheme (draft-sogomonian-ai-uri-scheme) defines a
   general-purpose scheme for identifying AI-addressable resources such
   as agents, tools, and tasks, with its own resolution protocol (AIIP).
   The "llm" scheme defined in this document is narrower in scope: it is
   a configuration-oriented URI for identifying a specific model
   endpoint and its inference parameters, intended to be resolved by
   client libraries into standard HTTPS API calls.  The two schemes are
   complementary and non-overlapping.

2.  Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

   LLM:  Large Language Model.  A neural network trained on large text
      corpora that generates text completions given an input prompt.

   Provider:  An organization or service that hosts and serves an LLM
      via an HTTP API (e.g., OpenAI, Anthropic, a self-hosted Ollama
      instance).

   Model:  A specific LLM offered by a provider, identified by a model
      name or version string (e.g., "gpt-5", "claude-sonnet-4",
      "llama3").

   Inference Parameters:  Configuration values that control the model's
      output behavior for a given request, such as temperature, top-p,
      and maximum token count.

   Connection String:  A URI conforming to this specification that
      encodes a complete LLM endpoint configuration.

3.  URI Syntax

3.1.  The "llm" Scheme

   The "llm" URI scheme uses the generic URI syntax defined in [RFC3986]
   Section 3, with all standard components:

   llm-URI = "llm:" "//" authority path-abempty
             [ "?" query ] [ "#" fragment ]





Levy                     Expires 10 August 2026                 [Page 4]

Internet-Draft               LLM URI Scheme                February 2026


   The double-slash "//" is used because the scheme defines a
   hierarchical structure with a naming authority (the provider host).

   Example:

   llm://api.openai.com/gpt-5?temp=0.7&max_tokens=1500

   This identifies a GPT-5 model endpoint at api.openai.com with
   temperature 0.7 and a maximum output length of 1500 tokens.

3.2.  ABNF

   Using the ABNF notation of [RFC5234] and the URI components defined
   in [RFC3986]:

   llm-URI       = "llm:" "//" llm-authority
                   llm-path [ "?" query ] [ "#" fragment ]

   llm-authority = [ userinfo "@" ] host [ ":" port ]

   llm-path      = "/" model-id *( "/" sub-path )

   model-id      = 1*( unreserved / pct-encoded / "." / "-" )

   sub-path      = 1*( unreserved / pct-encoded / "." / "-" )

   Where "userinfo", "host", "port", "query", "fragment", "unreserved",
   and "pct-encoded" are as defined in [RFC3986].

4.  URI Components

4.1.  Scheme

   The scheme component MUST be "llm" (case insensitive per [RFC3986]
   Section 3.1, but MUST be registered and SHOULD be written in
   lowercase).

4.2.  Authority

   The authority component identifies the LLM provider and follows the
   syntax defined in Section 3.2 of [RFC3986]:

   authority = [ userinfo "@" ] host [ ":" port ]

4.2.1.  Userinfo

   The userinfo subcomponent, if present, provides authentication
   credentials for the provider API:



Levy                     Expires 10 August 2026                 [Page 5]

Internet-Draft               LLM URI Scheme                February 2026


   userinfo = app-label [ ":" api-key ]

   app-label = 1*( unreserved / pct-encoded )
   api-key   = 1*( unreserved / pct-encoded )

   The "app-label" field identifies the application, project, or
   organization associated with the credential.  Client libraries SHOULD
   map this value to the provider's organization or project header where
   applicable.  For example, OpenAI-compatible APIs accept an OpenAI-
   Organization header; the app-label SHOULD be sent as the value of
   this header (or the provider's equivalent, such as Anthropic-
   Organization or X-Project-ID).  This enables providers to attribute
   usage, enforce per-project rate limits, and route billing.

   The "api-key" field contains the API secret or token.

   Example:

   llm://my-app:sk-proj-abc123@api.openai.com/gpt-5

   Implementations that produce or log LLM URIs MUST sanitize the
   userinfo component to prevent credential leakage (see Section 10).

4.2.2.  Host

   The host identifies the provider's API endpoint.

4.2.3.  Port

   The port subcomponent is OPTIONAL.  When omitted, clients SHOULD use
   port 443 (HTTPS) for remote providers and port 11434 for localhost
   (the conventional Ollama default).  Implementations MAY define other
   defaults.

4.3.  Path (Model Identifier)

   The path component identifies the model.  The first path segment MUST
   be the model identifier.  Additional path segments MAY be used for
   provider-specific namespacing (e.g., organization or deployment
   identifiers).

   The model identifier SHOULD correspond to the model name string
   accepted by the provider's API.

   Examples:






Levy                     Expires 10 August 2026                 [Page 6]

Internet-Draft               LLM URI Scheme                February 2026


   /gpt-5                          -- Simple model name
   /gpt-5.2                        -- Versioned model name
   /anthropic/claude-sonnet-4.5    -- Namespaced model (e.g., for
                                      aggregator providers like
                                      Bedrock or Vercel AI)
   /llama3                         -- Self-hosted model

   Clients MUST NOT interpret path segments beyond mapping them to the
   provider's model selection parameter.  Path semantics are defined by
   the provider.

4.4.  Query Parameters

   The query component carries inference parameters and provider-
   specific options as key=value pairs separated by "&", following
   standard URI query syntax.

   This document defines a set of well-known parameter names (Section 6)
   that implementations SHOULD recognize.  Providers and clients MAY
   define additional parameters.  Unrecognized parameters SHOULD be
   passed through to the provider API where possible and MUST NOT cause
   a client error.

   Parameter names are case-sensitive.  Values MUST be percent-encoded
   per [RFC3986] Section 2.1 when they contain reserved characters.

   Example:

   ?temp=0.7&max_tokens=1500&top_p=0.9&cache=true

   For simple structured parameters, implementations SHOULD use dot-
   notation to flatten nested keys:

   ?web_search.max_uses=3&web_search.enabled=true

   This avoids the brittleness of encoding full JSON objects in query
   strings.  When dot-notation is insufficient (e.g., deeply nested or
   array-valued configurations), implementations MAY percent-encode a
   JSON value:

   ?web_search=%7B%22maxUses%22%3A3%7D

   Which represents: web_search={"maxUses":3}

   Implementations that accept JSON-valued query parameters SHOULD
   impose a maximum length on individual parameter values and MUST be
   aware that many HTTP intermediaries (proxies, CDNs, load balancers)
   enforce URI length limits, commonly between 2,048 and 8,192 octets.



Levy                     Expires 10 August 2026                 [Page 7]

Internet-Draft               LLM URI Scheme                February 2026


   Complex configurations that would exceed practical URI length limits
   SHOULD be provided via out-of-band configuration files or environment
   variables rather than encoded in the URI.

4.5.  Fragment Identifier

   The fragment component, if present, is client-side and is NOT sent to
   the provider.  Fragments MAY be used by client applications for
   purposes such as identifying a specific system prompt variant, a
   named configuration profile, or a UI context.

   Example:

   llm://api.openai.com/gpt-5?temp=0.7#coding-assistant

   The semantics of the fragment are not defined by this specification
   and are left to the consuming application.

5.  Semantics and Operations

5.1.  Resolution

   An "llm" URI does not resolve to a resource in the traditional HTTP
   sense.  Instead, it is a configuration identifier that a client
   library parses to construct API requests to the identified provider.

   The resolution process is:

   1.  Parse the URI per [RFC3986].

   2.  Extract the host to determine the provider API base URL.  The
       transport protocol MUST be HTTPS for remote hosts.  Clients MAY
       use HTTP for localhost/loopback addresses only.

   3.  Extract the path to determine the model identifier.

   4.  Extract userinfo, if present, to obtain authentication
       credentials.

   5.  Extract query parameters and map them to the provider's API
       request format.

   6.  Construct an HTTPS request to the provider's inference endpoint
       (e.g., POST /v1/chat/completions for OpenAI-compatible APIs).

   Clients MUST NOT dereference an "llm" URI by making a GET request to
   the URI itself.  The URI is a configuration descriptor, not a
   resource locator.



Levy                     Expires 10 August 2026                 [Page 8]

Internet-Draft               LLM URI Scheme                February 2026


5.2.  Operations

   The primary operation associated with an LLM URI is inference:
   sending a prompt to the identified model and receiving a completion.
   This document does not define the prompt or response format, which
   varies by provider and is outside the scope of URI scheme definition.

   Implementations MAY support additional operations such as:

   *  Model metadata retrieval (listing capabilities, context window
      size, supported parameters).

   *  Streaming inference (server-sent events or chunked transfer).

   *  Batch inference.

   *  Embedding generation (when the identified model supports it).

6.  Well-Known Query Parameters

   The following query parameter names are RECOMMENDED for
   interoperability.  Implementations SHOULD map these to the
   corresponding provider API parameters.




























Levy                     Expires 10 August 2026                 [Page 9]

Internet-Draft               LLM URI Scheme                February 2026


     +============+========+=========================================+
     | Parameter  | Type   | Description                             |
     +============+========+=========================================+
     | temp       | float  | Sampling temperature (0.0 to 2.0)       |
     +------------+--------+-----------------------------------------+
     | max_tokens | int    | Maximum tokens in the completion        |
     +------------+--------+-----------------------------------------+
     | top_p      | float  | Nucleus sampling threshold (0.0 to 1.0) |
     +------------+--------+-----------------------------------------+
     | top_k      | int    | Top-k sampling parameter                |
     +------------+--------+-----------------------------------------+
     | stop       | string | Stop sequence(s), comma-separated       |
     +------------+--------+-----------------------------------------+
     | seed       | int    | Random seed for reproducibility         |
     +------------+--------+-----------------------------------------+
     | stream     | bool   | Enable streaming response               |
     +------------+--------+-----------------------------------------+
     | timeout    | int    | Request timeout in milliseconds         |
     +------------+--------+-----------------------------------------+
     | retries    | int    | Maximum retry attempts                  |
     +------------+--------+-----------------------------------------+
     | cache      | bool   | Enable response caching                 |
     +------------+--------+-----------------------------------------+
     | format     | string | Response format (e.g., "json")          |
     +------------+--------+-----------------------------------------+
     | system     | string | System prompt (percent-encoded)         |
     +------------+--------+-----------------------------------------+

                                  Table 1

   Boolean values MUST be represented as "true" or "false" (case-
   insensitive).  Numeric values MUST be represented in decimal
   notation.

   Provider-specific parameters (e.g., "frequency_penalty",
   "presence_penalty", "cacheControl") SHOULD use the parameter names
   defined by the provider's API documentation and MUST be percent-
   encoded if they contain reserved characters.

7.  Examples

   Basic usage with a cloud provider:

   llm://api.openai.com/gpt-5?temp=0.7&max_tokens=1500

   With authentication:

   llm://my-app:sk-proj-abc123@api.openai.com/gpt-5?temp=0.7



Levy                     Expires 10 August 2026                [Page 10]

Internet-Draft               LLM URI Scheme                February 2026


   Self-hosted model (Ollama):

   llm://localhost:11434/llama3

   Anthropic via aggregator (Bedrock):

   llm://bedrock.us-west-2.amazonaws.com/anthropic/claude-sonnet-4.5
     ?temp=0.8&cache=true

   Vercel AI SDK-style namespacing:

   llm://sdk.vercel.ai/anthropic/claude-sonnet-4.5
     ?temp=0.8&web_search=%7B%22maxUses%22%3A3%7D

   Local model with JSON output:

   llm://localhost:11434/mistral?format=json&temp=0

   With fragment for application context:

   llm://api.anthropic.com/claude-sonnet-4.5
     ?temp=0.3&max_tokens=4096#code-review

8.  Encoding Considerations

   LLM URIs MUST conform to the encoding rules defined in [RFC3986]
   Section 2.  In particular:

   *  The model identifier in the path component MUST percent-encode any
      characters not in the "unreserved" set or "/" delimiter.

   *  Query parameter values MUST percent-encode reserved characters per
      Section 2.2 of [RFC3986].

   *  The userinfo component MUST percent-encode ":" in API keys that
      contain literal colons, and "@" in any subcomponent.

   *  Structured values (JSON objects) in query parameters MUST be fully
      percent-encoded.

   Implementations SHOULD produce URIs using UTF-8 encoding for any non-
   ASCII characters, consistent with [RFC3987] IRI-to-URI mapping.









Levy                     Expires 10 August 2026                [Page 11]

Internet-Draft               LLM URI Scheme                February 2026


9.  Interoperability Considerations

   The "llm" scheme is designed to be consumed by LLM client libraries,
   CLI tools, middleware routers, and orchestration frameworks.  It is
   NOT designed to be resolved by web browsers or general-purpose HTTP
   clients.

   Known interoperability considerations include:

   *  *Provider API Variance:* LLM providers use different parameter
      names for equivalent concepts (e.g., "temperature" vs "temp",
      "max_tokens" vs "maxOutputTokens").  Client libraries SHOULD map
      the well-known parameter names defined in Section 6 to the
      provider's expected parameter names.

   *  *Model Name Formats:* Some providers use simple names ("gpt-5"),
      others use versioned names ("gpt-5-2025-01-15"), and aggregator
      services use namespaced paths ("anthropic/claude-sonnet-4.5").
      The path component accommodates all of these formats.

   *  *Authentication Methods:* While this scheme supports credential
      embedding via userinfo, some providers require additional
      authentication mechanisms (e.g., OAuth 2.0 tokens, IAM roles, mTLS
      certificates) that cannot be represented in a URI.  Clients MAY
      use query parameters or out-of-band configuration for such
      mechanisms.

10.  Security Considerations

   The "llm" URI scheme presents several security considerations that
   implementations MUST address.

10.1.  Credential Exposure

   LLM URIs MAY contain API keys in the userinfo component.  This
   creates a risk of credential leakage through:

   *  Application logs and monitoring systems.

   *  Browser history and bookmarks (if URIs are inadvertently used in
      web contexts).

   *  Shell history files.

   *  Error messages and stack traces.

   *  Version control systems (.env files committed to repositories).




Levy                     Expires 10 August 2026                [Page 12]

Internet-Draft               LLM URI Scheme                February 2026


   Implementations that produce, store, or transmit LLM URIs MUST:

   *  Sanitize the userinfo component before logging, by replacing the
      api-key portion with a redaction marker (e.g., "***").

   *  Treat LLM URIs containing credentials with the same care as any
      other bearer token or secret.

   Implementations SHOULD support credential resolution via environment
   variables or secret managers as an alternative to embedding
   credentials directly in the URI.

10.2.  Transport Security

   Clients MUST use HTTPS (TLS 1.2 or later, TLS 1.3 RECOMMENDED) for
   all remote provider communication.  Clients MUST NOT send credentials
   or inference requests over unencrypted HTTP to non-loopback
   addresses.

   For localhost and loopback addresses (127.0.0.0/8, ::1), clients MAY
   use HTTP, as this is common for self-hosted model servers.

10.3.  Injection Attacks

   Because query parameters are passed to provider APIs, implementations
   MUST validate and sanitize all parameter values before including them
   in API requests.  In particular:

   *  Numeric parameters MUST be validated as numeric.

   *  Boolean parameters MUST be validated as boolean.

   *  String parameters MUST be bounded in length.

   *  Structured values (JSON) MUST be parsed and validated before
      forwarding.

10.4.  Prompt Injection via URI

   The "system" query parameter, if supported, carries a percent-encoded
   system prompt.  This creates a vector for prompt injection if URIs
   are constructed from untrusted input.  Implementations SHOULD treat
   the "system" parameter as untrusted input and apply appropriate
   content filtering.







Levy                     Expires 10 August 2026                [Page 13]

Internet-Draft               LLM URI Scheme                February 2026


10.5.  Server-Side Request Forgery (SSRF)

   Applications that accept LLM URIs from untrusted sources (e.g., user-
   submitted bot configurations, webhook endpoints, or plugin manifests)
   are vulnerable to Server-Side Request Forgery.  An attacker could
   supply a URI such as llm://localhost:8080/admin or
   llm://169.254.169.254/latest/meta-data to cause the client library to
   issue HTTP requests to internal services, cloud metadata endpoints,
   or other resources that should not be externally reachable.

   Implementations that accept LLM URIs from untrusted input MUST:

   *  Maintain an allowlist of permitted provider hosts, OR maintain a
      blocklist that includes at minimum: loopback addresses
      (127.0.0.0/8, ::1), link-local addresses (169.254.0.0/16,
      fe80::/10), private network ranges (10.0.0.0/8, 172.16.0.0/12,
      192.168.0.0/16), and cloud metadata endpoints.

   *  Resolve hostnames and validate the resulting IP address against
      the allowlist/blocklist BEFORE establishing a connection, to
      prevent DNS rebinding attacks.

   *  Reject URIs with IP address literals in the host component when
      operating in production environments, unless explicitly configured
      to allow them.

   In production deployments, clients SHOULD default to blocking
   loopback and private network addresses.  Localhost access SHOULD only
   be permitted when explicitly enabled for local model serving (e.g.,
   Ollama).

11.  Privacy Considerations

   LLM URIs may reveal information about:

   *  Which AI providers and models a user or organization employs.

   *  Inference configuration choices that may indicate use case (e.g.,
      low temperature for code generation, high temperature for creative
      writing).

   *  Application identifiers embedded in the userinfo app-label.

   When LLM URIs are shared, logged, or transmitted, this metadata
   leakage SHOULD be considered.  Implementations SHOULD provide
   mechanisms to strip or redact URI components before sharing.





Levy                     Expires 10 August 2026                [Page 14]

Internet-Draft               LLM URI Scheme                February 2026


12.  IANA Considerations

12.1.  URI Scheme Registration: llm

   In accordance with [RFC7595], this section provides the registration
   request for the "llm" URI scheme.

   Scheme name:  llm

   Status:  Provisional

   Applications/protocols that use this scheme name:  LLM client
      libraries, AI application frameworks, CLI tools, and orchestration
      systems that connect to Large Language Model inference APIs.

   Contact:  Dan Levy <dan@danlevy.net>

   Change controller:  Dan Levy <dan@danlevy.net>

   References:  This document; RFC 3986; RFC 7595.

12.2.  Well-Known Query Parameters Registry

   This document requests IANA create a new registry titled "LLM URI
   Scheme Well-Known Query Parameters" under the "Uniform Resource
   Identifier (URI) Schemes" group.

   The initial contents of the registry are the parameters defined in
   Section 6 of this document.  New entries may be added via
   Specification Required [RFC8126] policy.

13.  References

13.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/rfc/rfc2119>.

   [RFC3986]  Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform
              Resource Identifier (URI): Generic Syntax", STD 66,
              RFC 3986, DOI 10.17487/RFC3986, January 2005,
              <https://www.rfc-editor.org/rfc/rfc3986>.







Levy                     Expires 10 August 2026                [Page 15]

Internet-Draft               LLM URI Scheme                February 2026


   [RFC5234]  Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
              Specifications: ABNF", STD 68, RFC 5234,
              DOI 10.17487/RFC5234, January 2008,
              <https://www.rfc-editor.org/rfc/rfc5234>.

   [RFC7595]  Thaler, D., Ed., Hansen, T., and T. Hardie, "Guidelines
              and Registration Procedures for URI Schemes", BCP 35,
              RFC 7595, DOI 10.17487/RFC7595, June 2015,
              <https://www.rfc-editor.org/rfc/rfc7595>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

13.2.  Informative References

   [LLM-CONN] Levy, D., "It's Time for LLM Connection Strings", January
              2026, <https://danlevy.net/llm-connection-strings/>.

   [OPENAI-API]
              OpenAI, "API Reference", n.d.,
              <https://platform.openai.com/docs/api-reference>.

   [RFC3987]  Duerst, M. and M. Suignard, "Internationalized Resource
              Identifiers (IRIs)", RFC 3987, DOI 10.17487/RFC3987,
              January 2005, <https://www.rfc-editor.org/rfc/rfc3987>.

   [RFC8126]  Cotton, M., Leiba, B., and T. Narten, "Guidelines for
              Writing an IANA Considerations Section in RFCs", BCP 26,
              RFC 8126, DOI 10.17487/RFC8126, June 2017,
              <https://www.rfc-editor.org/rfc/rfc8126>.

   [RFC8446]  Rescorla, E., "The Transport Layer Security (TLS) Protocol
              Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018,
              <https://www.rfc-editor.org/rfc/rfc8446>.

Appendix A.  Comparison with Database Connection URIs

   The following table illustrates the structural analogy between the
   "llm" scheme and established database connection URI schemes:











Levy                     Expires 10 August 2026                [Page 16]

Internet-Draft               LLM URI Scheme                February 2026


    +===========+=========================+===========================+
    | Component | Database URI (postgres) | LLM URI (llm)             |
    +===========+=========================+===========================+
    | Scheme    | postgres://             | llm://                    |
    +-----------+-------------------------+---------------------------+
    | Auth      | user:password           | app-label:api-key         |
    +-----------+-------------------------+---------------------------+
    | Host      | db.example.com          | api.openai.com            |
    +-----------+-------------------------+---------------------------+
    | Port      | :5432                   | :443 (default)            |
    +-----------+-------------------------+---------------------------+
    | Resource  | /database_name          | /model-name               |
    +-----------+-------------------------+---------------------------+
    | Options   | ?sslmode=require        | ?temp=0.7&max_tokens=1500 |
    +-----------+-------------------------+---------------------------+

                                  Table 2

   This structural equivalence means that existing URI parsing
   libraries, URL-safe storage mechanisms, and secret management tools
   that already handle database URIs can be extended to handle LLM URIs
   with minimal effort.

Acknowledgements

   The concept of LLM connection strings was first described in "It's
   Time for LLM Connection Strings" [LLM-CONN] published at danlevy.net.
   The author thanks Yakov Shafranovich for suggesting IANA registration
   and the commenters who provided early feedback.

Author's Address

   Dan Levy
   DanLevy Consulting
   Denver, CO,
   United States of America
   Email: dan@danlevy.net
   URI:   https://danlevy.net













Levy                     Expires 10 August 2026                [Page 17]


name: draft-condrey-rats-witnessd-enrollment
rev: 00
title: Trust Anchor Bootstrap Protocol for Proof of Process
time: 2026-02-07T10:31:35Z
group_uri: /api/v1/group/group/1027/
expires: 2026-08-11T10:31:34Z
states: ["/api/v1/doc/state/1/", "/api/v1/doc/state/150/"]
categories: ["Agent identity/auth", "Policy/governance", "Autonomous netops"]
tags: []
fetched_at: 2026-02-27T22:56:07.990779+00:00
abstract:
This document specifies a trust anchor bootstrap protocol for the
   witnessd Proof of Process framework.  The protocol defines how new
   devices enter the witnessd ecosystem, how device keys are provisioned
   and verified, and how Verifiers discover and validate device trust
   anchors.

   Three enrollment modes are defined: self-sovereign mode for
   individual users, organizational mode for enterprise deployments, and
   public registry mode for federated trust ecosystems.

full_text:
Remote ATtestation procedureS                                 D. Condrey
Internet-Draft                                          Writerslogic Inc
Intended status: Standards Track                         7 February 2026
Expires: 11 August 2026


          Trust Anchor Bootstrap Protocol for Proof of Process
               draft-condrey-rats-witnessd-enrollment-00

Abstract

   This document specifies a trust anchor bootstrap protocol for the
   witnessd Proof of Process framework.  The protocol defines how new
   devices enter the witnessd ecosystem, how device keys are provisioned
   and verified, and how Verifiers discover and validate device trust
   anchors.

   Three enrollment modes are defined: self-sovereign mode for
   individual users, organizational mode for enterprise deployments, and
   public registry mode for federated trust ecosystems.

About This Document

   This note is to be removed before publishing as an RFC.

   This is a companion document to draft-condrey-rats-pop, which defines
   the core Proof of Process evidence framework.  This document
   addresses the bootstrapping problem: how do Verifiers know which
   device keys to trust?

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 11 August 2026.





Condrey                  Expires 11 August 2026                 [Page 1]

Internet-Draft             witnessd Enrollment             February 2026


Copyright Notice

   Copyright (c) 2026 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Requirements Notation . . . . . . . . . . . . . . . . . .   3
   2.  Enrollment Modes  . . . . . . . . . . . . . . . . . . . . . .   3
     2.1.  Self-Sovereign Mode . . . . . . . . . . . . . . . . . . .   3
     2.2.  Organizational Mode . . . . . . . . . . . . . . . . . . .   4
       2.2.1.  Well-Known URI for Device Discovery . . . . . . . . .   4
     2.3.  Public Registry Mode  . . . . . . . . . . . . . . . . . .   4
   3.  Device Enrollment Protocol  . . . . . . . . . . . . . . . . .   5
     3.1.  Enrollment Request  . . . . . . . . . . . . . . . . . . .   5
     3.2.  Challenge-Response Flow . . . . . . . . . . . . . . . . .   5
     3.3.  Certificate Issuance  . . . . . . . . . . . . . . . . . .   6
   4.  Trust Anchor Discovery  . . . . . . . . . . . . . . . . . . .   7
     4.1.  Discovery Mechanisms  . . . . . . . . . . . . . . . . . .   7
     4.2.  Caching Policy  . . . . . . . . . . . . . . . . . . . . .   7
     4.3.  Revocation Checking Integration . . . . . . . . . . . . .   8
   5.  Key Lifecycle Management  . . . . . . . . . . . . . . . . . .   8
     5.1.  Key Rotation  . . . . . . . . . . . . . . . . . . . . . .   8
     5.2.  Device Migration  . . . . . . . . . . . . . . . . . . . .   9
     5.3.  Key Compromise Response . . . . . . . . . . . . . . . . .   9
     5.4.  Certificate Renewal . . . . . . . . . . . . . . . . . . .  10
   6.  Security Considerations . . . . . . . . . . . . . . . . . . .  10
     6.1.  Threat Model  . . . . . . . . . . . . . . . . . . . . . .  10
     6.2.  Compromise Impact Analysis  . . . . . . . . . . . . . . .  11
     6.3.  Mitigation Requirements . . . . . . . . . . . . . . . . .  11
   7.  Privacy Considerations  . . . . . . . . . . . . . . . . . . .  11
     7.1.  Device Identifier Privacy . . . . . . . . . . . . . . . .  12
     7.2.  Enrollment Metadata Minimization  . . . . . . . . . . . .  12
     7.3.  Registry Privacy Modes  . . . . . . . . . . . . . . . . .  12
   8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  13
   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  13
     9.1.  Normative References  . . . . . . . . . . . . . . . . . .  13
     9.2.  Informative References  . . . . . . . . . . . . . . . . .  13



Condrey                  Expires 11 August 2026                 [Page 2]

Internet-Draft             witnessd Enrollment             February 2026


   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  13

1.  Introduction

   The Proof of Process (PoP) specification [I-D.condrey-rats-pop]
   defines how Attesters generate Evidence during document authorship.
   Evidence packets are signed with device keys, but the core
   specification does not define how these keys are provisioned or how
   Verifiers establish trust in them.

   This document addresses the trust anchor bootstrap problem by
   defining:

   *  Device enrollment protocols for provisioning signing keys

   *  Trust anchor discovery mechanisms for Verifiers

   *  Multiple enrollment modes for different deployment scenarios

   *  Key lifecycle management including rotation and migration

1.1.  Requirements Notation

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

2.  Enrollment Modes

2.1.  Self-Sovereign Mode

   In self-sovereign mode, users generate their own device keys and
   manually provide trust anchors to Verifiers they wish to work with.
   This mode provides maximum privacy but requires out-of-band trust
   establishment.

   Workflow:

   1.  Device generates key pair in secure hardware (if available)

   2.  Device exports public key and self-signed certificate

   3.  User manually provides public key to Verifier (via secure
       channel)

   4.  Verifier stores trust anchor associated with user identity



Condrey                  Expires 11 August 2026                 [Page 3]

Internet-Draft             witnessd Enrollment             February 2026


   Use cases: Individual authors, privacy-conscious users, testing
   environments.

2.2.  Organizational Mode

   In organizational mode, device keys are signed by an organization's
   Certificate Authority (CA).  The organization publishes its CA
   certificate, and Verifiers trust devices whose keys chain to the
   organization's CA.

   Workflow:

   1.  Organization establishes CA infrastructure

   2.  Organization publishes CA certificate at well-known location

   3.  Device generates key pair and creates certificate signing request

   4.  Organization signs device certificate (may require proof of
       device identity)

   5.  Device stores signed certificate

   6.  Verifiers validate device certificates against organization CA

   Use cases: Enterprises, academic institutions, publishing houses.

2.2.1.  Well-Known URI for Device Discovery

   Organizations SHOULD publish device trust information at:

   /.well-known/witnessd-devices

   This endpoint returns a signed JSON document listing:

   *  Organization CA certificate chain

   *  Supported enrollment methods

   *  Device certificate validation policy

   *  Contact information for enrollment requests

2.3.  Public Registry Mode

   In public registry mode, devices register with a public discovery
   service, similar to ACME for TLS certificates.  This enables
   federated trust without requiring prior relationship establishment.



Condrey                  Expires 11 August 2026                 [Page 4]

Internet-Draft             witnessd Enrollment             February 2026


   Workflow:

   1.  Device generates key pair

   2.  Device proves control of user identifier (email, domain, etc.)

   3.  Registry issues device certificate bound to user identifier

   4.  Registry publishes certificate in transparency log

   5.  Verifiers query registry to validate device certificates

   Use cases: Open ecosystems, cross-organization verification, consumer
   applications.

3.  Device Enrollment Protocol

   The device enrollment protocol establishes trust between a new device
   and the enrollment authority.  The protocol uses a challenge-response
   flow to verify device identity and key possession.

3.1.  Enrollment Request

   Devices initiate enrollment by submitting a request containing:

   enrollment-request = {
     device-public-key: bstr,      ; Device's public key (COSE_Key)
     user-identifier: tstr,        ; Email, domain, or opaque ID
     enrollment-mode: mode-type,
     ? device-attestation: bstr,   ; TPM/Secure Enclave attestation
     ? proof-of-possession: bstr,  ; Signature over challenge
     nonce: bstr,                  ; Freshness nonce
     timestamp: time
   }

   mode-type = &(
     self-sovereign: 0,
     organizational: 1,
     public-registry: 2
   )

3.2.  Challenge-Response Flow

   For organizational and public registry modes, the enrollment
   authority issues a challenge:

   1.  Device submits initial enrollment request




Condrey                  Expires 11 August 2026                 [Page 5]

Internet-Draft             witnessd Enrollment             February 2026


   2.  Authority returns challenge (random nonce + user verification
       requirement)

   3.  Device signs challenge with private key (proof of possession)

   4.  User completes verification (email link, domain TXT record, etc.)

   5.  Device submits signed challenge and verification proof

   6.  Authority validates and issues device certificate

   enrollment-challenge = {
     challenge-nonce: bstr,
     verification-method: verification-type,
     ? verification-uri: tstr,    ; For email/web verification
     expires: time
   }

   verification-type = &(
     email-link: 0,
     domain-txt: 1,
     organizational-approval: 2,
     device-attestation-only: 3
   )

   enrollment-response = {
     challenge-nonce: bstr,
     challenge-signature: bstr,   ; Signed by device private key
     verification-token: tstr,    ; From email/domain verification
     timestamp: time
   }

3.3.  Certificate Issuance

   Upon successful verification, the authority issues a device
   certificate:















Condrey                  Expires 11 August 2026                 [Page 6]

Internet-Draft             witnessd Enrollment             February 2026


   device-certificate = {
     device-id: bstr,             ; Unique device identifier
     public-key: bstr,            ; COSE_Key
     user-identifier: tstr,
     issuer: tstr,                ; Authority identifier
     issued-at: time,
     expires-at: time,
     ? constraints: [* constraint],
     signature: bstr              ; Authority signature
   }

   constraint = &(
     max-evidence-per-day: uint,
     allowed-document-types: [* tstr],
     geographic-restriction: tstr
   )

   Certificates MUST be valid for no more than 1 year.  Devices MUST re-
   enroll before certificate expiration.

4.  Trust Anchor Discovery

   Verifiers must discover and maintain trust anchors for device
   certificate validation.  This section defines discovery mechanisms
   and caching policies.

4.1.  Discovery Mechanisms

   Verifiers discover trust anchors through:

   Well-known URI:  Query /.well-known/witnessd-devices at the
      organization's domain to retrieve CA certificates and enrollment
      policies.

   Public Registry:  Query the public registry service for device
      certificates bound to user identifiers.  The registry provides
      certificate chains and revocation status.

   Out-of-band provisioning:  For self-sovereign mode, trust anchors are
      manually configured by the Verifier operator.

4.2.  Caching Policy

   Verifiers SHOULD cache trust anchors to reduce latency and network
   load.  Caching policies:

   *  CA certificates: Cache for up to 24 hours or until the next-update
      field indicates refresh is needed



Condrey                  Expires 11 August 2026                 [Page 7]

Internet-Draft             witnessd Enrollment             February 2026


   *  Device certificates: Cache for the certificate validity period
      minus a safety margin (RECOMMENDED: 1 hour)

   *  Revocation status: Cache according to revocation list TTL (see
      draft-condrey-rats-witnessd-revocation)

   Verifiers MUST refresh cached trust anchors before expiration.  Stale
   cache entries MUST NOT be used for verification.

4.3.  Revocation Checking Integration

   Trust anchor discovery integrates with revocation checking:

   1.  Verifier receives Evidence packet

   2.  Verifier extracts device certificate from Evidence

   3.  Verifier validates certificate chain to trusted CA

   4.  Verifier checks certificate revocation status

   5.  Verifier checks Evidence revocation status

   6.  If all checks pass, Verifier proceeds with appraisal

   Certificate revocation and Evidence revocation are independent.  A
   revoked certificate invalidates all Evidence signed by that device.
   Evidence may be revoked even if the certificate remains valid.

5.  Key Lifecycle Management

   Device keys have a lifecycle from generation through retirement.
   This section defines key management operations.

5.1.  Key Rotation

   Devices SHOULD rotate signing keys periodically to limit exposure
   from potential compromise.  Key rotation workflow:

   1.  Device generates new key pair

   2.  Device requests certificate for new key (standard enrollment)

   3.  Device publishes key rotation notice signed by old key

   4.  Rotation notice links old key to new key

   5.  Old key enters grace period (RECOMMENDED: 30 days)



Condrey                  Expires 11 August 2026                 [Page 8]

Internet-Draft             witnessd Enrollment             February 2026


   6.  After grace period, old key is retired

   key-rotation-notice = {
     old-device-id: bstr,
     new-device-id: bstr,
     old-public-key: bstr,
     new-public-key: bstr,
     rotation-time: time,
     grace-period-end: time,
     old-key-signature: bstr,    ; Proves control of old key
     new-key-signature: bstr     ; Proves control of new key
   }

   Evidence signed during the grace period MAY use either key.  After
   the grace period, only the new key is valid.

5.2.  Device Migration

   When users migrate to a new device, Evidence continuity must be
   maintained.  Migration is treated as key rotation with additional
   verification:

   *  User initiates migration from old device (if available)

   *  Old device signs migration authorization for new device

   *  New device completes enrollment with migration authorization

   *  If old device is unavailable, user completes out-of-band
      verification (email, organizational approval, etc.)

   Migration without old device access requires stronger verification to
   prevent unauthorized device takeover.

5.3.  Key Compromise Response

   When a device key is compromised:

   1.  User reports compromise to enrollment authority

   2.  Authority revokes device certificate immediately

   3.  User enrolls new device with fresh key

   4.  User may re-endorse historical Evidence with new key (see draft-
       condrey-rats-witnessd-revocation)





Condrey                  Expires 11 August 2026                 [Page 9]

Internet-Draft             witnessd Enrollment             February 2026


   Hardware-backed keys (TPM, Secure Enclave) reduce compromise risk and
   are RECOMMENDED for high-security deployments.

5.4.  Certificate Renewal

   Devices MUST renew certificates before expiration.  Renewal may use
   the existing key (if not compromised) or coincide with key rotation.
   Renewal workflow:

   1.  Device submits renewal request with current certificate

   2.  Authority validates current certificate is still valid

   3.  Authority issues new certificate (same or new key)

   4.  New certificate validity period begins from issuance

   Devices SHOULD initiate renewal at least 7 days before certificate
   expiration to allow for processing delays.

6.  Security Considerations

6.1.  Threat Model

   The enrollment protocol considers the following threats:

   Unauthorized enrollment:  Attacker attempts to enroll a device under
      another user's identity.  Mitigated by user identifier
      verification (email, domain, organizational approval).

   Key extraction:  Attacker attempts to extract device private key.
      Mitigated by hardware-backed key storage (TPM, Secure Enclave)
      where available.

   Enrollment authority compromise:  Attacker compromises the enrollment
      authority and issues fraudulent certificates.  Mitigated by
      certificate transparency logs and short certificate validity
      periods.

   Man-in-the-middle:  Attacker intercepts enrollment messages.
      Mitigated by TLS for all enrollment communications and challenge-
      response binding.

   Replay attacks:  Attacker replays old enrollment requests.  Mitigated
      by nonces and timestamps in all protocol messages.






Condrey                  Expires 11 August 2026                [Page 10]

Internet-Draft             witnessd Enrollment             February 2026


6.2.  Compromise Impact Analysis

   The impact of various compromises:

   *  Single device key compromise: Only Evidence from that device is
      affected.  Other devices and users are unaffected.

   *  Organizational CA compromise: All devices enrolled under that CA
      are potentially affected.  Requires CA re-keying and mass device
      re-enrollment.

   *  Public registry compromise: All devices using that registry are
      potentially affected.  Requires registry recovery and certificate
      reissuance.

   Organizational deployments SHOULD use Hardware Security Modules
   (HSMs) for CA key protection.  Public registries MUST implement
   multi-party controls for certificate issuance.

6.3.  Mitigation Requirements

   Implementations MUST:

   *  Use TLS 1.3 or later for all enrollment communications

   *  Validate certificate chains completely before trusting

   *  Check certificate revocation status before accepting Evidence

   *  Implement rate limiting to prevent enrollment abuse

   *  Log enrollment events for audit purposes

   Implementations SHOULD:

   *  Use hardware-backed key storage where available

   *  Implement certificate transparency monitoring

   *  Support multiple enrollment authorities for redundancy

   *  Provide key compromise notification mechanisms

7.  Privacy Considerations







Condrey                  Expires 11 August 2026                [Page 11]

Internet-Draft             witnessd Enrollment             February 2026


7.1.  Device Identifier Privacy

   Device identifiers enable tracking across Evidence packets.  To
   protect user privacy:

   *  Device identifiers SHOULD be opaque random values, not derived
      from hardware identifiers or user information

   *  Users MAY request new device identifiers during key rotation to
      break tracking linkage

   *  Verifiers MUST NOT share device verification patterns with third
      parties

   Self-sovereign mode provides the strongest privacy as no central
   authority learns about device enrollment.  Public registry mode has
   the weakest privacy as the registry sees all enrolled devices.

7.2.  Enrollment Metadata Minimization

   Enrollment authorities collect metadata during device registration.
   To minimize privacy impact:

   *  Collect only the minimum information necessary for enrollment

   *  Do not retain IP addresses or location data beyond enrollment
      completion

   *  Do not share enrollment metadata with third parties

   *  Provide data deletion upon user request (subject to legal
      retention requirements)

   Enrollment authorities MUST publish a privacy policy describing data
   collection, retention, and sharing practices.

7.3.  Registry Privacy Modes

   Public registries MAY offer privacy modes:

   Public mode:  Device certificates are publicly discoverable.  Anyone
      can verify Evidence from the device.  Suitable for public authors.

   Unlisted mode:  Device certificates are not publicly listed but can
      be retrieved by packet-id lookup.  Provides some obscurity.

   Private mode:  Device certificates are only returned to authenticated




Condrey                  Expires 11 August 2026                [Page 12]

Internet-Draft             witnessd Enrollment             February 2026


      Verifiers with a legitimate need.  Suitable for sensitive
      contexts.

   Users SHOULD select the privacy mode appropriate for their use case.
   The default mode is deployment-specific.

8.  IANA Considerations

   This document requests registration of the following well-known URI:

   *  URI suffix: witnessd-devices

   *  Change controller: IETF

   *  Specification document: [this document]

9.  References

9.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, May 2017,
              <https://www.rfc-editor.org/info/rfc8174>.

9.2.  Informative References

   [I-D.condrey-rats-pop]
              Condrey, D., "Proof of Process: An Evidence Framework for
              Digital Authorship Attestation", Work in Progress,
              Internet-Draft, draft-condrey-rats-pop,
              <https://datatracker.ietf.org/doc/html/draft-condrey-rats-
              pop>.

Author's Address

   David Condrey
   Writerslogic Inc
   United States
   Email: david@writerslogic.com








Condrey                  Expires 11 August 2026                [Page 13]


