# drafts

name: draft-tjw-dbound2-problem-statement
rev: 01
title: Domain Boundaries 2.0 Problem Statement
time: 2024-01-11T08:14:05Z
group_uri: /api/v1/group/group/1027/
expires: 2024-01-11T20:18:20Z
states: ["/api/v1/doc/state/2/", "/api/v1/doc/state/150/"]
categories: []
tags: []
fetched_at: 2026-03-07T06:55:18.623738+00:00
source: ietf
abstract:
  Internet clients attempt to make inferences about the administrative
     relationship based on domain names.  Currently it is not possible to
     confirm organizational boundaries in the DNS.  Current mitigation
     strategies have there own issues.  This memo attempts to outline
     these issues.

full_text:
  DBOUND2                                                      T. Wicinski
  Internet-Draft                                              10 July 2023
  Intended status: Informational                                          
  Expires: 11 January 2024


                  Domain Boundaries 2.0 Problem Statement
                   draft-tjw-dbound2-problem-statement-01

  Abstract

     Internet clients attempt to make inferences about the administrative
     relationship based on domain names.  Currently it is not possible to
     confirm organizational boundaries in the DNS.  Current mitigation
     strategies have there own issues.  This memo attempts to outline
     these issues.

  Status of This Memo

     This Internet-Draft is submitted in full conformance with the
     provisions of BCP 78 and BCP 79.

     Internet-Drafts are working documents of the Internet Engineering
     Task Force (IETF).  Note that other groups may also distribute
     working documents as Internet-Drafts.  The list of current Internet-
     Drafts is at https://datatracker.ietf.org/drafts/current/.

     Internet-Drafts are draft documents valid for a maximum of six months
     and may be updated, replaced, or obsoleted by other documents at any
     time.  It is inappropriate to use Internet-Drafts as reference
     material or to cite them other than as "work in progress."

     This Internet-Draft will expire on 11 January 2024.

  Copyright Notice

     Copyright (c) 2023 IETF Trust and the persons identified as the
     document authors.  All rights reserved.

     This document is subject to BCP 78 and the IETF Trust's Legal
     Provisions Relating to IETF Documents (https://trustee.ietf.org/
     license-info) in effect on the date of publication of this document.
     Please review these documents carefully, as they describe your rights
     and restrictions with respect to this document.  Code Components
     extracted from this document must include Revised BSD License text as
     described in Section 4.e of the Trust Legal Provisions and are
     provided without warranty as described in the Revised BSD License.




  Wicinski                 Expires 11 January 2024                [Page 1]


  Internet-Draft               DBOUND2 Problem                   July 2023


  Table of Contents

     1.  Introduction and Motivation . . . . . . . . . . . . . . . . .   2
     2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
     3.  Simplifying the list of possible Use Cases  . . . . . . . . .   3
       3.1.  HTTP State management cookies . . . . . . . . . . . . . .   3
       3.2.  Service Boundaries in shared cloud environments . . . . .   4
       3.3.  Network resource boundaries in shared cloud
             environments  . . . . . . . . . . . . . . . . . . . . . .   4
     4.  Security Considerations . . . . . . . . . . . . . . . . . . .   5
     5.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   5
     6.  Normative References  . . . . . . . . . . . . . . . . . . . .   5
     7.  Informative References  . . . . . . . . . . . . . . . . . . .   5
     Appendix A.  Acknowledgements . . . . . . . . . . . . . . . . . .   5
     Appendix B.  Appendix . . . . . . . . . . . . . . . . . . . . . .   5
     Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .   5
     Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .   5

  1.  Introduction and Motivation

     Working off of the earlier problem statement
     [I-D.sullivan-dbound-problem-statement], which we still consider
     valid.  Various Internet protocols and applications require some
     mechanism for determining whether two domain names have some
     relation.

     The concept of an administrative boundary is by definition not
     present in the DNS.  Relying on the DNS to divine administrative
     structure thus renders such solutions unreliable and unnecessarily
     constrained.  For example, confirming or dismissing a relationship
     between two domain names based on the existence of a zone cut or
     common ancestry is often unfounded, and the notion of an upward "tree
     walk" as a search mechanism is, therefore, unacceptable.

     Currently, the most well known solution in existence is the Public
     Suffix List (PSL).  The PSL is maintained by and is kept current by
     volunteers on a best-effort basis.  It contains a list of points in
     the hierarchical namespace at which registrations take place, and is
     used to identify the boundary between so-called "public" names (below
     which registrations can occur, such as ".com" or ".org.uk") and the
     private names (organizational names) that domain registrars create
     within them.  When this list is inaccurate, it exposes a deviation
     from reality that degrades service to some and can be exploited by
     others.  As the PSL is the de-facto resource, and as there is not a
     more comprehensive, alternative solution for relationship
     identification, the PSL has often been misused to accomplish things
     beyond its capabilities.  For example, there is no way to confirm the
     relationship between two domain names -- the PSL may only signal that



  Wicinski                 Expires 11 January 2024                [Page 2]


  Internet-Draft               DBOUND2 Problem                   July 2023


     there is or is not a public boundary between the two.  Additionally,
     there are questions about the scalability, central management, and
     third-party management of the PSL as it currently exists.

     Applications and organizations impose policies and procedures that
     create additional structure in their use of domain names.  This
     creates many possible relationships that are not evident in the names
     themselves or in the operational, public representation of the names.

     (This document is currently being edited at
     https://github.com/moonshiner/draft-tjw-dbound2-problem-statement)
     (https://github.com/moonshiner/draft-tjw-dbound2-problem-statement))

  2.  Terminology

     The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
     "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
     "OPTIONAL" in this document are to be interpreted as described in BCP
     14 [RFC2119] [RFC8174] when, and only when, they appear in all
     capitals, as shown here.  DNS terminology is as described in
     [RFC8499].

  3.  Simplifying the list of possible Use Cases

     A main topic that immediately arises from this discussion is the
     replacement of the Public Suffix List (PSL).  Currently, this
     document is not looking at the problem space with regards to it.

     From the previous problem statement, the one use case which

  3.1.  HTTP State management cookies

     *  Cookies that have the same-site in browsers.  For example:

        allowing www.example.com to respond with a set-cookie for
        example.com so *.example.com will work.

        not allowing example1.co.uk to respond with a set-cookie for
        example2.co.uk.

     *  CA wildcards: it's OK to sign a cert for *.example.co.uk or
        *.example.com but not for .co.uk or *.com.

     Other applications and organizations impose policies and procedures
     that create additional structure to express many possible
     relationships, such as in first-party-sets or IDN-UA.  These are not
     always evident in the names themselves, and any solutions developed
     here may or may not suit these existing policies and procedures.



  Wicinski                 Expires 11 January 2024                [Page 3]


  Internet-Draft               DBOUND2 Problem                   July 2023


  3.2.  Service Boundaries in shared cloud environments

     A public cloud provider may have a large number of boundaries they
     need to publish.  Taking an example resource within an example
     service, that resource might have a domain name that follows the
     below pattern (2LD may vary):

     "resource-id.cluster-id.servicename.region-name.example.com"

     With the current PSL, the need may exist to publish 1+ records per
     region per service.  If there are many services across many more
     regions, these updates do not scale.  Putting this information into
     DNS allows us to publish records, without manual updates.

  3.3.  Network resource boundaries in shared cloud environments

     Network suffixes and resouces that are public, but not routable or
     resolvable outside of one's network (public-like-PSL, not public-
     like-pool).  A good example is an internal zone within a virtual
     private clouds (VPCs).

     VPCs are resources customers can create, which reserves them a
     logically-isolated portion of AWS's network.  Within each VPC, there
     is a VPC-internal DNS zone which contains DNS records for resources
     within the VPC (suffix zone of "compute.internal" or equivalent),
     which is separated per-VPC.  Every network interface in a VPC will
     have an associated per-interface record in this zone (for VPCs using
     a default configuration).

     These customers may chose to peer their VPC with another customer,
     and these VPC-internal zone would now have multiple different
     customers operating within it.

     *  Linking domains together

     In terms of specific use cases, within the realm of email there is a
     desire to link an arbitrary fully-qualified domain name (FQDN) to the
     organizational domain name (at some point in the namespace above it),
     in order to identify a deterministic location where some sort of
     statement of policy regarding that FQDN can be found.

     There is another growing use case within organizations that need to
     identify relationships between different FQDNs, but also different
     top level domains.  However, there is also desire to reliably
     identify relationships outside of the realm and constraints of the
     namespace tree.





  Wicinski                 Expires 11 January 2024                [Page 4]


  Internet-Draft               DBOUND2 Problem                   July 2023


  4.  Security Considerations

     None at this time.

  5.  IANA Considerations

     None at this time.

  6.  Normative References

     [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
                Requirement Levels", BCP 14, RFC 2119,
                DOI 10.17487/RFC2119, March 1997,
                <https://www.rfc-editor.org/info/rfc2119>.

  7.  Informative References

     [I-D.sullivan-dbound-problem-statement]
                Sullivan, A., Hodges, J., and J. R. Levine, "DBOUND: DNS
                Administrative Boundaries Problem Statement", Work in
                Progress, Internet-Draft, draft-sullivan-dbound-problem-
                statement-02, 18 February 2016,
                <https://datatracker.ietf.org/doc/html/draft-sullivan-
                dbound-problem-statement-02>.

     [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
                2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
                May 2017, <https://www.rfc-editor.org/info/rfc8174>.

     [RFC8499]  Hoffman, P., Sullivan, A., and K. Fujiwara, "DNS
                Terminology", BCP 219, RFC 8499, DOI 10.17487/RFC8499,
                January 2019, <https://www.rfc-editor.org/info/rfc8499>.

  Appendix A.  Acknowledgements

     The author leans heavily on the initial problem statement and thanks
     Andrew Sullivan, John Levine, Murray Kucherawy and Paul Vixie for
     comments and suggestions.

  Appendix B.  Appendix

  Acknowledgements

  Author's Address

     Tim Wicinski
     Elkins, WV 26241
     United States of America



  Wicinski                 Expires 11 January 2024                [Page 5]


  Internet-Draft               DBOUND2 Problem                   July 2023


     Email: tjw.ietf@gmail.com


















































  Wicinski                 Expires 11 January 2024                [Page 6]


name: draft-yao-tsvwg-cco-requirement-and-analysis
rev: 02
title: Collective Communication Optimizations: Requirement and Analysis
time: 2025-01-09T08:05:37Z
group_uri: /api/v1/group/group/1027/
expires: 2025-01-09T09:38:16Z
states: ["/api/v1/doc/state/2/", "/api/v1/doc/state/150/"]
categories: []
tags: []
fetched_at: 2026-03-07T06:55:18.623744+00:00
source: ietf
abstract:
  Gernerative AI applications depend on large scale parallel computing
     clusters for model training and inference.  Existing implementations
     of collective communication in parallel computing is built on top of
     RDMA, the most adoptable AI transport protocol.  However, One-to-
     Many, Many-to-One, and Many-to-Many collective operations all depend
     on point-to-point transport semantics of RDMA, which inevitably
     introduces more bandwidth occupancy and transmission overhead.
     Emerging approaches for collective communication optimization focus
     on network-assisted collective acceleration and can work compatibly
     with RDMA.  This document analyzes different technical schemes for
     network-assisted collective acceleration based on RDMA, and presents
     the gap between these work and current IETF standards, notably iWARP.
     Requirements for designing new standards are proposed accordingly.

full_text:
  Transport Area Working Group                                      K. Yao
  Internet-Draft                                                     S. Xu
  Intended status: Informational                                    C. Liu
  Expires: 9 January 2025                                     China Mobile
                                                                     Y. Li
                                                                  H. Huang
                                                       Huawei Technologies
                                                                   W. Wang
                                             New H3C Technologies Co., Ltd
                                                               D. KUTSCHER
            THE HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY (GUANGZHOU)
                                                               8 July 2024


      Collective Communication Optimizations: Requirement and Analysis
              draft-yao-tsvwg-cco-requirement-and-analysis-02

  Abstract

     Gernerative AI applications depend on large scale parallel computing
     clusters for model training and inference.  Existing implementations
     of collective communication in parallel computing is built on top of
     RDMA, the most adoptable AI transport protocol.  However, One-to-
     Many, Many-to-One, and Many-to-Many collective operations all depend
     on point-to-point transport semantics of RDMA, which inevitably
     introduces more bandwidth occupancy and transmission overhead.
     Emerging approaches for collective communication optimization focus
     on network-assisted collective acceleration and can work compatibly
     with RDMA.  This document analyzes different technical schemes for
     network-assisted collective acceleration based on RDMA, and presents
     the gap between these work and current IETF standards, notably iWARP.
     Requirements for designing new standards are proposed accordingly.

  Requirements Language

     The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
     "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
     document are to be interpreted as described in RFC 2119 [RFC2119].

  Status of This Memo

     This Internet-Draft is submitted in full conformance with the
     provisions of BCP 78 and BCP 79.

     Internet-Drafts are working documents of the Internet Engineering
     Task Force (IETF).  Note that other groups may also distribute
     working documents as Internet-Drafts.  The list of current Internet-
     Drafts is at https://datatracker.ietf.org/drafts/current/.



  Yao, et al.              Expires 9 January 2025                 [Page 1]


  Internet-Draft   Collective Communication Optimizations:       July 2024


     Internet-Drafts are draft documents valid for a maximum of six months
     and may be updated, replaced, or obsoleted by other documents at any
     time.  It is inappropriate to use Internet-Drafts as reference
     material or to cite them other than as "work in progress."

     This Internet-Draft will expire on 9 January 2025.

  Copyright Notice

     Copyright (c) 2024 IETF Trust and the persons identified as the
     document authors.  All rights reserved.

     This document is subject to BCP 78 and the IETF Trust's Legal
     Provisions Relating to IETF Documents (https://trustee.ietf.org/
     license-info) in effect on the date of publication of this document.
     Please review these documents carefully, as they describe your rights
     and restrictions with respect to this document.  Code Components
     extracted from this document must include Revised BSD License text as
     described in Section 4.e of the Trust Legal Provisions and are
     provided without warranty as described in the Revised BSD License.

  Table of Contents

     1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.  Definition of Terms . . . . . . . . . . . . . . . . . . . . .   3
     3.  Existing Work and Analysis  . . . . . . . . . . . . . . . . .   4
       3.1.  Classification and Analysis of Two Modes for NACA . . . .   4
         3.1.1.  NACA Based on Server-to-server RDMA Connection  . . .   4
         3.1.2.  NACA Based on Server-to-switch RDMA Connection  . . .   5
       3.2.  Gap Analysis of Existing Solutions  . . . . . . . . . . .   5
         3.2.1.  Infiniband SHARP  . . . . . . . . . . . . . . . . . .   6
         3.2.2.  RoCEv2 Solutions  . . . . . . . . . . . . . . . . . .   6
         3.2.3.  iWARP . . . . . . . . . . . . . . . . . . . . . . . .   9
     4.  Requirements  . . . . . . . . . . . . . . . . . . . . . . . .  10
       4.1.  NACA Function Design and Header Definition  . . . . . . .  10
       4.2.  Bridge RDMA Transport Semantics . . . . . . . . . . . . .  10
       4.3.  RDMA Transport Related Issues . . . . . . . . . . . . . .  11
       4.4.  Proper Data Input and Output Methods  . . . . . . . . . .  12
       4.5.  Joint Design of NACA Task Assignment and Routing
             Policies  . . . . . . . . . . . . . . . . . . . . . . . .  12
       4.6.  Security and Traffic Isolation  . . . . . . . . . . . . .  12
       4.7.  Fault Tolerance . . . . . . . . . . . . . . . . . . . . .  13
     5.  Security Considerations . . . . . . . . . . . . . . . . . . .  13
     6.  Operational Considerations  . . . . . . . . . . . . . . . . .  13
     7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  13
     8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  13
       8.1.  Normative References  . . . . . . . . . . . . . . . . . .  13
       8.2.  Informative References  . . . . . . . . . . . . . . . . .  14



  Yao, et al.              Expires 9 January 2025                 [Page 2]


  Internet-Draft   Collective Communication Optimizations:       July 2024


     Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  14

  1.  Introduction

     With the development of distributed applications, especially High
     Performance Computing (HPC) and Artificial Intelligence (AI), the
     scale of parallel computing clusters is constantly expanding, and the
     pressure brought by collective communication to the network is also
     increasing.  Existing implementations of collective communication are
     based on RDMA(Remote Direct Memory Access), however, the most obvious
     problem is that the point-to-point transmission semantic of RDMA is
     not well aligned with logical communication patterns defined in
     collective communication, which incurs more bandwidth occupancy,more
     memory copies at endpoints and more data movements, thus lowering the
     overall parallel computing efficiency.  Detailed use cases and
     problems are proposed in
     [I-D.yao-tsvwg-cco-problem-statement-and-usecases].

     Emerging collective communication optimization technical schemes
     focus on network assisted collective acceleration, which can greatly
     alleviate network pressure, improve transmission efficiency, and
     shorten flow completion time (FCT).  Some of these approaches can
     also work compatibly with RDMA, which raises new standardization
     design space for extended RDMA-based protocols for collective
     communication optimization.  In following sections, this document
     analyzes different technical schemes for network-assisted collective
     acceleration based on RDMA, and presents the gap between these work
     and current IETF standards.  Requirements for designing new standards
     are proposed accordingly.

  2.  Definition of Terms

     *Collective communication: A set of communication patterns that
     application processes follow to communicate with each other in a
     parallel computing cluster.  These patterns include One-to-Many,
     Many-to-one, or Many-to-Many delivery mode.

     *Network Assisted Collective Acceleration(NACA): Using network
     devices, like switches, to offload and perform collective operations,
     so as to improve the overall collective communication efficiency.











  Yao, et al.              Expires 9 January 2025                 [Page 3]


  Internet-Draft   Collective Communication Optimizations:       July 2024


  3.  Existing Work and Analysis

     NACA offloads collective operations to switch to implement.  For
     example, Allreduce is done in switch for aggregation.  Broadcast is
     offloaded to switch for data copy, and Scatter leverages switch for
     data tailoring.  Detailed collective operations are listed in
     [I-D.yao-tsvwg-cco-problem-statement-and-usecases].  NACA can be
     built on RDMA so as to optimize collective communication.  RDMA
     allows endpoints to directly read and write memory data from other
     endpoints at high speed without requiring kernel processing and CPU
     resources.  Memory zero copy and kernel bypass have gradually made
     RDMA the mainstream communication technology for HPC and AI
     applications in data centers.  This draft mainly focuses on the
     analysis of two different communication modes for RDMA-based network-
     assisted collective acceleration.

  3.1.  Classification and Analysis of Two Modes for NACA

     When using network devices to offload collective operations, RDMA
     communication modes can be divided into server-to-sever mode and
     server-to-switch mode.

  3.1.1.  NACA Based on Server-to-server RDMA Connection

     The server-to-server RDMA connection mode means not to change the
     logic of the existing applications. and switches that participate in
     collective communication cannot be seen by applications.  The
     destination of RDMA connection is set to be another server endpoint,
     but switches can participate in the collective operations during the
     data transmission.  In this communication mode, native transport
     protocols can cause false positives.  For example, when the switch
     help perform data aggregation during Allreduce, in each round, there
     will be only one aggregated packet sent from the switch to the
     destination server.  Packets from multiple senders are dropped after
     the aggregation.  And at the destination side, it will be judged as
     packet loss and trigger packet retransmission
     [I-D.yao-tsvwg-cco-problem-statement-and-usecases].  Some
     modification on reliability mechanisms of native RDMA transport
     should be improved.












  Yao, et al.              Expires 9 January 2025                 [Page 4]


  Internet-Draft   Collective Communication Optimizations:       July 2024


  +------------+           +----------------------+            +------------+
  |            <-----------------------------------------------> receiver 1 |
  |            |           |                      |            +------------+
  |            |           |        switch        |
  |   sender   |           | forwarding and NACA  |
  |            |           |                      |            +------------+
  |            <-----------------------------------------------> receiver 2 |
  |            |           |   RDMA connection    |            +------------+
  |            |           |                      |
  |            |           |                      |
  |            |           |                      |            +------------+
  |            <-----------------------------------------------> receiver 3 |
  +------------+           +----------------------+            +------------+

         Figure 1: NACA Based on Server-to-server RDMA Connection

  3.1.2.  NACA Based on Server-to-switch RDMA Connection

     The server-to-switch mode means that switches act as RDMA endpoints,
     and RDMA connection is built between the sender, i.e.  server, and
     the destination, i.e. switch.  In this case, hop-by-hop RDMA
     connections are built for end-to-end data transmission.  It is
     necessary to define what RDMA functions should be offloaded to
     switches, since offloading all RDMA functions to the network would
     bring heavy burdens to switches not only on memory and buffer space,
     but also on protocol complexity.

                                                 RDMA connection
                        +------------------------+             +------------+
                        |                        <-------------> receiver 1 |
                        |                        |             +------------+
                 RDMA   |                        |
              Connection|                        |
  +----------+          |         switch         |             +------------+
  |  sender  <---------->  forwarding and NACA   <-------------> receiver 1 |
  +----------+          |                        |             +------------+
                        |                        |
                        |                        |
                        |                        |             +------------+
                        |                        <-------------> receiver 1 |
                        +------------------------+             +------------+

         Figure 2: NACA Based on Server-to-switch RDMA Connection

  3.2.  Gap Analysis of Existing Solutions






  Yao, et al.              Expires 9 January 2025                 [Page 5]


  Internet-Draft   Collective Communication Optimizations:       July 2024


  3.2.1.  Infiniband SHARP

     Scalable Hierachical Aggregation and Reduction Protocol(SHARP)
     [SHARP]SHARP breaks the end-to-end transport rule by implementing
     Target Channel Adapter(TCA) in switches.  The TCA supports both
     Reliable Connection (RC) transport to enable reliable delivery of
     data through the aggregation tree as well as Unreliable Datagram (UD)
     transport to enable Multicast distribution of the aggregation result.
     SHARP has been realized in Infiniband commodity switches, but as is
     stated, SHARP is based on Infiniband architecture.  Currently it
     cannot work interoperably with the other network architectures, thus
     limiting its applicability.

     Figure 3 shows the SHARP protocol, it has two primary phases,
     Aggregation Request and Aggregation Response.  SHARP header is
     designed over IBA header, followed by aggregation operations and
     other data description information.

   +---------+---------+-------+------+-----------+--------+--------+----+
   |   IBA   |  SHARP  | Tuple | User | Operation | Target | SHARP  | CRC|
   | Header  |  Header | Header| Data |  Header   | Header | Payload|    |
   +---------+---------+--------------+-----------+--------+--------+----+

              Aggregation Request Pkt

   +---------+---------+-------+------+--------+----+
   |   IBA   |  SHARP  | Tuple | User | SHARP  | CRC|
   | Header  |  Header | Header| Data | Payload|    |
   +---------+---------+--------------+--------+----+

              Aggregation Response Pkt

                          Figure 3: SHARP Protocol

  3.2.2.  RoCEv2 Solutions

     RDMA over Converged Ethernet version 2(RoCEv2) is an RDMA scheme
     based on the UDP protocol over Ethernet.  Its core design uses
     InfiniBand's transport layer, where data is transmitted in sequence
     and retransmitted using go-back-n.  Therefore, a lossless and ordered
     network is required to achieve ideal performance.  The network has
     introduced Priority Flow Control (PFC) and IP based Explicit
     Congestion Notification (ECN) to ensure lossless transmission.
     Technical schemes of NACA based on RoCEv2 have been analyzed in both
     academia and industry, but compared with Infiniband SHARP, there is
     currently no commercial solutions, which means there are a lot of
     standardization space in this area.




  Yao, et al.              Expires 9 January 2025                 [Page 6]


  Internet-Draft   Collective Communication Optimizations:       July 2024


     Take [NetReduce] and [Cepheus] as two examples for server-to-server
     communication mode of RoCEv2-based NACA.

     [NetReduce] is designed to offload Allreduce to switches.  For ring-
     Allreduce, workers establish RDMA connections with front and rear
     workers, using RDMA write to send parameters, and RDMA read to
     receive aggregation results.  The switch is a man-in-the-middle who
     receives data and aggregates it locally, then returns the results to
     workers in RDMA read way.  This approach has little impact on
     applications.  And it improves the performance since it reduces
     aggregation rounds compared to traditional ring-Allreduce method.
     However, mechanisms such as transport reliability and flow control
     are designed based on an server-to-server communication model, so
     they need to be redesigned or adapted accordingly.

     Figure 5 shows the three phases of NetReduce protocol.  First packet,
     middle packet and the last packet.  NetReduce header is built over
     RoCEv2.  NetReduce has similar function as SHARP, but it is designed
     for aggregation used in ring-Allreduce, so it contains ring
     information, message information, and rank information.

     +---------------------------------------------------------+
     |                       Switch                            |
     |                  man-in-the-middle                      |
     |                                                         |
     |  +---------------------------------------------------+  |
     |  |                                                   |  |
     |  |      +--------------+      +--------------+       |  |
     |  |      |              |      |              |       |  |
     +---------------------------------------------------------+
        |      |              |      |              |       |
        |      |              |      |              |       |
        |      |              |      |              |       |
     +--v------+-+          +-v------+--+         +-v-------+-+
     |  worker 1 |          |  worker 2 |         |  worker 3 |
     +-----------+          +-----------+         +-----------+

                        Figure 4: NetReduce Illustration













  Yao, et al.              Expires 9 January 2025                 [Page 7]


  Internet-Draft   Collective Communication Optimizations:       July 2024


                             First Pkt
     +---------+---------+---------+---------------+----------+------+
     |   UDP   | IB BTH  | IB RETH | NetReduce Hdr |  Payload | ICRC |
     +---------+---------+---------+---------------+----------+------+

                            Middle Pkt
     +---------+---------+----------+------+
     |   UDP   | IB BTH  |  Payload | ICRC |
     +---------+---------+----------+------+

                            Last Pkt
     +---------+---------+---------+---------+------+
     |   UDP   | IB BTH  | IB IMM  | Payload | ICRC |
     +---------+---------+---------+---------+------+

                          Figure 5: NetReduce Protocol

     The design objective of [Cepheus] is for offloading Broadcast
     operations.  Multiple receivers first send RDMA related information,
     like Queue Pair(QP) number and destination address, to the sender
     host for registration, and Multicast Forwarding Tree(MFT) is built on
     these information.  Intermediate switches will make decisions based
     on their downstream connectors.  If leaf switch is directly connected
     with the receiver host, it will work as a RDMA bridge by modifying
     data packets.  In this way, multicast is done in the forward
     direction, and Acknowledge signals are aggregated in reverse
     direction to realize reliability.  This kind of implementation incur
     less modification to native RDMA and has better compatibility.

     Figure 6 shows the Cepheus Multicast Registration Protocol(MRP).
     Before starting to implement the Broadcast operations, the source of
     the multicast propagates the MRP into the entire fabric to install
     multicast forwarding table in each switch and build a MFT.
     Recevier's RDMA information, i.e, QP number and destination IP
     address are predefined before the MFT is set up.  During multicast,
     there is no real RDMA communication.  Switches that are directly
     connected with receivers will modify the packet header to make the
     logical RDMA connection complete.













  Yao, et al.              Expires 9 January 2025                 [Page 8]


  Internet-Draft   Collective Communication Optimizations:       July 2024


                  Cepheus MRP
     +---------+----------+--------------+
     |   UDP   | Metadata | Node Payload |
     +---------+----------+--------------+

                   Metadata
     +---------------+---------------+
     |  Total | Seq  |  Node Numbers |
     +---------------+---------------+

                 Node Payload
     +----------+---------+---------+
     | Node QPN | Node IP | Reserve |
     +----------+---------+---------+

               Figure 6: Cepheus Multicast Registration Protocol

     In RoCEv2 network ,there is also server-to-swtich mode where switches
     implement RDMA protocol stack and workers establish RDMA connections
     with switches.  This approach is similar to InfiniBand SHARP, but
     based on Ethernet.  Due to capacity limitations, network devices do
     not need to support complete RDMA transport protocol.  Similarly, the
     shortcoming of this mode is that it requires network devices to
     support RDMA.

  3.2.3.  iWARP

     iWARP[RFC5040] is another RDMA scheme for Ethernet based on TCP
     protocol.  Like RoCEv2, iWARP uses InfiniBand Verbs to interact with
     applications.RDMAP (Remote Direct Memory Access Protocol) provides
     RDMA semantic support for upper layer requests such as RDMA_Send,
     RDMA_Read, RDMA_Write.  DDP (Data Placement Protocol) implements zero
     copy function.  DDP Packet contains information describing the memory
     area.  Hardware can directly move data in the DDP Packet to the
     destination in memory through DMA based on the control information in
     the DDP Packet . The above process does not require the involvement
     of the CPU.  MPA (Marker Protocol Data Unit Aligned Framing) is
     responsible for adding control information to the TCP flow according
     to a certain algorithm at the sending end, so that the receiving end
     can recognize the boundaries of DDP Packet in the flow according to
     the algorithm.

  +--------------+--------------+--------------+-----------------+------------+----------+
  |  TCP Header  |  MPA Header  |  DDP Header  |   RDMA Header   |  Payload   |  MPA CRC |
  +--------------+--------------+--------------+-----------------+------------+----------+

                         Figure 7: iWARP Protocol




  Yao, et al.              Expires 9 January 2025                 [Page 9]


  Internet-Draft   Collective Communication Optimizations:       July 2024


     Due to TCP ensuring packet ordered delivering and transmission
     reliability, iWARP could adapt to larger network scales compared to
     RoCEv2, but its performance is lower. Because of the high cost of
     offloading complete TCP/IP stack to hardware and the resource
     intensive maintenance of TCP protocol status, the use of iWARP is not
     as widespread as RoCEv2.

     In the server-to-server NACA based on iWARP, any change in Payload
     may be considered as an interruption of the flow, and any packet loss
     must be retransmitted.  The transport layer mechanism is too complex
     and difficult to modify.

     In the server-to-switch NACA based on iWARP mode, due to resource
     limitations, network devices do not need to implement a complete
     protocol stack.  It is necessary to clarify which parts of existing
     protocols must be implemented.  Meanwhile, if network devices
     maintain TCP connections, they need to manage resources reasonably.

  4.  Requirements

  4.1.  NACA Function Design and Header Definition

     NACA offloads collective operations with low computational precision
     and high I/O communication to network device.  Network devices not
     only complete packet routing and forwarding, but also need to process
     collective messages.  Therefore, NACA functions should be designed to
     instruct network devices to distinguish and process different
     traffic.  Accordingly, an NACA header should be designed over the
     transport layer to complete the mapping mechanism between packets and
     collective messages.  Therefore, the following requirements are
     proposed to support collective communication optimization:

     R1: MUST define a NACA header to indicate what collective operations
     that switches need to offload, together with relevant information,
     for example, message id, sequence number, job id etc.

     R2: SHOULD support fallback mechanism, in case network devices are
     not sufficient for processing complete collective operations.

  4.2.  Bridge RDMA Transport Semantics

     As has been explained in previous sections, the major gap between
     native RDMA and NACA is reflected in the transport semantics.  There
     need mechanisms for transport semantic bridging in order to combine
     the high-performance transport capability of RDMA and NACA
     functionality.  Besides, NACA may not need full functionality of
     native RDMA, and it is not ideal to implement full RDMA functionality
     within switches, because of limited hardware resources.  For example,



  Yao, et al.              Expires 9 January 2025                [Page 10]


  Internet-Draft   Collective Communication Optimizations:       July 2024


     most of RDMA-based NACA solutions only call RDMA read, write, send,
     and receive operations.  Accordingly, the following requirements need
     to be met:

     R3: Transport layer MUST support RDMA function.

     R4: SHOULD allow for different RDMA communication modes for NACA as
     described in section 3.

     R5: In server-to-swtich mode, SHOULD clarify which part of the RDMA
     functions the switch supports, in order to establish a RDMA
     connection with the server and complete NACA.

  4.3.  RDMA Transport Related Issues

     As it has been analyzed in section 3 that IWARP solutions can not
     work well with NACA, because it builds RDMA functions on top of TCP
     which are too complex to implement in switches.  The most promising
     solution is RDMA over UDP, for example, RoCEv2.  However, native
     RoCEv2 has several limitations and can not work very well with NACA
     in large scale clusters.  These limitations are reflected in the
     mechanisms of reliability, flow control, and congestion control.  For
     reliability, go-back-n packet retransmission is low efficient, and it
     may incur much buffer occupancy in NACA switches.  Priority Flow
     Control(PFC) also has high requirement for buffer space, and for
     Many-to-one collective operations, PFC will take up even more buffer
     space.  As for congestion control, there are lots of algorithms and
     not all of them work well with NACA.  A common congestion control
     mechanism need to be designed.  Thus, there are following
     requirements:

     R6: NACA MUST be designed with reliability, and the reliability
     mechanism of RoCEv2 SHOULD be modified to be more efficient.

     R7: Flow control SHOULD be optimized in order to save more buffer and
     memory space for NACA functions.

     R8: The congestion control of NACA SHOULD work compatibly with other
     congestion control mechanisms applied for other network traffic that
     runs in the same fabric.











  Yao, et al.              Expires 9 January 2025                [Page 11]


  Internet-Draft   Collective Communication Optimizations:       July 2024


  4.4.  Proper Data Input and Output Methods

     As has been notified, switches or intermediate network nodes have
     relatively limited ressources, i.e., memories.  Thus, the resource
     management should be coordinated with data input and output methods
     on the dataplane, especially in the massive-scale distributed AI
     training application.  If they do not match, problemss like packet
     loss and high communication latency may happen.  Therefore,

     R9: Suitable data PUSH/PULL methods are REQUIRED to be designed for
     overcoming the limited-memory problem of switches.

  4.5.  Joint Design of NACA Task Assignment and Routing Policies

     Since AI model training tasks usually follow a predefined rule that
     task as well as the training group are settle, and once the training
     starts, there will be no more new comers to join the group.  On basis
     of this, NACA task assignment usually follows a centralized pattern.
     For example, NACA support Allreduce by following Aggregation tree,
     and support broadcast by building a multicast forwarding tree.  While
     some routing policies may follow distributed patterns.  For example,
     Adaptive Routing(AR) selects the optimal path at each network node
     distributedly.  These solutions may not co-exist with each other.  In
     order to better balance traffic management and task assignment:

     R10: NACA Task assignment SHOULD be co-designed with routing policies
     for joint optimization.

  4.6.  Security and Traffic Isolation

     Due to situations of multi-tenancy, a single switch may need to
     perform different NACA functions and forward normal traffic.  Since
     NACA header contains collective operations metadata and payload
     parameters, if the switch logic designed for NACA is incorrectly
     applied on normal traffic, there will be very severe security issues.
     Hence, security requirements are as follows:

     R11: Resources MUST be isolated on switches to ensure that different
     tasks do not interfere each other, and NACA functions do not operate
     on normal traffic.











  Yao, et al.              Expires 9 January 2025                [Page 12]


  Internet-Draft   Collective Communication Optimizations:       July 2024


  4.7.  Fault Tolerance

     Packet drops due to congestion can cause some negative impact in
     collective communication.  It needs to depend on design of
     retransmission mechanisms of transport layer to respond to this kind
     of errors, or design of congestion avoiding mechanisms.  These
     tolerant mechanisms usually are designed for transient errors, and
     are dependent on transport-related requirements.  While errors on one
     or some switches in aggregation-tree/broadcast-tree fabric in NACA
     may cause long-period or permanent failure, so there is a need to
     design some asynchronous notification mechanisms to mitigate the
     catastrophic impact of these errors on NACA.

     R12: The mechanism of choosing alternative node for implementing NACA
     functions MUST be designed, to ensure system robustness and
     reliability.  Some asynchronous notification mechanisms are REQUIRED
     to designed for mitigating errors on switches in NACA fabric.

  5.  Security Considerations

     Some security concerns have been described in the
     [I-D.yao-tsvwg-cco-problem-statement-and-usecases].

  6.  Operational Considerations

     Use cases like AI model training, distributed storage, and big data
     analysis usually need infrastructure to be deployed in clusters which
     are operated by single entities, for example, limited domain
     [RFC8799].  In this case, not only the compute and network
     infrastructure, but also the application could be owned by single
     service providers.  These use cases are typically performance-driven,
     which means they need application and infrastructure to be co-
     designed to reach optimization.  However, applications are not co-
     designed with underlying network protocols case-by-case, as long as
     the definition and realization of certain collective operations that
     would be offloaded can be reached in consensus across vendors, like
     unified primitives used for implementing the collective
     communication, applications can leverage on the standardized north
     bound API to improve performance, albeit the applications do not
     belong to the same service providers.

  7.  IANA Considerations

     TBD.

  8.  References

  8.1.  Normative References



  Yao, et al.              Expires 9 January 2025                [Page 13]


  Internet-Draft   Collective Communication Optimizations:       July 2024


     [I-D.yao-tsvwg-cco-problem-statement-and-usecases]
                Yao, K., Shiping, X., Li, Y., Huang, H., and D. KUTSCHER,
                "Collective Communication Optimization: Problem Statement
                and Use cases", Work in Progress, Internet-Draft, draft-
                yao-tsvwg-cco-problem-statement-and-usecases-00, 23
                October 2023, <https://datatracker.ietf.org/doc/html/
                draft-yao-tsvwg-cco-problem-statement-and-usecases-00>.

     [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
                Requirement Levels", BCP 14, RFC 2119,
                DOI 10.17487/RFC2119, March 1997,
                <https://www.rfc-editor.org/info/rfc2119>.

     [RFC5040]  Recio, R., Metzler, B., Culley, P., Hilland, J., and D.
                Garcia, "A Remote Direct Memory Access Protocol
                Specification", RFC 5040, DOI 10.17487/RFC5040, October
                2007, <https://www.rfc-editor.org/info/rfc5040>.

     [RFC8799]  Carpenter, B. and B. Liu, "Limited Domains and Internet
                Protocols", RFC 8799, DOI 10.17487/RFC8799, July 2020,
                <https://www.rfc-editor.org/info/rfc8799>.

  8.2.  Informative References

     [Cepheus]  Zhang, W. L. A. J., "Cepheus: Accelerating Datacenter
                Applications with High-Performance RoCE-Capable
                Multicast", 2024.

     [NetReduce]
                Liu, S., "In-Network Aggregation with Transport
                Transparency for Distributed Training",
                DOI 10.1145/3582016.3582037, 2023,
                <https://doi.org/10.1145/3582016.3582037>.

     [SHARP]    Graham, R. L., "Scalable Hierarchical Aggregation and
                Reduction Protocol (SHARP): A Hardware Architecture for
                Efficient Data Reduction", DOI 10.1109/COMHPC.2016.006,
                2023, <https://doi.org/10.1109/COMHPC.2016.006>.

  Authors' Addresses

     Kehan Yao
     China Mobile
     Beijing
     100053
     China
     Email: yaokehan@chinamobile.com




  Yao, et al.              Expires 9 January 2025                [Page 14]


  Internet-Draft   Collective Communication Optimizations:       July 2024


     Shiping Xu
     China Mobile
     Beijing
     100053
     China
     Email: xushiping@chinamobile.com


     Chang Liu
     China Mobile
     Beijing
     100053
     China
     Email: liuchangjc@chinamobile.com


     Yizhou Li
     Huawei Technologies
     Nanjing, Jiangsu
     China
     Email: liyizhou@huawei.com


     Hongyi Huang
     Huawei Technologies
     Beijing
     China
     Email: hongyi.huang@huawei.com


     Weifeng Wang
     New H3C Technologies Co., Ltd
     Beijing
     China
     Email: wangweifeng@h3c.com


     Dirk KUTSCHER
     THE HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY (GUANGZHOU)
     Guangzhou
     China
     Email: dku@hkust-gz.edu.cn









  Yao, et al.              Expires 9 January 2025                [Page 15]


name: draft-hong-nmrg-ai-deploy
rev: 07
title: Considerations of network/system for AI services
time: 2025-04-24T07:05:09Z
group_uri: /api/v1/group/group/1027/
expires: 2025-04-24T09:53:20Z
states: ["/api/v1/doc/state/2/", "/api/v1/doc/state/150/"]
categories: []
tags: []
fetched_at: 2026-03-07T06:55:18.623748+00:00
source: ietf
abstract:
  As the development of AI technology matured and AI technology began
     to be applied in various fields, AI technology is changed from
     running only on very high-performance servers with small hardware,
     including microcontrollers, low-performance CPUs and AI chipsets.  In
     this document, we consider how to configure the network and the
     system in terms of AI inference service to provide AI service in a
     distributed method.  Also, we describe the points to be considered in
     the environment where a client connects to a cloud server and an edge
     device and requests an AI service.  Some use cases of deploying
     network-based AI services, such as self-driving vehicles and network
     digital twins, are described.

full_text:
  Internet Research Task Force                                   Y-G. Hong
  Internet-Draft                                             Daejeon Univ.
  Intended status: Informational                                 J-S. Youn
  Expires: 24 April 2025                                    DONG-EUI Univ.
                                                                 S-W. Hong
                                                                 H-S. Yoon
                                                                      ETRI
                                                         P. Martinez-Julia
                                                                      NICT
                                                           21 October 2024


              Considerations of network/system for AI services
                        draft-hong-nmrg-ai-deploy-07

  Abstract

     As the development of AI technology matured and AI technology began
     to be applied in various fields, AI technology is changed from
     running only on very high-performance servers with small hardware,
     including microcontrollers, low-performance CPUs and AI chipsets.  In
     this document, we consider how to configure the network and the
     system in terms of AI inference service to provide AI service in a
     distributed method.  Also, we describe the points to be considered in
     the environment where a client connects to a cloud server and an edge
     device and requests an AI service.  Some use cases of deploying
     network-based AI services, such as self-driving vehicles and network
     digital twins, are described.

  Status of This Memo

     This Internet-Draft is submitted in full conformance with the
     provisions of BCP 78 and BCP 79.

     Internet-Drafts are working documents of the Internet Engineering
     Task Force (IETF).  Note that other groups may also distribute
     working documents as Internet-Drafts.  The list of current Internet-
     Drafts is at https://datatracker.ietf.org/drafts/current/.

     Internet-Drafts are draft documents valid for a maximum of six months
     and may be updated, replaced, or obsoleted by other documents at any
     time.  It is inappropriate to use Internet-Drafts as reference
     material or to cite them other than as "work in progress."

     This Internet-Draft will expire on 24 April 2025.






  Hong, et al.              Expires 24 April 2025                 [Page 1]


  Internet-Draft            Deploying AI services             October 2024


  Copyright Notice

     Copyright (c) 2024 IETF Trust and the persons identified as the
     document authors.  All rights reserved.

     This document is subject to BCP 78 and the IETF Trust's Legal
     Provisions Relating to IETF Documents (https://trustee.ietf.org/
     license-info) in effect on the date of publication of this document.
     Please review these documents carefully, as they describe your rights
     and restrictions with respect to this document.  Code Components
     extracted from this document must include Revised BSD License text as
     described in Section 4.e of the Trust Legal Provisions and are
     provided without warranty as described in the Revised BSD License.

  Table of Contents

     1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.  Procedure to provide AI services  . . . . . . . . . . . . . .   4
     3.  Network configuration structure to provide AI services  . . .   6
       3.1.  AI inference service on Local machine . . . . . . . . . .   6
       3.2.  AI inference service on Cloud server  . . . . . . . . . .   7
       3.3.  AI inference service on Edge device . . . . . . . . . . .   8
       3.4.  AI inference service on Cloud server and Edge device  . .   9
       3.5.  AI inference service on horizontal multiple servers . . .  10
       3.6.  Network-side utilization for AI learning  . . . . . . . .  11
     4.  Considerations of network/system for AI services  . . . . . .  12
       4.1.  Considerations of the functional characteristics of the
             hardware  . . . . . . . . . . . . . . . . . . . . . . . .  12
       4.2.  Considerations for the characteristics of the AI model  .  13
       4.3.  Considerations for the characteristics of the communication
             method  . . . . . . . . . . . . . . . . . . . . . . . . .  14
     5.  Addressing challenges for coupling AI and NM  . . . . . . . .  15
       5.1.  Low-level challenges  . . . . . . . . . . . . . . . . . .  15
       5.2.  High-level challenges . . . . . . . . . . . . . . . . . .  17
     6.  Use cases of deploying network-based AI services  . . . . . .  17
       6.1.  Deploying AI services for self-driving vehicles . . . . .  18
       6.2.  Deploying AI services for network digital twins . . . . .  19
     7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  22
     8.  Security Considerations . . . . . . . . . . . . . . . . . . .  22
     9.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  22
     10. References  . . . . . . . . . . . . . . . . . . . . . . . . .  22
       10.1.  Normative References . . . . . . . . . . . . . . . . . .  22
       10.2.  Informative References . . . . . . . . . . . . . . . . .  22
     Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  24







  Hong, et al.              Expires 24 April 2025                 [Page 2]


  Internet-Draft            Deploying AI services             October 2024


  1.  Introduction

     In the Internet of Things (IoT), the amount of data generated from
     IoT devices has exploded along with the number of IoT devices due to
     industrial digitization and the development and dissemination of new
     devices.  Various methods are being tried to effectively process the
     explosively increasing IoT devices and data of IoT devices.  One of
     them is to provide IoT services in a place located close to IoT
     devices and users, away from cloud computing that transmits all data
     generated from IoT devices to a cloud server [RFC9556].

     IoT services also started to break away from the traditional method
     of analyzing IoT data collected so far in the cloud and delivering
     the analyzed results back to IoT objects or devices.  In other words,
     AIoT (Artificial Intelligence of Things) technology, a combination of
     IoT technology and artificial intelligence (AI) technology, started
     to be discussed at international standardization organizations such
     as ITU-T.  AIoT technology, discussed by the ITU-T CG-AIoT group, is
     defined as a technology that combines AI technology and IoT
     infrastructure to achieve more efficient IoT operations, improve
     human-machine interaction, and improve data management and analysis
     [CG-AIoT].

     The first work started by the IETF to apply IoT technology to the
     Internet was to research a lightweight protocol stack instead of the
     existing TCP/IP protocol stack so that various types of IoT devices,
     not traditional Internet terminals, could access the Internet
     [RFC6574][RFC7452].  These technologies have been developed by
     6LoWPAN working group, 6lo working group, 6tisch working group, core
     working group, t2trg group, etc.  As the development of AI technology
     matured and AI technology began to be applied in various fields, just
     as IoT technology was mounted on resource-constrained devices and
     connected to the Internet, AI technology is also changed from running
     only on very high-performance servers.  The technology is being
     developed to run on small hardware, including microcontrollers, low-
     performance CPUs and AI chipsets.  This technology development
     direction is called On-device AI or TinyML[tinyML].

     In this document, we consider how to configure the network and system
     in terms of AI inference service to provide AI service in the IoT
     environment.  In the IoT environment, the technology of collecting
     sensing data from various sensors and delivering it to the cloud has
     already been studied by many standardization organizations including
     the IETF and many standards have been developed.  Now, after creating
     an AI model to provide AI services based on the collected data, how
     to configure this AI model as a system has become the main research
     goal.  Until now, it has been common to develop AI services that
     collect data and perform inferences from the trained servers, but in



  Hong, et al.              Expires 24 April 2025                 [Page 3]


  Internet-Draft            Deploying AI services             October 2024


     terms of the spread of AI services, it is not appropriate to use
     expensive servers to provide AI services.  In addition, since the
     server that collects and trains data mainly exists in the form of a
     cloud server, there are also many problems in proceeding in the form
     of requesting AI service by connecting a large number of terminals to
     these cloud servers to provide AI services.  Therefore, when an AI
     service is requested to an edge device located at a close distance,
     it may have effects such as real-time service support, network
     traffic reduction, and important data security rather than requesting
     an AI service to an AI server located in a distant cloud[RFC9556].

     Even if an edge device is used to serve AI services, it is still
     important to connect to an AI server in the cloud for tasks that take
     a lot of time or require a lot of data.  Therefore, an offloading
     technique for properly distributing the workload between the cloud
     server and the edge device is also a field that is being actively
     studied.  In this contribution, in the following proposed network
     structure, the points to be considered in the environment where a
     client connects to a server and an edge device and requests an AI
     service are derived and described.  That is, the following
     considerations and options could be derived.

     *  AI inference service execution entity

     *  Hardware specifications of the machine to perform AI inference
        services

     *  Selection of AI models to perform AI inference services

     *  A method of providing AI services from cloud servers or edge
        devices

     *  Communication method to transmit data to request AI inference
        service

     The proposed considerations and items could be used to describe the
     use case of self-driving vehicles and network digital twins.  Since
     providing AI services in a distributed method can provide various
     advantages, it is desirable to apply it to self-driving vehicles and
     network digital twins.

  2.  Procedure to provide AI services

     Since research on AI services has been started for a long time, there
     may be shapes to provide various types of AI services.  However, due
     to the nature of AI technology, in general, a system for providing AI
     services consists of the following steps [AI_inference_archtecture]
     [Google_cloud_iot].



  Hong, et al.              Expires 24 April 2025                 [Page 4]


  Internet-Draft            Deploying AI services             October 2024


  +-----------+  +-----------+  +-----------+  +-----------+  +-----------+
  | Collect & |  | Analysis &|  |   Train   |  |  Deploy & |  | Monitor & |
  |  Store    |->| Preprocess|->|  AI model |->| Inference |->|  Maintain |
  |   data    |  |    data   |  |           |  |  AI model |  |  Accuracy |
  +-----------+  +-----------+  +-----------+  +-----------+  +-----------+
  |<--------->|  |<------------------------>|  |<--------->|  |<--------->|
    Sensor, DB              AI Server              Target       AI Srvr &
                                                   machine    Tgt. machine
  |<---------------->|<--------------------->|<-------------->|<--------->|
        Interent              Local                Internet      Local &
                                                                Internet

                      Figure 1: AI service workflow


     *  Data collection & Store

     *  Data Analysis & Preprocess

     *  AI Model Training

     *  AI Model Deploy & Inference

     *  Monitor & Maintain Accuracy

     In the data collection step, data required for training is prepared
     by collecting data from sensors and IoT devices or by using data
     stored in a database.  Equipment involved in this step includes
     sensors, IoT devices and servers that store them, and database
     servers.  Since the operations performed at this step are conducted
     through the Internet, many IoT technologies studied by the IETF so
     far have developed technologies suitable for this step.

     In the data analysis and pre-processing step, the features of the
     prepared data are analyzed and pre-processing for training is
     performed.  Equipment involved in this step includes a high-
     performance server equipped with a GPU and a database server, and is
     mainly performed in a local network.

     In the model training step, a training model is created by applying
     an algorithm suitable for the characteristics of the data and the
     problem to be solved.  Equipment involved in this step includes a
     high-performance server equipped with a GPU, and is mainly performed
     on a local network.

     In the model deploying and inference service provision step, the
     problem to be solved (e.g., classification, regression problem) is
     solved using AI technology.  Equipment involved in this step may



  Hong, et al.              Expires 24 April 2025                 [Page 5]


  Internet-Draft            Deploying AI services             October 2024


     include a target machine, a client, a cloud, etc. that provide AI
     services, and since various equipment is involved in this stage, it
     is conducted through the Internet.  This document summarizes the
     factors to be considered at this step.

     In the accuracy monitoring step, if the performance deteriorates due
     to new data, a new model is created through re-training, and the AI
     service quality is maintained by using the newly created model.  This
     step is the same as described in the model training, model deploying,
     and inference service provision steps described in the previous step
     because re-training and model deploying are performed again.

  3.  Network configuration structure to provide AI services

     In general, after training a AI model, the AI model can be built on a
     local machine for AI model deploying and inference services to
     provide AI services.  Alternatively, we can place AI models on cloud
     servers or edge devices and make AI service requests remotely.  In
     addition, for overall service performance, some AI service requests
     to the cloud server and some AI service requests to edge devices can
     be performed through appropriate load balancing.

  3.1.  AI inference service on Local machine

     The following figure shows a case where a client module requesting AI
     service on the same local machine requests AI service from an AI
     server module on the same machine.

   +---------------------------------------------------------------------+
   |                                                                     |
   |   +-----------------+        Request AI      +-----------------+    |
   |   |  Client module  |    Inference service   |  Server module  |    |
   |   | for AI service  |----------------------->| for AI service  |    |
   |   |                 |<-----------------------|                 |    |
   |   +-----------------+        Reply AI        +-----------------+    |
   |                           Inference result                          |
   +---------------------------------------------------------------------+
                                Local machine













  Hong, et al.              Expires 24 April 2025                 [Page 6]


  Internet-Draft            Deploying AI services             October 2024


              Figure 2: AI inference service on Local machine


     This method is often used when configuring a system focused on
     training AI models to improve the inference accuracy and performance
     of AI models without considering AI services or AI model deploying
     and inference in particular.  In this case, since the client module
     that requests the AI inference service and the AI server module that
     directly performs the AI inference service are on the same machine,
     it is not necessary to consider the communication/network environment
     or service provision method too much.  Alternatively, this method can
     be used when we want to simply decorate the AI inference service on
     one machine without changing the AI service in the future, such as an
     embedded machine or a customized machine.

     In this case, a high level of hardware performance is not required to
     train the AI model, but hardware performance sufficient to run the AI
     inference service is required, so it is possible on a machine with a
     certain amount of hardware performance.

  3.2.  AI inference service on Cloud server

     The following figure shows the case where the client module that
     requests AI service and the AI server module that directly performs
     AI service run on different machines.

                                    +--------------------------------------+
  +------------------------+        |     +---------------------------+    |
  |   +-----------------+  |        |     |     +-----------------+   |    |
  |   |   Client module |<-+--------+-----+---->|   Server module |   |    |
  |   |  for AI service |  |        |     |     |  for AI service |   |    |
  |   +-----------------+  |        |     |     +-----------------+   |    |
  +------------------------+        |     + --------------------------+    |
         Local machine              |             Server machine           |
                                    +--------------------------------------+
                                                 Cloud (Internet)















  Hong, et al.              Expires 24 April 2025                 [Page 7]


  Internet-Draft            Deploying AI services             October 2024


              Figure 3: AI inference service on Cloud server


     In this case, the client module requesting the AI inference service
     runs on the local machine, and the AI server module that directly
     performs the AI inference service runs on a separate server machine,
     and this server machine is in the cloud network.  In this case, the
     performance of the local machine does not need to be high because the
     local machine simply needs to request the AI inference service and,
     if necessary, deliver only the data required for the AI service
     request.  For the AI server module that directly performs AI
     inference service, we can set up our own AI server, or we can use
     commercial clouds such as Amazon, Microsoft, and Google.

  3.3.  AI inference service on Edge device

     The following figure shows the case where the client module that
     requests AI service and the AI server module that directly performs
     AI service are separated, and the AI server module is located in the
     edge device.

                                    +--------------------------------------+
  +------------------------+        |     +---------------------------+    |
  |   +-----------------+  |        |     |     +-----------------+   |    |
  |   |   Client module |<-+--------+-----+---->|   Server module |   |    |
  |   |  for AI service |  |        |     |     |  for AI service |   |    |
  |   +-----------------+  |        |     |     +-----------------+   |    |
  +------------------------+        |     + --------------------------+    |
         Local machine              |                Edge device           |
                                    +--------------------------------------+
                                                    Edge network


              Figure 4: AI inference service on Edge device


     Even in this case, the client module that requests the AI inference
     service runs on the local machine, the AI server module that directly
     performs the AI inference service runs on the edge device, and the
     edge device is in the edge network.  Even in this case, the client
     module that requests the AI inference service runs on the local
     machine, the AI server module that directly performs the AI inference
     service runs on the edge device, and the edge device is in the edge
     network.  The AI module that directly performs the AI inference
     service on the edge device can directly configure the edge device or
     use a commercial edge computing module.





  Hong, et al.              Expires 24 April 2025                 [Page 8]


  Internet-Draft            Deploying AI services             October 2024


     The difference from the above case where the AI server module is in
     the cloud is that the edge device is usually close to the client,
     whereas the performance is lower than that of the server in the
     cloud, so there are advantages in data transfer time and inference
     time, but in unit time Inference service performance is poor.

  3.4.  AI inference service on Cloud server and Edge device

     The following figure shows the case where AI server modules that
     directly perform AI services are distributed in the cloud and edge
     devices.


                                    +--------------------------------------+
  +------------------------+        |     +---------------------------+    |
  |   +-----------------+  |        |     |     +-----------------+   |    |
  |   |   Client module |<-+---+----+-----+---->|   Server module |   |    |
  |   |  for AI service |<-+---+    |     |     |  for AI service |   |    |
  |   +-----------------+  |   |    |     |     +-----------------+   |    |
  +------------------------+   |    |     + --------------------------+    |
         Local machine         |    |                Edge device           |
                               |    +--------------------------------------+
                               |                    Edge network
                               |
                               |    +--------------------------------------+
                               |    |     +---------------------------+    |
                               |    |     |     +-----------------+   |    |
                               +----+-----+---->|   Server module |   |    |
                                    |     |     |  for AI service |   |    |
                                    |     |     +-----------------+   |    |
                                    |     + --------------------------+    |
                                    |              Server machine          |
                                    +--------------------------------------+
                                                  Cloud (Internet)


      Figure 5: AI inference service on Cloud sever and Edge device


     There is a difference between the AI server module performed in the
     cloud and the AI server module performed on the edge device in terms
     of AI inference service performance.  Therefore, the client
     requesting the AI inference service may request by distributing the
     AI inference service request to the cloud and edge device
     appropriately in order to perform the desired AI service.  In other
     words, in the case of an AI service with low inference accuracy but
     short inference time, we can request an AI inference service to the
     edge device.



  Hong, et al.              Expires 24 April 2025                 [Page 9]


  Internet-Draft            Deploying AI services             October 2024


  3.5.  AI inference service on horizontal multiple servers

     In the previous section, to provide AI inference service, the network
     configuration that consisted of local machines, edge devices, and
     cloud servers is a kind of vertical hierarchy.  Because the
     capabilities of each machine are different, the overall performance
     of the network using vertical hierarchy is dependent of each machine.
     Generally, a cloud server has a most powerful performance and then an
     edge device has the second powerful performance.

     In this network configuration, AI service may have different
     performance according to the load level of the server, computing
     capability of the server machine and link-state between the local
     machine and the server machines of the horizontal level.  Thus, to
     look for the server machine that can support the best AI service, it
     is necessary for the network element that can monitor network link-
     state and current state of the computing capability of the server
     machines and the network load-balance that can perform a scheduling
     policy of load balancing.  The following figure shows the case where
     the local machine that requests AI service to horizontal multiple
     cloud servers.






























  Hong, et al.              Expires 24 April 2025                [Page 10]


  Internet-Draft            Deploying AI services             October 2024


                                    +--------------------------------------+
                                    |     +---------------------------+    |
                                    |     |     +-----------------+   |    |
                               +----+-----+---->|   Server module |   |    |
                               |    |     |     |  for AI service |   |    |
                               |    |     |     +-----------------+   |    |
                               |    |     + --------------------------+    |
                               |    |              Server machine 1        |
                               |    +--------------------------------------+
                               |                  Cloud (Internet)
                               |
                               |    +--------------------------------------+
  +------------------------+   |    |     +---------------------------+    |
  |   +-----------------+  |   |    |     |     +-----------------+   |    |
  |   |   Client module |<-+---+----+-----+---->|   Server module |   |    |
  |   |  for AI service |<-+---+    |     |     |  for AI service |   |    |
  |   +-----------------+  |   |    |     |     +-----------------+   |    |
  +------------------------+   |    |     + --------------------------+    |
         Local machine         |    |              Server machine 2        |
                               |    +--------------------------------------+
                               |                  Cloud (Internet)
                               |
                               |    +--------------------------------------+
                               |    |     +---------------------------+    |
                               |    |     |     +-----------------+   |    |
                               +----+-----+---->|   Server module |   |    |
                                    |     |     |  for AI service |   |    |
                                    |     |     +-----------------+   |    |
                                    |     + --------------------------+    |
                                    |              Server machine 3        |
                                    +--------------------------------------+
                                                  Cloud (Internet)


      Figure 6: AI inference service on horizontal multiple servers


  3.6.  Network-side utilization for AI learning

     Collecting and preprocessing of data and training an AI model
     requires a high-performance resource such as CPU, GPU, Power, and
     Storage.  To mitigate this requirement, we can utilize a network-side
     configuration.  Typically, federating learning is a machine learning
     technique that trains an AI model across multiple decentralized
     servers.  It is a contrast to traditional centralized machine
     learning techniques where all the local datasets are uploaded to one
     server.  In this federated learning, it enables multiple network
     nodes to build a common machine learning model.



  Hong, et al.              Expires 24 April 2025                [Page 11]


  Internet-Draft            Deploying AI services             October 2024


     And, transfer learning is a machine learning technique that focuses
     on storing information gained while solving one problem and applying
     it to a different but related problem.  In this transfer learning, we
     can utilize a network configuration to transfer common information
     and knowledge between different network nodes.

  4.  Considerations of network/system for AI services

     As described in the previous chapter, the AI server module that
     directly performs AI inference services by utilizing AI models can be
     performed on a local machine or a cloud server or an edge device.

     In theory, if AI inference service is performed on a local machine,
     AI service can be provided without communication delay time or packet
     loss, but a certain amount of hardware performance is required to
     perform AI service inference.  So, in the future environment where AI
     services become popular, such as when various AI services are
     activated and AI services are disseminated, the cost of a machine
     that performs AI services is important

     If so, whether the AI inference service will be performed on the
     cloud server or the discount price on the edge device can be a
     determining factor in the system configuration.

  4.1.  Considerations of the functional characteristics of the hardware

     When AI inference service request is made to a distant cloud server,
     it may take a lot of time to transmit, but it has the advantage of
     being able to perform many AI inference service requests in a short
     time, and the accuracy of AI service inference increases.
     Conversely, when an AI service request is made to a nearby edge
     device, the transmission time is short, but many AI inference service
     requests cannot be performed at once, and the accuracy of AI service
     inference is lowered.

     Therefore, by analyzing the characteristics and requirements of the
     AI service to be performed, it is necessary to determine where to
     perform the AI inference service on a local machine, a cloud server,
     or an edge device.

     The hardware characteristics of the machine performing the AI service
     varies.  In general, machines on cloud servers are viewed as machines
     with higher performance than edge devices.  However, the performance
     of AI inference service varies depending on how the hardware such as
     CPU, RAM, GPU, and network interface is configured for each cloud
     server and edge device.  If we do not think about cost, it is good to
     configure a system for performing AI services with a machine with the
     best hardware performance, but in reality, we should always consider



  Hong, et al.              Expires 24 April 2025                [Page 12]


  Internet-Draft            Deploying AI services             October 2024


     the cost when configuring the system.  So, according to the
     characteristics and requirements of the AI service to be performed,
     the performance of the local machine, cloud server, and edge device
     must be determined.

     Performance evaluation is possible through the performance matrix
     presented in the standard of ETSI[MEC.IEG006].  The performance
     metrics suggested by the ETSI standard are as follows.  These metrics
     is divided into two groups, namely Functional metrics, which assess
     the user performance and include some classical indexes such as
     latency in task execution, device energy efficiency, bit-rate, loss
     rate, jitter, Quality of Service (QoS), etc.; and Non-functional
     metrics that, instead, focus on the MEC(Mobile Edge Computing)
     network deployment and management.  Non-functional metrics include
     the following indexes.  Service life-cycle(instantiation, service
     deployment, service provisioning, service update (e.g. service
     scalability and elasticity), service disposal), service availability
     and fault tolerance (aka reliability), service processing/
     computational load, global mobile equipment host load, number of API
     request (more generally number of events) processed/second on mobile
     equipment host, delay to process API request (north and south),
     number of failed API request.  The sum of service instantiation,
     service deployment, and service provisioning provide service boot-
     time.

  4.2.  Considerations for the characteristics of the AI model

     According to the characteristics of the AI service, although not
     directly related to communication/network, the biggest influence on
     AI inference services is the AI model to be used for AI inference
     service.  For example, in AI services such as image classification,
     there are various types of AI models such as ResNet, EfficientNet,
     VGG, and Inception.  These AI models differ in AI inference accuracy,
     but also in AI model file size and AI inference time.  AI models with
     the highest inference accuracy typically have very large file sizes
     and take a lot of AI inference time.  So, when constructing an AI
     service system, it is not always good to choose an AI model with the
     highest AI inference accuracy.  Again, it is important to select an
     AI model according to the characteristics and requirements of the AI
     service to be performed.

     Experimentally, it is recommended to use an AI model with high AI
     inference accuracy in the cloud server, and use an AI model that can
     provide fast AI inference service although the AI inference accuracy
     is slightly lower for the fast AI inference service in the edge
     device.





  Hong, et al.              Expires 24 April 2025                [Page 13]


  Internet-Draft            Deploying AI services             October 2024


     It might be a bit of an implementation issue, but we should also
     consider how we deliver AI services on cloud servers or edge devices.
     With the current technology, a traditional web server method or a
     server method specialized for AI service inference (e.g., Google's
     Tensorflow Serving) can be used.  Traditional web server methods such
     as Flask and Django have the advantage of running on various types of
     machines, but since they are designed to support general web
     services, the service execution time is not fast.  Tensorflow Serving
     uses the features of Tensorflow to make AI service inference services
     very fast and efficient.  However, older CPUs that do not support AVX
     cannot use the Tensorflow serving function because Google's
     Tensorflow does not run.  Therefore, rather than unconditionally
     using the server method specialized in AI service inference, it is
     necessary to decide the AI server module method that provides AI
     services in consideration of the hardware characteristics of the AI
     system that can be built.

  4.3.  Considerations for the characteristics of the communication method

     The communication method for transferring data to request AI
     inference service is also an important decision in constructing an AI
     system.  Using the traditional REST method, it can be used for
     various machines and services, but its performance is inferior to
     Google's gRPC.  There are many advantages to using gRPC for AI
     inference services because Google's gRPC enables large-capacity data
     transfer and efficient data transfer compared to REST.

     Cloud-edge collaboration-based AI service development is actively
     underway.  In particular, in the case of AI services that are
     sensitive to network delays, such as object recognition and
     autonomous vehicle services, (micro)services for inference are placed
     on edge devices to obtain fast inference results and provide
     services.  As such, in the development of intelligent IoT services,
     various devices that can provide computing services within the
     network, such as edge devices, are being added as network elements,
     and the number of IoT devices using them is rapidly increasing.
     Therefore, a new function for computing resource management and
     operation is required in terms of providing computing services within
     the network.  In addition, to operate distributed AI service on
     network, the network policy for collaboration between edge devices
     that support computing resource for AI service.

     In network policy, in order to efficiently support distributed AI
     services, existing networks must provide with the collaboration of AI
     service between edge devices such as multi-edge network configuration
     and AI service aware traffic steering in multi-edge network to
     receive distributed AI service support efficiently in various network
     environments that dynamically varies network resource and the



  Hong, et al.              Expires 24 April 2025                [Page 14]


  Internet-Draft            Deploying AI services             October 2024


     computing resource of edge device.  For example, in order to
     efficiently provide distributed AI inference service in multi-edge
     network environment, AI tasks message exchanges must be possible
     between edge devices.  Also, there are various delay sensitive AI
     services based on edge device on network.  They are divided into in-
     time delay AI services with a deadline limit and on-time delay AI
     services with the set of a time-range.  In particular on-time delay
     AI services want to return the results of prediction within the time
     range.  Therefore, distributed AI service must be able to is provided
     proper AI service in terms of the delay service for distributed AI
     service through network.  Therefore, the client of AI service should
     be able to be provided both in-time and on-time delay services and be
     interacted with edge devices where distributed AI service is built to
     provide both delay services.

  5.  Addressing challenges for coupling AI and NM

     The document [I-D.irtf-nmrg-ai-challenges] describes the challenges
     found in the application of AI to network management problems.  They
     are separated into low-level and high-level challenges.  This section
     describes how the concepts discussed in the present document are
     linked to those challenges.

     Some of the links between the structures presented in this document
     and AI challenges involve the alignment with AINEMA
     [I-D.pedro-nmrg-ai-framework].

     AINEMA [I-D.pedro-nmrg-ai-framework] is a framework of functions
     required for exploiting AI services and their interconnection.
     Intelligent management systems based on AINEMA, as shown in
     [TNSM-2018], introduce AI functions in the network management cycles,
     and are able to express management decisions in terms of network
     intents, as described in [I-D.pedro-ite].

  5.1.  Low-level challenges

     The first challenge (C1) concerns to the combinatorial explosion of
     the solution space in relation to the size of the problem.  The
     present document proposes to tackle part of such complexity by
     separating the AI work in multiple elements of the network.  The
     separation is asymmetrical, so that some elements (cloud side) will
     contribute more computation power to the distributed service.

     The second challenge (C2) regards the dimensional and context
     uncertainty and unpredictability.  The present document focus on
     cloistering the AI models and data within pre-defined client-edge-
     cloud-server structures.




  Hong, et al.              Expires 24 April 2025                [Page 15]


  Internet-Draft            Deploying AI services             October 2024


     The third challenge (C3) is to ensure that AI to is able to provide
     prompt responses and decisions to management questions.  The key
     aspect discussed in this document to resolve this challenge is the
     integration of end-node, in-network, edge, an cloud computing
     services.  This allows AI computations to be performed in the best
     place possible.  Time elapsed to get fast responses of easy
     operations will be minimized by executing them in the end-nodes
     (e.g., vehicles) and edge devices, while the time needed to compute
     more complex operations is minimized by offloading them to cloud
     computing services.

     The fourth challenge (C4) states the difficulties related to
     resolving data imperfections and scalability of techniques.  This
     challenge is specific to each information domain and problem.
     However, as presented in this document, the mentioned difficulties in
     C4 is relieved by exposing AINEMA [I-D.pedro-nmrg-ai-framework]
     functions instantiated in cloud continuum to the AI services, so that
     AI services transparently gain capabilities such as homogenization
     and scalability.  Instantiating a distributed AI system in a cloud
     continuum enables the AI system to have many functions to deal with
     data homogenization, resolving high data rates, etc.

     The fifth challenge (C5) concerns the integration of AI services with
     existing automation and human processes.  The flexibility of the
     structures presented in this document allow them to be connected to
     existing systems, being aligned and somewhat interconnected with
     AINEMA [I-D.pedro-nmrg-ai-framework].  The result is exemplified in
     the support for network digital twin and vehicle environments.

     Te sixth challenge (C6) exposes the need for cost-effective
     solutions.  Enabling AI services to be deployed in the cloud
     continuum supports the maximization of effectiveness by dynamically
     relocating services to the cheapest provider point that is able to
     accomplish the computation requirements.  This relocation is guided
     by intents, as described in [I-D.pedro-ite], and implemented by an
     independent management cycle, as described in AINEMA
     [I-D.pedro-nmrg-ai-framework] and [TNSM-2018].  The result is that
     the cost of a distributed AI system is continuously being checked and
     adjusted by requesting the underlying infrastructure to modify the
     point of instantiation of each part of the AI system.











  Hong, et al.              Expires 24 April 2025                [Page 16]


  Internet-Draft            Deploying AI services             October 2024


  5.2.  High-level challenges

     The first challenge (H1) relies in the observation that AI techniques
     were developed in a different area---imaging---, so they must be
     extensively tailored for network problems.  Although this is a quite
     complex challenge, this document supports its resolution by enabling
     AI models and algorithms to evolve separately.  Multiple providers
     will offer multiple solutions and they will be stitched together by
     following the structures introduced above, as well as the functions
     provided by the interconnection and alignment with AINEMA
     [I-D.pedro-nmrg-ai-framework].

     The second challenge (H2) conveys the mismatch that exists from the
     original data and internal data used in AI models.  Although the
     present document does cover this mismatch, the structures presented
     in this document help alleviate the burden by relocating some AI
     processes as close as possible to the end-points (e.g., vehicles and
     NDT).  The same structures support the implementation of measures to
     protect privacy.  The AI services that are closer to the edges will
     deal with sensitive data and ensure privacy protection by, for
     example, anonymizing it.  Those services are instantiated within the
     data domain, so private data does not crosses the boundaries of its
     data domain.

     The third challenge (H3) consists on the level of acceptance that an
     AI system experiences from administrators and operators.  It is
     agreed that giving full control of AI operations to administrators
     and operators increases such level of acceptance.  The structures
     presented in this document support the involvement of administrators
     and operators in AI system processes through the provision of
     policies and network intents.  On the one hand, aligning the
     distributed AI system specified in this document with AINEMA
     [I-D.pedro-nmrg-ai-framework], enables the enforcement high-level
     policies and management goals provided by administrators and
     operators.  On the other hand, aligning the distributed AI system
     specified in this document with intent-based networking systems,
     particularly intent translation [I-D.pedro-ite], enables
     administrators and operators to use high-level specifications (namely
     network intents) to communicate the policies, requirements, and goals
     that must be enforced to the AI system constructed over the cloud
     continuum.

  6.  Use cases of deploying network-based AI services








  Hong, et al.              Expires 24 April 2025                [Page 17]


  Internet-Draft            Deploying AI services             October 2024


  6.1.  Deploying AI services for self-driving vehicles

     Various sensors are used in self-driving vehicles, and the final
     judgment is made by combining these data.  Among them, camera data-
     based object detection solves parts that expensive equipment such as
     LiDAR and RADAR cannot solve.  Camera-based object detection performs
     various tasks, and in addition to lane recognition for maintaining
     driving lanes and changing lanes, it also supports safe driving and
     parking assistance by distinguishing shape information such as
     pedestrians, signs, and parking vehicles along the road.

     In order to perform such driving assistance and autonomous driving,
     object detection needs to be performed in real time.  The minimum
     FPS(Frames Per Second) to be considered real-time in autonomous
     driving is 30 FPS[Object_detection].  No matter how high the accuracy
     is, it cannot be used for autonomous driving if it does not meet the
     corresponding reference value.

     Task offloading refers to a technology or structure that transfers
     computing tasks to other processing devices or systems to perform
     them.  Task offloading can quickly process tasks that exceed the
     performance limits of devices that lack resources by delivering tasks
     from devices with limited computing power, storage space, and power
     to devices that are rich in computing resources.

     For devices with low hardware performance (e.g., NVIDIA Jetson Nano
     board, Qual-core ARM A57, 4GB RAM), all locally without task
     offloading results in 4.6 FPS, which is difficult to perform object
     detection-based autonomous driving.  On the other hand, if task
     offloading is applied to perform object detection on devices with
     high hardware performance (e.g., Intel i7, RTX 3060, 32GB RAM) and
     the rest of the work is performed on the client, 41.8 FPS will be
     obtained.  This is a result that satisfies 30 FPS, which is the
     reference FPS of object detection-based autonomous driving.

     In the case of AI services such as object detection, if it is
     difficult to perform on resource-constrained devices, it can be seen
     that the task offloading structure shows some efficiency.  However,
     without performing all operations locally, task offloading operations
     between network nodes can affect the entire time because the larger
     the size of the data, the greater the communication latency.
     Therefore, in such a network distributed environment, the provision
     of AI services should be designed in consideration of various
     variables.  The Figure 7 shows an example of distributed AI
     deployment in a self-driving vehicle when it does not have enough
     capabilities to proceed the object detection operation in real-time
     and it asks some tasks to edge devices and cloud servers.




  Hong, et al.              Expires 24 April 2025                [Page 18]


  Internet-Draft            Deploying AI services             October 2024


                                    +--------------------------------------+
  +------------------------+        |     +---------------------------+    |
  |   +-----------------+  |        |     |     +-----------------+   |    |
  |   |Object detection |<-+---+----+-----+---->|Object detection |   |    |
  |   |     service     |<-+---+    |     |     |     service     |   |    |
  |   +-----------------+  |   |    |     |     +-----------------+   |    |
  +------------------------+   |    |     + --------------------------+    |
           Vehicle             |    |                Edge device           |
                               |    +--------------------------------------+
                               |                    Edge network
                               |
                               |    +--------------------------------------+
                               |    |     +---------------------------+    |
                               |    |     |     +-----------------+   |    |
                               +----+-----+---->|Object detection |   |    |
                                    |     |     |     service     |   |    |
                                    |     |     +-----------------+   |    |
                                    |     + --------------------------+    |
                                    |              Server machine          |
                                    +--------------------------------------+
                                                  Cloud (Internet)


      Figure 7: Distributed object detection service in self-driving
                                 vehicle


  6.2.  Deploying AI services for network digital twins

     Network digital twin also need to build distributed AI services.  The
     purpose of a network digital twin is described in
     [I-D.irtf-nmrg-network-digital-twin-arch].  In particular, the
     network digital twin provides network operators with technology that
     enables stable operation of the physical network and stable execution
     of optimal network policies and deployment procedures.  To achieve
     this, the network digital twin will use AI capabilities for various
     purposes.

     Various AI functions will be applied for optimal network operation
     and management.  However, the actual physical network consists of
     many network devices and has a complex structure.  In addition, in a
     large-scale network environment, the network overhead is very large
     to collect and store information from many network devices in a
     centralized manner, and to create and operate network operation
     policies based on it.






  Hong, et al.              Expires 24 April 2025                [Page 19]


  Internet-Draft            Deploying AI services             October 2024


     Therefore, there is a need for a method to apply AI functions based
     on a distributed form for network operation and management.  In
     particular, the actual physical network structure is built in a
     logical hierarchical structure.  Therefore, it is necessary to apply
     a distributed AI method that considers the logical hierarchical
     network structure environment.

     In order to optimally perform network operation and management
     through distributed AI methods, it is necessary to generate AI
     function-based network operation and management policy models and an
     operational method to distribute the generated AI function-based
     network policies.  In particular, in order to operate a network
     digital twin in a large-scale network environment, it is necessary to
     generate AI-based network policy models in a distributed manner.  A
     federated learning algorithm or a transfer learning algorithm that
     can learn large-scale networks in a distributed manner can be
     applied.

            +-----------------------------------------------------+
            |                                                     |
            |        Distributed network learning model           |
            |        in large-scale network environment           |
            |                                                     |
            |            +-------------+------------+             |
            |            |          Master          |             |
            |            | (AI based Policy model)  |             |
            |            +-------------+------------+             |
            |                          |                          |
            |        +-----------+-----+-----+-----------+        |
            |        |           |           |           |        |
            |   +----+----+ +----+----+ +----+----+ +----+----+   |
            |   |  Worker | |  Worker | |  Worker | |  Worker |   |
            |   | (Agent) | | (Agent) | | (Agent) | | (Agent) |   |
            |   +----+----+ +----+----+ +----+----+ +----+----+   |
            |        |           |           |           |        |
            |   +----+----+ +----+----+ +----+----+ +----+----+   |
            |   |  Local  | |  Local  | |  Local  | |  Local  |   |
            |   |  Data   | |  Data   | |  Data   | |  Data   |   |
            |   |  Repo-  | |  Repo-  | |  Repo-  | |  Repo-  |   |
            |   |  sitory | |  sitory | |  sitory | |  sitory |   |
            |   +---------+ +---------+ +---------+ +---------+   |
            |                                                     |
            +-----------------------------------------------------+


          Figure 8: Distributed learning model of network learning for
                             network digital twins




  Hong, et al.              Expires 24 April 2025                [Page 20]


  Internet-Draft            Deploying AI services             October 2024


     As shown in Figure 8, in order to learn a large-scale network through
     a distributed learning method, a local data repository to store
     network data must be established in each region, for example, based
     on location or AS (Autonomous System).  Therefore, the distributed
     learning method learns through each worker (agent) based on the local
     network data stored in the local network data repository, and
     generates a large-scale network policy model through the master.
     This distributed learning method can reduce the network overhead of
     centralized data collection and storage, and reduce the time required
     to create AI models for network operation and management policies for
     large-scale networks.  In addition, the network policy model
     generated by the worker can be used as a locally optimized network
     policy model to provide AI-based network operation and management
     policy services optimized for local network operations.

     The distributed deployment of trained AI network policy models can be
     deployed on network devices that can manage and operate the local
     network to minimize network data movement.  For example, in a large-
     scale network consisting of multiple ASes, AI network policy models
     can be deployed per AS to optimize network operation and management.
     Figure 9 shows an example of operating and managing a network by
     distributing AI network policy models by AS.


          +---------------------------------------------------------+
          |              +-------------+------------+               |
          |              |          Master          |               |
          |              | (AI-based Network Policy |               |
          |              |     model management)    |               |
          |              +-------------+------------+               |
          |                            |                            |
          |              +-------------+-------------+              |
          |     Worker   |                 Worker    |              |
          |   +----------+----------+     +----------+----------+   |
          |   |    Network device   |     |    Network device   |   |
          |   |  (AI-based network  |     |  (AI-based network  |   |
          |   |     Policy model)   |     |     Policy model)   |   |
          |   +----------+----------+     +----------+----------+   |
          |              |                           |              |
          |   +----------+----------+     +----------+----------+   |
          |   |        AS_1         |     |        AS_2         |   |
          |   +---------------------+     +---------------------+   |
          +---------------------------------------------------------+


      Figure 9: Distributed deployment of trained AI network policy models





  Hong, et al.              Expires 24 April 2025                [Page 21]


  Internet-Draft            Deploying AI services             October 2024


  7.  IANA Considerations

     There are no IANA considerations related to this document.

  8.  Security Considerations

     When AI service is performed on a local machine, there is no security
     issue, but when AI service is provided through a cloud server or edge
     device, IP address and port number may be known to the outside can
     attack.  Therefore, when providing AI services by utilizing machines
     on the network such as cloud servers and edge devices, it is
     necessary to analyze the characteristics of the modules to be used
     well, identify vulnerabilities in security, and take countermeasures.

  9.  Acknowledgements

     TBA

  10.  References

  10.1.  Normative References

     [RFC6574]  Tschofenig, H. and J. Arkko, "Report from the Smart Object
                Workshop", RFC 6574, DOI 10.17487/RFC6574, April 2012,
                <https://www.rfc-editor.org/info/rfc6574>.

     [RFC7452]  Tschofenig, H., Arkko, J., Thaler, D., and D. McPherson,
                "Architectural Considerations in Smart Object Networking",
                RFC 7452, DOI 10.17487/RFC7452, March 2015,
                <https://www.rfc-editor.org/info/rfc7452>.

     [RFC9556]  Hong, J., Hong, Y., de Foy, X., Kovatsch, M., Schooler,
                E., and D. Kutscher, "Internet of Things (IoT) Edge
                Challenges and Functions", RFC 9556, DOI 10.17487/RFC9556,
                April 2024, <https://www.rfc-editor.org/info/rfc9556>.

  10.2.  Informative References

     [I-D.irtf-nmrg-network-digital-twin-arch]
                Zhou, C., Yang, H., Duan, X., Lopez, D., Pastor, A., Wu,
                Q., Boucadair, M., and C. Jacquenet, "Network Digital
                Twin: Concepts and Reference Architecture", Work in
                Progress, Internet-Draft, draft-irtf-nmrg-network-digital-
                twin-arch-08, 16 October 2024,
                <https://datatracker.ietf.org/doc/html/draft-irtf-nmrg-
                network-digital-twin-arch-08>.





  Hong, et al.              Expires 24 April 2025                [Page 22]


  Internet-Draft            Deploying AI services             October 2024


     [I-D.irtf-nmrg-ai-challenges]
                François, J., Clemm, A., Papadimitriou, D., Fernandes, S.,
                and S. Schneider, "Research Challenges in Coupling
                Artificial Intelligence and Network Management", Work in
                Progress, Internet-Draft, draft-irtf-nmrg-ai-challenges-
                03, 4 March 2024, <https://datatracker.ietf.org/doc/html/
                draft-irtf-nmrg-ai-challenges-03>.

     [I-D.pedro-nmrg-ai-framework]
                Martinez-Julia, P., Homma, S., and D. Lopez, "Artificial
                Intelligence Framework for Network Management", Work in
                Progress, Internet-Draft, draft-pedro-nmrg-ai-framework-
                04, 21 October 2023,
                <https://datatracker.ietf.org/doc/html/draft-pedro-nmrg-
                ai-framework-04>.

     [I-D.pedro-ite]
                Martinez-Julia, P. and J. P. Jeong, "Intent Translation
                Engine for Intent-Based Networking", Work in Progress,
                Internet-Draft, draft-pedro-ite-01, 4 March 2024,
                <https://datatracker.ietf.org/doc/html/draft-pedro-ite-
                01>.

     [CG-AIoT]  "ITU-T CG-AIoT", <https://www.itu.int/en/ITU-T/
                studygroups/2017-2020/20/Pages/ifa-structure.aspx>.

     [tinyML]   "tinyML Foundation", <https://www.tinyml.org/>.

     [AI_inference_archtecture]
                "IBM Systems, AI Infrastructure Reference Architecture",
                <https://www.ibm.com/downloads/cas/W1JQBNJV>.

     [Google_cloud_iot]
                "Bringing intelligence to the edge with Cloud IoT",
                <https://cloud.google.com/blog/products/gcp/bringing-
                intelligence-edge-cloud-iot>.

     [MEC.IEG006]
                ETSI, "Mobile Edge Computing; Market Acceleration; MEC
                Metrics Best Practice and Guidelines", Group
                Specification ETSI GS MEC-IEG 006 V1.1.1 (2017-01),
                January 2017.

     [Object_detection]
                Lewis, "Object Detection for Autonomous Vehicles Gene",
                2016.





  Hong, et al.              Expires 24 April 2025                [Page 23]


  Internet-Draft            Deploying AI services             October 2024


     [TNSM-2018]
                P. Martinez-Julia, V. P. Kafle, and H. Harai, "Exploiting
                External Events for Resource Adaptation in Virtual
                Computer and Network Systems, in IEEE Transactions on
                Network and Service Management. Vol. 15, n. 2, pp. 555--
                566, 2018.", 2018.

  Authors' Addresses

     Yong-Geun Hong
     Daejeon University
     62 Daehak-ro, Dong-gu
     Daejeon
     34520
     South Korea
     Phone: +82 42 280 4841
     Email: yonggeun.hong@gmail.com


     Joo-Sang Youn
     DONG-EUI University
     176 Eomgwangno Busan_jin_gu
     Busan
     614-714
     South Korea
     Phone: +82 51 890 1993
     Email: joosang.youn@gmail.com


     Seung-Woo Hong
     ETRI
     218 Gajeong-ro Yuseong-gu
     Daejeon
     34129
     South Korea
     Phone: +82 42 860 1041
     Email: swhong@etri.re.kr


     Ho-Sun Yoon
     ETRI
     218 Gajeong-ro Yuseong-gu
     Daejeon
     34129
     South Korea
     Phone: +82 42 860 5329
     Email: yhs@etri.re.kr




  Hong, et al.              Expires 24 April 2025                [Page 24]


  Internet-Draft            Deploying AI services             October 2024


     Pedro Martinez-Julia
     National Institute of Information and Communications Technology
     4-2-1, Nukui-Kitamachi, Koganei, Tokyo
     184-8795
     Japan
     Phone: +81 42 327 7293
     Email: pedro@nict.go.jp












































  Hong, et al.              Expires 24 April 2025                [Page 25]


name: draft-ai-traffic
rev: 00
title: Handling inter-DC/Edge AI-related network traffic: Problem statement
time: 2024-10-07T13:19:24Z
group_uri: /api/v1/group/group/1027/
expires: 2025-04-10T13:19:24Z
states: ["/api/v1/doc/state/4/", "/api/v1/doc/state/150/"]
categories: []
tags: []
fetched_at: 2026-03-07T06:55:18.623753+00:00
source: ietf
abstract:
  The growth in terms of number of parameters of LLM models as well as
     the need to use or train those models with private or protected data
     will require service providers operating LLM-based services to
     cooperate to train, specialize or serve LLM-based services accross
     datacenters.  Given their structure, the number of parameters they
     incorporate and the collective communication librairies they are
     built with, LLM training and inference (or serving) network traffic
     has specific characteristics.

     In that regard, understanding the specificities of AI-related
     workloads is critical to determine how to operate AI-based services
     in a federated setting across datacenters.

full_text:
  Working Group name                                       A. Fressancourt
  Internet-Draft                                                L. Iannone
  Intended status: Standards Track                                  D. Lou
  Expires: 10 April 2025                                        D. Trossen
                                                                    Huawei
                                                            7 October 2024


    Handling inter-DC/Edge AI-related network traffic: Problem statement
                            draft-ai-traffic-00

  Abstract

     The growth in terms of number of parameters of LLM models as well as
     the need to use or train those models with private or protected data
     will require service providers operating LLM-based services to
     cooperate to train, specialize or serve LLM-based services accross
     datacenters.  Given their structure, the number of parameters they
     incorporate and the collective communication librairies they are
     built with, LLM training and inference (or serving) network traffic
     has specific characteristics.

     In that regard, understanding the specificities of AI-related
     workloads is critical to determine how to operate AI-based services
     in a federated setting across datacenters.

  Status of This Memo

     This Internet-Draft is submitted in full conformance with the
     provisions of BCP 78 and BCP 79.

     Internet-Drafts are working documents of the Internet Engineering
     Task Force (IETF).  Note that other groups may also distribute
     working documents as Internet-Drafts.  The list of current Internet-
     Drafts is at https://datatracker.ietf.org/drafts/current/.

     Internet-Drafts are draft documents valid for a maximum of six months
     and may be updated, replaced, or obsoleted by other documents at any
     time.  It is inappropriate to use Internet-Drafts as reference
     material or to cite them other than as "work in progress."

     This Internet-Draft will expire on 10 April 2025.

  Copyright Notice

     Copyright (c) 2024 IETF Trust and the persons identified as the
     document authors.  All rights reserved.




  Fressancourt, et al.      Expires 10 April 2025                 [Page 1]


  Internet-Draft                 AI traffic                   October 2024


     This document is subject to BCP 78 and the IETF Trust's Legal
     Provisions Relating to IETF Documents (https://trustee.ietf.org/
     license-info) in effect on the date of publication of this document.
     Please review these documents carefully, as they describe your rights
     and restrictions with respect to this document.  Code Components
     extracted from this document must include Revised BSD License text as
     described in Section 4.e of the Trust Legal Provisions and are
     provided without warranty as described in the Revised BSD License.

  Table of Contents

     1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.  Applicability of AI . . . . . . . . . . . . . . . . . . . . .   4
       2.1.  xGPT-like Services: an example centralized training use
             case  . . . . . . . . . . . . . . . . . . . . . . . . . .   4
       2.2.  Decentralized training with centralized inference use
             cases . . . . . . . . . . . . . . . . . . . . . . . . . .   5
       2.3.  Federated building management: a decentralized inference
             use case  . . . . . . . . . . . . . . . . . . . . . . . .   6
       2.4.  Key Takeaways . . . . . . . . . . . . . . . . . . . . . .   6
     3.  How ML systems are built  . . . . . . . . . . . . . . . . . .   7
       3.1.  Lifecycle of a typical large language model . . . . . . .   7
       3.2.  The distributed nature of Machine Learning systems  . . .   8
       3.3.  The topology of distributed machine learning systems  . .  10
       3.4.  Deployment considerations for AI systems  . . . . . . . .  12
     4.  Challenges (in Networking for AI) . . . . . . . . . . . . . .  14
       4.1.  Network resource management at regional scale . . . . . .  14
       4.2.  Latency sensitivity of LLM training and inference . . . .  15
       4.3.  Aligning communications patterns to the Internet's
             constraints . . . . . . . . . . . . . . . . . . . . . . .  16
       4.4.  Managing incast traffic related to AI inference . . . . .  17
       4.5.  Securing and attesting AI-related traffic . . . . . . . .  18
     5.  Problem statement . . . . . . . . . . . . . . . . . . . . . .  19
     6.  Next Steps  . . . . . . . . . . . . . . . . . . . . . . . . .  19
     7.  Security Considerations . . . . . . . . . . . . . . . . . . .  20
     8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  20
     9.  Informative References  . . . . . . . . . . . . . . . . . . .  20
     Appendix A.  A primer on Machine learning (extended version of
             Section 3)  . . . . . . . . . . . . . . . . . . . . . . .  24
       A.1.  Machine learning model lifecycle  . . . . . . . . . . . .  25
       A.2.  System model  . . . . . . . . . . . . . . . . . . . . . .  26
       A.3.  Parallelization modes . . . . . . . . . . . . . . . . . .  28
       A.4.  Collective communication methods  . . . . . . . . . . . .  32
     Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  37







  Fressancourt, et al.      Expires 10 April 2025                 [Page 2]


  Internet-Draft                 AI traffic                   October 2024


  1.  Introduction

     AI has attracted a far bit of attention in the networking community,
     including the IETF, in regards to not just applications but also
     needed protocols and technologies for the realization of those
     applications.

     While AI is a large area, this document focuses on the method of
     (training and inferencing over) Large Language Models (LLM).  The
     starting point being the distributed nature of implementing LLMs,
     both for training and inferencing.  For this, a network of so-called
     'workers' is being realized, that over time will train a model, over
     which in turn inferencing can be performed.  This distributed nature
     involves a number of communication patterns to exchange suitable
     information, those patterns needing realization in suitable
     protocols.  Differences in those protocols emerge from the deployment
     choices for AI platforms and the challenges that arise from those
     deployments.

     The training of LLMs in ever growing large-scale data centres (DCs)
     is a current method to push the boundaries of key performance
     indicators, most notably the number of parameters included in the
     training model [LLMSize].  Observations in recent works [AIBackbone],
     however, point to the fact that distribution across more than one DC
     will quickly be necessary to continue the growth of LLM training.

     LLMs may also start inherently distributed in deployment itself, not
     because of their size, but because of their use case.  An example
     here is content moderation in fediverse environments [FEDI], where
     decentralized data governance and computation drives the equally
     decentralized realization of the workers developing a shared model
     over time.  Other examples of such decentralized use case stems from
     health (e.g., where independent health trust may train a shared model
     over strictly locally governed patient data), IoT (where AI-derived
     features may be derived from localized sensor data), or also network
     management (where operator data for training may not be possible or
     legally permitted to be disclosed to a centralized training entity).
     Realizations of platforms for those use cases often refer to
     'federated learning' [FLOWER] to capture the equal basis of
     participation, both at the data and processing level, of the
     participating entities of the platform.

     The intention of this document is to highlight the possible
     challenges in the spectrum of realizations for the distributed LLM
     training and inferencing.  For this, we provide more details and
     examples for use cases in Section 2, followed by a primer of key AI/
     LLM techniques in Section 3.  Then, a number of challenges to, e.g.,
     resource management, latency, and security, are identified in



  Fressancourt, et al.      Expires 10 April 2025                 [Page 3]


  Internet-Draft                 AI traffic                   October 2024


     Section 4 as a starting point for a focussed discussion in the IETF
     community on challenges AI may pose for networks, networks protocols
     and technologies.

     As the spectrum of realization ranges from centralized intra-DC to
     highly distributed, federated AI platforms, there is a strong
     relevance of solving the identified challenges within the IETF,
     leading to the formulation of a problem statement in Section 5 along
     those lines and proposing next steps in Section 6 for the IETF
     community to pursue.

  2.  Applicability of AI

     This section introduces the applicability of AI/LLMs across a
     spectrum of decentralization for realising machine learning training
     and inferencing tasks.  These two tasks are the most intensive in AI
     workflows, and they are inherently distributed compute problems given
     the amount of memory and processing power they require Section 3 will
     introduce the prominent techniques relevant to implementing those
     tasks in deployed systems.

     It can be observed upfront that the realization of AI/LLM use cases
     is driven by two main aspects, namely the decentralization of data
     and that of compute; during the training phase, the inference phase
     or both.  The proposed examples are introduced with regards to those
     two aspects, for reasons that are detailed in the text.

  2.1.  xGPT-like Services: an example centralized training use case

     Various GPT-like services have gained much attention recently,
     advocating the training of ultra-large language models, providing
     chat-like inferencing services to users for, _e.g._, text generation,
     generative AI-based image creation, and many others.

     The key success factor of those services is the ingestion of vast
     amounts of data into a centralized training entity.  This data may
     come from public sources, search input, through license arrangements
     with, _e.g._, video providers like YouTube, and many others.

     In this use case, centralization pertains to the ownership of the
     training resources under a single entity.  Until now, the prevalent
     deployment of such centralized training is within a single, large-
     scale DC with a growing number of GPUs to execute the necessary model
     training operations over a sustained period of time.  However, the
     growing need for more compute, together with physical and energy
     limitations in existing DCs, make a growing scale-out of those
     services beyond the reach of a single DC a likely needed path in the
     future.



  Fressancourt, et al.      Expires 10 April 2025                 [Page 4]


  Internet-Draft                 AI traffic                   October 2024


  2.2.  Decentralized training with centralized inference use cases

     In some cases, it is impossible to gather the amount of data
     necessary to properly train a machine learning model, either for
     security, privacy or regulatory reasons.  The necessity to tackle
     those cases triggered the development of Federated learning
     [FederatedLearning], in which several entities can collaborate in a
     decentralized way to the training of a single large model.

     Health is a traditional area for reasoning-based systems.  Its
     richness in data, both historical as well as current (_e.g._, from
     patients), lends itself for training LLMs over which inferencing can
     take place for, _e.g._, diagnosis, drug development, developing
     treatment plans and many others.

     Individual health organizations are often well equipped in terms of
     compute resources, albeit not at the scale that suffices to perform
     centralized LLM training on their own.  Thus, federating compute
     resources is useful for scale and incentivized through the sharing of
     the resulting LLM in the clinical pathway.

     Furthermore, data is also localized, since patients sign up to local
     health organizations, practises, or health trusts, which in turn
     manage their data.  In many countries, data sharing is performed,
     also for transferability (_e.g._, when patients change location and
     thus local health contacts) but also treatment across health
     organization.  Overall, data governance needs to be strictly adhered
     to for any application of the data, with a possible development of a
     LLM being just one such application.

     Network management is another use case for federated learning, where
     the federation is driven by the common goal to develop an
     increasingly refined model for, _e.g._, intrusion detection, network
     failure detection, and others, but where suitable training data is
     only shared in a limited manner or not at all for confidentiality
     reasons.

     In those use cases, the decentralization of the training stems from
     the constraints limiting the exchaneg of data between entities for
     technical feasibility, privacy, confidentiality or regulatory
     reasons.  Once trained, the models including the common knowledge
     gathered in the training phase can be used by each single entity
     without necessarily requiring collaboration during the inference
     phase.







  Fressancourt, et al.      Expires 10 April 2025                 [Page 5]


  Internet-Draft                 AI traffic                   October 2024


  2.3.  Federated building management: a decentralized inference use case

     Building management, where smart buildings are often equipped with
     micro-DC capabilities, can be federated to improve energy management
     in a larger geographic area, utilizing local yet private data to
     train a suitable (possibly locally limited) LLM.  Works like
     [AIConst] expand this use case into other areas like construction
     management, architectural design, material design, and others.

     Similar to the previous use case, the deployment across shared
     infrastructures (as well as the sharing of compute resources itself)
     is a key aspect, utilizing Internet technologies for realizing the
     necessary exchange within the training and inferencing scenario at
     hand.  Here, the decentralization of the inference task is a
     necessity given that the goal is to reach a global optimum (_e.g._,
     energy savings accross an aea or region) by clustering the
     capabilities of buildings while keeping the data used for training
     and inferencing local for security and privacy reasons.

  2.4.  Key Takeaways

     The following key takeaways can be derived from the proposed
     applicability examples:

     *  _LLM AI is inherently distributed_: This is due to its scale in
        required training nodes and model size.  Although the distribution
        may be handled entirely within a central, single data centre, the
        execution of tasks across, often many thousands, of workers still
        remains a key aspect in LLM AI.

     *  _Centralized LLM AI training does not need to touch the Internet_:
        This is because it may be performed entirely within the limits of
        a single DC, or at least among a few DCs interconnected by private
        lines under the control of a single entity.

     *  _Centralization of compute implies centralization of data_: This
        is a consequence of the model creation, based on the data, which
        is centralized and thus requires suitable data transfer towards
        the compute nodes.

     *  _Federation allows for decentralization of data_: This is possible
        with worker nodes that may individually train on data, while
        contributing to the global model.  However, centralization of the
        federation again leads to the observation in the second item,
        _i.e._, data is centralized, too.






  Fressancourt, et al.      Expires 10 April 2025                 [Page 6]


  Internet-Draft                 AI traffic                   October 2024


     *  _Inferencing inherently touches the Internet_: This is especially
        true in any applicability scenario involving end users residing on
        the Internet.  The impact may the creation of very large amounts
        of traffic to (_e.g._, centralized) nodes that hold the suitable
        model information for performing the desired inferencing.

     The next section outlines the system aspects of LLM AI, in light of
     the above takeaways, _e.g._, on centralization, while Section 4 more
     directly takes these takeaways into account when formulating key
     challenges for networking, stemming from the insights in this
     section.

  3.  How ML systems are built

  3.1.  Lifecycle of a typical large language model

     In the last few years, the impressive AI boom has been associated
     with the development of Large Language Models (LLMs).  LLMs are
     models, or representations of systems that are observed or measured
     with data.  Models have an underlying structure, consisting in a
     parametrized graph linking together small operators performing
     operations on *tensors*, or small arrays of finite dimensions.
     Models are built, or *trained*, using a reference data set (or
     training set), consisting in data labelled with its underlying
     structure or meaning.

     Before its use for training, the data is collected and pre-processed
     to structure it and chunk it into *tokens*, the basic data units used
     as input and outputs of models.  After the training data set has been
     prepared, the model is trained.  During this training phase, the
     model is parametrized, i.e., the parameters ruling the strength of
     the relationships between the tensor operators constituting the model
     are modified so the model is able to abstract the behaviour of the
     system from the dataset representing it.

     After the training phase, the model can be used during an *inference*
     phase to derive information or make predictions from new data
     presented in the form of a sequence of tokens.  Inference operations
     are not necessarily done by the same nodes that have trained the
     model.  Models can be transferred to other worker nodes to perfom
     inference tasks, sometimes placed close to the users making requests.
     Besides, those transferred models can be re-trained or fine-tuned in
     order to better serve requests in the context they are done.  This
     can be done for instance with private or locally relevant data.







  Fressancourt, et al.      Expires 10 April 2025                 [Page 7]


  Internet-Draft                 AI traffic                   October 2024


  3.2.  The distributed nature of Machine Learning systems

     To improve the accuracy of LLMs, those models incorporate an
     increasing amount of parameters, and are trained using ever larger
     datasets.  Modern LLMs use 125 million up to 405 billion parameters,
     where each parameters can be encoded with 4 bits to 4 bytes depending
     on the required precision.  This increase in models' sizes has
     important consequences on the power consumption and on the memory
     footprint of the systems used to train those models.

     From a power consumption perspective, the increase in computing power
     needed to accomplish the training tasks of large models requires the
     deployment of more powerful Tensor Processing Units (TPUs) of
     Graphics Processing Units (GPUs) with higher power and cooling
     demands.  It is becoming physically unsustainable to bring the power
     required to datacenters hosting those machine learning worker nodes.
     A possible way to address this challenge is to distribute machine
     learning workloads beyond the realm of a single datacenter, possibly
     between two or a few interconnected datacenters separated by long to
     mid range private interconnections.

     From a memory perspective, this means that storing a model and its
     parameters will cost from 62,5 MB (for a model using 125 million
     parameters with 4-bits parameters) to 1620 GB (for a model using 405
     billion parameters with 4 bytes parameters).  The memory footprint of
     modern large language models makes those models difficult to
     manipulate on a single node.  Besides, as mentionned in
     [SchedulingSharding], the pre-training of Llama3 with 405 billion
     parameters required from 8192 to 16384 GPUs, which were obviously
     hosted on multiple, connected nodes.

     To cope with the memory and computing needs of workloads associated
     with LLM operations, the data preparation, training and inference
     tasks mentionned in Section 3.1 can be distributed among nodes with
     specific functionnal roles (task execution, result aggregation,
     orchestration...).  The detailed description of those functional
     roles is given in Appendix A.2.

     From a networking perspective, some elements are important to
     mention:

     *  Some roles (aggregator or orchestrator) tend to concentrate
        network traffic, thus creating some issues in properly managing
        the incast traffic.







  Fressancourt, et al.      Expires 10 April 2025                 [Page 8]


  Internet-Draft                 AI traffic                   October 2024


     *  The parallel execution of training and inference tasks by worker
        nodes is following specific parallelism modes presented in
        Figure 1 and detailed in Appendix A.3.  Those modes have various
        requirements in terms of volume of data exchanged and latency (see
        [SchedulingSharding] for instance).

        a. *Data parallelism*, in which data is split into chunks to be
        used by different worker nodes to train the model.  This
        parallelism mode can cope with rather large latencies (several
        100s of ms) but requires exchanging a large volume of data.

        b. *Pipeline model-parallelism*, in which a model is separated
        into stages consisting in a few consecutive layers of the entire
        model.  Each stage is allocated to a separate worker node, and
        intermediate states are exchanged between successive worker nodes.
        This parallelism mode can cope with large latencies (10s of ms)
        and requires exchanging less data than data parallelism.

        c. *Tensor model-parallelism*, in which model layers are split
        into chunks that can be operated by different nodes.  This
        parallelism mode needs to operate in low latency networks (10s of
        us) and requires exchanging a lot of data.

        d. *Mixture of Expert parallelism*, in which nodes are holding
        smaller but specialized models trained over a smaller amount of
        data and holding fewer parameters.  This parallelism mode can cope
        with latencies in the ms range.

     *  Machine learning applications rely most of the time on collective
        communication libraries [xCCL] that use patterns presented in
        details in Appendix A.4 such as All-to-All or All-reduce.
        [I-D.yao-tsvwg-cco-problem-statement-and-usecases] has already
        introduced some networking challenges associated with the use of
        collective communication in machine learning systems.  From a
        networking perspective, those communication patterns translate in
        "on-off" communication patterns, with a few large flows starting
        simultaneously to allow nodes involved in a collective to exchange
        data and parameters.  The volume, synchronism and imbalance of the
        traffic generated by those communication pattern is a burden to
        the distribution of machine learning workloads, and a challenge to
        be addressed by the networking community.










  Fressancourt, et al.      Expires 10 April 2025                 [Page 9]


  Internet-Draft                 AI traffic                   October 2024


     +----+     +-+ +-+
     |    |     |P| |P|     +-+
     |    |     |.| |.|     |O|
     |Data| ==> |1|X|2| ==> |u|
     | 1. |     |-| |-|     |t|           +----+
     |    |     | | | |     +-+         +-+--+ |     +-+     +-+
     |    |     +-+ +-+                 |    | |     |P| +-+ |P|     +-+
     +----+                             |    | |     |.| |I| |.|     |O|
                                        |Data| | ==> |1|X|n|X|2| ==> |u|
     +----+     +-+ +-+                 |    | |     |-| |t| |-|     |t|
     |    |     |P| |P|     +-+         |    | |     | | +-+ | |     +-+
     |    |     |.| |.|     |O|         |    +-+     +-+     +-+
     |Data| ==> |1|X|2| ==> |u|         +----+
     | 2. |     |-| |-|     |t|
     |    |     | | | |     +-+
     |    |     +-+ +-+
     +----+
     (a) Data parallelism               (b) Pipeline model-parallelism

                  +-+ +-+                                +-+ +-+
                  |P| |P|     +-+                        |P| |P|
                  |.| |.|     |O|                        |.| |.|
       +----+     |1|X|2| ==> |-|         +----+       ^ |1|X|2|\
     +-+--+ |   ^ |-| |-|     |1|       +-+--+ |      /  |-| |-| \
     |    | |  /  |1| |1|     +-+ +-+   |    | |     /   |1| |1|  v  +-+
     |    | | /   +-+ +-+       \ |O|   |    | |   +-+   +-+ +-+     |O|
     |Data| |      |   |          |u|   |Data| |==>|R|  ----------   |u|
     |    | | \   +-+ +-+       / |t|   |    | |   +-+   +-+ +-+     |t|
     |    | |  \  |P| |P|     +-+ +-+   |    | |         |P| |P|     +-+
     |    +-+   v |.| |.|     |O|       |    +-+         |.| |.|
     +----+       |1|X|2| ==> |-|       +----+           |1|X|2|
                  |-| |-|     |2|                        |-| |-|
                  |2| |2|     +-+                        |2| |2|
                  +-+ +-+                                +-+ +-+
     (c) Tensor model-parallelism       (d) Mixture of Expert parallelism

         Figure 1: Parallelism models used in machine learning systems

  3.3.  The topology of distributed machine learning systems

     To boost the workload distribution efficiency, nodes participating in
     the common execution of machine learning workloads are interconnected
     following specific topologies.  Depending on the number of nodes
     involved, the nodes' capacities to connect directly with other nodes
     and the control of the machine learning application administrator
     over the physical connectivity substrate, difefrent topologies can be
     used.  Those topologies can inherit from the work done either in the
     high performance computing community if the nodes are homogeneous,



  Fressancourt, et al.      Expires 10 April 2025                [Page 10]


  Internet-Draft                 AI traffic                   October 2024


     high capacity computers under the control of a single administrator
     or in the peer-to-peer or swarm computing community if the nodes are
     more diverse and numerous.

     Topologies inspired from the HPC networking community are typically
     found in distributed systems in which all the nodes are located in
     the same datacenter.  In specific cases, those topologies can be
     extended beyond a single datacenter.  Then, the distribution of the
     workloads is done so as to avoid frequent and latency-bound
     communications over those links, and congestion control mechanisms
     are tuned to avoid deadlocks and bloats in network equipments manging
     the inter-datacenter link.  Among those HPC-inspired topologies, we
     can find the n-dimension torus, the Dragonfly topology,the fat tree
     topology and the hypercube topology.

     On the other end of the spectrum, topologies inspired from swarm
     computing and peer-to-peer networks have emerged as solutions
     connecting heteorgeneous nodes involved in distributed machine
     learning tasks outside datacenters, at the edge of the networks or
     among clusters of nodes hosted on public cloud platforms.  Those
     topologies can be built either from dynamic wireless connections,
     which can be rearranged easily, or as overlay links on top of an
     immutable physical connectivity substrate.  Among those swarm-
     inspired topologies presented in Figure 2, we can find the full mesh,
     the star topology, the Peer-to-Peer topology which presents a limited
     diameter without too many connections, the random topology and the
     clustered topology.  Note that in the case of the clustered topology,
     nodes can be gathered based on their proximity to a proxy or an
     aggregator, or based on their interest or capabilities.  Besides, the
     clustered topology can be structured according to a hierarchy.

        o-----o     o     o     o-----o        o     o      o o o   o o o
       / \   / \     \   /     / ____/ \       |    /        o o   / o o
      /   \ /   \     \ /     / /       \      o---o          o \ o
     o-----+-----o  o--o--o  o-+       +-o      \   \   o        o o
      \   / \   /     / \     \   ____/ /        o---o--+       / o \ o
       \ /   \ /     /   \     \ /     /    o   /          o o o     o o
        o-----o     o     o     o-----o     +--o            o o       o

     Full mesh      Star     Peer-to-Peer   Random         Clustered
     topology       topology topology       topology       topology

                       Figure 2: Decentralized topologies

     The topology connecting the nodes together is not anecdotical.  If it
     is under the control of the machine learning system administrator,
     the construction of a proper topology able to efficiently support the
     communication patterns generated by the parallelization model and the



  Fressancourt, et al.      Expires 10 April 2025                [Page 11]


  Internet-Draft                 AI traffic                   October 2024


     collective communication method used is a key performance factor.  If
     the topology is inherited from the environment, machine learning
     system administrators will need to adapt the communication patterns
     they use to the characteirstics of the topology.

  3.4.  Deployment considerations for AI systems

     Given the computing and memory requirements of machine learning
     workloads, modern machine learning systems are fundamentally
     distributed.  By combining functionnal roles together, distributing
     them using a combination of parallelization modes, and communicating
     following patterns, a variety of systems with different shades of
     decentralization can be built.  Indeed, distributing workloads across
     several workers does not necessarily means that the control and
     orchestration of those workloads is distributed.  In case a parameter
     server is used, the orchestrator role is played by a single node.

     Yet, following the Federated learning [FederatedLearning] approach,
     machine learning systems can be decentralized as shown in Figure 3.
































  Fressancourt, et al.      Expires 10 April 2025                [Page 12]


  Internet-Draft                 AI traffic                   October 2024


              +----+   +----+                     +----+   +----+
              | D  |   |  D |                     | DM |   | DM |
              +--+-+   +-+--+                     +--+-+   +-+--+
                  \     /                             \     /
                 +-+---+-+                           +-+---+-+
                 |Central|                           |Central|
                 | Inst. |                           | serv. |
                 | DM (Σ)|                           |  M (Σ)|
                 +-+---+-+                           +-+---+-+
                  /     \                             /     \
              +--+-+   +-+--+                     +--+-+   +-+--+
              | D  |   |  D |                     | DM |   | DM |
              +----+   +----+                     +----+   +----+

            Centralized learning          Centralized federated learning

               +-----------+                       +-----------+
               |   DM (Σ)  |                       |   DM (Σ)  |
               +-+---+---+-+                       +-+---+---+-+
                /    |    \                         /    |    \
     +---------+-+   |   +-+---------+   +---------+-+   |   +-+---------+
     |     DM    +---*---+     DM    |   |   DM (Σ)  +---*---+   DM (Σ)  |
     +--------+--+  / \  +--+--------+   +--------+--+  / \  +--+--------+
               \   /   \   /                       \   /   \   /
        +-------+-+-+ +-+-+-------+         +-------+-+-+ +-+-+-------+
        |     DM    +-+     DM    |         |   DM (Σ)  +-+   DM (Σ)  |
        +-----------+ +-----------+         +-----------+ +-----------+

             Semi-decentralized                Fully decentralized
             federated learning                federated learning

        Figure 3: Different centralization models in federated learning

     In federated learning, the coordination role of the orchestrator or
     the aggregation of parameters done by the aggregator can be done by a
     subset or all the worker nodes deployed in the system, using a
     distributed protocol to exchange information, synchronize parameters
     and agree on a reference.  Of course, this comes at a communication
     cost, but decentralization also has benefits.












  Fressancourt, et al.      Expires 10 April 2025                [Page 13]


  Internet-Draft                 AI traffic                   October 2024


     One of the first reason for decentralizing machine learning systems
     is to allow users to retain ownership of their data, and to control
     how their data is transferred or used.  In a centralized setting,
     users need to trust the central entity managing the system for
     respecting the data management agreed upon with the user.  In a
     decentralized setting, users can either infer from a copy of a
     generic model sent by a central entity (centralized federated
     learning), fine-tune the model and infer from it (semi-decentralized
     setting) or even cooperate with others to train together a model and
     use it afterwards (in a fully decentralized setting).

     Besides, in a decentralized machine learning system, users and worker
     nodes can be co-located, or workers can be placed close to the user
     in order to reduce the overall communication latency.  In such a
     decentralization scheme, if it is considered that edge nodes can
     benefit from less memory or computing power, a balance has to be
     found between the time spent in communicating between the user and
     worker nodes and the processing time to reach an optimal response
     time for user requests.  Worker nodes located at the edge of the
     network may also collaborate with other, more capable nodes located
     in other locations.  The tail latency of the flows associated with
     those tasks should be bounded to avoid degrading the user's
     experience.

  4.  Challenges (in Networking for AI)

  4.1.  Network resource management at regional scale

     In (large) model-based machine learning, training, fine-tuning and
     infering from the model are workloads that involve the transfer of a
     very large volume of data.  In [AITrainingTraffic], the authors
     estimate that the first iteration for training a GPT-3B model among
     32 GPU nodes generate roughly 85 GB of tensor-parallel data traffic,
     1 GB of pipeline-parallel data traffic and 741 MB of data-parallel
     data traffic.  The tensor-parallel and pipeline-parallel data traffic
     consist in 125 MB messages that are periodically exchanged during
     communication phases, while very little traffic is exchanged during
     computing phases.

     This traffic pattern is a good example of the characteristics of
     network traffic flows associated with machine learning workloads.
     Indeed, the network traffic generated by distributed machine learning
     systems consists in a relatively little number of large (elephant)
     flows, starting and stopping roughly simultaneously.  Such
     synchronous traffic patterns are coming from the use of collective
     communication libraries such as NCCL and associated collective
     communication primitive such as All-Reduce or All-Gather, as
     mentionned in [Burstiness].  Dealing with this synchronized traffic



  Fressancourt, et al.      Expires 10 April 2025                [Page 14]


  Internet-Draft                 AI traffic                   October 2024


     in a stochastic network is challenging, because it generates periodic
     traffic bursts that bloat network equipment buffers.  Besides, the
     network's capacity needs to meet peak requirements at the cost of an
     unefficient average utilization of the installed capacity.  Even if
     the network's capacity is sufficient in theory to accomodate the peak
     requirements of machine learning systems, the transport mechanisms
     used on the links between the nodes make it difficult to immediately
     use the full deployed capacity, resulting in inefficiencies at the
     network level [NetworkBottleneck] At last, in such a synchronous
     communication pattern, the failure of a link, delaying data
     transmission between two nodes in the system might delay the whole
     system after a few iterations.

     Mitigating the resource management challenges raised by machine
     learning traffic patterns is currently an open research area.  At the
     application level, machine learning system designers work to develop
     hybrid parallellization schemes combining the patterns presented in
     Section 3.2 (detailed in Appendix A.3) and orchestration methods
     aiming at better utilizing the deployed network capacity and at
     avoiding "on-off" behaviors.

     At the network level, the main challenge that the research community
     is trying to address is the proper balancing of the flows generated
     by machine learning workloads in the network.  Indeed, to address the
     need for bandwidth and avoid bottlenecks, machine learning system
     designers are often deploying their system on top of networks whose
     topology presents a large number of equal cost links between two
     nodes.  Yet, as mentionned earlier, machine learning traffic is
     constituted with a rather small number of large flows that have a
     very small entropy.  This makes applying classic load balancing
     techniques challenging.  To address this load balancing issue, most
     collective communication libraries use packet spraying strategies,
     which require specific tuning due to mentioned lack of entropy.  Yet,
     some researchers are questionning the relevance of this approach
     [ChallengingSpraying].

  4.2.  Latency sensitivity of LLM training and inference

     Training a (large) model in a distributed setting does not
     necessarily require data transfer to be operated at a controlled low
     latency, but large tail latencies related to packet losses or head of
     line blocking can delay the training of models considerably.  Indeed,
     problems arising from packet losses, link failures or excessive
     buffering can have cascading consequences since in most
     parallellization methods, data and model parameters are exchanged
     following synchronized patterns (see Section 4.1).





  Fressancourt, et al.      Expires 10 April 2025                [Page 15]


  Internet-Draft                 AI traffic                   October 2024


     The extend of the amplification of the latency effects of a soft
     failure on a connection between two nodes depends on the topology of
     the network connecting the nodes.  Besides, routing inefficiencies or
     failures to properly balance the load on some heavily-used links can
     also generate additional latencies.  Thus, the topology of the
     network supporting machine learning workloads needs specific care.

     In a large scale and decentralized AI context, heterogeneous links
     can be used between nodes participating in a decentralized machine
     learning system.  The specificities of the links need to be taken
     into account when orchestrating or distributing AI-related tasks.  As
     latency is affected by congestion, addressing the mismatch between
     links' bandwidth-delay products for efficient congestion management
     is an open challenge.  In the research community, some projects have
     proposed to introduce proxies to investigate new control loops taking
     into account link segments characteristics [SiteToSite].  In the
     IETF, CSIG draft (now expired) presented a model to expose a rich
     congestion signal [I-D.ravi-ippm-csig].

  4.3.  Aligning communications patterns to the Internet's constraints

     In the development and deployment of distributed machine learning
     systems within the real of an administrator's responsibility, it is
     feasible, and advised, to design the system to adapt the network's
     topology to the paralellization method used and to the collective
     communication patterns required to perform the training or inference
     task efficiently.  As we have seen in Section 3, several network
     topologies can be adopted, depending on the model's architecture and
     on the design choice made while designing the training system.

     In a decentralized or federated setting, such a co-design offers
     lmess freedom, as some adverserial choices can be made by people
     willing to cooperate on specific machine learning tasks.  Besides,
     when the nodes involved are deployed at the edge of the network, in a
     majority of the cases, the topology is following the access network's
     specificities rather than adapting to the requirements of machine
     learning tasks.

     As machine learning-related traffic is growing on the Internet, some
     challenges associated with the delivery of network flows related to
     the realization of decentralized training or inference tasks are
     appearing: How to inform a machine learning application about the
     topology of the network interconnecting involved nodes?  How to adapt
     the parallelization model or collective communication pattern to
     maximize efficiency?  In the IETF, the ALTO working group [ALTO] has
     investigated similar challenges related to the operation of peer-to-
     peer traffic.  As machine learning workloads have a different set of
     requiremenst, it may be time to revisit this work.



  Fressancourt, et al.      Expires 10 April 2025                [Page 16]


  Internet-Draft                 AI traffic                   October 2024


  4.4.  Managing incast traffic related to AI inference

     In a machine learning system, some specific nodes, such as the
     orchestrator of the aggregator, play a central role, and thus
     concentrate communications.  This concentration might be reinforced
     by the use of specific collective communication primitives, and by
     the synchronicity of traffic patterns.  The network traffic
     management challenges we previously mentionned (Packet losses,
     delays, congestion) can be amplified by this concentration on a few
     hotspots in the network.

     Decentralized systems tend to mitigate this functionnal centrality by
     distributing the responsibility for fullfilling those roles accross
     several nodes.  Yet, even in decentralized systems, it is difficult
     to completely avoid incast problems given the specificities of
     machine learning workloads.  For instance, due to the characteristics
     of model inference, first prompt requests addressed to a given node
     are bound to generate a large incast traffic related to the
     personalization and fine tuning of the model instance run by the
     node, which can represent a large amount of data.

     The research and operational communities dealing with scaling out
     machine learning systems are working on some solutions to address
     incast traffic management issues.  For instance, in the same way as
     media codecs structured as layers of increasing resolution, models
     used in training and inference can be layered, with coarse grained
     models using less parameters than finer grained ones.  Coarse grain
     models can be distributed more quickly, drafting a first answer to a
     request while finer grain models are retrieved and then used to build
     a more precise answer to a request.  Besides, as inference requests
     towards large models are often done in the form of a converstaion or
     contextualized exchange between a user and the node running inference
     tasks addressing its requests, the use of service routing to
     consistently route requests to the same instance can be used.
     In that extend, the work done in the CATS working group in the IETF
     is of particular relevance ([CATSWG]).  Specific work on the proper
     metrics to be applied to CATS in the context of large scale
     decentralized AI will need to be done, taking into consideration the
     large body of work on load balancing for machien learning workloads
     done in the research community.  In particular, one problem of
     importance in the selection of the instance serving a specific
     inference request is the trade-off to find between the need to serve
     the requests with a bounded latency, requiring to use a node that is
     located as close to the requesting user as possible, the necessity to
     bootstrap this node with the user's contextual model parameters and
     the uncertainty about the length of the session between the user and
     the instance, which puts a stress on the memory needed to keep the
     session's context at hand.



  Fressancourt, et al.      Expires 10 April 2025                [Page 17]


  Internet-Draft                 AI traffic                   October 2024


  4.5.  Securing and attesting AI-related traffic

     The distribution of machine learning workloads at a regional scale
     beyond the limits of a single datacenter and the decentralization of
     the control and management of those operations raises important
     security and privacy challenges.  Indeed, when those workloads are
     operated in a single tenant datacenter, data can be secured by means
     of perimeter security measures, or adopting encryption at least for
     data stored in the datacenter's realm.  In the same way, it is
     acceptable to exchange data between nodes in an unencrypted manner
     provided the network on which data is exchanged is isolated from the
     outside environment.

     When data, models and their associated parameters are exchanged on
     the Internet, or on public connections, data managers need to make
     sure that the data is exchanged securely, and following policies
     complying both with users preferences and local regulations.  As
     previously mentionned, data exchanges done during model training
     phases or directly before performing an inference task need to be
     done as quickly as possible, avoiding tail latencies as much as
     possible.  Even if encryption of large data flow has improved
     considerably in the last years, it is adding a latency overhead,
     considering that the computational overhead related to cryptographic
     operations is handled beyond the chip (often a GPU or a tensor
     processor) performing the machine learning tasks.  It would be
     interesting and benefitial to contribute to the efforts done in the
     IRTF and IETF to develop low latency, lightweight encryption schemes
     in order to reduce this overhead as much as possible.

     Furthermore, as private data is involved in data exchanges related to
     training and inference, specific care must be taken to respect
     regulations steering the way those data can be processed and
     transfered.  In particular, some data need to be processed in the
     same region or country they have been generated in.  To make sure
     those regulations are respected, attestation mechanisms need to be
     put in place.  By using those attestation mechanisms, data owners or
     managers can prove to local authorities that the data they are
     managing is being exchanged according to the policy they specified.
     In the IETF, the RATS working group [RATSWG], and the NASR initiative
     [NASR] are developping attestation mechanisms that could be adapted
     to address machine learning requirements, even if their work is still
     at the beginning.









  Fressancourt, et al.      Expires 10 April 2025                [Page 18]


  Internet-Draft                 AI traffic                   October 2024


  5.  Problem statement

     In today's LLM-based machine learning landscape, we observe a strong
     concentration of training abilities in the hands of a few
     hyperscalers, while several companies tend to propose innovative
     approaches to improve inference by making it faster, and done by
     machines located closer to the users.  Yet, there is a need to
     distribute both the training and inference workloads of AI.

     In the same way as the Internet, there is, and will be several
     incentives to decentralize some or all aspects of AI's lifecycle to
     address scalability, resilience, personalization or privacy concerns.
     This decentralization trend is exemplified by Federated Learning
     [FederatedLearning], with different decentralization models, as
     presented in Section 3.4.

     Given the challenges highlighted in Section 4, and the fact that
     multiple stakeholders are involved in properly adressing those
     challenges, AI-related network traffic will no longer be operated
     only on private infrastructure, but also on an *_open interconnected
     network_*.  Thus, it is desirable that the IETF community discuss
     networking challenges related to large scale decentralized AI to
     avoid the deployment of proprietary solutions, or of solutions
     putting the stability of the Internet at risk due to unfair resource
     mangement or competition between AI-related network traffic and other
     traffic in the Internet.

  6.  Next Steps

     While some work addressing the challenges highligted in this document
     is done in working groups related to congestion management,
     deterministic networking or service routing, there might be interest
     in the IETF community at large to aggregate a group of contributors
     interested in elaborating specific requirements and corresponding
     solutions to the challenges listed in Section 4.  In particular, the
     goal of such an initiative would be to:

     *  Formalize AI-related requirements for service routing: Indeed, it
        is needed to define the metrics to take into account for load
        balancing AI-related workloads, given the need for instance
        stickiness or the importance of latency aspects while addressing
        the specificities of AI-related traffic, consisting in elephant
        flows with little entropy.








  Fressancourt, et al.      Expires 10 April 2025                [Page 19]


  Internet-Draft                 AI traffic                   October 2024


     *  Formalize low latency / limited latency requirements associated
        with AI network traffic: As presented in this document, during the
        different phases of the AI lifecycle, it will be needed to enforce
        a different set of requirements in terms of latency, tolerance to
        jitter or packet loss for sustaining network traffic.

     *  Formalize congestion control aspects related to the operation of
        AI traffic at regional scale: The machine learning community is
        already engaged in improvements of the congestion aspects of AI
        workloads by working on application-level solutions taking for
        granted the underlying behavior of the network.  It would be
        interesting to determine what are the possible improvements that
        the network could benefit from in order to better solve the
        congestion and incast management issues related to the way AI
        network traffic is managed.

     *  Formalize coordination aspects of AI distributed systems: This
        topic is very important to the realization of decentralized, large
        scale AI systems, and to the emergence of an inter-AI network of
        entities collaborating in the execution of end to end AI workloads
        for end users.

  7.  Security Considerations

     Section 4.5 highlights privacy related challenges that AI operations
     at regional scale will have to address.  Beyond this section, no
     additional security concern is raised by the elements presented in
     the document.

  8.  IANA Considerations

     This document has no IANA actions.

  9.  Informative References

     [AIBackbone]
                Sundaresan, J., Gopalan, A., and Meta, "AI impact on
                backbone", <https://atscaleconference.com/videos/ai-
                impact-on-backbone/>.

     [AIConst]  Baduge, S., Thilakarathna, S., Perera, J., Arashpour, M.,
                Sharafi, P., Teodosio, B., Shringi, A., and P. Mendis,
                "Artificial intelligence and smart vision for building and
                construction 4.0: Machine and deep learning methods and
                applications", Elsevier BV, Automation in
                Construction vol. 141, pp. 104440,
                DOI 10.1016/j.autcon.2022.104440, September 2022,
                <https://doi.org/10.1016/j.autcon.2022.104440>.



  Fressancourt, et al.      Expires 10 April 2025                [Page 20]


  Internet-Draft                 AI traffic                   October 2024


     [AITrainingTraffic]
                Li, W., Liu, X., Li, Y., Jin, Y., Tian, H., Zhong, Z.,
                Liu, G., Zhang, Y., and K. Chen, "Understanding
                Communication Characteristics of Distributed Training",
                ACM, Proceedings of the 8th Asia-Pacific Workshop on
                Networking vol. 23, pp. 1-8, DOI 10.1145/3663408.3663409,
                August 2024, <https://doi.org/10.1145/3663408.3663409>.

     [ALTO]     "Application-Layer Traffic Optimization (alto)", n.d.,
                <https://datatracker.ietf.org/wg/alto/about/>.

     [Burstiness]
                Luangsomboon, N., Fazel, F., Liebeherr, J., Sobhani, A.,
                Guan, S., and X. Chu, "On the Burstiness of Distributed
                Machine Learning Traffic", arXiv,
                DOI 10.48550/ARXIV.2401.00329, 2024,
                <https://doi.org/10.48550/ARXIV.2401.00329>.

     [CATSWG]   "Computing-Aware Traffic Steering (cats) Working Group",
                n.d., <https://datatracker.ietf.org/group/cats/about/>.

     [ChallengingSpraying]
                Addanki, V., Goyal, P., and I. Marinos, "Challenging the
                Need for Packet Spraying in Large-Scale Distributed
                Training", arXiv, DOI 10.48550/ARXIV.2407.00550, 2024,
                <https://doi.org/10.48550/ARXIV.2407.00550>.

     [DataParallelism]
                Valiant, L., "A bridging model for parallel computation",
                Association for Computing Machinery (ACM), Communications
                of the ACM vol. 33, no. 8, pp. 103-111,
                DOI 10.1145/79173.79181, August 1990,
                <https://doi.org/10.1145/79173.79181>.

     [FederatedLearning]
                McMahan, H., Moore, E., Ramage, D., Hampson, S., and B.
                Arcas, "Communication-Efficient Learning of Deep Networks
                from Decentralized Data", arXiv,  arXiv,
                DOI 10.48550/ARXIV.1602.05629, 2016,
                <https://doi.org/10.48550/ARXIV.1602.05629>.

     [FEDI]     Anaobi, I., Raman, A., Castro, I., Zia, H., Ibosiola, D.,
                and G. Tyson, "Will Admins Cope? Decentralized Moderation
                in the Fediverse", ACM, Proceedings of the ACM Web
                Conference 2023, DOI 10.1145/3543507.3583487, April 2023,
                <https://doi.org/10.1145/3543507.3583487>.





  Fressancourt, et al.      Expires 10 April 2025                [Page 21]


  Internet-Draft                 AI traffic                   October 2024


     [FLOWER]   Flower Labs GmbH, "Flower - A Friendly Federated Learning
                Framework", n.d., <https://flower.ai/>.

     [I-D.ravi-ippm-csig]
                Ravi, A., Dukkipati, N., Mehta, N., and J. Kumar,
                "Congestion Signaling (CSIG)", Work in Progress, Internet-
                Draft, draft-ravi-ippm-csig-01, 2 February 2024,
                <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-
                csig-01>.

     [I-D.yao-tsvwg-cco-problem-statement-and-usecases]
                Yao, K., Shiping, X., Li, Y., Huang, H., and D. KUTSCHER,
                "Collective Communication Optimization: Problem Statement
                and Use cases", Work in Progress, Internet-Draft, draft-
                yao-tsvwg-cco-problem-statement-and-usecases-00, 23
                October 2023, <https://datatracker.ietf.org/doc/html/
                draft-yao-tsvwg-cco-problem-statement-and-usecases-00>.

     [LLMSize]  Zhang, B., Liu, Z., Cherry, C., and O. Firat, "When
                Scaling Meets LLM Finetuning: The Effect of Data, Model
                and Finetuning Method", arXiv,
                DOI 10.48550/ARXIV.2402.17193, 2024,
                <https://doi.org/10.48550/ARXIV.2402.17193>.

     [MoEParallelism1]
                Jacobs, R., Jordan, M., Nowlan, S., and G. Hinton,
                "Adaptive Mixtures of Local Experts", MIT Press -
                Journals, Neural Computation vol. 3, no. 1, pp. 79-87,
                DOI 10.1162/neco.1991.3.1.79, February 1991,
                <https://doi.org/10.1162/neco.1991.3.1.79>.

     [MoEParallelism2]
                Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le,
                Q., Hinton, G., and J. Dean, "Outrageously Large Neural
                Networks: The Sparsely-Gated Mixture-of-Experts Layer",
                arXiv, DOI 10.48550/ARXIV.1701.06538, 2017,
                <https://doi.org/10.48550/ARXIV.1701.06538>.

     [NASR]     "Network Attestation for Secure Routing (NASR)", n.d.,
                <https://datatracker.ietf.org/doc/bofreq-liu-nasr/>.

     [NCCL]     Nvidia, "NVIDIA Collective Communications Library (NCCL)",
                n.d., <https://developer.nvidia.com/nccl>.








  Fressancourt, et al.      Expires 10 April 2025                [Page 22]


  Internet-Draft                 AI traffic                   October 2024


     [NetworkBottleneck]
                Zhang, Z., Chang, C., Lin, H., Wang, Y., Arora, R., and X.
                Jin, "Is Network the Bottleneck of Distributed Training?",
                ACM, Proceedings of the Workshop on Network Meets AI & ML,
                DOI 10.1145/3405671.3405810, August 2020,
                <https://doi.org/10.1145/3405671.3405810>.

     [PipelineParallelism]
                Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V.,
                Devanur, N., Ganger, G., Gibbons, P., and M. Zaharia,
                "PipeDream: generalized pipeline parallelism for DNN
                training", ACM, Proceedings of the 27th ACM Symposium on
                Operating Systems Principles, DOI 10.1145/3341301.3359646,
                October 2019, <https://doi.org/10.1145/3341301.3359646>.

     [RATSWG]   "Remote ATtestation ProcedureS (rats) Working Group",
                n.d., <https://datatracker.ietf.org/wg/rats/about/>.

     [RCCL]     AMD, "AMD ROCm Software", n.d.,
                <https://github.com/ROCm/ROCm>.

     [SchedulingSharding]
                Chu, W., Choudhury, A., and Meta, "Scheduler and Sharding
                Considerations for Network Efficiency",
                <https://atscaleconference.com/videos/scheduler-and-
                sharding-considerations-for-network-efficiency/>.

     [SiteToSite]
                Cangialosi, F., Narayan, A., Goyal, P., Mittal, R.,
                Alizadeh, M., and H. Balakrishnan, "Site-to-site internet
                traffic control", ACM, Proceedings of the Sixteenth
                European Conference on Computer Systems,
                DOI 10.1145/3447786.3456260, April 2021,
                <https://doi.org/10.1145/3447786.3456260>.

















  Fressancourt, et al.      Expires 10 April 2025                [Page 23]


  Internet-Draft                 AI traffic                   October 2024


     [TensorParallelism]
                Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z.,
                Citro, C., Corrado, G., Davis, A., Dean, J., Devin, M.,
                Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard,
                M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M.,
                Levenberg, J., Mane, D., Monga, R., Moore, S., Murray, D.,
                Olah, C., Schuster, M., Shlens, J., Steiner, B.,
                Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V.,
                Vasudevan, V., Viegas, F., Vinyals, O., Warden, P.,
                Wattenberg, M., Wicke, M., Yu, Y., and X. Zheng,
                "TensorFlow: Large-Scale Machine Learning on Heterogeneous
                Distributed Systems", arXiv,
                DOI 10.48550/ARXIV.1603.04467, 2016,
                <https://doi.org/10.48550/ARXIV.1603.04467>.

     [xCCL]     Weingram, A., Li, Y., Qi, H., Ng, D., Dai, L., and X. Lu,
                "xCCL: A Survey of Industry-Led Collective Communication
                Libraries for Deep Learning", Springer Science and
                Business Media LLC, Journal of Computer Science and
                Technology vol. 38, no. 1, pp. 166-195,
                DOI 10.1007/s11390-023-2894-6, February 2023,
                <https://doi.org/10.1007/s11390-023-2894-6>.

  Appendix A.  A primer on Machine learning (extended version of
               Section 3)

     Along its development, Machine Learning (ML) involves the use of an
     increasing amount of computing and storage resources in the
     operations of algorithms taking decisions or deriving insights from
     data.  In recent generative AI algorithms, or in large language
     models, the amount of data used to train models of increasing size in
     terms of parameters has grown exponentially.  Besides, the size of
     large models translates in an increasing memory and computing
     footprint.  Given this evolution, ML algorithms can not be executed
     or trained on a single machine, and thus ML systems are "distributed"
     by nature, regardless of the architecture they adopt.

     This appendix section introduces the lifecycle of ML systems
     (Appendix A.1), then explains how ML systems can be split between
     entities fulfilling different roles (Appendix A.2) and introduces the
     methods designed to parallelize ML jobs (Appendix A.3) and the
     communication methods used between instances to communicate data and
     parameters (Appendix A.4).  It is an extended version of Section 3
     which focuses on the consequences on network traffic patterns and
     challenges of the elements presented here.






  Fressancourt, et al.      Expires 10 April 2025                [Page 24]


  Internet-Draft                 AI traffic                   October 2024


  A.1.  Machine learning model lifecycle

     In machine learning, two approaches can be adopted to algorithmically
     derive insights from a dataset: supervised learning and unsupervised
     learning.  In unsupervised learning, algorithms are developped and
     used to find patterns, clusters or relationships among data without
     previous knowledge or reference.  In supervised learning, algorithms
     use a reference data set (or training set), consisting in data
     labelled with answers or hints to the solution of the question being
     asked, to build (or train) a model to which data is compared later
     during the inference (or serving) phase.

     The model is the cornerstone of supervised machine learning.  It is a
     representation of a system that is observed and measured with data.
     When the model is trained, the model is parametrized or modified so
     it is able to model the behaviour of the system from the dataset used
     in the training phase.  Then, during the inference phase, data is
     presented to the trained model in order to derive information or make
     predictions about the system.

     +----------+    +----------+    +--------+    +------+    +---------+
     |   Data   |    |   Data   |    |  Model |    | Model|    |Inference|
     |          | => |   Pre-   | => |        | => | Fine-| => |    /    |
     |Collection|    |Processing|    |Training|    |Tuning|    | Serving |
     +----------+    +----------+    +--------+    +------+    +---------+

                Figure 4: Supervised machine learning lifecycle

     In the renewed interest in deep neural networks, and the strong
     development of generative AI, a supervised learning approach has been
     favoured, using reference models.  Supervised machine learning
     follows a cycle presented in Figure 4.

     To obtain a reference model, first, data is collected from a variety
     of sources.  This data is then gathered and pre-processed, in order
     to clean it (align formatting, erase obvious measurement errors,
     _etc._) and potentially to label it.  In particular, large data sets
     are divided into chunks, or tokens, which are the basic data units
     used as input or outputs of models.  For instance, if the data set
     consists in a corpus of texts, tokens are words, subwords or short
     sequences of words.










  Fressancourt, et al.      Expires 10 April 2025                [Page 25]


  Internet-Draft                 AI traffic                   October 2024


     The pre-processed data is used to train the reference model, _i.e._
     convert the knowledge to extract from the pre-processed data into a
     set of parameters including weights ruling how the model will answer
     future requests.  Parameters are variables adjusting the structure
     underlying the model that are set during the training phase.  Weights
     are a subste of the parameters, and determine how strong the ties are
     between some other parameters in the model.

     Once the model has been trained, it can be used to infer from a
     request, or, in other words, to serve an insight from a request's
     data.  Inference operations are not necessarily done by the same
     nodes that have trained the model.  Models can be transferred to
     other worker nodes to perfom inference tasks, sometimes placed close
     to the users making requests.  Besides, those transferred models can
     be re-trained or fine-tuned in order to better serve requests in the
     context they are done.  This can be done for instance with private or
     locally relevant data.

  A.2.  System model

     The machine learning systems built to meet the requirements of use
     cases presented in Section 2 are fundamentally distributed in order
     to meet scaling and time to answer requirements.  Indeed, if we
     consider modern lareg language models, they use from 125 million to
     175 billion parameters, where each parameters can be encoded with 4
     bytes to 4 bits depending on the precision required in the answer
     provided by the model.  From a memory perspective, this means that
     storing a model and its parameters will cost from 62,5 MB (for a
     model using 125 million parameters with 4-bits parameters) to 700 GB
     (for a model using 175 billion parameters with 4 bytes parameters).
     The memory footprint of modern large language models makes those
     models both difficult to manipulate on single nodes and to exchange
     data between nodes participating in a distributed task.

     In machine learning systems' distribution, nodes can play specific
     roles, depending on their (hardware) capabilities or their location
     in the network.

     Those roles, presented in Figure 5 are:

     *  *Data producer / Owner:*

        This entity is producing data from which users are willing to
        retrieve insights.  It can be a sensor in a network, a server
        producing logs and interacting with a set of users, etc.

     *  *Data manager:*




  Fressancourt, et al.      Expires 10 April 2025                [Page 26]


  Internet-Draft                 AI traffic                   October 2024


        This entity is in charge of managing the data produced by the data
        producer.  In that extend, it can pre-process, filter, add context
        (or metadata), store or transfer data to other entities in the
        system.  In its data management operations, the data manager has
        to take specific care of security and privacy aspects of data
        management.  In particular, it must make sure that data is stored
        securely, transferred to authorized entities and that user's
        consent and privacy preferences are respected.  To do so, it may
        use specialized encryption methods and protocols, meeting
        challenges presented in Section 4.5.

     *  *Worker:*

        This entity is in charge of processing data retrieved from the
        data manager in order to gain insights from it.  To do so, it can
        training a (potentially large) model during the model training
        phase; and personalize this model to better serve specific needs
        or requests from users during the inference phase.  In some
        specific cases, for instance, if the model from which the
        inference is done is small, the worker might be alone, but given
        the operational requirements of new approaches in machine
        learning, workers need to collaborate with other workers to
        complete a task together.

     *  *(Model) Aggregator:*

        In case the training, personalization or fine tuning of a model is
        done by several collaborating entities, the aggregator is
        responsible for gathering the results of the individual tasks
        performed by the workers and synthetize them in a general model.
        This model can be sent back to the workers for subsequent
        processing.

     *  *Inference engine / Server:*

        This entity is in charge of producing inferences, _i.e._ deriving
        insights from a request done against a trained model.  Those
        requests might be contextualized from a conversation, _i.e._ a
        sequence of requests and their answers.

     *  *Coordinator / Orchestrator:*

        This entity is in charge of orchestrating the execution of
        distributed tasks among the workers in charge.  This coordination
        role might be done by means of a centralized node (_e.g._ a
        parameter server), a federation of locally responsible nodes or in
        a coordinated fashion using a consensus protocol.




  Fressancourt, et al.      Expires 10 April 2025                [Page 27]


  Internet-Draft                 AI traffic                   October 2024


                    +----------------------------------+
                    |           Orchestrator           |
                    |                     +------------+  +------+   User
                    |                     | Aggregator |  |Server| <----
      +--------+    +---------------------+------------+  +------+   Req.
      |  Data  |        |         |||     ^  ^  ^  \  \     |||
      |Producer|        |         v||    /  /  /    \  \    ||v
      +--------+ \      v          v|   /  /  /   \  \  \   |v
        ...       v +-------+       v  /  /  /     \  \  v  v
      +--------+    |       |    +----+-+/  /       \  v+-+-------+
      |  Data  |    | Data  |--> |Worker+-+/         v+-+  Worker |
      |Producer|--> |Manager|--> +-+----+ +-+       +-+ +-------+-+
      +--------+    |       |-->   +-+----+ |       | +-------+-+
         ...      ^ +-------+        +------+       +---------+
      +--------+ /            :                :
      | Data   |              :                :
      |Producer|              : <--Training--> : <---  Inference --->
      +--------+              :                :  (with fine tuning)

                     Figure 5: Machine Learning syste model

     Those functional roles can be arranged, managed and executed in many
     different way in a ML system.  This arrangement determines how
     centralized and / or distributed a ML system is.

  A.3.  Parallelization modes

     In model-based machine learning, training and inference phases follow
     a pipeline similar to the pipeline presented in Figure 6.  As
     mentionned in previous sections, given the size of currently used
     models and the amount of data required to either train or infer from
     a model, machine learning workloads need to be distributed.  This
     distribution can follow different patterns (or a mix of those
     patterns): data parallelism, tensor parallelism, pipeline parallelism
     or mixture-of-expert parallelism.  Those patterns are presented in
     the following subsections.

           +------+
         +-+----+ |        +-+   +-+       +---+
       +-+----+ | |        |p|   |p|       | o |
     +-+----+ | | |       ^|a+   +a+       | u |
     |      | | | |      / |r|\ /|r|\      | t |
     |      | | | | ==> *  |a| * |a| * ==> | p |
     | Data | | | |      \ |m|/ \|m|/      | u |
     |      | | +-+       v|s+   +s+       | t |
     |      | +-+          |.|   |.|       |   |
     |      +-+            +-+   +-+       +---+
     +------+



  Fressancourt, et al.      Expires 10 April 2025                [Page 28]


  Internet-Draft                 AI traffic                   October 2024


                       Figure 6: Model-based AI pipeline

  A.3.1.  Data parallelism

     Data parallelism is the oldest among the parallelization patterns we
     highlighted.  It has been introduced in [DataParallelism].  As
     presented in Figure 7, data parallelism consists in the partitioning
     of the data used in a given machine learning task in a set of batches
     that are used to train or tune the model.  In data parallelism, a
     node manipulates a complete model and change its parameters using the
     data partition it has been allocated.  The result of the tasks
     performed in parallel by the workers are aggregated by an aggregator
     node at the end of the pipeline.  The model parameters resulting from
     this aggregation are transmitted back to the workers for future use
     of the model in order that each worker benefits from the work done by
     others in parallel.

                                       +-+   +-+       +---+
                       +-+----+        |p|   |p|       |i o|
                       |      |       ^|a+   +a+       |n u|
                       |      |      / |r|\ /|r|\      |t t|
           +------+    | Data | ==> *  |a| * |a| * ==> |e p|
         +-+----+ |    |  01  |      \ |m|/ \|m|/      |r u|    +---+
       +-+----+ | |  ^ |      |       v|s+   +s+       |i t| \  | o |
     +-+----+ | | | /  |      +        |.|   |.|       |m  |  v | u |
     |      | | | |    +------+        +-+   +-+       +---+    | t |
     |      | | | |                                             | p |
     | Data | | | |                    +-+   +-+       +---+    | u |
     |      | | +-+ \  +-+----+        |p|   |p|       |i o|  ^ | t |
     |      | +-+    v |      |       ^|a+   +a+       |n u| /  |   |
     |      +-+        |      |      / |r|\ /|r|\      |t t|    +---+
     +------+          | Data | ==> *  |a| * |a| * ==> |e p|
                       |  02  |      \ |m|/ \|m|/      |r u|
                       |      |       v|s+   +s+       |i t|
                       |      +        |.|   |.|       |m  |
                       +------+        +-+   +-+       +---+

                      Figure 7: Data-parallel AI pipeline

     In this pattern, the workers are only weakly synchronized, once the
     model has been aggregated and its parameters sent back.  In that
     extend, data parallelism can sustain up to seconds latency in the
     synchronization traffic.  Yet, as parameters are sent back by the
     aggregator to all the ndes for the entire model, the volume of data
     exchanged between training iterations of inference tasks can be quite
     large.  Besides, the aggregator is a focal point in the traffic
     between nodes, which can raise traffic management challenges.




  Fressancourt, et al.      Expires 10 April 2025                [Page 29]


  Internet-Draft                 AI traffic                   October 2024


  A.3.2.  Model parallelism

     For the last few years, models involved in machine learning tasks
     have increased in size, making their use by single nodes complex.  To
     allow the training and inference of larger models, some
     parallelization patterns have been designed to spilt models among
     several worker nodes: *pipeline parallelism* and *tensor
     parallelism*.

  A.3.2.1.  Pipeline parallelism

     Pipeline parallelism, described in [PipelineParallelism], takes
     advantage of the fact that models used in deep neural networks are
     structured in layers.  In pipeline parallelism, as shown in Figure 8,
     a model is separated into stages consisting in a few consecutive
     layers of the entire model.  Each stage is allocated to a separate
     worker node.  Each stage is executed using the intermediate results
     from the previous stage, and after each iteration, the parameters
     from adjacent stages are used to refine the model's stage held by the
     worker node.

           +------+
         +-+----+ |        +-+       +---+       +-+       +---+
       +-+----+ | |        |p|       |i o|       |p|       | o |
     +-+----+ | | |       ^|a+       |n u|       +a+       | u |
     |      | | | |      / |r|\      |t t|      /|r|\      | t |
     |      | | | | ==> *  |a| * ==> |e p| ==> * |a| * ==> | p |
     | Data | | | |      \ |m|/      |r u|      \|m|/      | u |
     |      | | +-+       v|s+       |i t|       +s+       | t |
     |      | +-+     :    |.|  :    |m  |    :  |.|    :  |   |
     |      +-+       :    +-+  :    +---+    :  +-+    :  +---+
     +------+         :         :             :         :
                      : Stage 1 :             : Stage 2 :

                 Figure 8: Pipeline model-parallel AI pipeline

     In this pattern, the communication volume between node can be less
     important than in data parallellism, as each node only communicates
     with nodes holding adjacent layers.  Yet, as stage iteration can be
     faster to execute, communications can be more frequent.











  Fressancourt, et al.      Expires 10 April 2025                [Page 30]


  Internet-Draft                 AI traffic                   October 2024


  A.3.2.2.  Tensor parallelism

     Tensor parallelism, stemming from the work presented in
     [TensorParallelism], is taking advantage of the fact that training
     and inference operations in model-based machine learning are using
     operations on matrixes that can be split according to several
     dimensions.  In comparison with pipeline parallellism, tensor
     parallelism is a pattern in which layers are split into chunks that
     can be operated by different nodes, as shown on Figure 9.  Tensor
     paralellism can be naively presented as a split in the model
     parameter space rather than along the layer / stage dimension.

                       +---+   +---+       +---+
                       |p p+   +p p+       |i o|
                       |a a|\ /|a a|\      |n u|
           +------+    |r r| * |r r| * ==> |t t|
         +-+----+ |    |a t|/ \|a t|/      |e p|    +---+
       +-+----+ | |  ^ |m 1+   +m 1+       |r .| \  | o |
     +-+----+ | | | /  +---+   +---+       +---+  v | u |
     |      | | | |      ^       ^                  | t |
     |      | | | |      |       |                  | p |
     | Data | | | |      v       v                  | u |
     |      | | +-+ \  +---+   +---+       +---+  ^ | t |
     |      | +-+    v |p p+   +p p+       |i o| /  |   |
     |      +-+        |a a|\ /|a a|\      |n u|    +---+
     +------+          |r r| * |r r| * ==> |t t|
                       |a t|/ \|a t|/      |e p|
                       |m 2+   +m 2+       |r .|
                       +---+   +---+       +---+

                  Figure 9: Tensor model-parallel AI pipeline

     In tensor parallellism, nodes holding chunks of the same model (or of
     the same stage) need to communicate during the execution of a model
     (or stage) iteration.  This involves even tighter latency
     requirements compared with pipeline parallelism for inter-node
     communications.

  A.3.3.  Mixture of Expert parallelism

     Mixture of experts (MoE) parallelism stems from an idea introduced in
     [MoEParallelism1] and adapted to deep neural networks in
     [MoEParallelism2].  In MoE parallellism, nodes are holding smaller
     but specialized models trained over a smaller amount of data and
     holding fewer parameters.  When a request or specific token is
     submitted to a MoE model, a gateway node, the router, takes a
     decision about which model instance to use against the specific token
     or request that has been submitted.  After the selected worker



  Fressancourt, et al.      Expires 10 April 2025                [Page 31]


  Internet-Draft                 AI traffic                   October 2024


     executed the task against the token or request, the result is given,
     and considered as the result given by the whole model.

                                  +---+   +---+
                                  |p s+   +p s+
           +------+               |a e|\ /|a e|\
         +-+----+ |               |r t| * |r t| *   +---+
       +-+----+ | |             ^ |a .|/ \|a .|/ \  | o |
     +-+----+ | | |            /  |m 1+   +m 1+   v | u |
     |      | | | |     +------+  +---+   +---+     | t |
     |      | | | | ==> |Router| ---------------    | p |
     | Data | | | |     +------+  +---+   +---+     | u |
     |      | | +-+               |p s+   +p s|     | t |
     |      | +-+                 |a e|\ /|a e|     |   |
     |      +-+                   |r t| * |r t|     +---+
     +------+                     |a .|/ \|a .|
                                  |m 2+   +m 2|
                                  +---+   +---+

               Figure 10: Mixture of Expert parallel AI pipeline

     In MoE parallellism, the router plays a major role, and is a
     communication focal point.

  A.4.  Collective communication methods

     In their development, distributed machine learning systems benefit
     from collective communication libraries such as NCCL ([NCCL]) or RCCL
     ([RCCL]) to abstract the setup and management of the connections used
     to exchange data between worker nodes involved in distributed
     training or inference tasks.  As presented in [xCCL] or in
     [I-D.yao-tsvwg-cco-problem-statement-and-usecases], those libraries
     introduce collective communication methodss, used in accordance with
     parallelization modes presented in Appendix A.3.

     To better explain what are the main collective communication methods,
     we consider a collective consisting in four nodes.  Those nodes
     exchange pieces of data in the frame of the execution of a
     distributed machine learning task.  The topology underlying the
     connections between those nodes is not detailed at this stage.  The
     major collective communication methods are:










  Fressancourt, et al.      Expires 10 April 2025                [Page 32]


  Internet-Draft                 AI traffic                   October 2024


  A.4.1.  Broadcast

     This collective communication method is very intuitive to understand
     for networking people, as its behaviour is the same as when a host
     broadcasts a packet in an IP-based network.  In the broadcast
     collective communication, method, a node (N.1) is sending the same
     information to all the nodes participating in the collective.  No
     data transformation is performed during this operation.

                                    +---+
                                    +N.1+
                                   /+---+\
                                  v   |   v
                              +---+   v   +---+
                              |N.2| +---+ |N.3|
                              +---+ |N.4| +---+
                                    +---+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | D.1 | | --- | | --- | | --- | ==> | D.1 | | D.1 | | D.1 | | D.1 |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+

                          Figure 11: Broadcast method

  A.4.2.  Reduce

     In this collective communication method, the data exchange is
     combined with an operation on the received data to produce an output.
     During a Reduce operation, the nodes involved in the collective send
     their data to an aggregator processing the received inputs _I.1_ ...
     _I.4_ using an operator _f_ to obtain _Out_.  _Out_ is then provided
     to one of the nodes in the collective (N.1).  Note that most of the
     time, the aggregation is done by one node in the collective, carrying
     the aggregator function.

















  Fressancourt, et al.      Expires 10 April 2025                [Page 33]


  Internet-Draft                 AI traffic                   October 2024


                           +---+ +---+ +---+ +---+
                           |N.1| |N.2| |N.3| |N.4|
                           +---+ +---+ +---+ +---+
                            | ^   |     |     |
                            | |   |     |     |
                            v |   v     v     v
                         +-------------------------+
                         |        Aggregator       |
                         | f(I.1,I.2,I.4,I.4)=Out  |
                         +-------------------------+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | I.1 | | I.2 | | I.3 | | I.4 | ==> | Out | | --- | | --- | | --- |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+

                            Figure 12: Reduce method

  A.4.3.  Scatter

     In this collective communication method, a node in the collective
     splits the data it has into equal size chunks, and distribute a
     different chunk to every single other node in the collective, in
     order to distribute the data evenly.

                                    +---+
                                    +N.1+
                                   /+---+\
                                  v   |   v
                              +---+   v   +---+
                              |N.2| +---+ |N.3|
                              +---+ |N.4| +---+
                                    +---+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | D.1 | | --- | | --- | | --- |     | D.1 | | --- | | --- | | --- |
     | D.2 | | --- | | --- | | --- | ==> | --- | | D.2 | | --- | | --- |
     | D.3 | | --- | | --- | | --- |     | --- | | --- | | D.3 | | --- |
     | D.4 | | --- | | --- | | --- |     | --- | | --- | | --- | | D.4 |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+

                           Figure 13: Scatter method










  Fressancourt, et al.      Expires 10 April 2025                [Page 34]


  Internet-Draft                 AI traffic                   October 2024


  A.4.4.  All-Gather

     This collective communication method can be seen as a simultaneous
     broadcast done by every node in the collective.  Indeed, in an All-
     gather operation, every node sends the data it has to every other
     nodes in the collective, so that in the end, every node has a copy of
     every piece of data held by the nodes in the collective before the
     operation.

                                    +---+
                                  ^ |N.1| ^
                                 /  +---+  \
                                v           v
                             +---+    ^    +---+
                             |N.2| <--|--> |N.3|
                             +---+    v    +---+
                                ^           ^
                                 \  +---+  /
                                  v |N.4| v
                                    +---+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | D.1 | | --- | | --- | | --- |     | D.1 | | D.1 | | D.1 | | D.1 |
     | --- | | D.2 | | --- | | --- | ==> | D.2 | | D.2 | | D.2 | | D.2 |
     | --- | | --- | | D.3 | | --- |     | D.3 | | D.3 | | D.3 | | D.3 |
     | --- | | --- | | --- | | D.4 |     | D.4 | | D.4 | | D.4 | | D.4 |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+

                          Figure 14: All-gather method

  A.4.5.  All-to-All

     This collective communication method can be seen as a simultaneous
     scatter operation done by every node in the collective.  Indeed, in
     an All-to-All operation, every node splits the data it holds in equal
     size chunks, and sends one chunk to every other node in the
     collective.  As a result, the data held by each node after the All-
     to-All operation is different, but every node holds roughly the same
     amount of data, even if the data volume was not balanced before the
     operation.











  Fressancourt, et al.      Expires 10 April 2025                [Page 35]


  Internet-Draft                 AI traffic                   October 2024


                                    +---+
                                  ^ |N.1| ^
                                 /  +---+  \
                                v           v
                             +---+    ^    +---+
                             |N.2| <--|--> |N.3|
                             +---+    v    +---+
                                ^           ^
                                 \  +---+  /
                                  v |N.4| v
                                    +---+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | A.1 | | B.1 | | C.1 | | D.1 |     | A.1 | | A.2 | | A.3 | | A.4 |
     | A.2 | | B.2 | | C.2 | | D.2 | ==> | B.1 | | B.2 | | B.3 | | B.4 |
     | A.3 | | B.3 | | C.3 | | D.3 |     | C.1 | | C.2 | | C.3 | | C.4 |
     | A.4 | | B.4 | | C.4 | | D.4 |     | D.1 | | D.2 | | D.3 | | D.4 |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+

                          Figure 15: All-to-All method

  A.4.6.  All-Reduce

     This collective communication method is very similar to the Reduce
     method.  During an All-Reduce operation, the nodes involved in the
     collective send their data to an aggregator processing the received
     inputs _I.1_ ... _I.4_ using an operator _f_ to obtain _Out_.  To the
     difference of the Reduce operation, _Out_ is sent back to every node
     in the collective.  In the implementation of this method, the
     aggregator function can be done by one node in the collective, or
     every single node applies the operator _f_ on the inputs received
     after an All-to-All operation.

                           +---+ +---+ +---+ +---+
                           |N.1| |N.2| |N.3| |N.4|
                           +---+ +---+ +---+ +---+
                            | ^   | ^   | ^   | ^
                            | |   | |   | |   | |
                            v |   v |   v |   v |
                         +-------------------------+
                         |        Aggregator       |
                         | f(I.1,I.2,I.3,I.4)=Out  |
                         +-------------------------+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | I.1 | | I.2 | | I.3 | | I.4 | ==> | Out | | Out | | Out | | Out |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+




  Fressancourt, et al.      Expires 10 April 2025                [Page 36]


  Internet-Draft                 AI traffic                   October 2024


                          Figure 16: All-Reduce method

  A.4.7.  Reduce-Scatter

     This collective communication method is a combination of the All-
     Reduce method with the Scatter method.  During a Reduce-Scatter
     operation, the nodes involved in the collective send their data to an
     aggregator processing the received inputs _I.1_ ... _I.4_ using an
     operator _f_ to obtain _Out_.  To the difference of the All-Reduce
     operation, _Out_ is split into equal size chunks _O.1_ ... _O.4_, and
     each chunk is sent to a different node in the collective.

                           +---+ +---+ +---+ +---+
                           |N.1| |N.2| |N.3| |N.4|
                           +---+ +---+ +---+ +---+
                            | ^   | ^   | ^   | ^
                            | |   | |   | |   | |
                            v |   v |   v |   v |
                   +---------------------------------------+
                   |              Aggregator               |
                   | f(I.1,I.2,I.3,I.4)=(O.1,O.2,O.3,O.4)  |
                   +---------------------------------------+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | I.1 | | I.2 | | I.3 | | I.4 | ==> | O.1 | | O.2 | | O.3 | | O.4 |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+

                        Figure 17: Reduce-Scatter method

  Authors' Addresses

     Antoine Fressancourt
     Huawei Technologies France S.A.S.U.
     18, Quai du Point du Jour
     92100 Boulogne-Billancourt
     France
     Email: antoine.fressancourt@huawei.com


     Luigi Iannone
     Huawei Technologies France S.A.S.U.
     18, Quai du Point du Jour
     92100 Boulogne-Billancourt
     France
     Email: luigi.iannone@huawei.com






  Fressancourt, et al.      Expires 10 April 2025                [Page 37]


  Internet-Draft                 AI traffic                   October 2024


     David Lou
     Huawei Technologies Duesseldorf GmbH
     Riesstrasse 25
     80992 Munich
     Germany
     Email: zhe.lou@huawei.com


     Dirk Trossen
     Huawei Technologies Duesseldorf GmbH
     Riesstrasse 25
     80992 Munich
     Germany
     Email: dirk.trossen@huawei.com





































  Fressancourt, et al.      Expires 10 April 2025                [Page 38]


name: draft-aft-ai-traffic
rev: 01
title: Handling inter-DC/Edge AI-related network traffic: Problem statement
time: 2025-10-13T07:09:01Z
group_uri: /api/v1/group/group/1027/
expires: 2025-10-13T17:48:27Z
states: ["/api/v1/doc/state/2/", "/api/v1/doc/state/150/"]
categories: []
tags: []
fetched_at: 2026-03-07T06:55:18.623757+00:00
source: ietf
abstract:
  The growth in terms of number of parameters of LLM models as well as
     the need to use or train those models with private or protected data
     will require service providers operating LLM-based services to
     cooperate to train, specialize or serve LLM-based services accross
     datacenters.  Given their structure, the number of parameters they
     incorporate and the collective communication librairies they are
     built with, LLM training and inference (or serving) network traffic
     has specific characteristics.

     In that regard, understanding the specificities of AI-related
     workloads is critical to determine how to operate AI-based services
     in a federated setting across datacenters.

full_text:
  Working Group name                                       A. Fressancourt
  Internet-Draft                                                L. Iannone
  Intended status: Standards Track                                  D. Lou
  Expires: 13 October 2025                                      D. Trossen
                                                                    Huawei
                                                             11 April 2025


    Handling inter-DC/Edge AI-related network traffic: Problem statement
                          draft-aft-ai-traffic-01

  Abstract

     The growth in terms of number of parameters of LLM models as well as
     the need to use or train those models with private or protected data
     will require service providers operating LLM-based services to
     cooperate to train, specialize or serve LLM-based services accross
     datacenters.  Given their structure, the number of parameters they
     incorporate and the collective communication librairies they are
     built with, LLM training and inference (or serving) network traffic
     has specific characteristics.

     In that regard, understanding the specificities of AI-related
     workloads is critical to determine how to operate AI-based services
     in a federated setting across datacenters.

  Status of This Memo

     This Internet-Draft is submitted in full conformance with the
     provisions of BCP 78 and BCP 79.

     Internet-Drafts are working documents of the Internet Engineering
     Task Force (IETF).  Note that other groups may also distribute
     working documents as Internet-Drafts.  The list of current Internet-
     Drafts is at https://datatracker.ietf.org/drafts/current/.

     Internet-Drafts are draft documents valid for a maximum of six months
     and may be updated, replaced, or obsoleted by other documents at any
     time.  It is inappropriate to use Internet-Drafts as reference
     material or to cite them other than as "work in progress."

     This Internet-Draft will expire on 13 October 2025.

  Copyright Notice

     Copyright (c) 2025 IETF Trust and the persons identified as the
     document authors.  All rights reserved.




  Fressancourt, et al.     Expires 13 October 2025                [Page 1]


  Internet-Draft                 AI traffic                     April 2025


     This document is subject to BCP 78 and the IETF Trust's Legal
     Provisions Relating to IETF Documents (https://trustee.ietf.org/
     license-info) in effect on the date of publication of this document.
     Please review these documents carefully, as they describe your rights
     and restrictions with respect to this document.  Code Components
     extracted from this document must include Revised BSD License text as
     described in Section 4.e of the Trust Legal Provisions and are
     provided without warranty as described in the Revised BSD License.

  Table of Contents

     1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.  Applicability of AI . . . . . . . . . . . . . . . . . . . . .   4
       2.1.  xGPT-like Services: an example centralized training use
             case  . . . . . . . . . . . . . . . . . . . . . . . . . .   4
       2.2.  Decentralized training with centralized inference use
             cases . . . . . . . . . . . . . . . . . . . . . . . . . .   5
       2.3.  Federated building management: a decentralized inference
             use case  . . . . . . . . . . . . . . . . . . . . . . . .   6
       2.4.  Key Takeaways . . . . . . . . . . . . . . . . . . . . . .   6
     3.  How ML systems are built  . . . . . . . . . . . . . . . . . .   7
       3.1.  Lifecycle of a typical large language model . . . . . . .   7
       3.2.  The distributed nature of Machine Learning systems  . . .   8
       3.3.  The topology of distributed machine learning systems  . .  10
       3.4.  Deployment considerations for AI systems  . . . . . . . .  12
     4.  Challenges (in Networking for AI) . . . . . . . . . . . . . .  14
       4.1.  Network resource management at regional scale . . . . . .  14
       4.2.  Latency sensitivity of LLM training and inference . . . .  15
       4.3.  Aligning communications patterns to the Internet's
             constraints . . . . . . . . . . . . . . . . . . . . . . .  16
       4.4.  Managing incast traffic related to AI inference . . . . .  17
       4.5.  Securing and attesting AI-related traffic . . . . . . . .  18
     5.  Problem statement . . . . . . . . . . . . . . . . . . . . . .  19
     6.  Next Steps  . . . . . . . . . . . . . . . . . . . . . . . . .  19
     7.  Security Considerations . . . . . . . . . . . . . . . . . . .  20
     8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  20
     9.  Informative References  . . . . . . . . . . . . . . . . . . .  20
     Appendix A.  A primer on Machine learning (extended version of
             Section 3)  . . . . . . . . . . . . . . . . . . . . . . .  24
       A.1.  Machine learning model lifecycle  . . . . . . . . . . . .  25
       A.2.  System model  . . . . . . . . . . . . . . . . . . . . . .  26
       A.3.  Parallelization modes . . . . . . . . . . . . . . . . . .  28
       A.4.  Collective communication methods  . . . . . . . . . . . .  32
     Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  37







  Fressancourt, et al.     Expires 13 October 2025                [Page 2]


  Internet-Draft                 AI traffic                     April 2025


  1.  Introduction

     AI has attracted a far bit of attention in the networking community,
     including the IETF, in regards to not just applications but also
     needed protocols and technologies for the realization of those
     applications.

     While AI is a large area, this document focuses on the method of
     (training and inferencing over) Large Language Models (LLM).  The
     starting point being the distributed nature of implementing LLMs,
     both for training and inferencing.  For this, a network of so-called
     'workers' is being realized, that over time will train a model, over
     which in turn inferencing can be performed.  This distributed nature
     involves a number of communication patterns to exchange suitable
     information, those patterns needing realization in suitable
     protocols.  Differences in those protocols emerge from the deployment
     choices for AI platforms and the challenges that arise from those
     deployments.

     The training of LLMs in ever growing large-scale data centres (DCs)
     is a current method to push the boundaries of key performance
     indicators, most notably the number of parameters included in the
     training model [LLMSize].  Observations in recent works [AIBackbone],
     however, point to the fact that distribution across more than one DC
     will quickly be necessary to continue the growth of LLM training.

     LLMs may also start inherently distributed in deployment itself, not
     because of their size, but because of their use case.  An example
     here is content moderation in fediverse environments [FEDI], where
     decentralized data governance and computation drives the equally
     decentralized realization of the workers developing a shared model
     over time.  Other examples of such decentralized use case stems from
     health (e.g., where independent health trust may train a shared model
     over strictly locally governed patient data), IoT (where AI-derived
     features may be derived from localized sensor data), or also network
     management (where operator data for training may not be possible or
     legally permitted to be disclosed to a centralized training entity).
     Realizations of platforms for those use cases often refer to
     'federated learning' [FLOWER] to capture the equal basis of
     participation, both at the data and processing level, of the
     participating entities of the platform.

     The intention of this document is to highlight the possible
     challenges in the spectrum of realizations for the distributed LLM
     training and inferencing.  For this, we provide more details and
     examples for use cases in Section 2, followed by a primer of key AI/
     LLM techniques in Section 3.  Then, a number of challenges to, e.g.,
     resource management, latency, and security, are identified in



  Fressancourt, et al.     Expires 13 October 2025                [Page 3]


  Internet-Draft                 AI traffic                     April 2025


     Section 4 as a starting point for a focussed discussion in the IETF
     community on challenges AI may pose for networks, networks protocols
     and technologies.

     As the spectrum of realization ranges from centralized intra-DC to
     highly distributed, federated AI platforms, there is a strong
     relevance of solving the identified challenges within the IETF,
     leading to the formulation of a problem statement in Section 5 along
     those lines and proposing next steps in Section 6 for the IETF
     community to pursue.

  2.  Applicability of AI

     This section introduces the applicability of AI/LLMs across a
     spectrum of decentralization for realising machine learning training
     and inferencing tasks.  These two tasks are the most intensive in AI
     workflows, and they are inherently distributed compute problems given
     the amount of memory and processing power they require Section 3 will
     introduce the prominent techniques relevant to implementing those
     tasks in deployed systems.

     It can be observed upfront that the realization of AI/LLM use cases
     is driven by two main aspects, namely the decentralization of data
     and that of compute; during the training phase, the inference phase
     or both.  The proposed examples are introduced with regards to those
     two aspects, for reasons that are detailed in the text.

  2.1.  xGPT-like Services: an example centralized training use case

     Various GPT-like services have gained much attention recently,
     advocating the training of ultra-large language models, providing
     chat-like inferencing services to users for, _e.g._, text generation,
     generative AI-based image creation, and many others.

     The key success factor of those services is the ingestion of vast
     amounts of data into a centralized training entity.  This data may
     come from public sources, search input, through license arrangements
     with, _e.g._, video providers like YouTube, and many others.

     In this use case, centralization pertains to the ownership of the
     training resources under a single entity.  Until now, the prevalent
     deployment of such centralized training is within a single, large-
     scale DC with a growing number of GPUs to execute the necessary model
     training operations over a sustained period of time.  However, the
     growing need for more compute, together with physical and energy
     limitations in existing DCs, make a growing scale-out of those
     services beyond the reach of a single DC a likely needed path in the
     future.



  Fressancourt, et al.     Expires 13 October 2025                [Page 4]


  Internet-Draft                 AI traffic                     April 2025


  2.2.  Decentralized training with centralized inference use cases

     In some cases, it is impossible to gather the amount of data
     necessary to properly train a machine learning model, either for
     security, privacy or regulatory reasons.  The necessity to tackle
     those cases triggered the development of Federated learning
     [FederatedLearning], in which several entities can collaborate in a
     decentralized way to the training of a single large model.

     Health is a traditional area for reasoning-based systems.  Its
     richness in data, both historical as well as current (_e.g._, from
     patients), lends itself for training LLMs over which inferencing can
     take place for, _e.g._, diagnosis, drug development, developing
     treatment plans and many others.

     Individual health organizations are often well equipped in terms of
     compute resources, albeit not at the scale that suffices to perform
     centralized LLM training on their own.  Thus, federating compute
     resources is useful for scale and incentivized through the sharing of
     the resulting LLM in the clinical pathway.

     Furthermore, data is also localized, since patients sign up to local
     health organizations, practises, or health trusts, which in turn
     manage their data.  In many countries, data sharing is performed,
     also for transferability (_e.g._, when patients change location and
     thus local health contacts) but also treatment across health
     organization.  Overall, data governance needs to be strictly adhered
     to for any application of the data, with a possible development of a
     LLM being just one such application.

     Network management is another use case for federated learning, where
     the federation is driven by the common goal to develop an
     increasingly refined model for, _e.g._, intrusion detection, network
     failure detection, and others, but where suitable training data is
     only shared in a limited manner or not at all for confidentiality
     reasons.

     In those use cases, the decentralization of the training stems from
     the constraints limiting the exchaneg of data between entities for
     technical feasibility, privacy, confidentiality or regulatory
     reasons.  Once trained, the models including the common knowledge
     gathered in the training phase can be used by each single entity
     without necessarily requiring collaboration during the inference
     phase.







  Fressancourt, et al.     Expires 13 October 2025                [Page 5]


  Internet-Draft                 AI traffic                     April 2025


  2.3.  Federated building management: a decentralized inference use case

     Building management, where smart buildings are often equipped with
     micro-DC capabilities, can be federated to improve energy management
     in a larger geographic area, utilizing local yet private data to
     train a suitable (possibly locally limited) LLM.  Works like
     [AIConst] expand this use case into other areas like construction
     management, architectural design, material design, and others.

     Similar to the previous use case, the deployment across shared
     infrastructures (as well as the sharing of compute resources itself)
     is a key aspect, utilizing Internet technologies for realizing the
     necessary exchange within the training and inferencing scenario at
     hand.  Here, the decentralization of the inference task is a
     necessity given that the goal is to reach a global optimum (_e.g._,
     energy savings accross an aea or region) by clustering the
     capabilities of buildings while keeping the data used for training
     and inferencing local for security and privacy reasons.

  2.4.  Key Takeaways

     The following key takeaways can be derived from the proposed
     applicability examples:

     *  _LLM AI is inherently distributed_: This is due to its scale in
        required training nodes and model size.  Although the distribution
        may be handled entirely within a central, single data centre, the
        execution of tasks across, often many thousands, of workers still
        remains a key aspect in LLM AI.

     *  _Centralized LLM AI training does not need to touch the Internet_:
        This is because it may be performed entirely within the limits of
        a single DC, or at least among a few DCs interconnected by private
        lines under the control of a single entity.

     *  _Centralization of compute implies centralization of data_: This
        is a consequence of the model creation, based on the data, which
        is centralized and thus requires suitable data transfer towards
        the compute nodes.

     *  _Federation allows for decentralization of data_: This is possible
        with worker nodes that may individually train on data, while
        contributing to the global model.  However, centralization of the
        federation again leads to the observation in the second item,
        _i.e._, data is centralized, too.






  Fressancourt, et al.     Expires 13 October 2025                [Page 6]


  Internet-Draft                 AI traffic                     April 2025


     *  _Inferencing inherently touches the Internet_: This is especially
        true in any applicability scenario involving end users residing on
        the Internet.  The impact may the creation of very large amounts
        of traffic to (_e.g._, centralized) nodes that hold the suitable
        model information for performing the desired inferencing.

     The next section outlines the system aspects of LLM AI, in light of
     the above takeaways, _e.g._, on centralization, while Section 4 more
     directly takes these takeaways into account when formulating key
     challenges for networking, stemming from the insights in this
     section.

  3.  How ML systems are built

  3.1.  Lifecycle of a typical large language model

     In the last few years, the impressive AI boom has been associated
     with the development of Large Language Models (LLMs).  LLMs are
     models, or representations of systems that are observed or measured
     with data.  Models have an underlying structure, consisting in a
     parametrized graph linking together small operators performing
     operations on *tensors*, or small arrays of finite dimensions.
     Models are built, or *trained*, using a reference data set (or
     training set), consisting in data labelled with its underlying
     structure or meaning.

     Before its use for training, the data is collected and pre-processed
     to structure it and chunk it into *tokens*, the basic data units used
     as input and outputs of models.  After the training data set has been
     prepared, the model is trained.  During this training phase, the
     model is parametrized, i.e., the parameters ruling the strength of
     the relationships between the tensor operators constituting the model
     are modified so the model is able to abstract the behaviour of the
     system from the dataset representing it.

     After the training phase, the model can be used during an *inference*
     phase to derive information or make predictions from new data
     presented in the form of a sequence of tokens.  Inference operations
     are not necessarily done by the same nodes that have trained the
     model.  Models can be transferred to other worker nodes to perfom
     inference tasks, sometimes placed close to the users making requests.
     Besides, those transferred models can be re-trained or fine-tuned in
     order to better serve requests in the context they are done.  This
     can be done for instance with private or locally relevant data.







  Fressancourt, et al.     Expires 13 October 2025                [Page 7]


  Internet-Draft                 AI traffic                     April 2025


  3.2.  The distributed nature of Machine Learning systems

     To improve the accuracy of LLMs, those models incorporate an
     increasing amount of parameters, and are trained using ever larger
     datasets.  Modern LLMs use 125 million up to 405 billion parameters,
     where each parameters can be encoded with 4 bits to 4 bytes depending
     on the required precision.  This increase in models' sizes has
     important consequences on the power consumption and on the memory
     footprint of the systems used to train those models.

     From a power consumption perspective, the increase in computing power
     needed to accomplish the training tasks of large models requires the
     deployment of more powerful Tensor Processing Units (TPUs) of
     Graphics Processing Units (GPUs) with higher power and cooling
     demands.  It is becoming physically unsustainable to bring the power
     required to datacenters hosting those machine learning worker nodes.
     A possible way to address this challenge is to distribute machine
     learning workloads beyond the realm of a single datacenter, possibly
     between two or a few interconnected datacenters separated by long to
     mid range private interconnections.

     From a memory perspective, this means that storing a model and its
     parameters will cost from 62,5 MB (for a model using 125 million
     parameters with 4-bits parameters) to 1620 GB (for a model using 405
     billion parameters with 4 bytes parameters).  The memory footprint of
     modern large language models makes those models difficult to
     manipulate on a single node.  Besides, as mentionned in
     [SchedulingSharding], the pre-training of Llama3 with 405 billion
     parameters required from 8192 to 16384 GPUs, which were obviously
     hosted on multiple, connected nodes.

     To cope with the memory and computing needs of workloads associated
     with LLM operations, the data preparation, training and inference
     tasks mentionned in Section 3.1 can be distributed among nodes with
     specific functionnal roles (task execution, result aggregation,
     orchestration...).  The detailed description of those functional
     roles is given in Appendix A.2.

     From a networking perspective, some elements are important to
     mention:

     *  Some roles (aggregator or orchestrator) tend to concentrate
        network traffic, thus creating some issues in properly managing
        the incast traffic.







  Fressancourt, et al.     Expires 13 October 2025                [Page 8]


  Internet-Draft                 AI traffic                     April 2025


     *  The parallel execution of training and inference tasks by worker
        nodes is following specific parallelism modes presented in
        Figure 1 and detailed in Appendix A.3.  Those modes have various
        requirements in terms of volume of data exchanged and latency (see
        [SchedulingSharding] for instance).

        a. *Data parallelism*, in which data is split into chunks to be
        used by different worker nodes to train the model.  This
        parallelism mode can cope with rather large latencies (several
        100s of ms) but requires exchanging a large volume of data.

        b. *Pipeline model-parallelism*, in which a model is separated
        into stages consisting in a few consecutive layers of the entire
        model.  Each stage is allocated to a separate worker node, and
        intermediate states are exchanged between successive worker nodes.
        This parallelism mode can cope with large latencies (10s of ms)
        and requires exchanging less data than data parallelism.

        c. *Tensor model-parallelism*, in which model layers are split
        into chunks that can be operated by different nodes.  This
        parallelism mode needs to operate in low latency networks (10s of
        us) and requires exchanging a lot of data.

        d. *Mixture of Expert parallelism*, in which nodes are holding
        smaller but specialized models trained over a smaller amount of
        data and holding fewer parameters.  This parallelism mode can cope
        with latencies in the ms range.

     *  Machine learning applications rely most of the time on collective
        communication libraries [xCCL] that use patterns presented in
        details in Appendix A.4 such as All-to-All or All-reduce.
        [I-D.yao-tsvwg-cco-problem-statement-and-usecases] has already
        introduced some networking challenges associated with the use of
        collective communication in machine learning systems.  From a
        networking perspective, those communication patterns translate in
        "on-off" communication patterns, with a few large flows starting
        simultaneously to allow nodes involved in a collective to exchange
        data and parameters.  The volume, synchronism and imbalance of the
        traffic generated by those communication pattern is a burden to
        the distribution of machine learning workloads, and a challenge to
        be addressed by the networking community.










  Fressancourt, et al.     Expires 13 October 2025                [Page 9]


  Internet-Draft                 AI traffic                     April 2025


     +----+     +-+ +-+
     |    |     |P| |P|     +-+
     |    |     |.| |.|     |O|
     |Data| ==> |1|X|2| ==> |u|
     | 1. |     |-| |-|     |t|           +----+
     |    |     | | | |     +-+         +-+--+ |     +-+     +-+
     |    |     +-+ +-+                 |    | |     |P| +-+ |P|     +-+
     +----+                             |    | |     |.| |I| |.|     |O|
                                        |Data| | ==> |1|X|n|X|2| ==> |u|
     +----+     +-+ +-+                 |    | |     |-| |t| |-|     |t|
     |    |     |P| |P|     +-+         |    | |     | | +-+ | |     +-+
     |    |     |.| |.|     |O|         |    +-+     +-+     +-+
     |Data| ==> |1|X|2| ==> |u|         +----+
     | 2. |     |-| |-|     |t|
     |    |     | | | |     +-+
     |    |     +-+ +-+
     +----+
     (a) Data parallelism               (b) Pipeline model-parallelism

                  +-+ +-+                                +-+ +-+
                  |P| |P|     +-+                        |P| |P|
                  |.| |.|     |O|                        |.| |.|
       +----+     |1|X|2| ==> |-|         +----+       ^ |1|X|2|\
     +-+--+ |   ^ |-| |-|     |1|       +-+--+ |      /  |-| |-| \
     |    | |  /  |1| |1|     +-+ +-+   |    | |     /   |1| |1|  v  +-+
     |    | | /   +-+ +-+       \ |O|   |    | |   +-+   +-+ +-+     |O|
     |Data| |      |   |          |u|   |Data| |==>|R|  ----------   |u|
     |    | | \   +-+ +-+       / |t|   |    | |   +-+   +-+ +-+     |t|
     |    | |  \  |P| |P|     +-+ +-+   |    | |         |P| |P|     +-+
     |    +-+   v |.| |.|     |O|       |    +-+         |.| |.|
     +----+       |1|X|2| ==> |-|       +----+           |1|X|2|
                  |-| |-|     |2|                        |-| |-|
                  |2| |2|     +-+                        |2| |2|
                  +-+ +-+                                +-+ +-+
     (c) Tensor model-parallelism       (d) Mixture of Expert parallelism

         Figure 1: Parallelism models used in machine learning systems

  3.3.  The topology of distributed machine learning systems

     To boost the workload distribution efficiency, nodes participating in
     the common execution of machine learning workloads are interconnected
     following specific topologies.  Depending on the number of nodes
     involved, the nodes' capacities to connect directly with other nodes
     and the control of the machine learning application administrator
     over the physical connectivity substrate, difefrent topologies can be
     used.  Those topologies can inherit from the work done either in the
     high performance computing community if the nodes are homogeneous,



  Fressancourt, et al.     Expires 13 October 2025               [Page 10]


  Internet-Draft                 AI traffic                     April 2025


     high capacity computers under the control of a single administrator
     or in the peer-to-peer or swarm computing community if the nodes are
     more diverse and numerous.

     Topologies inspired from the HPC networking community are typically
     found in distributed systems in which all the nodes are located in
     the same datacenter.  In specific cases, those topologies can be
     extended beyond a single datacenter.  Then, the distribution of the
     workloads is done so as to avoid frequent and latency-bound
     communications over those links, and congestion control mechanisms
     are tuned to avoid deadlocks and bloats in network equipments manging
     the inter-datacenter link.  Among those HPC-inspired topologies, we
     can find the n-dimension torus, the Dragonfly topology,the fat tree
     topology and the hypercube topology.

     On the other end of the spectrum, topologies inspired from swarm
     computing and peer-to-peer networks have emerged as solutions
     connecting heteorgeneous nodes involved in distributed machine
     learning tasks outside datacenters, at the edge of the networks or
     among clusters of nodes hosted on public cloud platforms.  Those
     topologies can be built either from dynamic wireless connections,
     which can be rearranged easily, or as overlay links on top of an
     immutable physical connectivity substrate.  Among those swarm-
     inspired topologies presented in Figure 2, we can find the full mesh,
     the star topology, the Peer-to-Peer topology which presents a limited
     diameter without too many connections, the random topology and the
     clustered topology.  Note that in the case of the clustered topology,
     nodes can be gathered based on their proximity to a proxy or an
     aggregator, or based on their interest or capabilities.  Besides, the
     clustered topology can be structured according to a hierarchy.

        o-----o     o     o     o-----o        o     o      o o o   o o o
       / \   / \     \   /     / ____/ \       |    /        o o   / o o
      /   \ /   \     \ /     / /       \      o---o          o \ o
     o-----+-----o  o--o--o  o-+       +-o      \   \   o        o o
      \   / \   /     / \     \   ____/ /        o---o--+       / o \ o
       \ /   \ /     /   \     \ /     /    o   /          o o o     o o
        o-----o     o     o     o-----o     +--o            o o       o

     Full mesh      Star     Peer-to-Peer   Random         Clustered
     topology       topology topology       topology       topology

                       Figure 2: Decentralized topologies

     The topology connecting the nodes together is not anecdotical.  If it
     is under the control of the machine learning system administrator,
     the construction of a proper topology able to efficiently support the
     communication patterns generated by the parallelization model and the



  Fressancourt, et al.     Expires 13 October 2025               [Page 11]


  Internet-Draft                 AI traffic                     April 2025


     collective communication method used is a key performance factor.  If
     the topology is inherited from the environment, machine learning
     system administrators will need to adapt the communication patterns
     they use to the characteirstics of the topology.

  3.4.  Deployment considerations for AI systems

     Given the computing and memory requirements of machine learning
     workloads, modern machine learning systems are fundamentally
     distributed.  By combining functionnal roles together, distributing
     them using a combination of parallelization modes, and communicating
     following patterns, a variety of systems with different shades of
     decentralization can be built.  Indeed, distributing workloads across
     several workers does not necessarily means that the control and
     orchestration of those workloads is distributed.  In case a parameter
     server is used, the orchestrator role is played by a single node.

     Yet, following the Federated learning [FederatedLearning] approach,
     machine learning systems can be decentralized as shown in Figure 3.
































  Fressancourt, et al.     Expires 13 October 2025               [Page 12]


  Internet-Draft                 AI traffic                     April 2025


              +----+   +----+                     +----+   +----+
              | D  |   |  D |                     | DM |   | DM |
              +--+-+   +-+--+                     +--+-+   +-+--+
                  \     /                             \     /
                 +-+---+-+                           +-+---+-+
                 |Central|                           |Central|
                 | Inst. |                           | serv. |
                 | DM (Σ)|                           |  M (Σ)|
                 +-+---+-+                           +-+---+-+
                  /     \                             /     \
              +--+-+   +-+--+                     +--+-+   +-+--+
              | D  |   |  D |                     | DM |   | DM |
              +----+   +----+                     +----+   +----+

            Centralized learning          Centralized federated learning

               +-----------+                       +-----------+
               |   DM (Σ)  |                       |   DM (Σ)  |
               +-+---+---+-+                       +-+---+---+-+
                /    |    \                         /    |    \
     +---------+-+   |   +-+---------+   +---------+-+   |   +-+---------+
     |     DM    +---*---+     DM    |   |   DM (Σ)  +---*---+   DM (Σ)  |
     +--------+--+  / \  +--+--------+   +--------+--+  / \  +--+--------+
               \   /   \   /                       \   /   \   /
        +-------+-+-+ +-+-+-------+         +-------+-+-+ +-+-+-------+
        |     DM    +-+     DM    |         |   DM (Σ)  +-+   DM (Σ)  |
        +-----------+ +-----------+         +-----------+ +-----------+

             Semi-decentralized                Fully decentralized
             federated learning                federated learning

        Figure 3: Different centralization models in federated learning

     In federated learning, the coordination role of the orchestrator or
     the aggregation of parameters done by the aggregator can be done by a
     subset or all the worker nodes deployed in the system, using a
     distributed protocol to exchange information, synchronize parameters
     and agree on a reference.  Of course, this comes at a communication
     cost, but decentralization also has benefits.












  Fressancourt, et al.     Expires 13 October 2025               [Page 13]


  Internet-Draft                 AI traffic                     April 2025


     One of the first reason for decentralizing machine learning systems
     is to allow users to retain ownership of their data, and to control
     how their data is transferred or used.  In a centralized setting,
     users need to trust the central entity managing the system for
     respecting the data management agreed upon with the user.  In a
     decentralized setting, users can either infer from a copy of a
     generic model sent by a central entity (centralized federated
     learning), fine-tune the model and infer from it (semi-decentralized
     setting) or even cooperate with others to train together a model and
     use it afterwards (in a fully decentralized setting).

     Besides, in a decentralized machine learning system, users and worker
     nodes can be co-located, or workers can be placed close to the user
     in order to reduce the overall communication latency.  In such a
     decentralization scheme, if it is considered that edge nodes can
     benefit from less memory or computing power, a balance has to be
     found between the time spent in communicating between the user and
     worker nodes and the processing time to reach an optimal response
     time for user requests.  Worker nodes located at the edge of the
     network may also collaborate with other, more capable nodes located
     in other locations.  The tail latency of the flows associated with
     those tasks should be bounded to avoid degrading the user's
     experience.

  4.  Challenges (in Networking for AI)

  4.1.  Network resource management at regional scale

     In (large) model-based machine learning, training, fine-tuning and
     infering from the model are workloads that involve the transfer of a
     very large volume of data.  In [AITrainingTraffic], the authors
     estimate that the first iteration for training a GPT-3B model among
     32 GPU nodes generate roughly 85 GB of tensor-parallel data traffic,
     1 GB of pipeline-parallel data traffic and 741 MB of data-parallel
     data traffic.  The tensor-parallel and pipeline-parallel data traffic
     consist in 125 MB messages that are periodically exchanged during
     communication phases, while very little traffic is exchanged during
     computing phases.

     This traffic pattern is a good example of the characteristics of
     network traffic flows associated with machine learning workloads.
     Indeed, the network traffic generated by distributed machine learning
     systems consists in a relatively little number of large (elephant)
     flows, starting and stopping roughly simultaneously.  Such
     synchronous traffic patterns are coming from the use of collective
     communication libraries such as NCCL and associated collective
     communication primitive such as All-Reduce or All-Gather, as
     mentionned in [Burstiness].  Dealing with this synchronized traffic



  Fressancourt, et al.     Expires 13 October 2025               [Page 14]


  Internet-Draft                 AI traffic                     April 2025


     in a stochastic network is challenging, because it generates periodic
     traffic bursts that bloat network equipment buffers.  Besides, the
     network's capacity needs to meet peak requirements at the cost of an
     unefficient average utilization of the installed capacity.  Even if
     the network's capacity is sufficient in theory to accomodate the peak
     requirements of machine learning systems, the transport mechanisms
     used on the links between the nodes make it difficult to immediately
     use the full deployed capacity, resulting in inefficiencies at the
     network level [NetworkBottleneck] At last, in such a synchronous
     communication pattern, the failure of a link, delaying data
     transmission between two nodes in the system might delay the whole
     system after a few iterations.

     Mitigating the resource management challenges raised by machine
     learning traffic patterns is currently an open research area.  At the
     application level, machine learning system designers work to develop
     hybrid parallellization schemes combining the patterns presented in
     Section 3.2 (detailed in Appendix A.3) and orchestration methods
     aiming at better utilizing the deployed network capacity and at
     avoiding "on-off" behaviors.

     At the network level, the main challenge that the research community
     is trying to address is the proper balancing of the flows generated
     by machine learning workloads in the network.  Indeed, to address the
     need for bandwidth and avoid bottlenecks, machine learning system
     designers are often deploying their system on top of networks whose
     topology presents a large number of equal cost links between two
     nodes.  Yet, as mentionned earlier, machine learning traffic is
     constituted with a rather small number of large flows that have a
     very small entropy.  This makes applying classic load balancing
     techniques challenging.  To address this load balancing issue, most
     collective communication libraries use packet spraying strategies,
     which require specific tuning due to mentioned lack of entropy.  Yet,
     some researchers are questionning the relevance of this approach
     [ChallengingSpraying].

  4.2.  Latency sensitivity of LLM training and inference

     Training a (large) model in a distributed setting does not
     necessarily require data transfer to be operated at a controlled low
     latency, but large tail latencies related to packet losses or head of
     line blocking can delay the training of models considerably.  Indeed,
     problems arising from packet losses, link failures or excessive
     buffering can have cascading consequences since in most
     parallellization methods, data and model parameters are exchanged
     following synchronized patterns (see Section 4.1).





  Fressancourt, et al.     Expires 13 October 2025               [Page 15]


  Internet-Draft                 AI traffic                     April 2025


     The extend of the amplification of the latency effects of a soft
     failure on a connection between two nodes depends on the topology of
     the network connecting the nodes.  Besides, routing inefficiencies or
     failures to properly balance the load on some heavily-used links can
     also generate additional latencies.  Thus, the topology of the
     network supporting machine learning workloads needs specific care.

     In a large scale and decentralized AI context, heterogeneous links
     can be used between nodes participating in a decentralized machine
     learning system.  The specificities of the links need to be taken
     into account when orchestrating or distributing AI-related tasks.  As
     latency is affected by congestion, addressing the mismatch between
     links' bandwidth-delay products for efficient congestion management
     is an open challenge.  In the research community, some projects have
     proposed to introduce proxies to investigate new control loops taking
     into account link segments characteristics [SiteToSite].  In the
     IETF, CSIG draft (now expired) presented a model to expose a rich
     congestion signal [I-D.ravi-ippm-csig].

  4.3.  Aligning communications patterns to the Internet's constraints

     In the development and deployment of distributed machine learning
     systems within the real of an administrator's responsibility, it is
     feasible, and advised, to design the system to adapt the network's
     topology to the paralellization method used and to the collective
     communication patterns required to perform the training or inference
     task efficiently.  As we have seen in Section 3, several network
     topologies can be adopted, depending on the model's architecture and
     on the design choice made while designing the training system.

     In a decentralized or federated setting, such a co-design offers
     lmess freedom, as some adverserial choices can be made by people
     willing to cooperate on specific machine learning tasks.  Besides,
     when the nodes involved are deployed at the edge of the network, in a
     majority of the cases, the topology is following the access network's
     specificities rather than adapting to the requirements of machine
     learning tasks.

     As machine learning-related traffic is growing on the Internet, some
     challenges associated with the delivery of network flows related to
     the realization of decentralized training or inference tasks are
     appearing: How to inform a machine learning application about the
     topology of the network interconnecting involved nodes?  How to adapt
     the parallelization model or collective communication pattern to
     maximize efficiency?  In the IETF, the ALTO working group [ALTO] has
     investigated similar challenges related to the operation of peer-to-
     peer traffic.  As machine learning workloads have a different set of
     requiremenst, it may be time to revisit this work.



  Fressancourt, et al.     Expires 13 October 2025               [Page 16]


  Internet-Draft                 AI traffic                     April 2025


  4.4.  Managing incast traffic related to AI inference

     In a machine learning system, some specific nodes, such as the
     orchestrator of the aggregator, play a central role, and thus
     concentrate communications.  This concentration might be reinforced
     by the use of specific collective communication primitives, and by
     the synchronicity of traffic patterns.  The network traffic
     management challenges we previously mentionned (Packet losses,
     delays, congestion) can be amplified by this concentration on a few
     hotspots in the network.

     Decentralized systems tend to mitigate this functionnal centrality by
     distributing the responsibility for fullfilling those roles accross
     several nodes.  Yet, even in decentralized systems, it is difficult
     to completely avoid incast problems given the specificities of
     machine learning workloads.  For instance, due to the characteristics
     of model inference, first prompt requests addressed to a given node
     are bound to generate a large incast traffic related to the
     personalization and fine tuning of the model instance run by the
     node, which can represent a large amount of data.

     The research and operational communities dealing with scaling out
     machine learning systems are working on some solutions to address
     incast traffic management issues.  For instance, in the same way as
     media codecs structured as layers of increasing resolution, models
     used in training and inference can be layered, with coarse grained
     models using less parameters than finer grained ones.  Coarse grain
     models can be distributed more quickly, drafting a first answer to a
     request while finer grain models are retrieved and then used to build
     a more precise answer to a request.  Besides, as inference requests
     towards large models are often done in the form of a converstaion or
     contextualized exchange between a user and the node running inference
     tasks addressing its requests, the use of service routing to
     consistently route requests to the same instance can be used.
     In that extend, the work done in the CATS working group in the IETF
     is of particular relevance ([CATSWG]).  Specific work on the proper
     metrics to be applied to CATS in the context of large scale
     decentralized AI will need to be done, taking into consideration the
     large body of work on load balancing for machien learning workloads
     done in the research community.  In particular, one problem of
     importance in the selection of the instance serving a specific
     inference request is the trade-off to find between the need to serve
     the requests with a bounded latency, requiring to use a node that is
     located as close to the requesting user as possible, the necessity to
     bootstrap this node with the user's contextual model parameters and
     the uncertainty about the length of the session between the user and
     the instance, which puts a stress on the memory needed to keep the
     session's context at hand.



  Fressancourt, et al.     Expires 13 October 2025               [Page 17]


  Internet-Draft                 AI traffic                     April 2025


  4.5.  Securing and attesting AI-related traffic

     The distribution of machine learning workloads at a regional scale
     beyond the limits of a single datacenter and the decentralization of
     the control and management of those operations raises important
     security and privacy challenges.  Indeed, when those workloads are
     operated in a single tenant datacenter, data can be secured by means
     of perimeter security measures, or adopting encryption at least for
     data stored in the datacenter's realm.  In the same way, it is
     acceptable to exchange data between nodes in an unencrypted manner
     provided the network on which data is exchanged is isolated from the
     outside environment.

     When data, models and their associated parameters are exchanged on
     the Internet, or on public connections, data managers need to make
     sure that the data is exchanged securely, and following policies
     complying both with users preferences and local regulations.  As
     previously mentionned, data exchanges done during model training
     phases or directly before performing an inference task need to be
     done as quickly as possible, avoiding tail latencies as much as
     possible.  Even if encryption of large data flow has improved
     considerably in the last years, it is adding a latency overhead,
     considering that the computational overhead related to cryptographic
     operations is handled beyond the chip (often a GPU or a tensor
     processor) performing the machine learning tasks.  It would be
     interesting and benefitial to contribute to the efforts done in the
     IRTF and IETF to develop low latency, lightweight encryption schemes
     in order to reduce this overhead as much as possible.

     Furthermore, as private data is involved in data exchanges related to
     training and inference, specific care must be taken to respect
     regulations steering the way those data can be processed and
     transfered.  In particular, some data need to be processed in the
     same region or country they have been generated in.  To make sure
     those regulations are respected, attestation mechanisms need to be
     put in place.  By using those attestation mechanisms, data owners or
     managers can prove to local authorities that the data they are
     managing is being exchanged according to the policy they specified.
     In the IETF, the RATS working group [RATSWG], and the NASR initiative
     [NASR] are developping attestation mechanisms that could be adapted
     to address machine learning requirements, even if their work is still
     at the beginning.









  Fressancourt, et al.     Expires 13 October 2025               [Page 18]


  Internet-Draft                 AI traffic                     April 2025


  5.  Problem statement

     In today's LLM-based machine learning landscape, we observe a strong
     concentration of training abilities in the hands of a few
     hyperscalers, while several companies tend to propose innovative
     approaches to improve inference by making it faster, and done by
     machines located closer to the users.  Yet, there is a need to
     distribute both the training and inference workloads of AI.

     In the same way as the Internet, there is, and will be several
     incentives to decentralize some or all aspects of AI's lifecycle to
     address scalability, resilience, personalization or privacy concerns.
     This decentralization trend is exemplified by Federated Learning
     [FederatedLearning], with different decentralization models, as
     presented in Section 3.4.

     Given the challenges highlighted in Section 4, and the fact that
     multiple stakeholders are involved in properly adressing those
     challenges, AI-related network traffic will no longer be operated
     only on private infrastructure, but also on an *_open interconnected
     network_*.  Thus, it is desirable that the IETF community discuss
     networking challenges related to large scale decentralized AI to
     avoid the deployment of proprietary solutions, or of solutions
     putting the stability of the Internet at risk due to unfair resource
     mangement or competition between AI-related network traffic and other
     traffic in the Internet.

  6.  Next Steps

     While some work addressing the challenges highligted in this document
     is done in working groups related to congestion management,
     deterministic networking or service routing, there might be interest
     in the IETF community at large to aggregate a group of contributors
     interested in elaborating specific requirements and corresponding
     solutions to the challenges listed in Section 4.  In particular, the
     goal of such an initiative would be to:

     *  Formalize AI-related requirements for service routing: Indeed, it
        is needed to define the metrics to take into account for load
        balancing AI-related workloads, given the need for instance
        stickiness or the importance of latency aspects while addressing
        the specificities of AI-related traffic, consisting in elephant
        flows with little entropy.








  Fressancourt, et al.     Expires 13 October 2025               [Page 19]


  Internet-Draft                 AI traffic                     April 2025


     *  Formalize low latency / limited latency requirements associated
        with AI network traffic: As presented in this document, during the
        different phases of the AI lifecycle, it will be needed to enforce
        a different set of requirements in terms of latency, tolerance to
        jitter or packet loss for sustaining network traffic.

     *  Formalize congestion control aspects related to the operation of
        AI traffic at regional scale: The machine learning community is
        already engaged in improvements of the congestion aspects of AI
        workloads by working on application-level solutions taking for
        granted the underlying behavior of the network.  It would be
        interesting to determine what are the possible improvements that
        the network could benefit from in order to better solve the
        congestion and incast management issues related to the way AI
        network traffic is managed.

     *  Formalize coordination aspects of AI distributed systems: This
        topic is very important to the realization of decentralized, large
        scale AI systems, and to the emergence of an inter-AI network of
        entities collaborating in the execution of end to end AI workloads
        for end users.

  7.  Security Considerations

     Section 4.5 highlights privacy related challenges that AI operations
     at regional scale will have to address.  Beyond this section, no
     additional security concern is raised by the elements presented in
     the document.

  8.  IANA Considerations

     This document has no IANA actions.

  9.  Informative References

     [AIBackbone]
                Sundaresan, J., Gopalan, A., and Meta, "AI impact on
                backbone", <https://atscaleconference.com/videos/ai-
                impact-on-backbone/>.

     [AIConst]  Baduge, S., Thilakarathna, S., Perera, J., Arashpour, M.,
                Sharafi, P., Teodosio, B., Shringi, A., and P. Mendis,
                "Artificial intelligence and smart vision for building and
                construction 4.0: Machine and deep learning methods and
                applications", Elsevier BV, Automation in
                Construction vol. 141, pp. 104440,
                DOI 10.1016/j.autcon.2022.104440, September 2022,
                <https://doi.org/10.1016/j.autcon.2022.104440>.



  Fressancourt, et al.     Expires 13 October 2025               [Page 20]


  Internet-Draft                 AI traffic                     April 2025


     [AITrainingTraffic]
                Li, W., Liu, X., Li, Y., Jin, Y., Tian, H., Zhong, Z.,
                Liu, G., Zhang, Y., and K. Chen, "Understanding
                Communication Characteristics of Distributed Training",
                ACM, Proceedings of the 8th Asia-Pacific Workshop on
                Networking pp. 1-8, DOI 10.1145/3663408.3663409, August
                2024, <https://doi.org/10.1145/3663408.3663409>.

     [ALTO]     "Application-Layer Traffic Optimization (alto)", n.d.,
                <https://datatracker.ietf.org/wg/alto/about/>.

     [Burstiness]
                Luangsomboon, N., Fazel, F., Liebeherr, J., Sobhani, A.,
                Guan, S., and X. Chu, "On the Burstiness of Distributed
                Machine Learning Traffic", arXiv,
                DOI 10.48550/ARXIV.2401.00329, 2024,
                <https://doi.org/10.48550/ARXIV.2401.00329>.

     [CATSWG]   "Computing-Aware Traffic Steering (cats) Working Group",
                n.d., <https://datatracker.ietf.org/group/cats/about/>.

     [ChallengingSpraying]
                Addanki, V., Goyal, P., Marinos, I., and S. Schmid,
                "Ethereal: Divide and Conquer Network Load Balancing in
                Large-Scale Distributed Training", arXiv,
                DOI 10.48550/ARXIV.2407.00550, 2024,
                <https://doi.org/10.48550/ARXIV.2407.00550>.

     [DataParallelism]
                Valiant, L., "A bridging model for parallel computation",
                Association for Computing Machinery (ACM), Communications
                of the ACM vol. 33, no. 8, pp. 103-111,
                DOI 10.1145/79173.79181, August 1990,
                <https://doi.org/10.1145/79173.79181>.

     [FederatedLearning]
                McMahan, H., Moore, E., Ramage, D., Hampson, S., and B.
                Arcas, "Communication-Efficient Learning of Deep Networks
                from Decentralized Data", arXiv,  arXiv,
                DOI 10.48550/ARXIV.1602.05629, 2016,
                <https://doi.org/10.48550/ARXIV.1602.05629>.

     [FEDI]     Anaobi, I., Raman, A., Castro, I., Zia, H., Ibosiola, D.,
                and G. Tyson, "Will Admins Cope? Decentralized Moderation
                in the Fediverse", ACM, Proceedings of the ACM Web
                Conference 2023 pp. 3109-3120,
                DOI 10.1145/3543507.3583487, April 2023,
                <https://doi.org/10.1145/3543507.3583487>.



  Fressancourt, et al.     Expires 13 October 2025               [Page 21]


  Internet-Draft                 AI traffic                     April 2025


     [FLOWER]   Flower Labs GmbH, "Flower - A Friendly Federated Learning
                Framework", n.d., <https://flower.ai/>.

     [I-D.ravi-ippm-csig]
                Ravi, A., Dukkipati, N., Mehta, N., and J. Kumar,
                "Congestion Signaling (CSIG)", Work in Progress, Internet-
                Draft, draft-ravi-ippm-csig-01, 2 February 2024,
                <https://datatracker.ietf.org/doc/html/draft-ravi-ippm-
                csig-01>.

     [I-D.yao-tsvwg-cco-problem-statement-and-usecases]
                Yao, K., Shiping, X., Li, Y., Huang, H., and D. KUTSCHER,
                "Collective Communication Optimization: Problem Statement
                and Use cases", Work in Progress, Internet-Draft, draft-
                yao-tsvwg-cco-problem-statement-and-usecases-00, 23
                October 2023, <https://datatracker.ietf.org/doc/html/
                draft-yao-tsvwg-cco-problem-statement-and-usecases-00>.

     [LLMSize]  Zhang, B., Liu, Z., Cherry, C., and O. Firat, "When
                Scaling Meets LLM Finetuning: The Effect of Data, Model
                and Finetuning Method", arXiv,
                DOI 10.48550/ARXIV.2402.17193, 2024,
                <https://doi.org/10.48550/ARXIV.2402.17193>.

     [MoEParallelism1]
                Jacobs, R., Jordan, M., Nowlan, S., and G. Hinton,
                "Adaptive Mixtures of Local Experts", MIT Press -
                Journals, Neural Computation vol. 3, no. 1, pp. 79-87,
                DOI 10.1162/neco.1991.3.1.79, February 1991,
                <https://doi.org/10.1162/neco.1991.3.1.79>.

     [MoEParallelism2]
                Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le,
                Q., Hinton, G., and J. Dean, "Outrageously Large Neural
                Networks: The Sparsely-Gated Mixture-of-Experts Layer",
                arXiv, DOI 10.48550/ARXIV.1701.06538, 2017,
                <https://doi.org/10.48550/ARXIV.1701.06538>.

     [NASR]     "Network Attestation for Secure Routing (NASR)", n.d.,
                <https://datatracker.ietf.org/doc/bofreq-liu-nasr/>.

     [NCCL]     Nvidia, "NVIDIA Collective Communications Library (NCCL)",
                n.d., <https://developer.nvidia.com/nccl>.








  Fressancourt, et al.     Expires 13 October 2025               [Page 22]


  Internet-Draft                 AI traffic                     April 2025


     [NetworkBottleneck]
                Zhang, Z., Chang, C., Lin, H., Wang, Y., Arora, R., and X.
                Jin, "Is Network the Bottleneck of Distributed Training?",
                ACM, Proceedings of the Workshop on Network Meets AI & ML,
                DOI 10.1145/3405671.3405810, August 2020,
                <https://doi.org/10.1145/3405671.3405810>.

     [PipelineParallelism]
                Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V.,
                Devanur, N., Ganger, G., Gibbons, P., and M. Zaharia,
                "PipeDream: generalized pipeline parallelism for DNN
                training", ACM, Proceedings of the 27th ACM Symposium on
                Operating Systems Principles pp. 1-15,
                DOI 10.1145/3341301.3359646, October 2019,
                <https://doi.org/10.1145/3341301.3359646>.

     [RATSWG]   "Remote ATtestation ProcedureS (rats) Working Group",
                n.d., <https://datatracker.ietf.org/wg/rats/about/>.

     [RCCL]     AMD, "AMD ROCm Software", n.d.,
                <https://github.com/ROCm/ROCm>.

     [SchedulingSharding]
                Chu, W., Choudhury, A., and Meta, "Scheduler and Sharding
                Considerations for Network Efficiency",
                <https://atscaleconference.com/videos/scheduler-and-
                sharding-considerations-for-network-efficiency/>.

     [SiteToSite]
                Cangialosi, F., Narayan, A., Goyal, P., Mittal, R.,
                Alizadeh, M., and H. Balakrishnan, "Site-to-site internet
                traffic control", ACM, Proceedings of the Sixteenth
                European Conference on Computer Systems pp. 574-589,
                DOI 10.1145/3447786.3456260, April 2021,
                <https://doi.org/10.1145/3447786.3456260>.
















  Fressancourt, et al.     Expires 13 October 2025               [Page 23]


  Internet-Draft                 AI traffic                     April 2025


     [TensorParallelism]
                Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z.,
                Citro, C., Corrado, G., Davis, A., Dean, J., Devin, M.,
                Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard,
                M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M.,
                Levenberg, J., Mane, D., Monga, R., Moore, S., Murray, D.,
                Olah, C., Schuster, M., Shlens, J., Steiner, B.,
                Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V.,
                Vasudevan, V., Viegas, F., Vinyals, O., Warden, P.,
                Wattenberg, M., Wicke, M., Yu, Y., and X. Zheng,
                "TensorFlow: Large-Scale Machine Learning on Heterogeneous
                Distributed Systems", arXiv,
                DOI 10.48550/ARXIV.1603.04467, 2016,
                <https://doi.org/10.48550/ARXIV.1603.04467>.

     [xCCL]     Weingram, A., Li, Y., Qi, H., Ng, D., Dai, L., and X. Lu,
                "xCCL: A Survey of Industry-Led Collective Communication
                Libraries for Deep Learning", Springer Science and
                Business Media LLC, Journal of Computer Science and
                Technology vol. 38, no. 1, pp. 166-195,
                DOI 10.1007/s11390-023-2894-6, February 2023,
                <https://doi.org/10.1007/s11390-023-2894-6>.

  Appendix A.  A primer on Machine learning (extended version of
               Section 3)

     Along its development, Machine Learning (ML) involves the use of an
     increasing amount of computing and storage resources in the
     operations of algorithms taking decisions or deriving insights from
     data.  In recent generative AI algorithms, or in large language
     models, the amount of data used to train models of increasing size in
     terms of parameters has grown exponentially.  Besides, the size of
     large models translates in an increasing memory and computing
     footprint.  Given this evolution, ML algorithms can not be executed
     or trained on a single machine, and thus ML systems are "distributed"
     by nature, regardless of the architecture they adopt.

     This appendix section introduces the lifecycle of ML systems
     (Appendix A.1), then explains how ML systems can be split between
     entities fulfilling different roles (Appendix A.2) and introduces the
     methods designed to parallelize ML jobs (Appendix A.3) and the
     communication methods used between instances to communicate data and
     parameters (Appendix A.4).  It is an extended version of Section 3
     which focuses on the consequences on network traffic patterns and
     challenges of the elements presented here.






  Fressancourt, et al.     Expires 13 October 2025               [Page 24]


  Internet-Draft                 AI traffic                     April 2025


  A.1.  Machine learning model lifecycle

     In machine learning, two approaches can be adopted to algorithmically
     derive insights from a dataset: supervised learning and unsupervised
     learning.  In unsupervised learning, algorithms are developped and
     used to find patterns, clusters or relationships among data without
     previous knowledge or reference.  In supervised learning, algorithms
     use a reference data set (or training set), consisting in data
     labelled with answers or hints to the solution of the question being
     asked, to build (or train) a model to which data is compared later
     during the inference (or serving) phase.

     The model is the cornerstone of supervised machine learning.  It is a
     representation of a system that is observed and measured with data.
     When the model is trained, the model is parametrized or modified so
     it is able to model the behaviour of the system from the dataset used
     in the training phase.  Then, during the inference phase, data is
     presented to the trained model in order to derive information or make
     predictions about the system.

     +----------+    +----------+    +--------+    +------+    +---------+
     |   Data   |    |   Data   |    |  Model |    | Model|    |Inference|
     |          | => |   Pre-   | => |        | => | Fine-| => |    /    |
     |Collection|    |Processing|    |Training|    |Tuning|    | Serving |
     +----------+    +----------+    +--------+    +------+    +---------+

                Figure 4: Supervised machine learning lifecycle

     In the renewed interest in deep neural networks, and the strong
     development of generative AI, a supervised learning approach has been
     favoured, using reference models.  Supervised machine learning
     follows a cycle presented in Figure 4.

     To obtain a reference model, first, data is collected from a variety
     of sources.  This data is then gathered and pre-processed, in order
     to clean it (align formatting, erase obvious measurement errors,
     _etc._) and potentially to label it.  In particular, large data sets
     are divided into chunks, or tokens, which are the basic data units
     used as input or outputs of models.  For instance, if the data set
     consists in a corpus of texts, tokens are words, subwords or short
     sequences of words.










  Fressancourt, et al.     Expires 13 October 2025               [Page 25]


  Internet-Draft                 AI traffic                     April 2025


     The pre-processed data is used to train the reference model, _i.e._
     convert the knowledge to extract from the pre-processed data into a
     set of parameters including weights ruling how the model will answer
     future requests.  Parameters are variables adjusting the structure
     underlying the model that are set during the training phase.  Weights
     are a subste of the parameters, and determine how strong the ties are
     between some other parameters in the model.

     Once the model has been trained, it can be used to infer from a
     request, or, in other words, to serve an insight from a request's
     data.  Inference operations are not necessarily done by the same
     nodes that have trained the model.  Models can be transferred to
     other worker nodes to perfom inference tasks, sometimes placed close
     to the users making requests.  Besides, those transferred models can
     be re-trained or fine-tuned in order to better serve requests in the
     context they are done.  This can be done for instance with private or
     locally relevant data.

  A.2.  System model

     The machine learning systems built to meet the requirements of use
     cases presented in Section 2 are fundamentally distributed in order
     to meet scaling and time to answer requirements.  Indeed, if we
     consider modern lareg language models, they use from 125 million to
     175 billion parameters, where each parameters can be encoded with 4
     bytes to 4 bits depending on the precision required in the answer
     provided by the model.  From a memory perspective, this means that
     storing a model and its parameters will cost from 62,5 MB (for a
     model using 125 million parameters with 4-bits parameters) to 700 GB
     (for a model using 175 billion parameters with 4 bytes parameters).
     The memory footprint of modern large language models makes those
     models both difficult to manipulate on single nodes and to exchange
     data between nodes participating in a distributed task.

     In machine learning systems' distribution, nodes can play specific
     roles, depending on their (hardware) capabilities or their location
     in the network.

     Those roles, presented in Figure 5 are:

     *  *Data producer / Owner:*

        This entity is producing data from which users are willing to
        retrieve insights.  It can be a sensor in a network, a server
        producing logs and interacting with a set of users, etc.

     *  *Data manager:*




  Fressancourt, et al.     Expires 13 October 2025               [Page 26]


  Internet-Draft                 AI traffic                     April 2025


        This entity is in charge of managing the data produced by the data
        producer.  In that extend, it can pre-process, filter, add context
        (or metadata), store or transfer data to other entities in the
        system.  In its data management operations, the data manager has
        to take specific care of security and privacy aspects of data
        management.  In particular, it must make sure that data is stored
        securely, transferred to authorized entities and that user's
        consent and privacy preferences are respected.  To do so, it may
        use specialized encryption methods and protocols, meeting
        challenges presented in Section 4.5.

     *  *Worker:*

        This entity is in charge of processing data retrieved from the
        data manager in order to gain insights from it.  To do so, it can
        training a (potentially large) model during the model training
        phase; and personalize this model to better serve specific needs
        or requests from users during the inference phase.  In some
        specific cases, for instance, if the model from which the
        inference is done is small, the worker might be alone, but given
        the operational requirements of new approaches in machine
        learning, workers need to collaborate with other workers to
        complete a task together.

     *  *(Model) Aggregator:*

        In case the training, personalization or fine tuning of a model is
        done by several collaborating entities, the aggregator is
        responsible for gathering the results of the individual tasks
        performed by the workers and synthetize them in a general model.
        This model can be sent back to the workers for subsequent
        processing.

     *  *Inference engine / Server:*

        This entity is in charge of producing inferences, _i.e._ deriving
        insights from a request done against a trained model.  Those
        requests might be contextualized from a conversation, _i.e._ a
        sequence of requests and their answers.

     *  *Coordinator / Orchestrator:*

        This entity is in charge of orchestrating the execution of
        distributed tasks among the workers in charge.  This coordination
        role might be done by means of a centralized node (_e.g._ a
        parameter server), a federation of locally responsible nodes or in
        a coordinated fashion using a consensus protocol.




  Fressancourt, et al.     Expires 13 October 2025               [Page 27]


  Internet-Draft                 AI traffic                     April 2025


                    +----------------------------------+
                    |           Orchestrator           |
                    |                     +------------+  +------+   User
                    |                     | Aggregator |  |Server| <----
      +--------+    +---------------------+------------+  +------+   Req.
      |  Data  |        |         |||     ^  ^  ^  \  \     |||
      |Producer|        |         v||    /  /  /    \  \    ||v
      +--------+ \      v          v|   /  /  /   \  \  \   |v
        ...       v +-------+       v  /  /  /     \  \  v  v
      +--------+    |       |    +----+-+/  /       \  v+-+-------+
      |  Data  |    | Data  |--> |Worker+-+/         v+-+  Worker |
      |Producer|--> |Manager|--> +-+----+ +-+       +-+ +-------+-+
      +--------+    |       |-->   +-+----+ |       | +-------+-+
         ...      ^ +-------+        +------+       +---------+
      +--------+ /            :                :
      | Data   |              :                :
      |Producer|              : <--Training--> : <---  Inference --->
      +--------+              :                :  (with fine tuning)

                     Figure 5: Machine Learning syste model

     Those functional roles can be arranged, managed and executed in many
     different way in a ML system.  This arrangement determines how
     centralized and / or distributed a ML system is.

  A.3.  Parallelization modes

     In model-based machine learning, training and inference phases follow
     a pipeline similar to the pipeline presented in Figure 6.  As
     mentionned in previous sections, given the size of currently used
     models and the amount of data required to either train or infer from
     a model, machine learning workloads need to be distributed.  This
     distribution can follow different patterns (or a mix of those
     patterns): data parallelism, tensor parallelism, pipeline parallelism
     or mixture-of-expert parallelism.  Those patterns are presented in
     the following subsections.

           +------+
         +-+----+ |        +-+   +-+       +---+
       +-+----+ | |        |p|   |p|       | o |
     +-+----+ | | |       ^|a+   +a+       | u |
     |      | | | |      / |r|\ /|r|\      | t |
     |      | | | | ==> *  |a| * |a| * ==> | p |
     | Data | | | |      \ |m|/ \|m|/      | u |
     |      | | +-+       v|s+   +s+       | t |
     |      | +-+          |.|   |.|       |   |
     |      +-+            +-+   +-+       +---+
     +------+



  Fressancourt, et al.     Expires 13 October 2025               [Page 28]


  Internet-Draft                 AI traffic                     April 2025


                       Figure 6: Model-based AI pipeline

  A.3.1.  Data parallelism

     Data parallelism is the oldest among the parallelization patterns we
     highlighted.  It has been introduced in [DataParallelism].  As
     presented in Figure 7, data parallelism consists in the partitioning
     of the data used in a given machine learning task in a set of batches
     that are used to train or tune the model.  In data parallelism, a
     node manipulates a complete model and change its parameters using the
     data partition it has been allocated.  The result of the tasks
     performed in parallel by the workers are aggregated by an aggregator
     node at the end of the pipeline.  The model parameters resulting from
     this aggregation are transmitted back to the workers for future use
     of the model in order that each worker benefits from the work done by
     others in parallel.

                                       +-+   +-+       +---+
                       +-+----+        |p|   |p|       |i o|
                       |      |       ^|a+   +a+       |n u|
                       |      |      / |r|\ /|r|\      |t t|
           +------+    | Data | ==> *  |a| * |a| * ==> |e p|
         +-+----+ |    |  01  |      \ |m|/ \|m|/      |r u|    +---+
       +-+----+ | |  ^ |      |       v|s+   +s+       |i t| \  | o |
     +-+----+ | | | /  |      +        |.|   |.|       |m  |  v | u |
     |      | | | |    +------+        +-+   +-+       +---+    | t |
     |      | | | |                                             | p |
     | Data | | | |                    +-+   +-+       +---+    | u |
     |      | | +-+ \  +-+----+        |p|   |p|       |i o|  ^ | t |
     |      | +-+    v |      |       ^|a+   +a+       |n u| /  |   |
     |      +-+        |      |      / |r|\ /|r|\      |t t|    +---+
     +------+          | Data | ==> *  |a| * |a| * ==> |e p|
                       |  02  |      \ |m|/ \|m|/      |r u|
                       |      |       v|s+   +s+       |i t|
                       |      +        |.|   |.|       |m  |
                       +------+        +-+   +-+       +---+

                      Figure 7: Data-parallel AI pipeline

     In this pattern, the workers are only weakly synchronized, once the
     model has been aggregated and its parameters sent back.  In that
     extend, data parallelism can sustain up to seconds latency in the
     synchronization traffic.  Yet, as parameters are sent back by the
     aggregator to all the ndes for the entire model, the volume of data
     exchanged between training iterations of inference tasks can be quite
     large.  Besides, the aggregator is a focal point in the traffic
     between nodes, which can raise traffic management challenges.




  Fressancourt, et al.     Expires 13 October 2025               [Page 29]


  Internet-Draft                 AI traffic                     April 2025


  A.3.2.  Model parallelism

     For the last few years, models involved in machine learning tasks
     have increased in size, making their use by single nodes complex.  To
     allow the training and inference of larger models, some
     parallelization patterns have been designed to spilt models among
     several worker nodes: *pipeline parallelism* and *tensor
     parallelism*.

  A.3.2.1.  Pipeline parallelism

     Pipeline parallelism, described in [PipelineParallelism], takes
     advantage of the fact that models used in deep neural networks are
     structured in layers.  In pipeline parallelism, as shown in Figure 8,
     a model is separated into stages consisting in a few consecutive
     layers of the entire model.  Each stage is allocated to a separate
     worker node.  Each stage is executed using the intermediate results
     from the previous stage, and after each iteration, the parameters
     from adjacent stages are used to refine the model's stage held by the
     worker node.

           +------+
         +-+----+ |        +-+       +---+       +-+       +---+
       +-+----+ | |        |p|       |i o|       |p|       | o |
     +-+----+ | | |       ^|a+       |n u|       +a+       | u |
     |      | | | |      / |r|\      |t t|      /|r|\      | t |
     |      | | | | ==> *  |a| * ==> |e p| ==> * |a| * ==> | p |
     | Data | | | |      \ |m|/      |r u|      \|m|/      | u |
     |      | | +-+       v|s+       |i t|       +s+       | t |
     |      | +-+     :    |.|  :    |m  |    :  |.|    :  |   |
     |      +-+       :    +-+  :    +---+    :  +-+    :  +---+
     +------+         :         :             :         :
                      : Stage 1 :             : Stage 2 :

                 Figure 8: Pipeline model-parallel AI pipeline

     In this pattern, the communication volume between node can be less
     important than in data parallellism, as each node only communicates
     with nodes holding adjacent layers.  Yet, as stage iteration can be
     faster to execute, communications can be more frequent.











  Fressancourt, et al.     Expires 13 October 2025               [Page 30]


  Internet-Draft                 AI traffic                     April 2025


  A.3.2.2.  Tensor parallelism

     Tensor parallelism, stemming from the work presented in
     [TensorParallelism], is taking advantage of the fact that training
     and inference operations in model-based machine learning are using
     operations on matrixes that can be split according to several
     dimensions.  In comparison with pipeline parallellism, tensor
     parallelism is a pattern in which layers are split into chunks that
     can be operated by different nodes, as shown on Figure 9.  Tensor
     paralellism can be naively presented as a split in the model
     parameter space rather than along the layer / stage dimension.

                       +---+   +---+       +---+
                       |p p+   +p p+       |i o|
                       |a a|\ /|a a|\      |n u|
           +------+    |r r| * |r r| * ==> |t t|
         +-+----+ |    |a t|/ \|a t|/      |e p|    +---+
       +-+----+ | |  ^ |m 1+   +m 1+       |r .| \  | o |
     +-+----+ | | | /  +---+   +---+       +---+  v | u |
     |      | | | |      ^       ^                  | t |
     |      | | | |      |       |                  | p |
     | Data | | | |      v       v                  | u |
     |      | | +-+ \  +---+   +---+       +---+  ^ | t |
     |      | +-+    v |p p+   +p p+       |i o| /  |   |
     |      +-+        |a a|\ /|a a|\      |n u|    +---+
     +------+          |r r| * |r r| * ==> |t t|
                       |a t|/ \|a t|/      |e p|
                       |m 2+   +m 2+       |r .|
                       +---+   +---+       +---+

                  Figure 9: Tensor model-parallel AI pipeline

     In tensor parallellism, nodes holding chunks of the same model (or of
     the same stage) need to communicate during the execution of a model
     (or stage) iteration.  This involves even tighter latency
     requirements compared with pipeline parallelism for inter-node
     communications.

  A.3.3.  Mixture of Expert parallelism

     Mixture of experts (MoE) parallelism stems from an idea introduced in
     [MoEParallelism1] and adapted to deep neural networks in
     [MoEParallelism2].  In MoE parallellism, nodes are holding smaller
     but specialized models trained over a smaller amount of data and
     holding fewer parameters.  When a request or specific token is
     submitted to a MoE model, a gateway node, the router, takes a
     decision about which model instance to use against the specific token
     or request that has been submitted.  After the selected worker



  Fressancourt, et al.     Expires 13 October 2025               [Page 31]


  Internet-Draft                 AI traffic                     April 2025


     executed the task against the token or request, the result is given,
     and considered as the result given by the whole model.

                                  +---+   +---+
                                  |p s+   +p s+
           +------+               |a e|\ /|a e|\
         +-+----+ |               |r t| * |r t| *   +---+
       +-+----+ | |             ^ |a .|/ \|a .|/ \  | o |
     +-+----+ | | |            /  |m 1+   +m 1+   v | u |
     |      | | | |     +------+  +---+   +---+     | t |
     |      | | | | ==> |Router| ---------------    | p |
     | Data | | | |     +------+  +---+   +---+     | u |
     |      | | +-+               |p s+   +p s|     | t |
     |      | +-+                 |a e|\ /|a e|     |   |
     |      +-+                   |r t| * |r t|     +---+
     +------+                     |a .|/ \|a .|
                                  |m 2+   +m 2|
                                  +---+   +---+

               Figure 10: Mixture of Expert parallel AI pipeline

     In MoE parallellism, the router plays a major role, and is a
     communication focal point.

  A.4.  Collective communication methods

     In their development, distributed machine learning systems benefit
     from collective communication libraries such as NCCL ([NCCL]) or RCCL
     ([RCCL]) to abstract the setup and management of the connections used
     to exchange data between worker nodes involved in distributed
     training or inference tasks.  As presented in [xCCL] or in
     [I-D.yao-tsvwg-cco-problem-statement-and-usecases], those libraries
     introduce collective communication methodss, used in accordance with
     parallelization modes presented in Appendix A.3.

     To better explain what are the main collective communication methods,
     we consider a collective consisting in four nodes.  Those nodes
     exchange pieces of data in the frame of the execution of a
     distributed machine learning task.  The topology underlying the
     connections between those nodes is not detailed at this stage.  The
     major collective communication methods are:










  Fressancourt, et al.     Expires 13 October 2025               [Page 32]


  Internet-Draft                 AI traffic                     April 2025


  A.4.1.  Broadcast

     This collective communication method is very intuitive to understand
     for networking people, as its behaviour is the same as when a host
     broadcasts a packet in an IP-based network.  In the broadcast
     collective communication, method, a node (N.1) is sending the same
     information to all the nodes participating in the collective.  No
     data transformation is performed during this operation.

                                    +---+
                                    +N.1+
                                   /+---+\
                                  v   |   v
                              +---+   v   +---+
                              |N.2| +---+ |N.3|
                              +---+ |N.4| +---+
                                    +---+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | D.1 | | --- | | --- | | --- | ==> | D.1 | | D.1 | | D.1 | | D.1 |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+

                          Figure 11: Broadcast method

  A.4.2.  Reduce

     In this collective communication method, the data exchange is
     combined with an operation on the received data to produce an output.
     During a Reduce operation, the nodes involved in the collective send
     their data to an aggregator processing the received inputs _I.1_ ...
     _I.4_ using an operator _f_ to obtain _Out_.  _Out_ is then provided
     to one of the nodes in the collective (N.1).  Note that most of the
     time, the aggregation is done by one node in the collective, carrying
     the aggregator function.

















  Fressancourt, et al.     Expires 13 October 2025               [Page 33]


  Internet-Draft                 AI traffic                     April 2025


                           +---+ +---+ +---+ +---+
                           |N.1| |N.2| |N.3| |N.4|
                           +---+ +---+ +---+ +---+
                            | ^   |     |     |
                            | |   |     |     |
                            v |   v     v     v
                         +-------------------------+
                         |        Aggregator       |
                         | f(I.1,I.2,I.4,I.4)=Out  |
                         +-------------------------+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | I.1 | | I.2 | | I.3 | | I.4 | ==> | Out | | --- | | --- | | --- |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+

                            Figure 12: Reduce method

  A.4.3.  Scatter

     In this collective communication method, a node in the collective
     splits the data it has into equal size chunks, and distribute a
     different chunk to every single other node in the collective, in
     order to distribute the data evenly.

                                    +---+
                                    +N.1+
                                   /+---+\
                                  v   |   v
                              +---+   v   +---+
                              |N.2| +---+ |N.3|
                              +---+ |N.4| +---+
                                    +---+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | D.1 | | --- | | --- | | --- |     | D.1 | | --- | | --- | | --- |
     | D.2 | | --- | | --- | | --- | ==> | --- | | D.2 | | --- | | --- |
     | D.3 | | --- | | --- | | --- |     | --- | | --- | | D.3 | | --- |
     | D.4 | | --- | | --- | | --- |     | --- | | --- | | --- | | D.4 |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+

                           Figure 13: Scatter method










  Fressancourt, et al.     Expires 13 October 2025               [Page 34]


  Internet-Draft                 AI traffic                     April 2025


  A.4.4.  All-Gather

     This collective communication method can be seen as a simultaneous
     broadcast done by every node in the collective.  Indeed, in an All-
     gather operation, every node sends the data it has to every other
     nodes in the collective, so that in the end, every node has a copy of
     every piece of data held by the nodes in the collective before the
     operation.

                                    +---+
                                  ^ |N.1| ^
                                 /  +---+  \
                                v           v
                             +---+    ^    +---+
                             |N.2| <--|--> |N.3|
                             +---+    v    +---+
                                ^           ^
                                 \  +---+  /
                                  v |N.4| v
                                    +---+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | D.1 | | --- | | --- | | --- |     | D.1 | | D.1 | | D.1 | | D.1 |
     | --- | | D.2 | | --- | | --- | ==> | D.2 | | D.2 | | D.2 | | D.2 |
     | --- | | --- | | D.3 | | --- |     | D.3 | | D.3 | | D.3 | | D.3 |
     | --- | | --- | | --- | | D.4 |     | D.4 | | D.4 | | D.4 | | D.4 |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+

                          Figure 14: All-gather method

  A.4.5.  All-to-All

     This collective communication method can be seen as a simultaneous
     scatter operation done by every node in the collective.  Indeed, in
     an All-to-All operation, every node splits the data it holds in equal
     size chunks, and sends one chunk to every other node in the
     collective.  As a result, the data held by each node after the All-
     to-All operation is different, but every node holds roughly the same
     amount of data, even if the data volume was not balanced before the
     operation.











  Fressancourt, et al.     Expires 13 October 2025               [Page 35]


  Internet-Draft                 AI traffic                     April 2025


                                    +---+
                                  ^ |N.1| ^
                                 /  +---+  \
                                v           v
                             +---+    ^    +---+
                             |N.2| <--|--> |N.3|
                             +---+    v    +---+
                                ^           ^
                                 \  +---+  /
                                  v |N.4| v
                                    +---+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | A.1 | | B.1 | | C.1 | | D.1 |     | A.1 | | A.2 | | A.3 | | A.4 |
     | A.2 | | B.2 | | C.2 | | D.2 | ==> | B.1 | | B.2 | | B.3 | | B.4 |
     | A.3 | | B.3 | | C.3 | | D.3 |     | C.1 | | C.2 | | C.3 | | C.4 |
     | A.4 | | B.4 | | C.4 | | D.4 |     | D.1 | | D.2 | | D.3 | | D.4 |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+

                          Figure 15: All-to-All method

  A.4.6.  All-Reduce

     This collective communication method is very similar to the Reduce
     method.  During an All-Reduce operation, the nodes involved in the
     collective send their data to an aggregator processing the received
     inputs _I.1_ ... _I.4_ using an operator _f_ to obtain _Out_.  To the
     difference of the Reduce operation, _Out_ is sent back to every node
     in the collective.  In the implementation of this method, the
     aggregator function can be done by one node in the collective, or
     every single node applies the operator _f_ on the inputs received
     after an All-to-All operation.

                           +---+ +---+ +---+ +---+
                           |N.1| |N.2| |N.3| |N.4|
                           +---+ +---+ +---+ +---+
                            | ^   | ^   | ^   | ^
                            | |   | |   | |   | |
                            v |   v |   v |   v |
                         +-------------------------+
                         |        Aggregator       |
                         | f(I.1,I.2,I.3,I.4)=Out  |
                         +-------------------------+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | I.1 | | I.2 | | I.3 | | I.4 | ==> | Out | | Out | | Out | | Out |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+




  Fressancourt, et al.     Expires 13 October 2025               [Page 36]


  Internet-Draft                 AI traffic                     April 2025


                          Figure 16: All-Reduce method

  A.4.7.  Reduce-Scatter

     This collective communication method is a combination of the All-
     Reduce method with the Scatter method.  During a Reduce-Scatter
     operation, the nodes involved in the collective send their data to an
     aggregator processing the received inputs _I.1_ ... _I.4_ using an
     operator _f_ to obtain _Out_.  To the difference of the All-Reduce
     operation, _Out_ is split into equal size chunks _O.1_ ... _O.4_, and
     each chunk is sent to a different node in the collective.

                           +---+ +---+ +---+ +---+
                           |N.1| |N.2| |N.3| |N.4|
                           +---+ +---+ +---+ +---+
                            | ^   | ^   | ^   | ^
                            | |   | |   | |   | |
                            v |   v |   v |   v |
                   +---------------------------------------+
                   |              Aggregator               |
                   | f(I.1,I.2,I.3,I.4)=(O.1,O.2,O.3,O.4)  |
                   +---------------------------------------+
       N.1     N.2     N.3     N.4         N.1     N.2     N.3     N.4
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+
     | I.1 | | I.2 | | I.3 | | I.4 | ==> | O.1 | | O.2 | | O.3 | | O.4 |
     +-----+ +-----+ +-----+ +-----+     +-----+ +-----+ +-----+ +-----+

                        Figure 17: Reduce-Scatter method

  Authors' Addresses

     Antoine Fressancourt
     Huawei Technologies France S.A.S.U.
     18, Quai du Point du Jour
     92100 Boulogne-Billancourt
     France
     Email: antoine.fressancourt@huawei.com


     Luigi Iannone
     Huawei Technologies France S.A.S.U.
     18, Quai du Point du Jour
     92100 Boulogne-Billancourt
     France
     Email: luigi.iannone@huawei.com






  Fressancourt, et al.     Expires 13 October 2025               [Page 37]


  Internet-Draft                 AI traffic                     April 2025


     David Lou
     Huawei Technologies Duesseldorf GmbH
     Riesstrasse 25
     80992 Munich
     Germany
     Email: zhe.lou@huawei.com


     Dirk Trossen
     Huawei Technologies Duesseldorf GmbH
     Riesstrasse 25
     80992 Munich
     Germany
     Email: dirk.trossen@huawei.com





































  Fressancourt, et al.     Expires 13 October 2025               [Page 38]


name: draft-irtf-nmrg-ai-deploy
rev: 02
title: Considerations of network/system for AI services
time: 2025-10-20T13:42:18Z
group_uri: /api/v1/group/group/38/
expires: 2026-04-23T13:42:18Z
states: ["/api/v1/doc/state/1/", "/api/v1/doc/state/150/"]
categories: []
tags: []
fetched_at: 2026-03-07T06:55:18.623762+00:00
source: ietf
abstract:
  As the development of AI technology has matured and AI technology has
     begun to be applied in various fields, it has changed from running
     only on very high-performance servers to running on commodity
     servers, with affordable, small-scale hardware, including
     microcontrollers, low-performance CPUs, and AI chipsets.  This
     document outlines how to configure the network and system for an AI
     inference service, providing AI services in a distributed manner.  It
     also outlines the factors to consider when a client connects to a
     cloud server and an edge device to requests an AI service.  It
     describes some use cases for deploying network-based AI services,
     such as self-driving vehicles and network digital twins.

full_text:
  Internet Research Task Force                                   Y-G. Hong
  Internet-Draft                                             Daejeon Univ.
  Intended status: Informational                                 J-S. Youn
  Expires: 23 April 2026                                    DONG-EUI Univ.
                                                                 S-W. Hong
                                                                      ETRI
                                                         P. Martinez-Julia
                                                                      NICT
                                                                     Q. Wu
                                                                    Huawei
                                                           20 October 2025


              Considerations of network/system for AI services
                        draft-irtf-nmrg-ai-deploy-02

  Abstract

     As the development of AI technology has matured and AI technology has
     begun to be applied in various fields, it has changed from running
     only on very high-performance servers to running on commodity
     servers, with affordable, small-scale hardware, including
     microcontrollers, low-performance CPUs, and AI chipsets.  This
     document outlines how to configure the network and system for an AI
     inference service, providing AI services in a distributed manner.  It
     also outlines the factors to consider when a client connects to a
     cloud server and an edge device to requests an AI service.  It
     describes some use cases for deploying network-based AI services,
     such as self-driving vehicles and network digital twins.

  Status of This Memo

     This Internet-Draft is submitted in full conformance with the
     provisions of BCP 78 and BCP 79.

     Internet-Drafts are working documents of the Internet Engineering
     Task Force (IETF).  Note that other groups may also distribute
     working documents as Internet-Drafts.  The list of current Internet-
     Drafts is at https://datatracker.ietf.org/drafts/current/.

     Internet-Drafts are draft documents valid for a maximum of six months
     and may be updated, replaced, or obsoleted by other documents at any
     time.  It is inappropriate to use Internet-Drafts as reference
     material or to cite them other than as "work in progress."

     This Internet-Draft will expire on 23 April 2026.





  Hong, et al.              Expires 23 April 2026                 [Page 1]


  Internet-Draft            Deploying AI services             October 2025


  Copyright Notice

     Copyright (c) 2025 IETF Trust and the persons identified as the
     document authors.  All rights reserved.

     This document is subject to BCP 78 and the IETF Trust's Legal
     Provisions Relating to IETF Documents (https://trustee.ietf.org/
     license-info) in effect on the date of publication of this document.
     Please review these documents carefully, as they describe your rights
     and restrictions with respect to this document.  Code Components
     extracted from this document must include Revised BSD License text as
     described in Section 4.e of the Trust Legal Provisions and are
     provided without warranty as described in the Revised BSD License.

  Table of Contents

     1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.  Procedure to provide AI services  . . . . . . . . . . . . . .   5
     3.  Network configuration structure to provide AI services  . . .   6
       3.1.  AI inference service on Local machine . . . . . . . . . .   6
       3.2.  AI inference service on Cloud server  . . . . . . . . . .   7
       3.3.  AI inference service on Edge device . . . . . . . . . . .   8
       3.4.  AI inference service on Cloud server and Edge device  . .   9
       3.5.  AI inference service on horizontal multiple servers . . .  11
       3.6.  Network-side utilization for AI learning  . . . . . . . .  12
     4.  Considerations of network/system for AI services  . . . . . .  13
       4.1.  Considerations of the functional characteristics of the
             hardware  . . . . . . . . . . . . . . . . . . . . . . . .  13
       4.2.  Considerations for the characteristics of the AI model  .  14
       4.3.  Considerations for the characteristics of the communication
             method  . . . . . . . . . . . . . . . . . . . . . . . . .  15
     5.  Addressing challenges for coupling AI and NM  . . . . . . . .  16
       5.1.  Low-level challenges  . . . . . . . . . . . . . . . . . .  17
       5.2.  High-level challenges . . . . . . . . . . . . . . . . . .  18
     6.  Use cases of deploying network-based AI services  . . . . . .  19
       6.1.  Deploying AI services for self-driving vehicles . . . . .  19
       6.2.  Deploying AI services for network digital twins . . . . .  21
     7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  23
     8.  Security Considerations . . . . . . . . . . . . . . . . . . .  23
     9.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  24
     10. References  . . . . . . . . . . . . . . . . . . . . . . . . .  24
       10.1.  Normative References . . . . . . . . . . . . . . . . . .  24
       10.2.  Informative References . . . . . . . . . . . . . . . . .  24
     Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  25







  Hong, et al.              Expires 23 April 2026                 [Page 2]


  Internet-Draft            Deploying AI services             October 2025


  1.  Introduction

     In the Internet of Things (IoT), the amount of data generated from
     IoT devices has exploded along with the number of IoT devices due to
     industrial digitization and the development and dissemination of new
     devices.  Various methods are being tried to effectively process the
     explosively increasing IoT devices and data of IoT devices.  One of
     them is to provide IoT services in a place located close to IoT
     devices and users, away from cloud computing that transmits all data
     generated from IoT devices to a cloud server [RFC9556].

     IoT services also started to break away from the traditional method
     of analyzing IoT data collected so far in the cloud and delivering
     the analyzed results back to IoT objects or devices.  In other words,
     AIoT (Artificial Intelligence of Things) technology, a combination of
     IoT technology and artificial intelligence (AI) technology, started
     to be discussed at international standardization organizations such
     as ITU-T.  AIoT technology, discussed by the ITU-T CG-AIoT group, is
     defined as a technology that combines AI technology and IoT
     infrastructure to achieve more efficient IoT operations, improve
     human-machine interaction, and improve data management and analysis
     [CG-AIoT].

     The first work started by the IETF to apply IoT technology to the
     Internet was to research a lightweight protocol stack instead of the
     existing TCP/IP protocol stack so that various types of IoT devices,
     not traditional Internet terminals, could access the Internet
     [RFC6574][RFC7452].  These technologies have been developed by
     6LoWPAN working group, 6lo working group, 6tisch working group, core
     working group, t2trg group, etc.  As the development of AI technology
     matured and AI technology began to be applied in various fields, just
     as IoT technology was mounted on resource-constrained devices and
     connected to the Internet, AI technology is also changed from running
     only on very high-performance servers.  The technology is being
     developed to run on small hardware, including microcontrollers, low-
     performance CPUs and AI chipsets.  This technology development
     direction is called On-device AI or TinyML[tinyML].

     In this document, we consider how to configure the network and system
     in terms of AI inference service to provide AI service in the IoT
     environment.  In the IoT environment, the technology of collecting
     sensing data from various sensors and delivering it to the cloud has
     already been studied by many standardization organizations including
     the IETF and many standards have been developed.  Now, after creating
     an AI model to provide AI services based on the collected data, how
     to configure this AI model as a system has become the main research
     goal.  Until now, it has been common to develop AI services that
     collect data and perform inferences from the trained servers, but in



  Hong, et al.              Expires 23 April 2026                 [Page 3]


  Internet-Draft            Deploying AI services             October 2025


     terms of the spread of AI services, it is not appropriate to use
     expensive servers to provide AI services.  In addition, since the
     server that collects and trains data mainly exists in the form of a
     cloud server, there are also many problems in proceeding in the form
     of requesting AI service by connecting a large number of terminals to
     these cloud servers to provide AI services.  Therefore, when an AI
     service is requested to an edge device located at a close distance,
     it may have effects such as real-time service support, network
     traffic reduction, and important data security rather than requesting
     an AI service to an AI server located in a distant cloud[RFC9556].

     Even if an edge device is used to serve AI services, it is still
     important to connect to an AI server in the cloud for tasks that take
     a lot of time or require a lot of data.  Therefore, an offloading
     technique for properly distributing the workload between the cloud
     server and the edge device is also a field that is being actively
     studied.

     Furthermore, beyond the deployment of basic AI inference on
     distributed devices (On-device lightweight AI or TinyML), the concept
     of Agentic AI is rapidly emerging.  Agentic AI refers to an AI system
     capable of autonomous goal setting, complex task decomposition,
     planning, execution, and continuous learning and reflection, rather
     than simple one-off inference responses.  Deploying such Agentic AI
     systems in a distributed environment (Cloud-Edge-Local) is essential
     for realizing truly autonomous and intelligent network management and
     service provision.  This document's considerations for distributed AI
     deployment are directly applicable to the foundational network
     infrastructure required for Agentic AI.

     In this contribution, in the following proposed network structure,
     the points to be considered in the environment where a client
     connects to a server and an edge device and requests an AI service
     are derived and described.  That is, the following considerations and
     options could be derived.

     *  AI inference service execution entity

     *  Hardware specifications of the machine to perform AI inference
        services

     *  Selection of AI models to perform AI inference services

     *  A method of providing AI services from cloud servers or edge
        devices

     *  Communication method to transmit data to request AI inference
        service



  Hong, et al.              Expires 23 April 2026                 [Page 4]


  Internet-Draft            Deploying AI services             October 2025


     The proposed considerations and items could be used to describe the
     use case of self-driving vehicles and network digital twins.  Since
     providing AI services in a distributed method can provide various
     advantages, it is desirable to apply it to self-driving vehicles and
     network digital twins.

  2.  Procedure to provide AI services

     Since research on AI services has been started for a long time, there
     may be shapes to provide various types of AI services.  However, due
     to the nature of AI technology, in general, a system for providing AI
     services consists of the following steps [AI_inference_archtecture]
     [Google_cloud_iot].

  +-----------+  +-----------+  +-----------+  +-----------+  +-----------+
  | Collect & |  | Analysis &|  |   Train   |  |  Deploy & |  | Monitor & |
  |  Store    |->| Preprocess|->|  AI model |->| Inference |->|  Maintain |
  |   data    |  |    data   |  |           |  |  AI model |  |  Accuracy |
  +-----------+  +-----------+  +-----------+  +-----------+  +-----------+
  |<--------->|  |<------------------------>|  |<--------->|  |<--------->|
    Sensor, DB              AI Server              Target       AI Srvr &
                                                   machine    Tgt. machine
  |<---------------->|<--------------------->|<-------------->|<--------->|
        Internet              Local                Internet      Local &
                                                                Internet

                      Figure 1: AI service workflow


     *  Data collection & Store

     *  Data Analysis & Preprocess

     *  AI Model Training

     *  AI Model Deploy & Inference

     *  Monitor & Maintain Accuracy

     In the data collection step, data required for training is prepared
     by collecting data from sensors and IoT devices or by using data
     stored in a database.  Equipment involved in this step includes
     sensors, IoT devices and servers that store them, and database
     servers.  Since the operations performed at this step are conducted
     through the Internet, many IoT technologies studied by the IETF so
     far have developed technologies suitable for this step.





  Hong, et al.              Expires 23 April 2026                 [Page 5]


  Internet-Draft            Deploying AI services             October 2025


     In the data analysis and pre-processing step, the features of the
     prepared data are analyzed and pre-processing for training is
     performed.  Equipment involved in this step includes a high-
     performance server equipped with a GPU and a database server, and is
     mainly performed in a local network.

     In the model training step, a training model is created by applying
     an algorithm suitable for the characteristics of the data and the
     problem to be solved.  Equipment involved in this step includes a
     high-performance server equipped with a GPU, and is mainly performed
     on a local network.

     In the model deploying and inference service provision step, the
     problem to be solved (e.g., classification, regression problem) is
     solved using AI technology.  Equipment involved in this step may
     include a target machine, a client, a cloud, etc. that provide AI
     services, and since various equipment is involved in this stage, it
     is conducted through the Internet.  This document summarizes the
     factors to be considered at this step.

     In the accuracy monitoring step, if the performance deteriorates due
     to new data, a new model is created through re-training, and the AI
     service quality is maintained by using the newly created model.  This
     step is the same as described in the model training, model deploying,
     and inference service provision steps described in the previous step
     because re-training and model deploying are performed again.

  3.  Network configuration structure to provide AI services

     In general, after training a AI model, the AI model can be built on a
     local machine for AI model deploying and inference services to
     provide AI services.  Alternatively, we can place AI models on cloud
     servers or edge devices and make AI service requests remotely.  In
     addition, for overall service performance, some AI service requests
     to the cloud server and some AI service requests to edge devices can
     be performed through appropriate load balancing.

  3.1.  AI inference service on Local machine

     The following figure shows a case where a client module requesting AI
     service on the same local machine requests AI service from an AI
     server module on the same machine.









  Hong, et al.              Expires 23 April 2026                 [Page 6]


  Internet-Draft            Deploying AI services             October 2025


   +---------------------------------------------------------------------+
   |                                                                     |
   |   +-----------------+        Request AI      +-----------------+    |
   |   |  Client module  |    Inference service   |  Server module  |    |
   |   | for AI service  |----------------------->| for AI service  |    |
   |   |                 |<-----------------------|                 |    |
   |   +-----------------+        Reply AI        +-----------------+    |
   |                           Inference result                          |
   +---------------------------------------------------------------------+
                                Local machine


              Figure 2: AI inference service on Local machine


     This method is often used when configuring a system focused on
     training AI models to improve the inference accuracy and performance
     of AI models without considering AI services or AI model deploying
     and inference in particular.  In this case, since the client module
     that requests the AI inference service and the AI server module that
     directly performs the AI inference service are on the same machine,
     it is not necessary to consider the communication/network environment
     or service provision method too much.  Alternatively, this method can
     be used when we want to simply decorate the AI inference service on
     one machine without changing the AI service in the future, such as an
     embedded machine or a customized machine.

     In this case, a high level of hardware performance is not required to
     train the AI model, but hardware performance sufficient to run the AI
     inference service is required, so it is possible on a machine with a
     certain amount of hardware performance.

  3.2.  AI inference service on Cloud server

     The following figure shows the case where the client module that
     requests AI service and the AI server module that directly performs
     AI service run on different machines.














  Hong, et al.              Expires 23 April 2026                 [Page 7]


  Internet-Draft            Deploying AI services             October 2025


                                    +--------------------------------------+
  +------------------------+        |     +---------------------------+    |
  |   +-----------------+  |        |     |     +-----------------+   |    |
  |   |   Client module |<-+--------+-----+---->|   Server module |   |    |
  |   |  for AI service |  |        |     |     |  for AI service |   |    |
  |   +-----------------+  |        |     |     +-----------------+   |    |
  +------------------------+        |     + --------------------------+    |
         Local machine              |             Server machine           |
                                    +--------------------------------------+
                                                 Cloud (Internet)


              Figure 3: AI inference service on Cloud server


     In this case, the client module requesting the AI inference service
     runs on the local machine, and the AI server module that directly
     performs the AI inference service runs on a separate server machine,
     and this server machine is in the cloud network.  In this case, the
     performance of the local machine does not need to be high because the
     local machine simply needs to request the AI inference service and,
     if necessary, deliver only the data required for the AI service
     request.  For the AI server module that directly performs AI
     inference service, we can set up our own AI server, or we can use
     commercial clouds such as Amazon, Microsoft, and Google.

  3.3.  AI inference service on Edge device

     The following figure shows the case where the client module that
     requests the AI service and the AI server module that directly
     performs the AI service are separated, and the AI server module is
     located on the edge device.  As more and more data is generated at
     the edge, it is necessary to perform AI processing at the edge due to
     performance, privacy, and cost of moving data to a central location,
     for example, some enterprises that own both local machine and edge
     network will not feel comfortable uploading their confidential
     datasets to the central cloud, instead these enterprises retrieve AI
     model from the cloud and use such AI model for AI inference locally
     at the edge device.












  Hong, et al.              Expires 23 April 2026                 [Page 8]


  Internet-Draft            Deploying AI services             October 2025


                                    +--------------------------------------+
  +------------------------+        |     +---------------------------+    |
  |   +-----------------+  |        |     |     +-----------------+   |    |
  |   |   Client module |<-+--------+-----+---->|   Server module |   |    |
  |   |  for AI service |  |        |     |     |  for AI service |   |    |
  |   +-----------------+  |        |     |     +-----------------+   |    |
  +------------------------+        |     + --------------------------+    |
         Local machine              |                Edge device           |
                                    +--------------------------------------+
                                                    Edge network


              Figure 4: AI inference service on Edge device


     Even in this case, the client module that requests the AI inference
     service runs on the local machine, the AI server module that directly
     performs the AI inference service runs on the edge device, and the
     edge device is in the edge network.  Even in this case, the client
     module that requests the AI inference service runs on the local
     machine, the AI server module that directly performs the AI inference
     service runs on the edge device, and the edge device is in the edge
     network.  The AI module that directly performs the AI inference
     service on the edge device can directly configure the edge device or
     use a commercial edge computing module.

     The difference from the above case where the AI server module is in
     the cloud is that the edge device is usually close to the client,
     whereas the performance is lower than that of the server in the
     cloud, so there are advantages in data transfer time and inference
     time, but in unit time Inference service performance is poor.

  3.4.  AI inference service on Cloud server and Edge device

     The following figure shows the case where AI server modules that
     directly perform AI services are distributed in the cloud and edge
     devices.














  Hong, et al.              Expires 23 April 2026                 [Page 9]


  Internet-Draft            Deploying AI services             October 2025


                                    +--------------------------------------+
  +------------------------+        |     +---------------------------+    |
  |   +-----------------+  |        |     |     +-----------------+   |    |
  |   |   Client module |<-+---+----+-----+---->|   Server module |   |    |
  |   |  for AI service |<-+---+    |     |     |  for AI service |   |    |
  |   +-----------------+  |   |    |     |     +-----------------+   |    |
  +------------------------+   |    |     + --------------------------+    |
         Local machine         |    |                Edge device           |
                               |    +--------------------------------------+
                               |                    Edge network
                               |
                               |    +--------------------------------------+
                               |    |     +---------------------------+    |
                               |    |     |     +-----------------+   |    |
                               +----+-----+---->|   Server module |   |    |
                                    |     |     |  for AI service |   |    |
                                    |     |     +-----------------+   |    |
                                    |     + --------------------------+    |
                                    |              Server machine          |
                                    +--------------------------------------+
                                                  Cloud (Internet)


      Figure 5: AI inference service on Cloud sever and Edge device


     There is a difference between the AI server module performed in the
     cloud and the AI server module performed on the edge device in terms
     of AI inference service performance.  Therefore, the client
     requesting the AI inference service may request by distributing the
     AI inference service request to the cloud and edge device
     appropriately in order to perform the desired AI service.  In other
     words, in the case of an AI service with low inference accuracy but
     short inference time, we can request an AI inference service to the
     edge device.

     The distribution of AI inference service requests between the cloud
     and edge devices is a critical decision.  In the context of Agentic
     AI, the AI client module itself evolves into an AI Agent Planning
     Module.  This module autonomously determines the optimal execution
     point (Local, Edge, or Cloud) based on a dynamic assessment of
     inference accuracy requirements, latency constraints (e.g., in-time/
     on-time delay sensitivity), and resource costs.  This autonomous
     decision-making process is a core characteristic of Agentic AI.







  Hong, et al.              Expires 23 April 2026                [Page 10]


  Internet-Draft            Deploying AI services             October 2025


  3.5.  AI inference service on horizontal multiple servers

     In the previous section, to provide AI inference service, the network
     configuration that consisted of local machines, edge devices, and
     cloud servers is a kind of vertical hierarchy.  Because the
     capabilities of each machine are different, the overall performance
     of the network using vertical hierarchy is dependent of each machine.
     Generally, a cloud server has a most powerful performance and then an
     edge device has the second powerful performance.

     In this network configuration, AI service may have different
     performance according to the load level of the server, computing
     capability of the server machine and link-state between the local
     machine and the server machines of the horizontal level.  Thus, to
     look for the server machine that can support the best AI service, it
     is necessary for the network element that can monitor network link-
     state and current state of the computing capability of the server
     machines and the network load-balance that can perform a scheduling
     policy of load balancing.  Therefore, when a client requests an AI
     inference service, those requests are shared between edge devices and
     cloud servers, and routed to a single cloud server machine in the
     network based on the load balancer's decision.  Alternatively, these
     requests are processed by all the Cloud Server machines that
     contribute to the final inference result.

     When utilizing horizontal multiple servers, the integration of Multi-
     Agent Systems (MAS) based on Agentic AI becomes crucial.  Instead of
     relying solely on a centralized load balancer, multiple AI agents,
     each deployed on a separate server machine (or edge device), can
     autonomously negotiate tasks and collaborate toward a common
     objective.  This multi-agent collaboration structure allows for the
     decomposition of a complex AI task into sub-tasks, with each agent
     specializing in a part, leading to a collectively processed final
     inference result and improved resilience and dynamic resource
     optimization.

     The following figure shows the case where the local machine that
     requests AI service to horizontal multiple cloud servers.













  Hong, et al.              Expires 23 April 2026                [Page 11]


  Internet-Draft            Deploying AI services             October 2025


                                    +--------------------------------------+
                                    |     +---------------------------+    |
                                    |     |     +-----------------+   |    |
                               +----+-----+---->|   Server module |   |    |
                               |    |     |     |  for AI service |   |    |
                               |    |     |     +-----------------+   |    |
                               |    |     + --------------------------+    |
                               |    |              Server machine 1        |
                               |    +--------------------------------------+
                               |                  Cloud (Internet)
                               |
                               |    +--------------------------------------+
  +------------------------+   |    |     +---------------------------+    |
  |   +-----------------+  |   |    |     |     +-----------------+   |    |
  |   |   Client module |<-+---+----+-----+---->|   Server module |   |    |
  |   |  for AI service |<-+---+    |     |     |  for AI service |   |    |
  |   +-----------------+  |   |    |     |     +-----------------+   |    |
  +------------------------+   |    |     + --------------------------+    |
         Local machine         |    |              Server machine 2        |
                               |    +--------------------------------------+
                               |                  Cloud (Internet)
                               |
                               |    +--------------------------------------+
                               |    |     +---------------------------+    |
                               |    |     |     +-----------------+   |    |
                               +----+-----+---->|   Server module |   |    |
                                    |     |     |  for AI service |   |    |
                                    |     |     +-----------------+   |    |
                                    |     + --------------------------+    |
                                    |              Server machine 3        |
                                    +--------------------------------------+
                                                  Cloud (Internet)


      Figure 6: AI inference service on horizontal multiple servers


  3.6.  Network-side utilization for AI learning

     Collecting and preprocessing of data and training an AI model
     requires a high-performance resource such as CPU, GPU, Power, and
     Storage.  To mitigate this requirement, we can utilize a network-side
     configuration.  Typically, federating learning is a machine learning
     technique that trains an AI model across multiple decentralized
     servers.  It is a contrast to traditional centralized machine
     learning techniques where all the local datasets are uploaded to one
     server.  In this federated learning, it enables multiple network
     nodes to build a common machine learning model.  These network nodes



  Hong, et al.              Expires 23 April 2026                [Page 12]


  Internet-Draft            Deploying AI services             October 2025


     or servers can be located in the same data center network or multiple
     data center network.

     And, transfer learning is a machine learning technique that focuses
     on storing information gained while solving one problem and applying
     it to a different but related problem.  In this transfer learning, we
     can utilize a network configuration to transfer common information
     and knowledge between different network nodes within one data center
     network or across multiple data center networks.

  4.  Considerations of network/system for AI services

     As described in the previous chapter, the AI server module that
     directly performs AI inference services by utilizing AI models can be
     performed on a local machine or a cloud server or an edge device.

     In theory, if AI inference service is performed on a local machine,
     AI service can be provided without communication delay time or packet
     loss, but a certain amount of hardware performance is required to
     perform AI service inference.  So, in the future environment where AI
     services become popular, such as when various AI services are
     activated and AI services are disseminated, the cost of a machine
     that performs AI services is important

     If so, whether the AI inference service will be performed on the
     cloud server or the discount price on the edge device can be a
     determining factor in the system configuration.

  4.1.  Considerations of the functional characteristics of the hardware

     When AI inference service request is made to a distant cloud server,
     it may take a lot of time to transmit, but it has the advantage of
     being able to perform many AI inference service requests in a short
     time, and the accuracy of AI service inference increases.
     Conversely, when an AI service request is made to a nearby edge
     device, the transmission time is short, but many AI inference service
     requests cannot be performed at once, and the accuracy of AI service
     inference is lowered.

     Therefore, by analyzing the characteristics and requirements of the
     AI service to be performed, it is necessary to determine where to
     perform the AI inference service on a local machine, a cloud server,
     or an edge device.

     The hardware characteristics of the machine performing the AI service
     varies.  In general, machines on cloud servers are viewed as machines
     with higher performance than edge devices.  However, the performance
     of AI inference service varies depending on how the hardware such as



  Hong, et al.              Expires 23 April 2026                [Page 13]


  Internet-Draft            Deploying AI services             October 2025


     CPU, RAM, GPU, and network interface is configured for each cloud
     server and edge device.  If we do not think about cost, it is good to
     configure a system for performing AI services with a machine with the
     best hardware performance, but in reality, we should always consider
     the cost when configuring the system.  So, according to the
     characteristics and requirements of the AI service to be performed,
     the performance of the local machine, cloud server, and edge device
     must be determined.

     Performance evaluation is possible through the performance matrix
     presented in the standard of ETSI[MEC.IEG006].  The performance
     metrics suggested by the ETSI standard are as follows.  These metrics
     is divided into two groups, namely Functional metrics, which assess
     the user performance and include some classical indexes such as
     latency in task execution, device energy efficiency, bit-rate, loss
     rate, jitter, Quality of Service (QoS), etc.; and Non-functional
     metrics that, instead, focus on the MEC(Mobile Edge Computing)
     network deployment and management.  Non-functional metrics include
     the following indexes.  Service life-cycle(instantiation, service
     deployment, service provisioning, service update (e.g. service
     scalability and elasticity), service disposal), service availability
     and fault tolerance (aka reliability), service processing/
     computational load, global mobile equipment host load, number of API
     request (more generally number of events) processed/second on mobile
     equipment host, delay to process API request (north and south),
     number of failed API request.  The sum of service instantiation,
     service deployment, and service provisioning provide service boot-
     time.

  4.2.  Considerations for the characteristics of the AI model

     According to the characteristics of the AI service, although not
     directly related to communication/network, the biggest influence on
     AI inference services is the AI model to be used for AI inference
     service.  For example, in AI services such as image classification,
     there are various types of AI models such as ResNet, EfficientNet,
     VGG, and Inception.  These AI models differ in AI inference accuracy,
     but also in AI model file size and AI inference time.  AI models with
     the highest inference accuracy typically have very large file sizes
     and take a lot of AI inference time.  So, when constructing an AI
     service system, it is not always good to choose an AI model with the
     highest AI inference accuracy.  Again, it is important to select an
     AI model according to the characteristics and requirements of the AI
     service to be performed.







  Hong, et al.              Expires 23 April 2026                [Page 14]


  Internet-Draft            Deploying AI services             October 2025


     Experimentally, it is recommended to use an AI model with high AI
     inference accuracy in the cloud server, and use an AI model that can
     provide fast AI inference service although the AI inference accuracy
     is slightly lower for the fast AI inference service in the edge
     device.

     It might be a bit of an implementation issue, but we should also
     consider how we deliver AI services on cloud servers or edge devices.
     With the current technology, a traditional web server method or a
     server method specialized for AI service inference (e.g., Google's
     Tensorflow Serving) can be used.  Traditional web server methods such
     as Flask and Django have the advantage of running on various types of
     machines, but since they are designed to support general web
     services, the service execution time is not fast.  Tensorflow Serving
     uses the features of Tensorflow to make AI service inference services
     very fast and efficient.  However, older CPUs that do not support AVX
     cannot use the Tensorflow serving function because Google's
     Tensorflow does not run.  Therefore, rather than unconditionally
     using the server method specialized in AI service inference, it is
     necessary to decide the AI server module method that provides AI
     services in consideration of the hardware characteristics of the AI
     system that can be built.

  4.3.  Considerations for the characteristics of the communication method

     The communication method for transferring data to request AI
     inference service is also an important decision in constructing an AI
     system.  Using the traditional REST method, it can be used for
     various machines and services, but its performance is inferior to
     gRPC.  There are many advantages to using gRPC for AI inference
     services, as gRPC uses HTTP/2 and protocol buffers for communication,
     providing low latency and efficient data serialization, and enabling
     large capacity and efficient data transfer compared to REST.
     Alternatively, the QUIC protocol or other future transport protocol
     can also be used to request an AI inference service.  Which transport
     protocol is used is beyond the scope of this document.

     Cloud-edge-endpoint collaboration-based AI service development is
     actively underway.  In particular, in the case of AI services that
     are sensitive to network delays, such as object recognition and
     autonomous vehicle services, (micro)services for inference are placed
     on edge devices close to the client to obtain fast inference results
     and provide services.  As such, in the development of intelligent IoT
     services, various devices that can provide computing services within
     the network, such as edge devices, are being added as network
     elements, and the number of IoT devices using them is rapidly
     increasing.  Therefore, a new function for computing resource
     management and operation is required in terms of providing computing



  Hong, et al.              Expires 23 April 2026                [Page 15]


  Internet-Draft            Deploying AI services             October 2025


     services within the network.  In addition, to operate distributed AI
     service on network, the network policy for collaboration is required
     between edge devices or between edge device and endpoint device that
     support computing resource for AI service.

     In network policy, in order to efficiently support distributed AI
     services, existing networks must be provided with the collaboration
     of AI service between edge devices such as multi-edge network
     configuration for AI service aware traffic steering in multi-edge
     network, so that the client can receive distributed AI service
     efficiently in various network environments and the multi-edge
     network configuration enables dynamically vary network resource and
     the computing resource of edge device.  For example, AI tasks message
     exchanges must be possible between edge devices or between edge
     devices and endpoint devices to provide collective AI inference.
     Another example is to place caching at the edge devices between the
     client and the central cloud to allow make a copy of an accessed/
     requested file at the edge devices, which are then served for
     subsequent requests from the same client or other clients.  Also,
     there are various delay sensitive AI services based on edge device in
     the network.  They are divided into in-time delay sensitive AI
     services with a deadline limit and on-time delay sensitive AI
     services with the set of a time-range.  In particular on-time delay
     sensitive AI services want to return the results of prediction within
     the time range.  Therefore, distributed AI service must be able to
     provide proper AI service in terms of the delay performance for
     distributed AI service through network.  Therefore, the client of AI
     service should be able to be provided both in-time and on-time delay
     sensitive services and be interacted with edge devices where
     distributed AI service is built.

  5.  Addressing challenges for coupling AI and NM

     The document [I-D.irtf-nmrg-ai-challenges] describes the challenges
     found in the application of AI to network management problems.  They
     are separated into low-level and high-level challenges.  This section
     describes how the concepts discussed in the present document are
     linked to those challenges.

     Some of the links between the structures presented in this document
     and AI challenges involve the alignment with AINEMA
     [I-D.pedro-nmrg-ai-framework].









  Hong, et al.              Expires 23 April 2026                [Page 16]


  Internet-Draft            Deploying AI services             October 2025


     AINEMA [I-D.pedro-nmrg-ai-framework] is a framework of functions
     required for exploiting AI services and their interconnection.
     Intelligent management systems based on AINEMA, as shown in
     [TNSM-2018], introduce AI functions in the network management cycles,
     and are able to express management decisions in terms of network
     intents, as described in [I-D.pedro-ite].

  5.1.  Low-level challenges

     The first challenge (C1) concerns to the combinatorial explosion of
     the solution space in relation to the size of the problem.  The
     present document proposes to tackle part of such complexity by
     separating the AI work in multiple elements of the network.  The
     separation is asymmetrical, so that some elements (cloud side) will
     contribute more computation power to the distributed service.

     The second challenge (C2) regards the dimensional and context
     uncertainty and unpredictability.  The present document focus on
     cloistering the AI models and data within pre-defined client-edge-
     cloud-server structures.

     The third challenge (C3) is to ensure that AI to is able to provide
     prompt responses and decisions to management questions.  The key
     aspect discussed in this document to resolve this challenge is the
     integration of end-node, in-network, edge, an cloud computing
     services.  This allows AI computations to be performed in the best
     place possible.  Time elapsed to get fast responses of easy
     operations will be minimized by executing them in the end-nodes
     (e.g., vehicles) and edge devices, while the time needed to compute
     more complex operations is minimized by offloading them to cloud
     computing services.

     The fourth challenge (C4) states the difficulties related to
     resolving data imperfections and scalability of techniques.  This
     challenge is specific to each information domain and problem.
     However, as presented in this document, the mentioned difficulties in
     C4 is relieved by exposing AINEMA [I-D.pedro-nmrg-ai-framework]
     functions instantiated in cloud continuum to the AI services, so that
     AI services transparently gain capabilities such as homogenization
     and scalability.  Instantiating a distributed AI system in a cloud
     continuum enables the AI system to have many functions to deal with
     data homogenization, resolving high data rates, etc.









  Hong, et al.              Expires 23 April 2026                [Page 17]


  Internet-Draft            Deploying AI services             October 2025


     The fifth challenge (C5) concerns the integration of AI services with
     existing automation and human processes.  The flexibility of the
     structures presented in this document allow them to be connected to
     existing systems, being aligned and somewhat interconnected with
     AINEMA [I-D.pedro-nmrg-ai-framework].  The result is exemplified in
     the support for network digital twin and vehicle environments.

     The sixth challenge (C6) exposes the need for cost-effective
     solutions.  Enabling AI services to be deployed in the cloud
     continuum supports the maximization of effectiveness by dynamically
     relocating services to the cheapest provider point that is able to
     accomplish the computation requirements.  This relocation is guided
     by intents, as described in [I-D.pedro-ite], and implemented by an
     independent management cycle, as described in AINEMA
     [I-D.pedro-nmrg-ai-framework] and [TNSM-2018].  The result is that
     the cost of a distributed AI system is continuously being checked and
     adjusted by requesting the underlying infrastructure to modify the
     point of instantiation of each part of the AI system.

  5.2.  High-level challenges

     The first challenge (H1) relies in the observation that AI techniques
     were developed in a different area---imaging---, so they must be
     extensively tailored for network problems.  Although this is a quite
     complex challenge, this document supports its resolution by enabling
     AI models and algorithms to evolve separately.  Multiple providers
     will offer multiple solutions and they will be stitched together by
     following the structures introduced above, as well as the functions
     provided by the interconnection and alignment with AINEMA
     [I-D.pedro-nmrg-ai-framework].

     The second challenge (H2) conveys the mismatch that exists from the
     original data and internal data used in AI models.  Although the
     present document does cover this mismatch, the structures presented
     in this document help alleviate the burden by relocating some AI
     processes as close as possible to the end-points (e.g., vehicles and
     NDT).  The same structures support the implementation of measures to
     protect privacy.  The AI services that are closer to the edges will
     deal with sensitive data and ensure privacy protection by, for
     example, anonymizing it.  Those services are instantiated within the
     data domain, so private data does not crosses the boundaries of its
     data domain.

     The third challenge (H3) consists on the level of acceptance that an
     AI system experiences from administrators and operators.  It is
     agreed that giving full control of AI operations to administrators
     and operators increases such level of acceptance.  The structures
     presented in this document support the involvement of administrators



  Hong, et al.              Expires 23 April 2026                [Page 18]


  Internet-Draft            Deploying AI services             October 2025


     and operators in AI system processes through the provision of
     policies and network intents.  On the one hand, aligning the
     distributed AI system specified in this document with AINEMA
     [I-D.pedro-nmrg-ai-framework], enables the enforcement high-level
     policies and management goals provided by administrators and
     operators.  On the other hand, aligning the distributed AI system
     specified in this document with intent-based networking systems,
     particularly intent translation [I-D.pedro-ite], enables
     administrators and operators to use high-level specifications (namely
     network intents) to communicate the policies, requirements, and goals
     that must be enforced to the AI system constructed over the cloud
     continuum.

  6.  Use cases of deploying network-based AI services

  6.1.  Deploying AI services for self-driving vehicles

     Various sensors are used in self-driving vehicles, and the final
     judgment is made by combining these data.  Among them, camera data-
     based object detection solves parts that expensive equipment such as
     LiDAR and RADAR cannot solve.  Camera-based object detection performs
     various tasks, and in addition to lane recognition for maintaining
     driving lanes and changing lanes, it also supports safe driving and
     parking assistance by distinguishing shape information such as
     pedestrians, signs, and parking vehicles along the road.

     In order to perform such driving assistance and autonomous driving,
     object detection needs to be performed in real time.  The minimum
     FPS(Frames Per Second) to be considered real-time in autonomous
     driving is 30 FPS[Object_detection].  No matter how high the accuracy
     is, it cannot be used for autonomous driving if it does not meet the
     corresponding reference value.

     Task offloading refers to a technology or structure that transfers
     computing tasks to other processing devices or systems to perform
     them.  Task offloading can quickly process tasks that exceed the
     performance limits of devices that lack resources by delivering tasks
     from devices with limited computing power, storage space, and power
     to devices that are rich in computing resources.












  Hong, et al.              Expires 23 April 2026                [Page 19]


  Internet-Draft            Deploying AI services             October 2025


     For devices with low hardware performance (e.g., NVIDIA Jetson Nano
     board, Qual-core ARM A57, 4GB RAM), all locally without task
     offloading results in 4.6 FPS, which is difficult to perform object
     detection-based autonomous driving.  On the other hand, if task
     offloading is applied to perform object detection on devices with
     high hardware performance (e.g., Intel i7, RTX 3060, 32GB RAM) and
     the rest of the work is performed on the client, 41.8 FPS will be
     obtained.  This is a result that satisfies 30 FPS, which is the
     reference FPS of object detection-based autonomous driving.

     In the case of AI services such as object detection, if it is
     difficult to perform on resource-constrained devices, it can be seen
     that the task offloading structure shows some efficiency.  However,
     without performing all operations locally, task offloading operations
     between network nodes can affect the entire time because the larger
     the size of the data, the greater the communication latency.
     Therefore, in such a network distributed environment, the provision
     of AI services should be designed in consideration of various
     variables.  The Figure 7 shows an example of distributed AI
     deployment in a self-driving vehicle when it does not have enough
     capabilities to proceed the object detection operation in real-time
     and it asks some tasks to edge devices and cloud servers.


                                    +--------------------------------------+
  +------------------------+        |     +---------------------------+    |
  |   +-----------------+  |        |     |     +-----------------+   |    |
  |   |Object detection |<-+---+----+-----+---->|Object detection |   |    |
  |   |     service     |<-+---+    |     |     |     service     |   |    |
  |   +-----------------+  |   |    |     |     +-----------------+   |    |
  +------------------------+   |    |     + --------------------------+    |
           Vehicle             |    |                Edge device           |
                               |    +--------------------------------------+
                               |                    Edge network
                               |
                               |    +--------------------------------------+
                               |    |     +---------------------------+    |
                               |    |     |     +-----------------+   |    |
                               +----+-----+---->|Object detection |   |    |
                                    |     |     |     service     |   |    |
                                    |     |     +-----------------+   |    |
                                    |     + --------------------------+    |
                                    |              Server machine          |
                                    +--------------------------------------+
                                                  Cloud (Internet)






  Hong, et al.              Expires 23 April 2026                [Page 20]


  Internet-Draft            Deploying AI services             October 2025


      Figure 7: Distributed object detection service in self-driving
                                 vehicle


  6.2.  Deploying AI services for network digital twins

     Network digital twin also need to build distributed AI services.  The
     purpose of a network digital twin is described in
     [I-D.irtf-nmrg-network-digital-twin-arch].  In particular, the
     network digital twin provides network operators with technology that
     enables data driven network management and allows real time
     interaction between physical network and twin network.  To achieve
     this, the network digital twin will use AI capabilities for various
     purposes such as scenario planning, impact analysis and change
     management.

     Various AI functions will be applied for optimal network operation
     and management.  However, the actual physical network consists of
     various different network devices and has a complex structure for
     various different type of data such as topology data, configuration
     data, state data.  In addition, in a large-scale network environment,
     the network overhead is very large and expensive to collect and store
     information from many network devices in a centralized manner, and to
     create and change network management policies based on it.

     Therefore, there is a need for a method to apply AI functions based
     on a distributed form for network operation and management.  In
     particular, the actual physical network structure is built in a
     logical hierarchical structure.  Therefore, it is necessary to apply
     a distributed AI method that considers the logical hierarchical
     network structure environment.

     In order to optimally perform network operation and management
     through distributed AI methods, it is necessary to generate AI
     function-based network operation and management topology model and
     policy models and an operational method to distribute the generated
     AI function-based network policies across different networks or
     administrative domains.  In particular, in order to operate a network
     digital twin in a large-scale network environment, it is necessary to
     generate AI-based network policy models in a distributed manner.  A
     federated learning algorithm or a transfer learning algorithm that
     can learn large-scale networks in a distributed manner can be
     applied.








  Hong, et al.              Expires 23 April 2026                [Page 21]


  Internet-Draft            Deploying AI services             October 2025


            +-----------------------------------------------------+
            |                                                     |
            |        Distributed network learning model           |
            |        in large-scale network environment           |
            |                                                     |
            |            +-------------+------------+             |
            |            |          Master          |             |
            |            | (AI based Policy model)  |             |
            |            +-------------+------------+             |
            |                          |                          |
            |        +-----------+-----+-----+-----------+        |
            |        |           |           |           |        |
            |   +----+----+ +----+----+ +----+----+ +----+----+   |
            |   |  Worker | |  Worker | |  Worker | |  Worker |   |
            |   | (Agent) | | (Agent) | | (Agent) | | (Agent) |   |
            |   +----+----+ +----+----+ +----+----+ +----+----+   |
            |        |           |           |           |        |
            |   +----+----+ +----+----+ +----+----+ +----+----+   |
            |   |  Local  | |  Local  | |  Local  | |  Local  |   |
            |   |  Data   | |  Data   | |  Data   | |  Data   |   |
            |   |  Repo-  | |  Repo-  | |  Repo-  | |  Repo-  |   |
            |   |  sitory | |  sitory | |  sitory | |  sitory |   |
            |   +---------+ +---------+ +---------+ +---------+   |
            |                                                     |
            +-----------------------------------------------------+


          Figure 8: Distributed learning model of network learning for
                             network digital twins


     As shown in Figure 8, in order to learn a large-scale network through
     a distributed learning method, a local data repository to store
     network data must be established in each region, for example, based
     on location or AS (Autonomous System).  Therefore, the distributed
     learning method learns through each worker (agent) based on the local
     network data stored in the local network data repository, and
     generates a large-scale network policy model through the master.
     This distributed learning method can reduce the network overhead of
     centralized data collection and storage, and reduce the time required
     to create AI models for network operation and management policies for
     large-scale networks.  In addition, the network policy model
     generated by the worker can be used as a locally optimized network
     policy model to provide AI-based network operation and management
     policy services optimized for local network operations.






  Hong, et al.              Expires 23 April 2026                [Page 22]


  Internet-Draft            Deploying AI services             October 2025


     The distributed deployment of trained AI network policy models can be
     deployed on network devices that can manage and operate the local
     network to minimize network data movement.  For example, in a large-
     scale network consisting of multiple ASes, AI network policy models
     can be deployed per AS to optimize network operation and management.
     Figure 9 shows an example of operating and managing a network by
     distributing AI network policy models by AS.


          +---------------------------------------------------------+
          |              +-------------+------------+               |
          |              |          Master          |               |
          |              | (AI-based Network Policy |               |
          |              |     model management)    |               |
          |              +-------------+------------+               |
          |                            |                            |
          |              +-------------+-------------+              |
          |     Worker   |                 Worker    |              |
          |   +----------+----------+     +----------+----------+   |
          |   |    Network device   |     |    Network device   |   |
          |   |  (AI-based network  |     |  (AI-based network  |   |
          |   |     Policy model)   |     |     Policy model)   |   |
          |   +----------+----------+     +----------+----------+   |
          |              |                           |              |
          |   +----------+----------+     +----------+----------+   |
          |   |        AS_1         |     |        AS_2         |   |
          |   +---------------------+     +---------------------+   |
          +---------------------------------------------------------+


      Figure 9: Distributed deployment of trained AI network policy models


  7.  IANA Considerations

     There are no IANA considerations related to this document.

  8.  Security Considerations

     When AI service is performed on a local machine, there is no security
     issue, but when AI service is provided through a cloud server or edge
     device, IP address and port number may be known to the outside can
     attack.  Therefore, when providing AI services by utilizing machines
     on the network such as cloud servers and edge devices, it is
     necessary to analyze the characteristics of the modules to be used
     well, identify vulnerabilities in security, and take countermeasures.





  Hong, et al.              Expires 23 April 2026                [Page 23]


  Internet-Draft            Deploying AI services             October 2025


  9.  Acknowledgements

     TBA

  10.  References

  10.1.  Normative References

     [RFC6574]  Tschofenig, H. and J. Arkko, "Report from the Smart Object
                Workshop", RFC 6574, DOI 10.17487/RFC6574, April 2012,
                <https://www.rfc-editor.org/info/rfc6574>.

     [RFC7452]  Tschofenig, H., Arkko, J., Thaler, D., and D. McPherson,
                "Architectural Considerations in Smart Object Networking",
                RFC 7452, DOI 10.17487/RFC7452, March 2015,
                <https://www.rfc-editor.org/info/rfc7452>.

     [RFC9556]  Hong, J., Hong, Y., de Foy, X., Kovatsch, M., Schooler,
                E., and D. Kutscher, "Internet of Things (IoT) Edge
                Challenges and Functions", RFC 9556, DOI 10.17487/RFC9556,
                April 2024, <https://www.rfc-editor.org/info/rfc9556>.

  10.2.  Informative References

     [I-D.irtf-nmrg-network-digital-twin-arch]
                Zhou, C., Yang, H., Duan, X., Lopez, D., Pastor, A., Wu,
                Q., Boucadair, M., and C. Jacquenet, "Network Digital
                Twin: Concepts and Reference Architecture", Work in
                Progress, Internet-Draft, draft-irtf-nmrg-network-digital-
                twin-arch-11, 6 July 2025,
                <https://datatracker.ietf.org/doc/html/draft-irtf-nmrg-
                network-digital-twin-arch-11>.

     [I-D.irtf-nmrg-ai-challenges]
                François, J., Clemm, A., Papadimitriou, D., Fernandes, S.,
                and S. Schneider, "Research Challenges in Coupling
                Artificial Intelligence and Network Management", Work in
                Progress, Internet-Draft, draft-irtf-nmrg-ai-challenges-
                05, 18 March 2025, <https://datatracker.ietf.org/doc/html/
                draft-irtf-nmrg-ai-challenges-05>.

     [I-D.pedro-nmrg-ai-framework]
                Martinez-Julia, P., Homma, S., and D. Lopez, "Artificial
                Intelligence Framework for Network Management", Work in
                Progress, Internet-Draft, draft-pedro-nmrg-ai-framework-
                05, 20 October 2024,
                <https://datatracker.ietf.org/doc/html/draft-pedro-nmrg-
                ai-framework-05>.



  Hong, et al.              Expires 23 April 2026                [Page 24]


  Internet-Draft            Deploying AI services             October 2025


     [I-D.pedro-ite]
                Martinez-Julia, P. and J. P. Jeong, "Intent Translation
                Engine for Intent-Based Networking", Work in Progress,
                Internet-Draft, draft-pedro-ite-01, 4 March 2024,
                <https://datatracker.ietf.org/doc/html/draft-pedro-ite-
                01>.

     [CG-AIoT]  "ITU-T CG-AIoT", <https://www.itu.int/en/ITU-T/
                studygroups/2017-2020/20/Pages/ifa-structure.aspx>.

     [tinyML]   "tinyML Foundation", <https://www.tinyml.org/>.

     [AI_inference_archtecture]
                "IBM Systems, AI Infrastructure Reference Architecture",
                <https://www.ibm.com/downloads/cas/W1JQBNJV>.

     [Google_cloud_iot]
                "Bringing intelligence to the edge with Cloud IoT",
                <https://cloud.google.com/blog/products/gcp/bringing-
                intelligence-edge-cloud-iot>.

     [MEC.IEG006]
                ETSI, "Mobile Edge Computing; Market Acceleration; MEC
                Metrics Best Practice and Guidelines", Group
                Specification ETSI GS MEC-IEG 006 V1.1.1 (2017-01),
                January 2017.

     [Object_detection]
                Lewis, "Object Detection for Autonomous Vehicles Gene",
                2016.

     [TNSM-2018]
                P. Martinez-Julia, V. P. Kafle, and H. Harai, "Exploiting
                External Events for Resource Adaptation in Virtual
                Computer and Network Systems, in IEEE Transactions on
                Network and Service Management. Vol. 15, n. 2, pp. 555--
                566, 2018.", 2018.

  Authors' Addresses

     Yong-Geun Hong
     Daejeon University
     62 Daehak-ro, Dong-gu
     Daejeon
     34520
     South Korea
     Phone: +82 42 280 4841
     Email: yonggeun.hong@gmail.com



  Hong, et al.              Expires 23 April 2026                [Page 25]


  Internet-Draft            Deploying AI services             October 2025


     Joo-Sang Youn
     DONG-EUI University
     176 Eomgwangno Busan_jin_gu
     Busan
     614-714
     South Korea
     Phone: +82 51 890 1993
     Email: joosang.youn@gmail.com


     Seung-Woo Hong
     ETRI
     218 Gajeong-ro Yuseong-gu
     Daejeon
     34129
     South Korea
     Phone: +82 42 860 1041
     Email: swhong@etri.re.kr


     Pedro Martinez-Julia
     National Institute of Information and Communications Technology
     4-2-1, Nukui-Kitamachi, Koganei, Tokyo
     184-8795
     Japan
     Phone: +81 42 327 7293
     Email: pedro@nict.go.jp


     Qin Wu
     Huawei
     101 Software Avenue, Yuhua District
     Nanjing
     210012
     China
     Email: bill.wu@huawei.com















  Hong, et al.              Expires 23 April 2026                [Page 26]


name: draft-dunbar-neotec-ac-te-applicability
rev: 04
title: Applying Attachmet Circuit and Traffic Engineering YANG Data Model to Edge AI Use Case
time: 2025-06-23T05:44:13Z
group_uri: /api/v1/group/group/1027/
expires: 2025-12-25T05:44:13Z
states: ["/api/v1/doc/state/4/", "/api/v1/doc/state/150/"]
categories: []
tags: []
fetched_at: 2026-03-07T06:55:18.623780+00:00
source: ietf
abstract:
  This document explores how existing IETF YANG data models,
     specifically the Attachment Circuit (AC)-as-a-Service (ACaaS) and
     Traffic Engineering (TE) topology data models, can be applied to
     support a use case involving dynamic AI model placement at Edge Cloud
     sites.  The use case involves selecting optimal Edge locations for
     real-time AI inference based on end-to-end network performance
     between street-level cameras and Edge Cloud compute nodes.  By
     mapping the use case across multiple network segments and applying
     relevant YANG data models to retrieve and request specific services
     objectives such as bandwidth, latency, and reliability, this document
     serves as a practical exercise to evaluate model applicability and
     identify gaps, if any.

full_text:
  NeoTec                                                    L. Dunbar, Ed.
  Internet-Draft                                                 Futurewei
  Intended status: Informational                                    Q. Sun
  Expires: 24 December 2025                                  China Telecom
                                                                B. Wu, Ed.
                                                                    Huawei
                                                  L. CONTRERASMURILLO, Ed.
                                                               Telefornica
                                                                    C. Xie
                                                             China Telecom
                                                              22 June 2025


   Applying Attachmet Circuit and Traffic Engineering YANG Data Model to
                              Edge AI Use Case
                 draft-dunbar-neotec-ac-te-applicability-04

  Abstract

     This document explores how existing IETF YANG data models,
     specifically the Attachment Circuit (AC)-as-a-Service (ACaaS) and
     Traffic Engineering (TE) topology data models, can be applied to
     support a use case involving dynamic AI model placement at Edge Cloud
     sites.  The use case involves selecting optimal Edge locations for
     real-time AI inference based on end-to-end network performance
     between street-level cameras and Edge Cloud compute nodes.  By
     mapping the use case across multiple network segments and applying
     relevant YANG data models to retrieve and request specific services
     objectives such as bandwidth, latency, and reliability, this document
     serves as a practical exercise to evaluate model applicability and
     identify gaps, if any.

  Status of This Memo

     This Internet-Draft is submitted in full conformance with the
     provisions of BCP 78 and BCP 79.

     Internet-Drafts are working documents of the Internet Engineering
     Task Force (IETF).  Note that other groups may also distribute
     working documents as Internet-Drafts.  The list of current Internet-
     Drafts is at https://datatracker.ietf.org/drafts/current/.

     Internet-Drafts are draft documents valid for a maximum of six months
     and may be updated, replaced, or obsoleted by other documents at any
     time.  It is inappropriate to use Internet-Drafts as reference
     material or to cite them other than as "work in progress."

     This Internet-Draft will expire on 24 December 2025.



  Dunbar, et al.          Expires 24 December 2025                [Page 1]


  Internet-Draft                 Neotec API                      June 2025


  Copyright Notice

     Copyright (c) 2025 IETF Trust and the persons identified as the
     document authors.  All rights reserved.

     This document is subject to BCP 78 and the IETF Trust's Legal
     Provisions Relating to IETF Documents (https://trustee.ietf.org/
     license-info) in effect on the date of publication of this document.
     Please review these documents carefully, as they describe your rights
     and restrictions with respect to this document.  Code Components
     extracted from this document must include Revised BSD License text as
     described in Section 4.e of the Trust Legal Provisions and are
     provided without warranty as described in the Revised BSD License.

  Table of Contents

     1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.  Conventions used in this document . . . . . . . . . . . . . .   4
     3.  YANG Data Models and APIs to Support Dynamic AI Model
             Placement . . . . . . . . . . . . . . . . . . . . . . . .   5
       3.1.  Using AC YANG Data Model to Evaluate Access and Edge
             Connectivity  . . . . . . . . . . . . . . . . . . . . . .   9
       3.2.  Using IETF YANG Data Models to Evaluate PE to PE
             Connectivity  . . . . . . . . . . . . . . . . . . . . . .  10
       3.3.  Non-TE-Based PE to PE Connectivity  . . . . . . . . . . .  13
     4.  Potential APIs for Kubernetes to Query Network Conditions . .  13
       4.1.  Behavior and Semantics  . . . . . . . . . . . . . . . . .  14
       4.2.  YANG Model Integration  . . . . . . . . . . . . . . . . .  15
     5.  Dynamic UCMP Load Balancing for Periodic Inter Site AI
             Traffic . . . . . . . . . . . . . . . . . . . . . . . . .  15
       5.1.  Edge Cloud Site to Local PE (Access Segment)  . . . . . .  15
         5.1.1.  Time-Scoped UCMP Policy Enforcement . . . . . . . . .  19
         5.1.2.  UCMP Enforcement for SRv6 based PE to PE segment  . .  20
       5.2.  Cloud-Initiated UCMP Activation API . . . . . . . . . . .  21
     6.  Gaps Between Existing IETF Specifications and Neotec Use Case
             Needs . . . . . . . . . . . . . . . . . . . . . . . . . .  21
       6.1.  Consumption Model Mismatch  . . . . . . . . . . . . . . .  22
       6.2.  Lack of Time-Bounded, Cloud-Triggered Policy
             Interfaces  . . . . . . . . . . . . . . . . . . . . . . .  22
       6.3.  Absence of Intent and Workload-to-Flow Mapping  . . . . .  22
       6.4.  Lack of Abstracted, Queryable Interfaces  . . . . . . . .  22
       6.5.  Feedback and Acknowledgment Gaps  . . . . . . . . . . . .  23
       6.6.  Security and Access Control Framework Alignment . . . . .  23
     7.  Need for Technology-Agnostic Modeling and Justification for a
             Separate Neotec WG  . . . . . . . . . . . . . . . . . . .  23
     8.  Security Considerations . . . . . . . . . . . . . . . . . . .  24
     9.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  24
     10. References  . . . . . . . . . . . . . . . . . . . . . . . . .  24



  Dunbar, et al.          Expires 24 December 2025                [Page 2]


  Internet-Draft                 Neotec API                      June 2025


       10.1.  Normative References . . . . . . . . . . . . . . . . . .  24
       10.2.  Informative References . . . . . . . . . . . . . . . . .  24
     Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  25
     Contributors  . . . . . . . . . . . . . . . . . . . . . . . . . .  26
     Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  26

  1.  Introduction

     This document explores the applicability of the Attachment Circuits
     YANG data model [I-D.ietf-opsawg-teas-attachment-circuit] and TE
     topology YANG data model, to support a simplified version of the use
     case described in China telecom's Neotec side meeting during
     IETF122(Cloud aware Network Operation for AI Services).  Also, the
     document specifies the APIs needed to support the simplified use
     case.

     Simplifed Use Case:

     Let's assume there are 10 Edge Cloud sites.  An City Surveillance AI
     model (e.g., detecting traffic congestion or garbage classification)
     needs to be deployed dynamically to some of these sites in response
     to real time events.

     High level steps for selecting Edge sites to instantiate the City
     Surveillance AI model:

     - Step 1: A Cloud Manager needs to query the network connectivity
     characteristics (bandwidth, latency, topology constraints, etc.)
     between street cameras (or gateways, eNB that connect to those street
     cameras) and candidate Edge Cloud sites in order to determine the
     optimal locations for the City Surveillance AI model deployment.

     - Step 2: Based on the information gathered, the Cloud Manager
     decides to deploy the City Surveillance AI model in a subset of the
     Edge Cloud sites (e.g., 4 sites among 10).

     High level steps to support the following desired outcome:

     - Suppose that the City Surveillance AI model instances in the 4 Edge
     Cloud sites need to exchange large volumes of data with strict
     performance constraints (e.g., XX Gbps bandwidth and YY ms one-way
     delay along the entire end-to-end paths between edge sites).  This
     request is to a network controller to dynamically adjust UCMP
     (Unequal Cost Multipath) load-balancing algorithms [I-D. ietf-bess-
     evpn-unequal-lb][I-D.ietf-idr-link-bandwidth] on all the nodes along
     the paths interconnecting those 4 sites.

     Disclaimer



  Dunbar, et al.          Expires 24 December 2025                [Page 3]


  Internet-Draft                 Neotec API                      June 2025


     The use of specific YANG data models (e.g., Attachment Circuit and TE
     topology) in this section is intended as a provisional exercise to
     explore how existing IETF models might address aspects of such a use
     case.  These examples are not exclusive or exhaustive.  Other data
     models, such as Network Slicing Service Model (NSSM) or service
     function chaining models, could also be relevant depending on the
     network architecture and service requirements.  The intent is to
     assess the applicability and identify gaps (if any), not to pre-
     define the final solution set.

  2.  Conventions used in this document

     Cloud Manager:  An entity that is primarily responsible for placing
        service instances and managing compute resources across Edge Cloud
        sites.  It monitors the health and scaling status of VMs,
        containers, or services, and makes infrastructure-level decisions,
        such as where to host service instances based on latency, CPU, GPU
        availability, or other constraints.  In addition, the Cloud
        Manager may provide abstracted status about the cloud environment
        (e.g., resource availability) to external entities.

     Neotec:  Network Operation for Telco Cloud.

     Network Orchestrator (or Orchestrator):  A logical entity that
        interfaces with the Cloud Manager to receive service requests or
        queries and coordinates the end-to-end connectivity across
        multiple network domains.  It abstracts underlying domain-specific
        technologies (e.g., L2/L3 VPNs, SR paths, TE tunnels) and
        disseminates policies to individual Network Controllers, enabling
        seamless stitching of diverse network segments to meet service-
        level requirements.

     Network Controller:  A domain specific control entity responsible for
        managing and configuring network resources within a single
        administrative or technological domain (e.g., IP/MPLS core, access
        network).  It receives high-level intent or service instructions
        from a Network Orchestrator and translates them into device-level
        configurations using protocols such as NETCONF, BGP, or PCEP.

     UE:  User Equipment

     UPF:  User Plane Function [TS.23.501-3GPP]

     The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
     "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
     "OPTIONAL" in this document are to be interpreted as described in BCP
     14 [RFC2119] [RFC8174] when, and only when, they appear in all
     capitals, as shown here.



  Dunbar, et al.          Expires 24 December 2025                [Page 4]


  Internet-Draft                 Neotec API                      June 2025


  3.  YANG Data Models and APIs to Support Dynamic AI Model Placement

     Multiple Edge Cloud sites are located in close geographic proximity,
     allowing any of them to potentially host AI inference model instances
     for city surveillance tasks such as traffic congestion detection or
     garbage classification.  A Cloud Manager must evaluate the end-to-end
     data paths between the street-level cameras and each Edge Cloud site
     to determine which sites offer sufficient compute resources and the
     required network performance (e.g., low latency and high throughput).
     This assessment enables dynamic placement of AI inference models in
     response to real-time events while ensuring reliable data delivery
     from the field.

     This path typically spans three segments:

     - The first is the access segment from the cameras local access node,
     such as a WiFi Access Point or an eNB, to a PE router.

     - The second segment traverses the provider transport network between
     that Access PE and the PE router serving the candidate Edge Cloud
     site.

     - The third segment connects that Edge PE to the Edge Cloud Gateway
     or compute node where the AI workload is deployed.

     There are two primary types of access connectivity for the cameras:
     via cellular networks (e.g., through eNBs/gNBs and User Plane
     Functions (UPFs)) or via WiFi Access Points.  In cellular based
     access, cameras connect to the network through eNBs, which forward
     user data to UPFs via GTP-U tunnels.  The UPFs serve as termination
     points for GTP-U tunnels and are often co-located with or adjacent to
     Edge Cloud sites.  When the Cloud Manager selects Edge Cloud sites
     for hosting AI inference modules, it must ensure that the
     corresponding nearby UPFs are assigned as the serving UPFs for the
     associated eNBs.  This enables the establishment of GTP-U tunnels
     from the eNBs directly to the selected Edge Cloud locations,
     minimizing latency and improving efficiency for real-time AI
     processing.

     For cameras connected through WiFi Access Points, no GTP tunneling is
     involved.  These access points typically connect to PEs through Layer
     2 or Layer 3 links.  In this case, the Attachment Circuit (AC) YANG
     model can be directly used to represent the logical connectivity
     between the WiFi AP and the Access PE.  The Cloud Manager or
     orchestrator can query the AC model to evaluate operational status,
     available bandwidth, latency, and packet loss, and determine whether
     the path is capable of supporting the target AI workload.




  Dunbar, et al.          Expires 24 December 2025                [Page 5]


  Internet-Draft                 Neotec API                      June 2025


     In both access scenarios, after determining candidate Edge Cloud
     sites, the Cloud Manager evaluates the transport network (Segment 2)
     between the access side PE (or the PE adjacent to the UPF) and the
     Edge side PE.  This segment may be traffic engineered and can be
     modeled using the IETF TE topology model [RFC8795] and the TE tunnel
     model [RFC8776].  These data models expose metrics such as available
     bandwidth, TE delay, and administrative attributes, which the
     orchestrator can use to assess whether the underlying transport paths
     meet the end-to-end service constraints (captured in an SLA.

     Finally, the last segment (Segment 3: from the PE to the Edge Cloud
     Gateway or compute node) is again modeled using the AC YANG data
     model.  By combining insights from the AC and TE data models across
     all three segments, the Cloud Manager and orchestrator can select
     Edge Cloud sites that not only have available compute capacity but
     also meet the network performance requirements to support low
     latency, high bandwidth AI model inference.

     Important Clarification:

     The Attachment Circuit (AC) and Traffic Engineering (TE) Topology
     YANG models described above are internal network models used by the
     network controller and orchestrator.  They are not directly exposed
     to external entities like the Cloud Manager.  Instead, the network
     controller abstracts relevant information from these internal models
     and presents it through simplified, service-oriented APIs (e.g.,
     Neotec APIs), allowing the Cloud Manager to query network performance
     metrics (such as latency and available bandwidth) without accessing
     raw topology or low-level link state.






















  Dunbar, et al.          Expires 24 December 2025                [Page 6]


  Internet-Draft                 Neotec API                      June 2025


                              +---------------------+
                              |     Street Cameras  |
                              +---------------------+
                                  /             \
                                 /               \
                 (Cellular Access)           (WiFi Access)
                               /                 \
                         +--------+         +-------------+
                         |  eNB   |         |  WiFi AP   |
                         +--------+         +-------------+
                               |                   |
                               |                   |
                 +----------------------+    +----------------------+
                 |        UPF           |    |    Access PE Router  |
                 +----------------------+    +----------------------+
                         |                             |
                         |       Segment 1             |
                         |<--------------------------->|
                                 (Access Segment)

                                 \                 /
                                  \               /
                                   \             /
                                 +----------------------+
                                 |  Provider Transport   |
                                 |     Network (TE)      |
                                 +----------------------+
                                             |
                                  Segment 2  | (PE-to-PE)
                                             v
                               +---------------------------+
                               |   Edge PE Router (per site)|
                               +---------------------------+
                                             |
                                             |
                                Segment 3    v
                                      +--------------------+
                                      |  Edge Cloud Gateway|
                                      |  + AI Model Module |
                                      +--------------------+


                           Figure 1: Network Segments

     For the first and third segments, the Attachment Circuit (AC) YANG
     model, defined in [I-D:ietf-opsawg-teas-attachment-circuit], provides
     a standardized abstraction of the logical link between a service
     access point and the provider's routing infrastructure.  This model



  Dunbar, et al.          Expires 24 December 2025                [Page 7]


  Internet-Draft                 Neotec API                      June 2025


     allows the Cloud Manager to retrieve configuration and operational
     status of these links, including encapsulation type, bandwidth, and
     link status.  While the Bearer Service Model [I-D:ietf-opsawg-teas-
     attachment-circuit] can provide additional contextual information,
     such as interface type, location, and administrative details, it is
     not the primary focus of this document.  For the purposes of real-
     time workload placement and path performance assessment, the emphasis
     here is on network performance metrics (e.g., latency, available
     bandwidth) exposed via AC and TE models.  Integration with bearer-
     level metadata (e.g., location) may be considered in future
     enhancements or broader orchestration frameworks.By querying this
     information, the orchestrator can determine whether each access
     circuit is operational and capable of delivering real-time video
     streams to or from the AI inference point.

     The second segment, the PE to PE transport path across the provider
     network, requires querying a different set of YANG models.  The base
     network and topology models defined in [RFC8345] provide the
     foundation, while the Traffic Engineering Topology model [RFC8795]
     adds details such as TE metrics, link delay, bandwidth availability,
     and administrative constraints.  These models are typically
     maintained by network controllers and made accessible via NETCONF or
     RESTCONF interfaces.  The Cloud Manager can initiate a query to
     evaluate the performance characteristics of available transport paths
     between Access PEs and candidate Edge PEs.  This query is handled by
     the Network Orchestrator, which interacts with the Network
     Controllers managing each domain.  The orchestrator uses IETF defined
     topology and TE models, such as the TE Topology model [RFC8795] and
     the TE Tunnel model [RFC8776], to retrieve current path attributes,
     including available bandwidth and latency.  In architectures that
     support a Path Computation Element (PCE), the orchestrator may also
     issue path computation requests via PCEP or a REST API to identify
     transport paths that meet the performance objectives required for AI
     inference workloads.

     While L2SM [RFC8466] and L3SM [RFC8299] offer useful abstractions for
     provisioning Layer 2 and Layer 3 VPN services, they are primarily
     focused on expressing service level intent rather than evaluating
     real time transport performance.  As such, they are not directly
     applicable to the dynamic, performance sensitive selection process
     described in the Neotec use case.

     By combining the Attachment Circuit and TE topology models, the Cloud
     Manager can construct an end to end view of network connectivity from
     each camera access point to every potential Edge Cloud site.  It can
     then select the subset of Edge Cloud locations that not only have
     sufficient computing resources but also meet the required network
     performance criteria for serving the selected set of street cameras.



  Dunbar, et al.          Expires 24 December 2025                [Page 8]


  Internet-Draft                 Neotec API                      June 2025


     This allows dynamic, network aware placement of AI inference models,
     ensuring efficient use of edge infrastructure while meeting service
     quality requirements.

  3.1.  Using AC YANG Data Model to Evaluate Access and Edge Connectivity

     In the context of dynamic AI model placement at Edge Cloud sites, the
     AC data model is useful for querying the health and performance of
     the access (Segment 1) which connects Access Points or eNBs to Access
     PE routers, and edge (Segment 3) which links Edge PEs to Edge Cloud
     Gateways or compute nodes.

     Each attachment circuit is modeled with attributes such as
     administrative state, operational status, encapsulation type, and
     configured bandwidth.  When augmented with telemetry or traffic
     engineering extensions, the AC model also supports real time status
     such as current utilization, available bandwidth, latency, and even
     packet loss.

     However, the Cloud Manager does not directly interact with the AC
     model.  Instead, the network controller queries and interprets the AC
     YANG data model and exposes only the relevant service-level metrics
     through the Neotec API interface.  This ensures that the Cloud
     Manager can assess whether a given access or edge segment supports
     the required performance (e.g., latency less than 10 ms, throughput
     greater than 500 Mbps) without needing detailed knowledge of the
     network.

     For example, if the Cloud Manager specifies a target Edge Cloud site,
     the network controller may internally retrieve attachment circuit
     metrics and present a simple API response indicating available
     bandwidth and latency thresholds met or not met:


     GET https://<controller>/restconf/data/ietf-ac:attachment-circuits
     /ac[pe-address='192.0.2.11']

                            Figure 2: Get AC Status













  Dunbar, et al.          Expires 24 December 2025                [Page 9]


  Internet-Draft                 Neotec API                      June 2025


     {
       "ietf-ac:ac": [
         {
           "name": "ac-to-eNB-001",
           "pe-address": "192.0.2.11",
           "ce-interface": "ge-0/0/1",
           "oper-status": "up",
           "admin-status": "enabled",
           "encapsulation-type": "dot1q",
           "bandwidth": {
             "configured": 1000,
             "available": 850,
             "unit": "Mbps"
           },
           "performance-metrics": {
             "latency": {
               "average": 5,
               "max": 10,
               "unit": "ms"
             },
             "packet-loss": {
               "percentage": 0.01
             }
           }
         }
       ]
     }

                       Figure 3: Access Segments Response

     In this example, the attachment circuit is operational (oper-status:
     up), with 850 Mbps of available bandwidth and average latency of 5
     ms, making it a strong candidate for supporting a real time AI
     application.  The Cloud Manager may apply similar queries to all PE
     addresses associated with candidate Edge Cloud sites and their
     corresponding access points, filtering the results to identify which
     circuits meet both latency and throughput thresholds.

  3.2.  Using IETF YANG Data Models to Evaluate PE to PE Connectivity

     Segment 2 of the end-to-end path connects the Access PE to the Edge
     PE router serving a candidate Edge Cloud site.  This segment
     typically traverses the provider's core or aggregation network and is
     crucial for ensuring that real-time AI inference traffic can meet
     stringent latency and bandwidth requirements.






  Dunbar, et al.          Expires 24 December 2025               [Page 10]


  Internet-Draft                 Neotec API                      June 2025


     The network controller uses internal models such as the IETF Network
     Topology YANG model [RFC8345] and the Traffic Engineering Topology
     YANG model [RFC8795] to monitor transport network characteristics.
     These models describe nodes, TE links, available bandwidth, delay,
     SRLGs, and other performance attributes.  Similarly, TE tunnels can
     be modeled using [RFC8776] to represent established LSPs or segment
     routing paths.

     Importantly, the Cloud Manager does not directly access these
     detailed TE models.  Instead, the controller or orchestrator
     processes this information and presents it via an abstracted API
     interface, exposing key metrics like end-to-end latency, available
     bandwidth, and path health between specified endpoints (e.g., Access
     Node to Edge Site).

     This abstraction enables the Cloud Manager to make workload placement
     decisions based on network performance, without having to manage or
     interpret the detailed topology or traffic engineering constructs.

     For example, to query available transport links from an Access PE at
     PE1 to a set of Edge PEs, the Cloud Manager may retrieve the list of
     TE links and their attributes using the following RESTCONF-style
     request:


     GET https://<controller>/restconf/data/ietf-te-topology:te-topologies
     /topology=default/te-node=PE1


                        Figure 4: Get PE-PE Path Status





















  Dunbar, et al.          Expires 24 December 2025               [Page 11]


  Internet-Draft                 Neotec API                      June 2025


     {
       "te-links": [
         {
           "name": "PE1-to-PE5",
           "link-id": "link-001",
           "oper-status": "up",
           "te-default-metric": 30,
           "te-bandwidth": {
             "max-link-bandwidth": 10000,
             "available-bandwidth": 7000,
             "unit": "Mbps"
           },
           "delay": {
             "unidirectional-delay": 8,
             "unit": "ms"
           },
           "adjacent-te-node-id": "PE5"
         }
       ]
     }

                            Figure 5: PE-PE Segment

     In this example, the link from PE1 to PE5 offers 7 Gbps of available
     bandwidth with 8 ms unidirectional delay, which may satisfy a 500
     Mbps, sub-10 ms latency requirement for AI data ingestion.  The Cloud
     Manager can evaluate similar TE links or tunnels to other Edge PEs
     (e.g., PE6, PE7) and compare their performance characteristics.

     Alternatively, if the network has PCE deployed, the Cloud Manager can
     issue a path computation request, using PCEP to query the end to end
     path metrics and validate whether a PE to PE segment can meet
     specified SLA constraints.  If Segment Routing (SR-MPLS or SRv6) is
     deployed, these models can also include SR label stacks or SID lists
     needed for forwarding decisions.

     By querying the TE topology and tunnel models, the orchestrator can
     build a filtered set of feasible transport segments that support the
     expected latency and bandwidth for the AI workload.  This insight,
     combined with data from the Attachment Circuit models for Segments 1
     and 3, allows the orchestrator to make holistic decisions about AI
     workload placement and optimal traffic steering across the network.









  Dunbar, et al.          Expires 24 December 2025               [Page 12]


  Internet-Draft                 Neotec API                      June 2025


  3.3.  Non-TE-Based PE to PE Connectivity

     The previous sections assume that the transport network between
     Access PEs and Edge PEs is traffic-engineered and that the
     orchestrator has access to detailed models, such as ietf-TE-topology,
     ietf-TE, or SR policy models.  However, in some deployments,
     particularly those relying on Internet transit or best effort IP
     core, traffic engineering is not explicitly available.

     In these cases, performance visibility and decision-making must rely
     on more general network telemetry.  Useful data can still be obtained
     through IETF defined YANG models, including:

     - RFC 8343 (Interface Management YANG Model) and RFC 8344 (IP
     Management YANG Model) for retrieving interface status and link-level
     attributes

     - Operational telemetry models, such as ietf-system-telemetry or real
     time telemetry via YANG Push [RFC8641][RFC8639], to gather metrics
     such as interface utilization, packet loss, round trip time (RTT),
     and jitter, especially if measured using active probes or synthetic
     monitoring.

     The Cloud Manager's query can trigger the network orchestrator to
     collect and aggregate per-hop or per-segment metrics using these
     models or active measurements (similar to IP SLA).  The resulting
     data allows the Cloud Manager to estimate end-to-end path performance
     and make workload placement decisions accordingly, even if
     deterministic path selection or policy enforcement is not possible.

     While this approach lacks the fine-grained path control of TE-based
     networks, it still enables adaptive, network-aware service placement
     in less structured or loosely-managed environments.

  4.  Potential APIs for Kubernetes to Query Network Conditions

     This section outlines a potential Neotec API interface to enable
     Kubernetes (or its external workload orchestrators) to make network-
     aware workload placement decisions, such as selecting appropriate
     Edge Cloud sites for deploying latency-sensitive AI inference
     modules.

     Cloud-native platforms such as Kubernetes do not consume YANG-modeled
     data directly.  Instead, they rely on REST-style APIs or gRPC
     interfaces with JSON or protobuf-encoded payloads.  To support Neotec
     use cases, the network infrastructure can expose an abstracted API
     that translates YANG-modeled topology and performance data into a
     form consumable by Kubernetes or its scheduling extensions.



  Dunbar, et al.          Expires 24 December 2025               [Page 13]


  Internet-Draft                 Neotec API                      June 2025


     A representative Neotec API endpoint could take the form:


     GET /network-advisor/query-path-performance?
     source-node=<access-node-id>&target-node=<edge-site-id>


                            Figure 6: REST style API

     This API allows Kubernetes to query the performance of the network
     path between a street camera's access node (e.g., a WiFi AP or eNB/
     UPF) and a candidate Edge Cloud site.  The API response includes
     metrics such as available bandwidth, average path latency, and
     operational status.  A sample response might be:



     {
       "source-node": "AP-235",
       "target-node": "EdgeSite-4",
       "path-latency-ms": 6.5,
       "available-bandwidth-mbps": 920,
       "path-status": "healthy"
     }

                            Figure 7: JSON Response

  4.1.  Behavior and Semantics

     This API provides a read-only, low-latency interface for workload
     schedulers to assess whether a given path meets predefined service-
     level thresholds (e.g., latency less than 10 ms, bandwidth larger
     than 500 Mbps).  The source node and target node identifiers
     correspond to access nodes (e.g., PE routers adjacent to UPFs or WiFi
     APs) and Edge Cloud PE routers respectively.  These identifiers are
     assumed to be mapped within the operator domain.

     The semantics of the fields are as follows:

     - path-latency-ms: Derived from YANG-modeled metrics in ietf-TE-
     topology, representing end to end unidirectional delay.

     - available-bandwidth-mbps: Aggregated from TE-bandwidth in the same
     topology model.

     - path-status: Reflects operational state derived from oper-status
     fields in the TE and AC models.




  Dunbar, et al.          Expires 24 December 2025               [Page 14]


  Internet-Draft                 Neotec API                      June 2025


  4.2.  YANG Model Integration

     The backend implementation of this API is expected to query IETF
     defined YANG models using RESTCONF or NETCONF.  Specifically:

     - Topology and path delay metrics are sourced from the ietf-te-
     topology and ietf-network-topology models [RFC8795], [RFC8345].

     - Access circuit status and available bandwidth can be derived from
     the AC model [opsawg-teas-attachment-circuit].

     A shim layer exposes REST or gRPC APIs that accept requests from
     cloud managers (e.g., Kubernetes) and translates them into queries
     over IETF defined YANG data models using NETCONF or RESTCONF.  This
     architecture enables real-time network path evaluations without
     requiring cloud systems to interact directly with YANG or underlying
     network protocols.

  5.  Dynamic UCMP Load Balancing for Periodic Inter Site AI Traffic

     In the Neotec use case described in Section 1, AI inference modules
     are deployed across four Edge Cloud sites to support distributed city
     surveillance.  These modules periodically exchange large volumes of
     data, for instance, during result aggregation or synchronized event
     analysis.  These data exchanges are not continuous but are periodic
     and event driven, requiring guaranteed bandwidth and low latency for
     short time windows.

     The underlying network connecting these Edge Cloud sites typically
     includes multiple paths between nodes and across multiple network
     segments.An end-to-end path between Edge Cloud Site A and B spans at
     least three segments:

     - The first is the access segment from the Edge Cloud A to its
     closest PE.  There could be multipe PEs to Edge Cloud A for multi-
     homing.

     - The second segment traverses the provider transport network between
     the PE serving Edge Cloud A and the PE serving the Edge Cloud B.

     - The third segment connects that Edge PE to the Edge Cloud B's
     Gateway

  5.1.  Edge Cloud Site to Local PE (Access Segment)







  Dunbar, et al.          Expires 24 December 2025               [Page 15]


  Internet-Draft                 Neotec API                      June 2025


                             +-----------------------------+
                         |      Edge Cloud Site        |
                         | +------------------------+  |
                         | |   AI Inference Module  |  |
                         | +------------------------+  |
                         |             |               |
                         |     +----------------+      |
                         |     |  Edge Cloud GW |      |
                         |     +----------------+      |
                         +-------------|---------------+
                                       |
                          Multiple Attachment Circuits
                                       |
        +----------------+   +----------------+   +----------------+
        |     PE1        |   |      PE2       |   |      PE3       |
        +----------------+   +----------------+   +----------------+
               |                     |                    |
        Provider Network     Provider Network      Provider Network

                      Figure 8: Edge Cloud Access Segment

     The Edge Cloud Gateway has multiple logical links (attachment
     circuits) to a set of PEs (e.g., PE1, PE2, PE3).  These links may
     vary in latency, bandwidth, or current load.  During periodic AI data
     bursts, the Edge Cloud GW must push all other traffic away from PE1,
     PE2, and PE3, reserving their full available bandwidth for AI flows.
     Or it could push all other traffic to one of the PEs that has the
     lowest bandwidth and highest latency.  This can be implemented by
     dynamically updating forwarding policies or QoS profiles to
     deprioritize or reroute non-AI traffic, ensuring that the AI
     inference module has uncontested access to network capacity during
     critical synchronization windows.  Depending on the request from the
     Cloud Manager, the Network Orchestrator can determine what exact
     policies to push to the PEs and the Edge GW..

     YANG Data Models for Policy Enforcement:

     To support dynamic UCMP-based traffic steering across PE1, PE2, and
     PE3, the Network Orchestrator can utilize the following IETF YANG
     models:

     - ietf-routing [RFC8349] enables configuration of routing instances
     and static routes.  The data model allows per-prefix route entries
     with multiple weighted next hops, supporting unequal cost paths.  It
     can be used to define static next-hop policies from the Edge GW
     toward PE1, PE2, and PE3.





  Dunbar, et al.          Expires 24 December 2025               [Page 16]


  Internet-Draft                 Neotec API                      June 2025


     - ietf-qos-policy [I-D.claise-opsawg-qos-packet-marking] defines
     traffic classification, marking, and treatment policies.  It can
     enforce rate limits or scheduling priority for non-AI traffic routed
     to PE3.

     - ietf-ac [I-D.ietf-opsawg-teas-attachment-circuit] can provides
     performance characteristics (bandwidth, latency, packet loss), which
     inform the decision to assign traffic classes.

     - ietf-traffic-classifier / ietf-packet-policy (or OpenConfig
     equivalents) can be used to match traffic classes (e.g., AI vs non-AI
     flows), enables forwarding decisions at the Edge GW and ingress of
     PEs.

     Policy Example: Prioritized AI Flow Assignment and Non-AI Rerouting:

     Assuming PE1 has the highest available capacity and best latency
     toward the remote site:

     The orchestrator uses ietf-routing [RFC8349] to define weighted
     static next hops at the Edge GW:

     - PE1: 2/3 of AI traffic.

     - PE2: 1/3 of AI traffic.

     - PE3: no AI traffic.
























  Dunbar, et al.          Expires 24 December 2025               [Page 17]


  Internet-Draft                 Neotec API                      June 2025


  {
    "routing:routing": {
      "routing-instance": [
        {
          "name": "ai-routing-instance",
          "routing-protocols": {
            "routing-protocol": [
              {
                "type": "static",
                "name": "ai-policy",
                "static-routes": {
                  "ipv4": {
                    "route": [
                      {
                        "destination-prefix": "198.51.100.0/24",
                        "next-hop": [
                          { "outgoing-interface": "to-PE1", "weight": 66 },
                          { "outgoing-interface": "to-PE2", "weight": 34 }
                        ]
                      }
                    ]
                  }
                }
              }
            ]
          }
        }
      ]
    }
  }

                      Figure 9: UCMP policy example

     Non-AI traffic is matched by classifiers and redirected entirely to
     PE3:

     - Apply QoS policy using ietf-qos-policy [I-D.claise-opsawg-qos-
     packet-marking] to limit available bandwidth on PE3 to non-AI
     traffic.

     - PE3 may be tagged "degraded" in the ietf-ac model to signal its
     backup nature.

     This ensures that the AI traffic is prioritized through the most
     capable paths (PE1 and PE2) with precise weight.  All non-critical
     traffic is offloaded to the least-desired path (PE3), reducing
     congestion.  Orchestrator needs to dynamically adapt this logic per
     service request.



  Dunbar, et al.          Expires 24 December 2025               [Page 18]


  Internet-Draft                 Neotec API                      June 2025


     Fallback Logic: If available capacity on PE1 and PE2 is not
     sufficient, AI traffic still receives best possible routing via PE1
     and PE2, and Non-AI traffic may be allowed limited access to PE2 with
     reduced weight (e.g., 10%) while still primarily routed via PE3.

     This tiered policy enforcement ensures strict adherence to SLA goals
     for AI inference while maintaining service continuity for other
     traffic classes using standard YANG-based configuration interfaces.

  5.1.1.  Time-Scoped UCMP Policy Enforcement

     One critical limitation in existing IETF YANG models is the lack of
     native support for time-scoped UCMP policy activation.  To support
     the bursty nature of AI traffic, the following strategies can be
     applied:

     - Define policy activation via external triggers from the Cloud
     Manager using an API call to the orchestrator.

     - Orchestrator maintains a mapping between the requested activation
     time window and the temporary configuration to be applied.

     - Extend existing YANG models (e.g., ietf-routing or ietf-sr-policy)
     with optional augmentation for: start-time, duration, and expiration-
     action

     Example augmentation (conceptual):


     augment "/routing:routing/routing-instance/static-routes/route" {
       leaf burst-policy-start-time {
         type yang:date-and-time;
       }
       leaf burst-policy-duration-sec {
         type uint32;
       }
       leaf expiration-action {
         type enumeration {
           enum revert-to-default;
           enum retain;
         }
       }
     }


                        Figure 10: Example Augmentation





  Dunbar, et al.          Expires 24 December 2025               [Page 19]


  Internet-Draft                 Neotec API                      June 2025


     In the absence of standard YANG support, this behavior need to be
     implemented in the orchestrator logic by maintaining policy lifecycle
     state and timers, pushing temporary configuration via NETCONF/
     RESTCONF at start-time, and reverting after duration-sec.

  5.1.2.  UCMP Enforcement for SRv6 based PE to PE segment

     Assuming an SRv6 underlay among the PEs, the network controller can
     use the ietf-sr-policy YANG model to update the traffic distribution
     weights across pre-established paths.  For example, if three SRv6
     paths exist between EdgeSite-A and EdgeSite-C, the controller can
     push the following configuration to the ingress node:


     sr-policy {
       color 4001;
       endpoint "2001:db8:100::1";
       candidate-paths {
         preference 100;
         path {
           weight 70;
           sid-list [2001:db8:10::1, 2001:db8:11::2];
         }
         path {
           weight 20;
           sid-list [2001:db8:20::1, 2001:db8:21::2];
         }
         path {
           weight 10;
           sid-list [2001:db8:30::1, 2001:db8:31::2];
         }
       }
     }

                           Figure 11: Using SR Policy

     This UCMP configuration tells the network to distribute traffic
     unequally across the three paths based on their capability.  The
     underlying topology and metrics are derived from ietf-TE-topology and
     ietf-TE models, which expose bandwidth, latency, and available
     resources for each link.

     Similar UCMP behavior can also be implemented over SR-MPLS, MPLS-TE,
     or enhanced IP networks, using the corresponding IETF YANG models
     (ietf-TE, ietf-routing, etc.).  The key point is that the network
     paths are preexisting, and the only dynamic action is adjusting how
     traffic is forwarded among them in response to a cloud service
     request.



  Dunbar, et al.          Expires 24 December 2025               [Page 20]


  Internet-Draft                 Neotec API                      June 2025


  5.2.  Cloud-Initiated UCMP Activation API

     A simplified example of a cloud-initiated API call to the network
     controller might look like:



     POST /network-policy/ucmp-activation
     {
       "source-sites": ["EdgeSite-A", "EdgeSite-B"],
       "dest-sites": ["EdgeSite-C", "EdgeSite-D"],
       "start-time": "2025-05-01T10:00:00Z",
       "duration-sec": 300,
       "min-bandwidth-mbps": 5000,
       "max-latency-ms": 10
     }

                        Figure 12: Burst Network Request

     This request informs the network controller that a high-volume, low-
     latency data exchange will occur and that UCMP forwarding policies
     should be applied to optimize transport between the specified sites
     for the specified duration.

  6.  Gaps Between Existing IETF Specifications and Neotec Use Case Needs

     The Neotec use case, supporting real time, event driven placement and
     coordination of AI inference workloads across Edge Cloud sites,
     requires close interaction between cloud orchestration platforms and
     programmable transport networks.  While IETF has standardized robust
     YANG models for traffic engineering (e.g., ietf-te-topology, ietf-sr-
     policy, and ietf-ac), these models are network-internal, and fall
     short in addressing cloud-driven, time-scoped network adaptation
     requirements.

     This document evaluates two core capabilities against existing IETF
     YANG models:

     - Network-aware workload placement at Edge sites

     - Dynamic UCMP policy activation for inter-site AI data exchange

     From these exercises, the following gaps have been identified:








  Dunbar, et al.          Expires 24 December 2025               [Page 21]


  Internet-Draft                 Neotec API                      June 2025


  6.1.  Consumption Model Mismatch

     Most IETF YANG models are accessed via NETCONF/RESTCONF, and are
     designed for network operator tools.  In contrast, cloud-native
     environments rely on REST/gRPC APIs, JSON payloads, and declarative
     interfaces (e.g., Kubernetes CRDs).  There is no standardized
     translation layer that exposes network topology or path performance
     in a form consumable by external cloud orchestrators or AI services.

  6.2.  Lack of Time-Bounded, Cloud-Triggered Policy Interfaces

     In the UCMP use case, the cloud controller must be able to request
     changes in traffic distribution policy across existing network paths
     for a specific time window, such as when AI model instances begin
     inter-site synchronization.  Current IETF models (e.g., ietf-sr-
     policy) allow weighted path configuration but do not support time-
     scoping, activation triggers, or scheduling.  These functions are
     essential for on-demand, just-in-time optimization and must be added.

  6.3.  Absence of Intent and Workload-to-Flow Mapping

     There is no YANG model or API that allows the cloud controller to
     associate a workload (e.g., "AI inference service X") with network
     traffic that should receive enhanced treatment.  While SR policies
     can be assigned to colors or DSCPs, there is no abstracted intent
     interface for service-aware flow classification or network SLA
     expression based on application context.

  6.4.  Lack of Abstracted, Queryable Interfaces

     Even for read-only functions like path selection, there is no IETF-
     standardized API to answer high-level queries such as:

     - "Which Edge sites have less than 10ms latency from these access
     nodes?"

     - "What is the bandwidth-latency profile of the path from Access PE A
     to Edge PE B?"

     Operators must instead manually interpret TE link state and construct
     custom tooling.  A standardized API to query the aggregated path
     metrics between logical service endpoints is missing.









  Dunbar, et al.          Expires 24 December 2025               [Page 22]


  Internet-Draft                 Neotec API                      June 2025


  6.5.  Feedback and Acknowledgment Gaps

     There is no defined mechanism for the network controller to confirm
     whether a policy request (e.g., UCMP activation) was accepted or
     enforced, nor to notify the cloud when SLA targets are not being met
     during the activation window.  A bi-directional feedback channel is
     required to support closed-loop coordination between the cloud and
     the network.

  6.6.  Security and Access Control Framework Alignment

     While the IETF has defined robust identity and access control
     mechanisms (e.g., OAuth2, RPKI, TLS), integrating these mechanisms
     with cloud-native identity systems (such as Kubernetes RBAC, SPIFFE/
     SPIRE, or cloud IAM services) is still ad hoc.  A standard framework
     for mutual trust establishment and token exchange between cloud and
     network domains is needed to support secure Neotec APIs under Zero
     Trust principles.

  7.  Need for Technology-Agnostic Modeling and Justification for a
      Separate Neotec WG

     The modeling challenges discussed in Section 6 highlight the need for
     standardized mechanisms that enable dynamic coordination between
     cloud service platforms and transport networks.  These requirements
     are not specific to any one transport technology (e.g., SRv6, SR-
     MPLS, MPLS-TE, or native IP), but instead call for common, technology
     agnostic abstractions that can be applied across diverse deployments.

     Existing IETF working groups, such as TEAS, SPRING, IDR, and OPSAWG,
     are focused on protocol specific data modeling and operational
     behavior.  While these groups have developed rich YANG models for
     their respective technologies, they are not scoped to define cross
     domain control inputs or service layer integration mechanisms that
     enable the network to adapt dynamically based on evolving service
     demands.

     The proposed Neotec Working Group provides a focused venue to address
     this gap.  Neotec's scope includes:

     - Defining transport independent policy triggers and control inputs
     that allow applications and orchestration platforms to request
     network behavior aligned with workload demands (e.g., bandwidth
     guarantees, latency constraints, traffic prioritization).

     - Developing modular YANG model extensions to allow temporary, per-
     service adjustments to traffic treatment across existing underlay
     technologies.



  Dunbar, et al.          Expires 24 December 2025               [Page 23]


  Internet-Draft                 Neotec API                      June 2025


     - Specifying interoperable API interfaces consistent with the
     approach outlined in the Neotec API strategy [neotec-api-strategy],
     where IETF documents define the API behavior and semantics, and
     developer-facing details (e.g., OpenAPI specifications) are
     maintained via open-source collaboration.

     - Establishing a framework for policy lifecycle management, including
     activation, expiration, and real-time status feedback, to support
     cloud-network coordination at runtime.

     By concentrating on service-driven, SLA-sensitive network behavior,
     Neotec aims to bridge the operational divide between cloud-native
     service orchestration and IETF-modeled transport infrastructure. its
     output will support reusable, vendor-neutral interfaces applicable
     across technologies and adaptable to dynamic service changes.

     A dedicated WG will prevent fragmented development across protocol
     specific groups and promote architectural consistency.  It will also
     provide a platform for broader engagement, enabling collaboration
     with the Kubernetes ecosystem, open source orchestration projects,
     and network operator communities.  Neotec documents may explore
     solution frameworks and prototype workflows that extend the
     applicability of existing IETF models rather than replacing them.

     Ultimately, Neotec seeks to define complementary APIs and interaction
     patterns that strengthen the cloud-network interface, enabling
     effective, scalable coordination of service placement, network
     telemetry, and policy enforcement across domains.

  8.  Security Considerations

     To be added

  9.  IANA Considerations

     None

  10.  References

  10.1.  Normative References

  10.2.  Informative References

     [Neotec-Zerotrust-Access]
                L. Dunbar and H. Chihi, "Neotec-Zerotrust-Access",
                December 2024, <https://datatracker.ietf.org/doc/draft-
                dunchihi-neotec-zerotrust-access-dm/>.




  Dunbar, et al.          Expires 24 December 2025               [Page 24]


  Internet-Draft                 Neotec API                      June 2025


     [opsawg-teas-attachment-circuit]
                M. Boucadair, et al, "opsawg-teas-attachment-circuit",
                January 2025, <https://datatracker.ietf.org/doc/draft-
                ietf-opsawg-teas-attachment-circuit/>.

     [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
                Requirement Levels", BCP 14, RFC 2119,
                DOI 10.17487/RFC2119, March 1997,
                <https://www.rfc-editor.org/info/rfc2119>.

     [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
                2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
                May 2017, <https://www.rfc-editor.org/info/rfc8174>.

     [RFC8345]  Clemm, A., Medved, J., Varga, R., Bahadur, N.,
                Ananthakrishnan, H., and X. Liu, "A YANG Data Model for
                Network Topologies", RFC 8345, DOI 10.17487/RFC8345, March
                2018, <https://www.rfc-editor.org/info/rfc8345>.

     [RFC8639]  Voit, E., Clemm, A., Gonzalez Prieto, A., Nilsen-Nygaard,
                E., and A. Tripathy, "Subscription to YANG Notifications",
                RFC 8639, DOI 10.17487/RFC8639, September 2019,
                <https://www.rfc-editor.org/info/rfc8639>.

     [RFC8641]  Clemm, A. and E. Voit, "Subscription to YANG Notifications
                for Datastore Updates", RFC 8641, DOI 10.17487/RFC8641,
                September 2019, <https://www.rfc-editor.org/info/rfc8641>.

     [RFC8776]  Saad, T., Gandhi, R., Liu, X., Beeram, V., and I. Bryskin,
                "Common YANG Data Types for Traffic Engineering",
                RFC 8776, DOI 10.17487/RFC8776, June 2020,
                <https://www.rfc-editor.org/info/rfc8776>.

     [RFC8795]  Liu, X., Bryskin, I., Beeram, V., Saad, T., Shah, H., and
                O. Gonzalez de Dios, "YANG Data Model for Traffic
                Engineering (TE) Topologies", RFC 8795,
                DOI 10.17487/RFC8795, August 2020,
                <https://www.rfc-editor.org/info/rfc8795>.

     [TS.23.501-3GPP]
                3rd Generation Partnership Project (3GPP), "System
                Architecture for 5G System; Stage 2, 3GPP TS 23.501
                v2.0.1", December 2017.

  Acknowledgements

     The authors would like to thank for following for discussions and
     providing input to this document: xxx.



  Dunbar, et al.          Expires 24 December 2025               [Page 25]


  Internet-Draft                 Neotec API                      June 2025


  Contributors

  Authors' Addresses

     Linda Dunbar (editor)
     Futurewei
     United States of America
     Email: ldunbar@futurewei.com


     Qiong
     China Telecom
     China
     Email: sunqiong@chinatelecom.cn


     Wu Bo (editor)
     Huawei
     China
     Email: lana.wubo@huawei.com


     LUIS MIGUEL CONTRERAS MURILLO (editor)
     Telefornica
     Spain
     Email: luismiguel.contrerasmurillo@telefonica.com


     ChongFeng
     China Telecom
     China
     Email: xiechf@chinatelecom.cn



















  Dunbar, et al.          Expires 24 December 2025               [Page 26]


