rfc9971.original   rfc9971.txt 
Benchmarking Working Group M. Konstantynowicz Internet Engineering Task Force (IETF) M. Konstantynowicz
Internet-Draft V. Polak Request for Comments: 9971 V. Polak
Intended status: Informational Cisco Systems Category: Informational Cisco Systems
Expires: 8 May 2026 4 November 2025 ISSN: 2070-1721 May 2026
Multiple Loss Ratio Search Multiple Loss Ratio Search
draft-ietf-bmwg-mlrsearch-15
Abstract Abstract
This document describes an alternative to "Benchmarking Methodology This document describes an alternative to throughput in "Benchmarking
for Network Interconnect Devices" (RFC 2544) throughput by defining a Methodology for Network Interconnect Devices" (RFC 2544) by defining
new methodology called Multiple Loss Ratio search (MLRsearch). a new methodology called Multiple Loss Ratio search (MLRsearch).
MLRsearch aims to minimize search duration, support multiple loss MLRsearch aims to minimize search duration, support multiple loss
ratio searches, and improve result repeatability and comparability. ratio searches, and improve result repeatability and comparability.
MLRsearch is motivated by the pressing need to address the challenges MLRsearch is motivated by the pressing need to address the challenges
of evaluating and testing the various data plane solutions, of evaluating and testing the various data plane solutions,
especially in software-based networking systems based on Commercial especially in software-based networking systems based on Commercial
Off-the-Shelf (COTS) CPU hardware vs purpose-built ASIC / NPU / FPGA Off-the-Shelf (COTS) CPU hardware vs. purpose-built Application-
hardware. Specific Integrated Circuit (ASIC), Network Processing Unit (NPU),
and Field-Programmable Gate Array (FPGA) hardware.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This document is not an Internet Standards Track specification; it is
provisions of BCP 78 and BCP 79. published for informational purposes.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are candidates for any level of Internet
Standard; see Section 2 of RFC 7841.
This Internet-Draft will expire on 8 May 2026. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc9971.
Copyright Notice Copyright Notice
Copyright (c) 2025 IETF Trust and the persons identified as the Copyright (c) 2026 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/ Provisions Relating to IETF Documents
license-info) in effect on the date of publication of this document. (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
Please review these documents carefully, as they describe your rights carefully, as they describe your rights and restrictions with respect
and restrictions with respect to this document. Code Components to this document. Code Components extracted from this document must
extracted from this document must include Revised BSD License text as include Revised BSD License text as described in Section 4.e of the
described in Section 4.e of the Trust Legal Provisions and are Trust Legal Provisions and are provided without warranty as described
provided without warranty as described in the Revised BSD License. in the Revised BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 1. Introduction
1.1. Purpose . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1. Purpose
1.2. Positioning within BMWG Methodologies . . . . . . . . . . 6 1.2. Positioning Within BMWG Methodologies
2. Overview of RFC 2544 Problems . . . . . . . . . . . . . . . . 7 2. Overview of RFC 2544 Problems
2.1. Binary Search . . . . . . . . . . . . . . . . . . . . . . 7 2.1. Binary Search
2.2. Long Search Duration . . . . . . . . . . . . . . . . . . 8 2.2. Long Search Duration
2.3. DUT in SUT . . . . . . . . . . . . . . . . . . . . . . . 9 2.3. DUT in SUT
2.4. Repeatability and Comparability . . . . . . . . . . . . . 11 2.4. Repeatability and Comparability
2.5. Throughput with Non-Zero Loss . . . . . . . . . . . . . . 12 2.5. Throughput with Non-Zero Loss
2.6. Inconsistent Trial Results . . . . . . . . . . . . . . . 13 2.6. Inconsistent Trial Results
3. Requirements Language . . . . . . . . . . . . . . . . . . . . 14 3. Requirements Language
4. MLRsearch Specification . . . . . . . . . . . . . . . . . . . 14 4. MLRsearch Specification
4.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4.1. Scope
4.1.1. Relationship to RFC 2544 . . . . . . . . . . . . . . 15 4.1.1. Relationship to RFC 2544
4.1.2. Applicability of Other Specifications . . . . . . . . 16 4.1.2. Applicability of Other Specifications
4.1.3. Out of Scope . . . . . . . . . . . . . . . . . . . . 16 4.1.3. Out of Scope
4.2. Architecture Overview . . . . . . . . . . . . . . . . . . 16 4.2. Architecture Overview
4.2.1. Test Report . . . . . . . . . . . . . . . . . . . . . 18 4.2.1. Test Report
4.2.2. Behavior Correctness . . . . . . . . . . . . . . . . 18 4.2.2. Behavior Correctness
4.3. Quantities . . . . . . . . . . . . . . . . . . . . . . . 18 4.3. Quantities
4.3.1. Current and Final Values . . . . . . . . . . . . . . 19 4.3.1. Current and Final Values
4.4. Existing Terms . . . . . . . . . . . . . . . . . . . . . 19 4.4. Existing Terms
4.4.1. SUT . . . . . . . . . . . . . . . . . . . . . . . . . 19 4.4.1. SUT
4.4.2. DUT . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.4.2. DUT
4.4.3. Trial . . . . . . . . . . . . . . . . . . . . . . . . 20 4.4.3. Trial
4.5. Trial Terms . . . . . . . . . . . . . . . . . . . . . . . 22 4.5. Trial Terms
4.5.1. Trial Duration . . . . . . . . . . . . . . . . . . . 22 4.5.1. Trial Duration
4.5.2. Trial Load . . . . . . . . . . . . . . . . . . . . . 22 4.5.2. Trial Load
4.5.3. Trial Input . . . . . . . . . . . . . . . . . . . . . 24 4.5.3. Trial Input
4.5.4. Traffic Profile . . . . . . . . . . . . . . . . . . . 24 4.5.4. Traffic Profile
4.5.5. Trial Forwarding Ratio . . . . . . . . . . . . . . . 26 4.5.5. Trial Forwarding Ratio
4.5.6. Trial Loss Ratio . . . . . . . . . . . . . . . . . . 27 4.5.6. Trial Loss Ratio
4.5.7. Trial Forwarding Rate . . . . . . . . . . . . . . . . 27 4.5.7. Trial Forwarding Rate
4.5.8. Trial Effective Duration . . . . . . . . . . . . . . 28 4.5.8. Trial Effective Duration
4.5.9. Trial Output . . . . . . . . . . . . . . . . . . . . 28 4.5.9. Trial Output
4.5.10. Trial Result . . . . . . . . . . . . . . . . . . . . 29 4.5.10. Trial Result
4.6. Goal Terms . . . . . . . . . . . . . . . . . . . . . . . 29 4.6. Goal Terms
4.6.1. Goal Final Trial Duration . . . . . . . . . . . . . . 30 4.6.1. Goal Final Trial Duration
4.6.2. Goal Duration Sum . . . . . . . . . . . . . . . . . . 30 4.6.2. Goal Duration Sum
4.6.3. Goal Loss Ratio . . . . . . . . . . . . . . . . . . . 31 4.6.3. Goal Loss Ratio
4.6.4. Goal Exceed Ratio . . . . . . . . . . . . . . . . . . 31 4.6.4. Goal Exceed Ratio
4.6.5. Goal Width . . . . . . . . . . . . . . . . . . . . . 32 4.6.5. Goal Width
4.6.6. Goal Initial Trial Duration . . . . . . . . . . . . . 32 4.6.6. Goal Initial Trial Duration
4.6.7. Search Goal . . . . . . . . . . . . . . . . . . . . . 33 4.6.7. Search Goal
4.6.8. Controller Input . . . . . . . . . . . . . . . . . . 33 4.6.8. Controller Input
4.7. Auxiliary Terms . . . . . . . . . . . . . . . . . . . . . 35 4.7. Auxiliary Terms
4.7.1. Trial Classification . . . . . . . . . . . . . . . . 35 4.7.1. Trial Classification
4.7.2. Load Classification . . . . . . . . . . . . . . . . . 36 4.7.2. Load Classification
4.8. Result Terms . . . . . . . . . . . . . . . . . . . . . . 38 4.8. Result Terms
4.8.1. Relevant Upper Bound . . . . . . . . . . . . . . . . 38 4.8.1. Relevant Upper Bound
4.8.2. Relevant Lower Bound . . . . . . . . . . . . . . . . 39 4.8.2. Relevant Lower Bound
4.8.3. Conditional Throughput . . . . . . . . . . . . . . . 39 4.8.3. Conditional Throughput
4.8.4. Goal Results . . . . . . . . . . . . . . . . . . . . 40 4.8.4. Goal Results
4.8.5. Search Result . . . . . . . . . . . . . . . . . . . . 42 4.8.5. Search Result
4.8.6. Controller Output . . . . . . . . . . . . . . . . . . 42 4.8.6. Controller Output
4.9. Architecture Terms . . . . . . . . . . . . . . . . . . . 42 4.9. Architecture Terms
4.9.1. Measurer . . . . . . . . . . . . . . . . . . . . . . 43 4.9.1. Measurer
4.9.2. Controller . . . . . . . . . . . . . . . . . . . . . 44 4.9.2. Controller
4.9.3. Manager . . . . . . . . . . . . . . . . . . . . . . . 44 4.9.3. Manager
4.10. Compliance . . . . . . . . . . . . . . . . . . . . . . . 45 4.10. Compliance
4.10.1. Test Procedure Compliant with MLRsearch . . . . . . 45 4.10.1. Test Procedure Compliant with MLRsearch
4.10.2. MLRsearch Compliant with RFC 2544 . . . . . . . . . 46 4.10.2. MLRsearch Compliant with RFC 2544
4.10.3. MLRsearch Compliant with TST009 . . . . . . . . . . 47 4.10.3. MLRsearch Compliant with TST009
5. Methodology Rationale and Design Considerations . . . . . . . 47 5. Methodology Rationale and Design Considerations
5.1. Binary Search Commonalities . . . . . . . . . . . . . . . 48 5.1. Binary Search Commonalities
5.2. Stopping Conditions and Precision . . . . . . . . . . . . 48 5.2. Stopping Conditions and Precision
5.3. Loss Ratios and Loss Inversion . . . . . . . . . . . . . 49 5.3. Loss Ratios and Loss Inversion
5.3.1. Single Goal and Hard Bounds . . . . . . . . . . . . . 49 5.3.1. Single Goal and Hard Bounds
5.3.2. Multiple Goals and Loss Inversion . . . . . . . . . . 49 5.3.2. Multiple Goals and Loss Inversion
5.3.3. Conservativeness and Relevant Bounds . . . . . . . . 50 5.3.3. Conservativeness and Relevant Bounds
5.3.4. Consequences . . . . . . . . . . . . . . . . . . . . 50 5.3.4. Consequences
5.4. Exceed Ratio and Multiple Trials . . . . . . . . . . . . 50 5.4. Exceed Ratio and Multiple Trials
5.5. Short Trials and Duration Selection . . . . . . . . . . . 51 5.5. Short Trials and Duration Selection
5.6. Generalized Throughput . . . . . . . . . . . . . . . . . 52 5.6. Generalized Throughput
5.6.1. Hard Performance Limit . . . . . . . . . . . . . . . 52 5.6.1. Hard Performance Limit
5.6.2. Performance Variability . . . . . . . . . . . . . . . 53 5.6.2. Performance Variability
6. MLRsearch Logic and Example . . . . . . . . . . . . . . . . . 53 6. MLRsearch Logic and Example
6.1. Load Classification Logic . . . . . . . . . . . . . . . . 54 6.1. Load Classification Logic
6.2. Conditional Throughput Logic . . . . . . . . . . . . . . 55 6.2. Conditional Throughput Logic
6.2.1. Conditional Throughput and Load Classification . . . 56 6.2.1. Conditional Throughput and Load Classification
6.3. SUT Behaviors . . . . . . . . . . . . . . . . . . . . . . 57 6.3. SUT Behaviors
6.3.1. Expert Predictions . . . . . . . . . . . . . . . . . 57 6.3.1. Expert Predictions
6.3.2. Exceed Probability . . . . . . . . . . . . . . . . . 57 6.3.2. Exceed Probability
6.3.3. Trial Duration Dependence . . . . . . . . . . . . . . 57 6.3.3. Trial Duration Dependence
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 58 7. IANA Considerations
8. Security Considerations . . . . . . . . . . . . . . . . . . . 58 8. Security Considerations
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 59 9. Acknowledgements
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 59 10. References
10.1. Normative References . . . . . . . . . . . . . . . . . . 59 10.1. Normative References
10.2. Informative References . . . . . . . . . . . . . . . . . 60 10.2. Informative References
Appendix A. Load Classification Code . . . . . . . . . . . . . . 61 Appendix A. Load Classification Code
Appendix B. Conditional Throughput Code . . . . . . . . . . . . 63 Appendix B. Conditional Throughput Code
Appendix C. Example Search . . . . . . . . . . . . . . . . . . . 65 Appendix C. Example Search
C.1. Example Goals . . . . . . . . . . . . . . . . . . . . . . 65 C.1. Example Goals
C.2. Example Trial Results . . . . . . . . . . . . . . . . . . 66 C.2. Example Trial Results
C.3. Load Classification Computations . . . . . . . . . . . . 68 C.3. Load Classification Computations
C.3.1. Point 1 . . . . . . . . . . . . . . . . . . . . . . . 68 C.3.1. Point 1
C.3.2. Point 2 . . . . . . . . . . . . . . . . . . . . . . . 69 C.3.2. Point 2
C.3.3. Point 3 . . . . . . . . . . . . . . . . . . . . . . . 70 C.3.3. Point 3
C.3.4. Point 4 . . . . . . . . . . . . . . . . . . . . . . . 72 C.3.4. Point 4
C.3.5. Point 5 . . . . . . . . . . . . . . . . . . . . . . . 73 C.3.5. Point 5
C.3.6. Point 6 . . . . . . . . . . . . . . . . . . . . . . . 74 C.3.6. Point 6
C.4. Conditional Throughput Computations . . . . . . . . . . . 76 C.4. Conditional Throughput Computations
C.4.1. Goal 2 . . . . . . . . . . . . . . . . . . . . . . . 76 C.4.1. Goal 2
C.4.2. Goal 3 . . . . . . . . . . . . . . . . . . . . . . . 77 C.4.2. Goal 3
C.4.3. Goal 4 . . . . . . . . . . . . . . . . . . . . . . . 78 C.4.3. Goal 4
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 79 Authors' Addresses
1. Introduction 1. Introduction
This document describes the Multiple Loss Ratio search (MLRsearch) This document describes the Multiple Loss Ratio search (MLRsearch)
methodology, optimized for determining data plane throughput in methodology, optimized for determining data plane throughput in
software-based networking functions running on commodity systems with software-based networking functions running on commodity systems with
generic CPUs (vs purpose-built ASIC / NPU / FPGA). Such network generic CPUs (vs. purpose-built ASICs, NPUs, and FPGAs). Such
functions can be deployed on dedicated physical appliance (e.g., a network functions can be deployed on a dedicated physical appliance
standalone hardware device) or as virtual appliance (e.g., Virtual (e.g., a standalone hardware device) or as virtual appliance (e.g., a
Network Function running on shared servers in the compute cloud). Virtual Network Function running on shared servers in the compute
cloud).
This document tightly couples terminology and methodology aspects. This document tightly couples terminology and methodology aspects.
Instead of a separate terminology section, Table of Contents Instead of a separate terminology section, the subsections of
subsections of MLRsearch Specification (Section 4) act as list of "MLRsearch Specification" (Section 4) act as a list of newly defined
newly defined terms. If a phrase appears with first letters terms. If a term appears with the first letter capitalized, it
capitalized, it likely refers to a specific term defined in eponymous likely refers to a specific term defined in an eponymous subsection
subsection of MLRsearch Specification (Section 4). of "MLRsearch Specification" (Section 4).
For first time readers, the information in MLRsearch Specification For first-time readers, the information in "MLRsearch Specification"
(Section 4) might feel dense and lacking motivation. Subsequent (Section 4) might feel dense and lacking motivation. Subsequent
sections are there to provide explanations, making MLRsearch sections provide explanations, making "MLRsearch Specification"
Specification (Section 4) more approachable on repeated reads. (Section 4) more approachable on repeated reads.
1.1. Purpose 1.1. Purpose
The purpose of this document is to describe the Multiple Loss Ratio The purpose of this document is to describe the Multiple Loss Ratio
search (MLRsearch) methodology, optimized for determining data plane search (MLRsearch) methodology, optimized for determining data plane
throughput in software-based networking devices and functions. throughput in software-based networking devices and functions.
Applying the Binary Search (Section 2.1) to software devices under Applying the Binary Search (Section 2.1) to software Devices Under
test (DUTs) results in several problems: Test (DUTs) results in several problems:
* Binary search takes a long time as most trials are done far from * Binary search takes a long time, as most trials are done far from
the eventually found throughput. the eventually found throughput.
* The required final trial duration and pauses between trials * The required final trial duration and pauses between trials
prolong the overall search duration. prolong the overall search duration.
* Software DUTs show noisy trial results, leading to a big spread of * Software DUTs show noisy trial results, leading to a big spread of
possible discovered throughput values. possible discovered throughput values.
* Throughput requires a loss of exactly zero frames, but the * Throughput requires a loss of exactly zero frames, but the
industry best practices frequently allow for low but non-zero industry best practices frequently allow for tolerance of low but
losses tolerance ([Y.1564], test-equipment manuals). non-zero losses ([Y.1564], test-equipment manuals).
* The definition of throughput is not clear when trial results are * The definition of throughput is not clear when trial results are
inconsistent. (e.g., when successive trials at the same - or even inconsistent (e.g., when successive trials at the same -- or even
a higher - offered load yield different loss ratios, the classical a higher -- offered load yield different loss ratios, the
[RFC1242] / [RFC2544] throughput metric can no longer be pinned to classical throughput metric in [RFC1242] and [RFC2544] can no
a single, unambiguous value.) longer be pinned to a single, unambiguous value.)
To address these problems, early MLRsearch implementations employed To address these problems, early MLRsearch implementations employed
the following enhancements: the following enhancements:
1. Allow multiple short trials instead of one big trial per load. 1. Allow multiple short trials instead of one big trial per load.
* Optionally, tolerate a percentage of trial results with higher * Optionally, tolerate a percentage of trial results with higher
loss. loss.
2. Allow searching for multiple Search Goals (Section 4.6.7), with 2. Allow searching for multiple Search Goals (Section 4.6.7), with
differing loss ratios. differing loss ratios.
* Any trial result can affect each Search Goal in principle. * Any trial result can affect each Search Goal in principle.
3. Insert multiple coarse targets for each Search Goal, earlier ones 3. Insert multiple coarse targets for each Search Goal; earlier ones
need to spend less time on trials. need to spend less time on trials.
* Earlier targets also aim for lesser precision. * Earlier targets also aim for lesser precision.
* Use Forwarding Rate (FR) at Maximum Offered Load (FRMOL), as * Use Forwarding Rate at Maximum Offered Load (FRMOL), as
defined in Section 3.6.2 of [RFC2285], to initialize bounds. defined in Section 3.6.2 of [RFC2285], to initialize bounds.
4. Clarify handling of inconsistent trial results. 4. Clarify handling of inconsistent trial results.
* Reported throughput should be smaller than the smallest load * Reported throughput should be smaller than the smallest load
with high loss. with high loss.
* Measure smaller load candidates first. * Measure smaller load candidates first.
5. Apply several time-saving load selection heuristics that 5. Apply several time-saving load selection heuristics that
deliberately prevent the bounds from narrowing unnecessarily. deliberately prevent the bounds from narrowing unnecessarily.
Enhancements 1, 2 and partly 4 are formalized as MLRsearch Enhancements 1, 2, and partly 4 are formalized as the MLRsearch
Specification within this document, other implementation details are Specification within this document; other implementation details are
out the scope. out the scope.
The remaining enhancements are treated as implementation details, The remaining enhancements are treated as implementation details,
thus achieving high comparability without limiting future thus achieving high comparability without limiting future
improvements. improvements.
MLRsearch configuration supports both conservative settings and MLRsearch configuration supports both conservative settings and
aggressive settings. Results unconditionally compliant with aggressive settings. Results unconditionally compliant with
[RFC2544] are possible with conservative enough settings, but without [RFC2544] are possible with conservative enough settings but without
much improvement on search duration and repeatability - see MLRsearch much improvement on search duration and repeatability -- see
Compliant with RFC 2544 (Section 4.10.2). Conversely, aggressive "MLRsearch Compliant with RFC 2544" (Section 4.10.2). Conversely,
settings lead to shorter search durations and better repeatability, aggressive settings lead to shorter search durations and better
but the results are not compliant with [RFC2544]. Exact settings are repeatability, but the results are not compliant with [RFC2544].
not specified, but see the discussion in Overview of RFC 2544 Exact settings are not specified, but see the discussion in "Overview
Problems (Section 2) for the impact of different settings on result of RFC 2544 Problems" (Section 2) for the impact of different
quality. settings on result quality.
This document does not change or obsolete any part of [RFC2544]. This document does not change or obsolete any part of [RFC2544].
1.2. Positioning within BMWG Methodologies 1.2. Positioning Within BMWG Methodologies
The Benchmarking Methodology Working Group (BMWG) produces The Benchmarking Methodology Working Group (BMWG) produces
recommendations (RFCs) that describe various benchmarking recommendations (RFCs) that describe various benchmarking
methodologies for use in a controlled laboratory environment. A methodologies for use in a controlled laboratory environment. A
large number of these benchmarks are based on the terminology from large number of these benchmarks are based on the terminology from
[RFC1242] and the foundational methodology from [RFC2544]. A common [RFC1242] and the foundational methodology from [RFC2544]. A common
pattern has emerged where BMWG documents reference the methodology of pattern has emerged where BMWG documents reference the methodology of
[RFC2544] and augment it with specific requirements for testing [RFC2544] and augment it with specific requirements for testing
particular network systems or protocols, without modifying the core particular network systems or protocols, without modifying the core
benchmark definitions. benchmark definitions.
While BMWG documents are formally recommendations, they are widely While BMWG documents are formal recommendations, they are widely
treated as industry norms to ensure the comparability of results treated as industry norms to ensure the comparability of results
between different labs. The set of benchmarks defined in [RFC2544], between different labs. The set of benchmarks defined in [RFC2544],
in particular, became a de facto standard for performance testing. in particular, became a de facto standard for performance testing.
In this context, the MLRsearch Specification formally defines a new In this context, the MLRsearch Specification formally defines a new
class of benchmarks that fits within the wider [RFC2544] framework class of benchmarks that fits within the wider framework of
(see Scope (Section 4.1)). [RFC2544]; see Section 4.1 ("Scope").
A primary consideration in the design of MLRsearch is the trade-off A primary consideration in the design of MLRsearch is the trade-off
between configurability and comparability. The methodology's between configurability and comparability. The methodology's
flexibility, especially the ability to define various sets of Search flexibility, especially the ability to define various sets of Search
Goals, supporting both single-goal and multiple-goal benchmarks in an Goals, in supporting both single-goal and multiple-goal benchmarks in
unified way is powerful for detailed characterization and internal a unified way is powerful for detailed characterization and internal
testing. However, this same flexibility is detrimental to inter-lab testing. However, this same flexibility is detrimental to inter-lab
comparability unless a specific, common set of Search Goals is agreed comparability unless a specific, common set of Search Goals is agreed
upon. upon.
Therefore, MLRsearch should not be seen as a direct extension nor a Therefore, MLRsearch should not be seen as a direct extension nor a
replacement for the [RFC2544] Throughput benchmark. Instead, this replacement for the [RFC2544] Throughput benchmark. Instead, this
document provides a foundational methodology that future BMWG document provides a foundational methodology that future BMWG
documents can use to define new, specific, and comparable benchmarks documents can use to define new, specific, and comparable benchmarks
by mandating particular Search Goal configurations. For operators of by mandating particular Search Goal configurations. For operators of
existing test procedures, it is worth noting that many test setups existing test procedures, it is worth noting that many test setups
measuring [RFC2544] Throughput can be adapted to produce results measuring [RFC2544] Throughput can be adapted to produce results
compliant with the MLRsearch Specification, often without affecting compliant with the MLRsearch Specification, often without affecting
Trials, merely by augmenting the content of the final test report. Trials, merely by augmenting the content of the final test report.
2. Overview of RFC 2544 Problems 2. Overview of RFC 2544 Problems
This section describes the problems affecting usability of various This section describes the problems affecting usability of various
performance testing methodologies, mainly the Binary Search performance testing methodologies, mainly the Binary Search
(Section 2.1) for [RFC2544] unconditionally compliant throughput. (Section 2.1) for unconditionally compliant [RFC2544] throughput.
2.1. Binary Search 2.1. Binary Search
While [RFC2544] offers some flexibility when searching for While [RFC2544] offers some flexibility when searching for
throughput, a particular algorithm is frequently used as a starting throughput, a particular algorithm is frequently used as a starting
point, as it is the simplest one among those that offer reasonable point, as it is the simplest one among those that offer reasonable
effectivity. effectivity.
This algorithm is based on (balanced) binary search over sorted This algorithm is based on (balanced) binary search over sorted
arrays, but does not have a specific name when searching for arrays but does not have a specific name when searching for
throughput. Section 24. Trial duration of [RFC2544] mentions binary throughput. "Trial duration" (Section 24 of [RFC2544]) mentions
search only in quotes, without providing specifics. In this document binary search only in quotes, without providing specifics. In this
we call that algorithm the Binary Search, as that is the title of document, we call that algorithm the Binary Search, as that is the
Section 12.3.2 of [TST009] describing a variant of it. title of Section 12.3.2 of [TST009], which describes a variant of it.
Here is a simplified description of the algorithm: Here is a simplified description of the algorithm:
* Initialize lower-bound variable to line-rate value. * Initialize the lower-bound variable to a line-rate value.
* Initialize upper-bound variable to a loss-free load value. * Initialize the upper-bound variable to a loss-free load value.
* Compute a midpoint, the arithmetic mean of current bound values. * Compute a midpoint, the arithmetic mean of current bound values.
* Run a single 60-second trial at the midpoint (for [RFC2544] * Run a single 60-second trial at the midpoint (for [RFC2544]
unconditional compliance). unconditional compliance).
* If loss is zero, set lower-bound to midpoint value; else set * If loss is zero, set the lower-bound to a midpoint value; else,
upper-bound to midpoint value. set the upper bound to a midpoint value.
* Repeat (computing new midpoint) until the gap between the bounds * Repeat (computing new midpoint) until the gap between the bounds
meets the desired precision. meets the desired precision.
* Return the final lower-bound value as the throughput. * Return the final lower-bound value as the throughput.
The [TST009] description had two more requirements (stopping The description in [TST009] has two more requirements (stopping
condition and rounding, both based on Offered Load Step Size condition and rounding, both based on the Offered Load Step Size
Parameter) but those are not required in this document. Parameter), but those are not required in this document.
Small modifications related to initial bound values are also allowed. Small modifications related to initial bound values are also allowed.
The load values currently held in the two variables are called The load values currently held in the two variables are called
"tightest bounds", especially when discussing older trial results "tightest bounds", especially when discussing older trial results
(logically still bounds). (logically still bounds).
2.2. Long Search Duration 2.2. Long Search Duration
The proliferation of software DUTs, with frequent software updates The proliferation of software DUTs, with frequent software updates
and a number of different frame processing modes and configurations, and a number of different frame processing modes and configurations,
has increased both the number of performance tests required to verify has increased both the number of performance tests required to verify
the DUT update and the frequency of running those tests. This makes the DUT update and the frequency of running those tests. This makes
the overall test execution time even more important than before. the overall test execution time even more important than before.
The definition of throughput test methodology per [RFC2544] restricts The definition of throughput test methodology per [RFC2544] restricts
the potential for time-efficiency improvements. The Binary Search the potential for time-efficiency improvements. The Binary Search
(Section 2.1), when used in a manner unconditionally compliant with (Section 2.1), when used in a manner unconditionally compliant with
[RFC2544], is excessively slow due to two main factors. [RFC2544], is excessively slow due to two main factors.
Firstly, a significant amount of time is spent on trials with loads First, a significant amount of time is spent on trials with loads
that, in retrospect, are far from the final determined throughput. that, in retrospect, are far from the final determined throughput.
Secondly, [RFC2544] does not specify any stopping condition for Second, [RFC2544] does not specify any stopping condition for
throughput search, so users of testing equipment implementing the throughput search, so users of testing equipment implementing the
procedure already have access to a limited trade-off between search procedure already have access to a limited trade-off between search
duration and achieved precision, as each one of the full 60-second duration and achieved precision, as each one of the full 60-second
trials halves the interval of possible results. trials halves the interval of possible results.
As such, not many trials can be removed without a substantial loss of As such, not many trials can be removed without a substantial loss of
precision. precision.
2.3. DUT in SUT 2.3. DUT in SUT
Section 19 of [RFC2544] specifies a test setup with an external Section 19 of [RFC2544] specifies a test setup with an external
tester stimulating the networking system, treating it either as a tester stimulating the networking system, treating it either as a
single Device Under Test (DUT), or as a system of devices, a System single Device Under Test (DUT) or as a system of devices, a System
Under Test (SUT). Under Test (SUT).
[RFC2285] defines: [RFC2285] defines these terms as follows.
DUT as:
* The network frame forwarding device to which stimulus is offered
and response measured Section 3.1.1 of [RFC2285].
SUT as: DUT: The network frame forwarding device to which stimulus is
offered and response measured (Section 3.1.1 of [RFC2285]).
* The collective set of network devices as a single entity to which SUT: The collective set of network devices to which stimulus is
stimulus is offered and response measured Section 3.1.2 of offered as a single entity and response measured
[RFC2285]. (Section 3.1.2 of [RFC2285]).
For software-based data-plane forwarding running on commodity x86/ARM For software-based data plane forwarding running on commodity x86/ARM
CPUs, the SUT comprises not only the forwarding application itself, CPUs, the SUT comprises not only the forwarding application itself,
the DUT, but the entire execution environment: host hardware, the DUT, but also the entire execution environment: host hardware,
firmware and kernel/hypervisor services, as well as any other firmware and kernel/hypervisor services, as well as any other
software workloads that share the same CPUs, memory and I/O software workloads that share the same CPUs, memory, and I/O
resources. resources.
Given that a SUT is a shared multi-tenant environment, the DUT might Given that a SUT is a shared multi-tenant environment, the DUT might
inadvertently experience interference from the operating system or inadvertently experience interference from the operating system or
from other software operating on the same server. from other software operating on the same server.
Some of this interference can be mitigated. For instance, in multi- Some of this interference can be mitigated. For instance, in multi-
core CPU systems, pinning DUT program threads to specific CPU cores core CPU systems, pinning DUT program threads to specific CPU cores
and isolating those cores can prevent context switching. and isolating those cores can prevent context switching.
Despite taking all feasible precautions, some adverse effects may Despite taking all feasible precautions, some adverse effects may
still impact the DUT's network performance. In this document, these still impact the DUT's network performance. In this document, these
effects are collectively referred to as SUT noise, even if the effects are collectively referred to as SUT noise, even if the
effects are not as unpredictable as what other engineering effects are not as unpredictable as what other engineering
disciplines call noise. disciplines call noise.
A DUT can also exhibit fluctuating performance itself, for reasons A DUT can also exhibit fluctuating performance itself, for reasons
not related to the rest of SUT. For example, this can be due to not related to the rest of SUT. For example, this can be due to
pauses in execution as needed for internal stateful processing. In pauses in execution as needed for internal stateful processing. In
many cases this may be an expected per-design behavior, as it would many cases, this may be an expected per-design behavior, as it would
be observable even in a hypothetical scenario where all sources of be observable even in a hypothetical scenario where all sources of
SUT noise are eliminated. Such behavior affects trial results in a SUT noise are eliminated. Such behavior affects trial results in a
way similar to SUT noise. As the two phenomena are hard to way similar to SUT noise. As the two phenomena are hard to
distinguish, in this document the term 'noise' is used to encompass distinguish, in this document, the term "noise" is used to encompass
both the internal performance fluctuations of the DUT and the genuine both the internal performance fluctuations of the DUT and the genuine
noise of the SUT. noise of the SUT.
A simple model of SUT performance consists of an idealized noiseless A simple model of SUT performance consists of an idealized noiseless
performance, and additional noise effects. For a specific SUT, the performance and additional noise effects. For a specific SUT, the
noiseless performance is assumed to be constant, with all observed noiseless performance is assumed to be constant, with all observed
performance variations being attributed to noise. The impact of the performance variations being attributed to noise. The impact of the
noise can vary in time, sometimes wildly, even within a single trial. noise can vary in time, sometimes wildly, even within a single trial.
The noise can sometimes be negligible, but frequently it lowers the The noise can sometimes be negligible, but it frequently lowers the
observed SUT performance as observed in trial results. observed SUT performance as observed in trial results.
In this simple model, a SUT does not have a single performance value, In this simple model, a SUT does not have a single performance value;
it has a spectrum. One end of the spectrum is the idealized it has a spectrum. One end of the spectrum is the idealized
noiseless performance value, the other end can be called a noiseful noiseless performance value, and the other end can be called a
performance. In practice, trial results close to the noiseful end of noiseful performance. In practice, trial results close to the
the spectrum happen only rarely. The worse a possible performance noiseful end of the spectrum happen only rarely. The worse a
value is, the more rarely it is seen in a trial. Therefore, the possible performance value is, the more rarely it is seen in a trial.
extreme noiseful end of the SUT spectrum is not observable among Therefore, the extreme noiseful end of the SUT spectrum is not
trial results. observable among trial results.
Furthermore, the extreme noiseless end of the SUT spectrum is Furthermore, the extreme noiseless end of the SUT spectrum is
unlikely to be observable, this time because minor noise events unlikely to be observable, this time because minor noise events
almost always occur during each trial, nudging the measured almost always occur during each trial, nudging the measured
performance slightly below the theoretical maximum. performance slightly below the theoretical maximum.
Unless specified otherwise, this document's focus is on the Unless specified otherwise, this document's focus is on the
potentially observable ends of the SUT performance spectrum, as potentially observable ends of the SUT performance spectrum, as
opposed to the extreme ones. opposed to the extreme ones.
When focusing on the DUT, the benchmarking effort should ideally aim When focusing on the DUT, the benchmarking effort should ideally aim
to eliminate only the SUT noise from SUT measurements. However, this to eliminate only the SUT noise from SUT measurements. However, this
is currently not feasible in practice, as there are no realistic is currently not feasible in practice, as there are no realistic
enough models that would be capable to distinguish SUT noise from DUT enough models that would be capable to distinguish SUT noise from DUT
fluctuations (based on the available literature at the time of fluctuations (based on the available literature at the time of
writing). writing).
Provided SUT execution environment and any co-resident workloads If the SUT execution environments and any co-resident workloads place
place only negligible demands on SUT shared resources, so that the only negligible demands on SUT shared resources, so that the DUT
DUT remains the principal performance limiter, the DUT's ideal remains the principal performance limiter, the DUT's ideal noiseless
noiseless performance is defined as the noiseless end of the SUT performance is defined as the noiseless end of the SUT performance
performance spectrum. spectrum.
Note that by this definition, DUT noiseless performance also Note that by this definition, DUT noiseless performance also
minimizes the impact of DUT fluctuations, as much as realistically minimizes the impact of DUT fluctuations, as much as realistically
possible for a given trial duration. possible for a given trial duration.
The MLRsearch methodology aims to solve the DUT in SUT problem by The MLRsearch methodology aims to solve the DUT-in-SUT problem by
estimating the noiseless end of the SUT performance spectrum using a estimating the noiseless end of the SUT performance spectrum using a
limited number of trial results. limited number of trial results.
Improvements to the throughput search algorithm, aimed at better Improvements to the throughput search algorithm, aimed at better
dealing with software networking SUT and DUT setups, should adopt dealing with software networking SUT and DUT setups, should adopt
methods that explicitly model SUT-generated noise, enabling to derive methods that explicitly model SUT-generated noise, deriving surrogate
surrogate metrics that approximate the (proxies for) DUT noiseless metrics that approximate the (proxies for) DUT noiseless performance
performance across a range of SUT noise-tolerance levels. across a range of SUT noise-tolerance levels.
2.4. Repeatability and Comparability 2.4. Repeatability and Comparability
[RFC2544] does not suggest repeating throughput search. Also, note [RFC2544] does not suggest repeating throughput search. Also, note
that from simply one discovered throughput value, it cannot be that from simply one discovered throughput value, it cannot be
determined how repeatable that value is. Unsatisfactory determined how repeatable that value is. Unsatisfactory
repeatability then leads to unacceptable comparability, as different repeatability then leads to unacceptable comparability, as different
benchmarking teams may obtain varying throughput values for the same benchmarking teams may obtain varying throughput values for the same
SUT, exceeding the expected differences from search precision. SUT, exceeding the expected differences from search precision.
Repeatability is important also when the test procedure is kept the Repeatability is also important when the test procedure is kept the
same, but SUT is varied in small ways. For example, during same, but SUT is varied in small ways. For example, during
development of software-based DUTs, repeatability is needed to detect development of software-based DUTs, repeatability is needed to detect
small regressions. small regressions.
[RFC2544] throughput requirements (60-second trial and no tolerance [RFC2544] throughput requirements (60-second trial and no tolerance
of a single frame loss) affect the throughput result as follows: of a single frame loss) affect the throughput result as follows.
The SUT behavior close to the noiseful end of its performance The SUT behavior close to the noiseful end of its performance
spectrum consists of rare occasions of significantly low performance, spectrum consists of rare occasions of significantly low performance,
but the long trial duration makes those occasions not so rare on the but the long trial duration makes those occasions not so rare on the
trial level. Therefore, the Binary Search (Section 2.1) results tend trial level. Therefore, the Binary Search (Section 2.1) results tend
to spread away from the noiseless end of SUT performance spectrum, to spread away from the noiseless end of SUT performance spectrum
more frequently and more widely than shorter trials would, thus more frequently and more widely than shorter trials would, thus
causing unacceptable throughput repeatability. causing unacceptable throughput repeatability.
The repeatability problem can be better addressed by defining a The repeatability problem can be better addressed by defining a
search procedure that identifies a consistent level of performance, search procedure that identifies a consistent level of performance,
even if it does not meet the strict definition of throughput test even if it does not meet the strict definition of throughput test
methodology in [RFC2544]. methodology in [RFC2544].
According to the SUT performance spectrum model, better repeatability According to the SUT performance spectrum model, better repeatability
will be at the noiseless end of the spectrum. Therefore, solutions will be at the noiseless end of the spectrum. Therefore, solutions
to the DUT in SUT problem will help also with the repeatability to the DUT-in-SUT problem will also help with the repeatability
problem. problem.
Conversely, any alteration to [RFC2544] throughput search that Conversely, any alteration to [RFC2544] throughput search that
improves repeatability should be considered as less dependent on the improves repeatability should be considered as less dependent on the
SUT noise. SUT noise.
An alternative option is to simply run a search multiple times, and An alternative option is to simply run a search multiple times and
report some statistics (e.g., average and standard deviation, and/or report some statistics (e.g., average and standard deviation and/or
percentiles like p95). percentiles like p95).
This can be used for a subset of tests deemed more important, but it This can be used for a subset of tests deemed more important, but it
makes the search duration problem even more pronounced. makes the search duration problem even more pronounced.
2.5. Throughput with Non-Zero Loss 2.5. Throughput with Non-Zero Loss
Section 3.17 of [RFC1242] defines throughput as: The maximum rate at Section 3.17 of [RFC1242] defines throughput as:
which none of the offered frames are dropped by the device.
Then, it says: Since even the loss of one frame in a data stream can | The maximum rate at which none of the offered frames are dropped
cause significant delays while waiting for the higher-level | by the device.
protocols to time out, it is useful to know the actual maximum
data rate that the device can support. Then, it says:
| Since even the loss of one frame in a data stream can cause
| significant delays while waiting for the higher level protocols to
| time out, it is useful to know the actual maximum data rate that
| the device can support.
However, many benchmarking teams accept a low, non-zero loss ratio as However, many benchmarking teams accept a low, non-zero loss ratio as
the goal for their load search. the goal for their load search.
Motivations are many: There are many motivations:
* Networking protocols tolerate frame loss better, compared to the * Networking protocols tolerate frame loss better, compared to the
time when [RFC1242] and [RFC2544] were specified. time when [RFC1242] and [RFC2544] were specified.
* Increased link speeds require trials sending more frames within * Increased link speeds require trials sending more frames within
the same duration, increasing the chance of a small SUT the same duration, increasing the chance of a small SUT
performance fluctuation being enough to cause frame loss. performance fluctuation being enough to cause frame loss.
* Because noise-related drops usually arrive in small bursts, their * Because noise-related drops usually arrive in small bursts, their
impact on the trial's overall frame loss ratio is diluted by the impact on the trial's overall frame loss ratio is diluted by the
longer intervals in which the SUT operates close to its noiseless longer intervals in which the SUT operates close to its noiseless
performance; consequently, the averaged Trial Loss Ratio can still performance; consequently, the averaged Trial Loss Ratio can still
end up below the specified Goal Loss Ratio value. end up below the specified Goal Loss Ratio value.
* If an approximation of the SUT noise impact on the Trial Loss * If an approximation of the SUT noise impact on the Trial Loss
Ratio is known, it can be set as the Goal Loss Ratio (see Ratio is known, it can be set as the Goal Loss Ratio (see
definitions of Trial and Goal terms in Trial Terms (Section 4.5) definitions of Trial and Goal terms in "Trial Terms" (Section 4.5)
and Goal Terms (Section 4.6)). and "Goal Terms" (Section 4.6)).
For more information, see Section 5 of an earlier draft For more information, see Section 5 of [Lencze-Shima] (and the
[Lencze-Shima] (and references there) for few synthetic examples, references there) for a few synthetic examples, confirming that each
confirming that each protocol and application can have different protocol and application can have different realistic loss ratio
realistic loss ratio value. values.
Regardless of the validity of all similar motivations, support for Regardless of the validity of all similar motivations, support for
non-zero loss goals makes a search algorithm applicable for a wider non-zero loss goals makes a search algorithm applicable for a wider
range of use cases than the approach defined in [RFC2544]. range of use cases than the approach defined in [RFC2544].
Furthermore, allowing users to specify multiple loss ratio values, Furthermore, allowing users to specify multiple loss ratio values,
and enabling a single search to find all relevant bounds, and enabling a single search to find all relevant bounds,
significantly enhances the usefulness of the search algorithm. significantly enhances the usefulness of the search algorithm.
Searching for multiple Search Goals also helps to describe the SUT Searching for multiple Search Goals also helps to describe the SUT
performance spectrum better than the result of a single Search Goal. performance spectrum better than the result of a single Search Goal.
For example, the repeated wide gap between zero and non-zero loss For example, the repeated wide gap between zero and non-zero loss
loads indicates the noise has a large impact on the observed loads indicates the noise has a large impact on the observed
performance, which is not evident from a single goal load search performance, which is not evident from the result of a single goal
procedure result. load search procedure.
It is easy to modify the Binary Search (Section 2.1) to find a lower It is easy to modify the Binary Search (Section 2.1) to find a lower
bound for the load that satisfies a non-zero Goal Loss Ratio. But it bound for the load that satisfies a non-zero Goal Loss Ratio. But
is not that obvious how to search for multiple goals at once, hence how to search for multiple goals at once is not that obvious; hence,
the support for multiple Search Goals remains a problem. the support for multiple Search Goals remains a problem.
At the time of writing there does not seem to be a consensus in the At the time of writing, there does not seem to be a consensus in the
industry on which loss ratio value is the best. For users, industry on which loss ratio value is the best. For users,
performance of higher protocol layers is important, for example, performance of higher protocol layers is important, for example,
goodput of TCP connection (TCP throughput, [RFC6349]), but goodput of TCP connection (TCP throughput [RFC6349]), but the
relationship between goodput and loss ratio is not simple. Refer to relationship between goodput and loss ratio is not simple. Refer to
[Lencze-Kovacs-Shima] for examples of various corner cases, Section 3 [Lencze-Kovacs-Shima] for examples of various corner cases, Section 3
of [RFC6349] for loss ratios acceptable for an accurate measurement of [RFC6349] for loss ratios acceptable for an accurate measurement
of TCP throughput, and [Ott-Mathis-Semke-Mahdavi] for models and of TCP throughput, and [Ott-Mathis-Semke-Mahdavi] for models and
calculations of TCP performance in presence of packet loss. calculations of TCP performance in presence of packet loss.
2.6. Inconsistent Trial Results 2.6. Inconsistent Trial Results
While performing throughput search by executing a sequence of While performing throughput search by executing a sequence of
measurement trials, there is a risk of encountering inconsistencies measurement trials, there is a risk of encountering inconsistencies
between trial results. between trial results.
Examples include, but are not limited to: Examples include but are not limited to:
* A trial at the same load (same or different trial duration) * A trial at the same load (same or different trial duration)
results in a different Trial Loss Ratio. results in a different Trial Loss Ratio.
* A trial at a larger load (same or different trial duration) * A trial at a larger load (same or different trial duration)
results in a lower Trial Loss Ratio. results in a lower Trial Loss Ratio.
The Binary Search (Section 2.1) never encounters inconsistent trials. The Binary Search (Section 2.1) never encounters inconsistent trials.
But [RFC2544] hints about the possibility of inconsistent trial But [RFC2544] hints about the possibility of inconsistent trial
results, in two places in its text. The first place is Section 24 of results in two places in its text. The first place is Section 24 of
[RFC2544], where full trial durations are required, presumably [RFC2544], where full trial durations are required, presumably
because they can be inconsistent with the results from short trial because they can be inconsistent with the results from short trial
durations. The second place is Section 26.3 of [RFC2544], where two durations. The second place is Section 26.3 of [RFC2544], where two
successive zero-loss trials are recommended, presumably because after successive zero-loss trials are recommended, presumably because after
one zero-loss trial there can be a subsequent inconsistent non-zero- one zero-loss trial, there can be a subsequent inconsistent non-zero-
loss trial. loss trial.
A robust throughput search algorithm needs to decide how to continue A robust throughput search algorithm needs to decide how to continue
the search in the presence of such inconsistencies. Definitions of the search in the presence of such inconsistencies. Definitions of
throughput and its test methodology in [RFC1242] and [RFC2544] are throughput and its test methodology in [RFC1242] and [RFC2544] are
not specific enough to imply a unique way of handling such not specific enough to imply a unique way of handling such
inconsistencies. inconsistencies.
Ideally, there will be a definition of a new quantity which both Ideally, there will be a definition of a new quantity that both
generalizes throughput for non-zero Goal Loss Ratio values (and other generalizes throughput for non-zero Goal Loss Ratio values (and other
possible repeatability enhancements), while being precise enough to possible repeatability enhancements) while being precise enough to
force a specific way to resolve trial result inconsistencies. But force a specific way to resolve trial result inconsistencies. But
until such a definition is agreed upon, the correct way to handle until such a definition is agreed upon, the correct way to handle
inconsistent trial results remains an open problem. inconsistent trial results remains an open problem.
Relevant Lower Bound is the MLRsearch term that addresses this "Relevant Lower Bound" is the MLRsearch term that addresses this
problem. problem.
3. Requirements Language 3. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP "OPTIONAL" in this document are to be interpreted as described in
14, [RFC2119] and [RFC8174] when, and only when, they appear in all BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
This document is categorized as an Informational RFC. While it does This document is categorized as an Informational RFC. While it does
not mandate the adoption of the MLRsearch methodology, it uses the not mandate the adoption of the MLRsearch methodology, it uses the
normative language of BCP 14 to provide an unambiguous specification. normative language of BCP 14 [RFC2119] [RFC8174] to provide an
This ensures that if a test procedure or test report claims unambiguous specification. This ensures that if a test procedure or
compliance with the MLRsearch Specification, it MUST adhere to all test report claims compliance with the MLRsearch Specification, it
the absolute requirements defined herein. The use of normative MUST adhere to all the absolute requirements defined herein. The use
language is intended to promote repeatable and comparable results of normative language is intended to promote repeatable and
among those who choose to implement this methodology. comparable results among those who choose to implement this
methodology.
4. MLRsearch Specification 4. MLRsearch Specification
This chapter provides all technical definitions needed for evaluating This section provides all technical definitions needed for evaluating
whether a particular test procedure complies with MLRsearch whether a particular test procedure complies with the MLRsearch
Specification. Specification.
Some terms used in the specification are capitalized. It is just a Some terms used in the specification are capitalized. This is a
stylistic choice for this document, reminding the reader this term is stylistic choice for this document, reminding the reader that the
introduced, defined or explained elsewhere in the document. Lower term is introduced, defined, or explained elsewhere in the document.
case variants are equally valid. Lowercase variants are equally valid.
This document does not separate terminology from methodology. Terms This document does not separate terminology from methodology. Terms
are fully specified and discussed in their own subsections, under are fully specified and discussed in their own subsections, under
sections titled "Terms". This way, the list of terms is visible in sections with the word "Terms" in the title. This way, the list of
table of contents. terms is visible in the table of contents.
Each per term subsection contains a short _Definition_ paragraph Each per term subsection contains a short "Definition" paragraph
containing a minimal definition and all strict requirements, followed containing a minimal definition and all strict requirements, followed
by _Discussion_ paragraphs focusing on important consequences and by "Discussion" paragraphs focusing on important consequences and
recommendations. Requirements about how other components can use the recommendations. Requirements about how other components can use the
defined quantity are also included in the discussion. defined quantity are also included in the discussion.
4.1. Scope 4.1. Scope
This document specifies the Multiple Loss Ratio search (MLRsearch) This document specifies the Multiple Loss Ratio search (MLRsearch)
methodology. The MLRsearch Specification details a new class of methodology. The MLRsearch Specification details a new class of
benchmarks by listing all terminology definitions and methodology benchmarks by listing all terminology definitions and methodology
requirements. The definitions support "multi-goal" benchmarks, with requirements. The definitions support "multi-goal" benchmarks, with
"single-goal" as a subset. "single-goal" as a subset.
The normative scope of this specification includes: The normative scope of this specification includes:
* The terminology for all required quantities and their attributes. * The terminology for all required quantities and their attributes.
* An abstract architecture consisting of functional components * An abstract architecture consisting of functional components
(Manager, Controller, Measurer) and the requirements for their (Manager, Controller, and Measurer) and the requirements for their
inputs and outputs. inputs and outputs.
* The required structure and attributes of the Controller Input, * The required structure and attributes of the Controller Input,
including one or more Search Goal instances. including one or more Search Goal instances.
* The required logic for Load Classification, which determines * The required logic for Load Classification, which determines
whether a given Trial Load qualifies as a Lower Bound or an Upper whether a given Trial Load qualifies as a Lower Bound or an Upper
Bound for a Search Goal. Bound for a Search Goal.
* The required structure and attributes of the Controller Output, * The required structure and attributes of the Controller Output,
including a Goal Result for each Search Goal. including a Goal Result for each Search Goal.
4.1.1. Relationship to RFC 2544 4.1.1. Relationship to RFC 2544
MLRsearch Specification is an independent methodology and does not The MLRsearch Specification is an independent methodology and does
change or obsolete any part of [RFC2544]. not change or obsolete any part of [RFC2544].
This specification permits deviations from the Trial procedure as This specification permits deviations from the Trial procedure as
described in [RFC2544]. Any deviation from the [RFC2544] procedure described in [RFC2544]. Any deviation from the procedure in
must be documented explicitly in the Test Report, and such variations [RFC2544] must be documented explicitly in the Test Report, and such
remain outside the scope of the original [RFC2544] benchmarks. variations remain outside the scope of the original benchmarks in
[RFC2544].
A specific single-goal MLRsearch benchmark can be configured to be A specific single-goal MLRsearch benchmark can be configured to be
compliant with [RFC2544] Throughput, and most procedures reporting compliant with [RFC2544] Throughput, and most procedures reporting
[RFC2544] Throughput can be adapted to satisfy also MLRsearch [RFC2544] Throughput can be adapted to also satisfy MLRsearch
requirements for specific search goal. requirements for a specific search goal.
4.1.2. Applicability of Other Specifications 4.1.2. Applicability of Other Specifications
Methodology extensions from other BMWG documents that specify details Methodology extensions from other BMWG documents that specify details
for testing particular DUTs, configurations, or protocols (e.g., by for testing particular DUTs, configurations, or protocols (e.g., by
defining a particular Traffic Profile) are considered orthogonal to defining a particular Traffic Profile) are considered orthogonal to
MLRsearch and are applicable to a benchmark conducted using MLRsearch MLRsearch and are applicable to a benchmark conducted using MLRsearch
methodology. methodology.
4.1.3. Out of Scope 4.1.3. Out of Scope
skipping to change at page 16, line 43 skipping to change at line 749
* The internal heuristics or algorithms used by the Controller to * The internal heuristics or algorithms used by the Controller to
select Trial Input values (e.g., the load selection strategy) are select Trial Input values (e.g., the load selection strategy) are
considered implementation details. considered implementation details.
* The potential for, and the effects of, interference between * The potential for, and the effects of, interference between
different Search Goal instances within a multiple-goal search are different Search Goal instances within a multiple-goal search are
considered outside the normative scope of this specification. considered outside the normative scope of this specification.
4.2. Architecture Overview 4.2. Architecture Overview
Although the normative text references only terminology that has Although the normative text only references terminology that has
already been introduced, explanatory passages beside it sometimes already been introduced, explanatory passages sometimes profit from
profit from terms that are defined later in the document. To keep terms that are defined later in the document. To keep the initial
the initial read-through clear, this informative section offers a read-through clear, this informative section offers a concise, top-
concise, top-down sketch of the complete MLRsearch architecture. down sketch of the complete MLRsearch architecture.
The architecture is modelled as a set of abstract, interacting The architecture is modelled as a set of abstract, interacting
components. Information exchange between components is expressed in components. Information exchange between components is expressed in
an imperative-programming style: one component "calls" another, an imperative-programming style: One component "calls" another,
supplying inputs (arguments) and receiving outputs (return values). supplying inputs (arguments) and receiving outputs (return values).
This notation is purely conceptual; actual implementations need not This notation is purely conceptual; actual implementations need not
exchange explicit messages. When the text contrasts alternative exchange explicit messages. When the text contrasts alternative
behaviours, it refers to the different implementations of the same behaviors, it refers to the different implementations of the same
component. component.
A test procedure is considered compliant with the MLRsearch A test procedure is considered compliant with the MLRsearch
Specification if it can be conceptually decomposed into the abstract Specification if it can be conceptually decomposed into the abstract
components defined herein, and each component satisfies the components defined herein and if each component satisfies the
requirements defined for its corresponding MLRsearch element. requirements defined for its corresponding MLRsearch element.
The Measurer component is tasked to perform Trials, the Controller The Measurer component is tasked to perform Trials, the Controller
component is tasked to select Trial Durations and Loads, the Manager component is tasked to select Trial Durations and Loads, and the
component is tasked to pre-configure involved entities and to produce Manager component is tasked to preconfigure involved entities and to
the Test Report. The Test Report explicitly states Search Goals (as produce the Test Report. The Test Report explicitly states Search
Controller Input) and corresponding Goal Results (Controller Output). Goals (as Controller Input) and corresponding Goal Results
(Controller Output).
This constitutes one benchmark (single-goal or multi-goal). Repeated This constitutes one benchmark (single-goal or multi-goal). Repeated
or slightly differing benchmarks are realized by calling Controller or slightly differing benchmarks are realized by calling the
once for each benchmark. Controller once for each benchmark.
The Manager calls a Controller once, and the Controller then invokes The Manager calls a Controller once, and the Controller then invokes
the Measurer repeatedly until Controller decides it has enough the Measurer repeatedly until the Controller decides it has enough
information to return outputs. information to return outputs.
The part during which the Controller invokes the Measurer is termed The part during which the Controller invokes the Measurer is termed
the Search. Any work the Manager performs either before invoking the the "Search". Any work the Manager performs, either before invoking
Controller or after Controller returns, falls outside the scope of the Controller or after Controller returns, falls outside the scope
the Search. of the Search.
MLRsearch Specification prescribes Regular Goal Results The MLRsearch Specification prescribes Regular Goal Results
(Section 4.8.4.1) and recommends corresponding search completion (Section 4.8.4.1) and recommends corresponding search completion
conditions. Irregular Goal Results (Section 4.8.4.2) are also conditions. Irregular Goal Results (Section 4.8.4.2) are also
allowed, they have different requirements and their corresponding allowed; they have different requirements, and their corresponding
stopping conditions are out of scope. The Search Result is the stopping conditions are out of scope. The Search Result is the
combination of regular and irregular results for each goal. combination of regular and irregular results for each goal.
Search Results are based on Load Classification. When measured Search Results are based on Load Classification. When measured
enough, a chosen Load can either achieve or fail each Search Goal enough, a chosen Load can either achieve or fail each Search Goal
(separately), thus becoming a Lower Bound or an Upper Bound for that (separately), thus becoming a Lower Bound or an Upper Bound for that
Search Goal. Search Goal.
When the Relevant Lower Bound is close enough to Relevant Upper Bound When the Relevant Lower Bound is close enough to the Relevant Upper
according to Goal Width, the Regular Goal Result is found. Search Bound according to Goal Width, the Regular Goal Result is found.
stops when all Regular Goal Results are found, or when some Search Search stops when all Regular Goal Results are found or when some
Goals are proven to have only Irregular Goal Results. Search Goals are proven to have only Irregular Goal Results.
4.2.1. Test Report 4.2.1. Test Report
A primary responsibility of the Manager is to produce a Test Report, A primary responsibility of the Manager is to produce a Test Report,
which serves as the final and formal output of the test procedure. which serves as the final and formal output of the test procedure.
This document does not provide a single, complete, normative This document does not provide a single, complete, normative
definition for the structure of the Test Report. For example, Test definition for the structure of the Test Report. For example, the
Report may contain results for a single benchmark, or it could Test Report may contain results for a single benchmark, or it could
aggregate results of many benchmarks. aggregate results of many benchmarks.
Instead, normative requirements for the content of the Test Report Instead, normative requirements for the content of the Test Report
are specified throughout this document in conjunction with the are specified throughout this document in conjunction with the
definitions of the quantities and procedures to which they apply. definitions of the quantities and procedures to which they apply.
Readers should note that any clause requiring a value to be Readers should note that any clause requiring a value to be
"reported" or "stated in the test report" constitutes a normative "reported" or "stated in the test report" constitutes a normative
requirement on the content of this final artifact. requirement on the content of this final artifact.
Even where not stated explicitly, the "Reporting format" paragraphs Even where not stated explicitly, the "Reporting format" paragraphs
in [RFC2544] sections are still requirements on Test Report if they in [RFC2544] are still requirements on Test Reports if they apply to
apply to a MLRsearch benchmark. an MLRsearch benchmark.
4.2.2. Behavior Correctness 4.2.2. Behavior Correctness
MLRsearch Specification by itself does not guarantee that the Search The MLRsearch Specification by itself does not guarantee that the
ends in finite time, as the freedom the Controller has for Load Search ends in finite time, as the freedom the Controller has for
selection also allows for clearly deficient choices. Load selection also allows for clearly deficient choices.
For deeper insights on these matters, refer to [FDio-CSIT-MLRsearch]. For deeper insights on these matters, refer to [FDio-CSIT-MLRsearch].
The primary MLRsearch implementation, used as the prototype for this The primary MLRsearch implementation, used as the prototype for this
specification, is [PyPI-MLRsearch]. specification, is [PyPI-MLRsearch].
4.3. Quantities 4.3. Quantities
MLRsearch Specification uses a number of specific quantities, some of The MLRsearch Specification uses a number of specific quantities;
them can be expressed in several different units. some of them can be expressed in several different units.
In general, MLRsearch Specification does not require particular units In general, the MLRsearch Specification does not require particular
to be used, but it is REQUIRED for the test report to state all the units to be used, but it is REQUIRED for the test report to state all
units. For example, ratio quantities can be dimensionless numbers the units. For example, ratio quantities can be dimensionless
between zero and one, but may be expressed as percentages instead. numbers between zero and one but may be expressed as percentages
instead.
For convenience, a group of quantities can be treated as a composite For convenience, a group of quantities can be treated as a composite
quantity. One constituent of a composite quantity is called an quantity. One constituent of a composite quantity is called an
attribute. A group of attribute values is called an instance of that attribute. A group of attribute values is called an instance of that
composite quantity. composite quantity.
Some attributes may depend on others and can be calculated from other Some attributes may depend on others and can be calculated from other
attributes. Such quantities are called derived quantities. attributes. Such quantities are called derived quantities.
4.3.1. Current and Final Values 4.3.1. Current and Final Values
Some quantities are defined in a way that makes it possible to Some quantities are defined in a way that makes it possible to
compute their values in the middle of a Search. Other quantities are compute their values in the middle of a Search. Other quantities are
specified so that their values can be computed only after a Search specified so that their values can be computed only after a Search
ends. Some quantities are important only after a Search ended, but ends. Some quantities are important only after a Search ends, but
their values are computable also before a Search ends. their values are also computable before a Search ends.
For a quantity that is computable before a Search ends, the adjective For a quantity that is computable before a Search ends, the adjective
*current* is used to mark a value of that quantity available before "current" is used to mark a value of that quantity available before
the Search ends. When such value is relevant for the search result, the Search ends. When such value is relevant for the search result,
the adjective *final* is used to denote the value of that quantity at the adjective "final" is used to denote the value of that quantity at
the end of the Search. the end of the Search.
If a time evolution of such a dynamic quantity is guided by If a time evolution of such a dynamic quantity is guided by
configuration quantities, those adjectives can be used to distinguish configuration quantities, those adjectives can be used to distinguish
quantities. For example, if the current value of "duration" (dynamic quantities. For example, if the current value of "duration" (dynamic
quantity) increases from "initial duration" to "final duration" quantity) increases from "initial duration" to "final duration"
(configuration quantities), all the quoted names denote separate but (configuration quantities), all the quoted names denote separate but
related quantities. As the naming suggests, the final value of related quantities. As the naming suggests, the final value of
"duration" is expected to be equal to "final duration" value. "duration" is expected to be equal to the "final duration" value.
4.4. Existing Terms 4.4. Existing Terms
This specification relies on the following three documents that This specification relies on the following three documents that
should be consulted before attempting to make use of this document: should be consulted before attempting to make use of this document:
* "Benchmarking Terminology for Network Interconnect Devices" * "Benchmarking Terminology for Network Interconnection Devices"
[RFC1242] contains basic term definitions. [RFC1242] contains basic term definitions.
* "Benchmarking Terminology for LAN Switching Devices" [RFC2285] * "Benchmarking Terminology for LAN Switching Devices" [RFC2285]
adds more terms and discussions, describing some known network includes more terms and discussions, and it describes some known
benchmarking situations in a more precise way. network benchmarking situations in a more precise way.
* "Benchmarking Methodology for Network Interconnect Devices" * "Benchmarking Methodology for Network Interconnect Devices"
[RFC2544] contains discussions about terms and additional [RFC2544] contains discussions about terms and additional
methodology requirements. methodology requirements.
Definitions of some central terms from above documents are copied and Definitions of some central terms from the above documents are copied
discussed in the following subsections. and discussed in the following subsections.
4.4.1. SUT 4.4.1. SUT
Defined in Section 3.1.2 of [RFC2285] as follows. SUT is defined in Section 3.1.2 of [RFC2285] as follows.
Definition: Definition:
The collective set of network devices to which stimulus is offered The collective set of network devices to which stimulus is offered
as a single entity and response measured. as a single entity and response measured.
Discussion: Discussion:
A SUT consisting of a single network device is allowed by this
An SUT consisting of a single network device is allowed by this
definition. definition.
In software-based networking SUT may comprise multitude of In software-based networking, SUT may comprise a multitude of
networking applications and the entire host hardware and software networking applications and the entire host hardware and software
execution environment. execution environment.
SUT is the only entity that can be benchmarked directly, even SUT is the only entity that can be benchmarked directly, even
though only the performance of some sub-components are of though only the performance of some sub-components are of
interest. interest.
4.4.2. DUT 4.4.2. DUT
Defined in Section 3.1.1 of [RFC2285] as follows. DUT is defined in Section 3.1.1 of [RFC2285] as follows.
Definition: Definition:
The network forwarding device to which stimulus is offered and The network forwarding device to which stimulus is offered and
response measured. response measured.
Discussion: Discussion:
Contrary to SUT, the DUT stimulus and response are frequently Contrary to SUT, the DUT stimulus and response are frequently
initiated and observed only indirectly, on different parts of SUT. initiated and observed only indirectly, on different parts of SUT.
DUT, as a sub-component of SUT, is only indirectly mentioned in DUT, as a sub-component of SUT, is only indirectly mentioned in
MLRsearch Specification, but is of key relevance for its the MLRsearch Specification but is of key relevance for its
motivation. The device can represent a software-based networking motivation. The device can represent a software-based networking
functions running on commodity x86/ARM CPUs (vs purpose-built ASIC function running on commodity x86/ARM CPUs (vs. purpose-built
/ NPU / FPGA). ASICs, NPUs, and FPGAs).
A well-designed SUTs should have the primary DUT as their A well-designed SUT should have the primary DUT as their
performance bottleneck. The ways to achieve that are outside of performance bottleneck. The ways to achieve that are outside the
MLRsearch Specification scope. scope of the MLRsearch Specification.
4.4.3. Trial 4.4.3. Trial
A trial is the part of the test described in Section 23 of [RFC2544]. A trial is the part of the test described in Section 23 of [RFC2544].
Definition: Definition:
A particular test consists of multiple trials. Each trial returns A particular test consists of multiple trials. Each trial returns
one piece of information, for example the loss rate at a one piece of information, for example, the loss rate at a
particular input frame rate. Each trial consists of a number of particular input frame rate. Each trial consists of a number of
phases: phases:
a) If the DUT is a router, send the routing update to the "input" a) If the DUT is a router, send the routing update to the "input"
port and pause two seconds to be sure that the routing has port and pause two seconds to be sure that the routing has
settled. settled.
b) Send the "learning frames" to the "output" port and wait 2 b) Send the "learning frames" to the "output" port and wait 2
seconds to be sure that the learning has settled. Bridge learning seconds to be sure that the learning has settled. Bridge learning
frames are frames with source addresses that are the same as the frames are frames with source addresses that are the same as the
skipping to change at page 21, line 29 skipping to change at line 969
tables in the DUT. The formats of the learning frame that should tables in the DUT. The formats of the learning frame that should
be used are shown in the Test Frame Formats document. be used are shown in the Test Frame Formats document.
c) Run the test trial. c) Run the test trial.
d) Wait for two seconds for any residual frames to be received. d) Wait for two seconds for any residual frames to be received.
e) Wait for at least five seconds for the DUT to restabilize. e) Wait for at least five seconds for the DUT to restabilize.
Discussion: Discussion:
The traffic is sent only in phase c) and received in phases c) and The traffic is sent only in phase c) and received in phases c) and
d). d).
Trials are the only stimuli the SUT is expected to experience Trials are the only stimuli the SUT is expected to experience
during the Search. during the Search.
In some discussion paragraphs, it is useful to consider the In some "Discussion" paragraphs, it is useful to consider the
traffic as sent and received by a tester, as implicitly defined in traffic as sent and received by a tester, as implicitly defined in
Section 6 of [RFC2544]. Section 6 of [RFC2544].
The definition describes some traits, not using capitalized verbs The definition describes some traits, not using capitalized verbs
to signify strength of the requirements. For the purposes of the to signify the strength of the requirements. For the purposes of
MLRsearch Specification, the test procedure MAY deviate from the the MLRsearch Specification, the test procedure MAY deviate from
[RFC2544] description, but any such deviation MUST be described the description in [RFC2544], but any such deviation MUST be
explicitly in the Test Report. It is still RECOMMENDED to not described explicitly in the Test Report. It is still RECOMMENDED
deviate from the description, as any deviation weakens to not deviate from the description, as any deviation weakens
comparability. comparability.
An example of deviation from [RFC2544] is using shorter wait An example of deviation from [RFC2544] is using shorter wait
times, compared to those described in phases a), b), d) and e). times, compared to those described in phases a), b), d), and e).
The [RFC2544] document itself seems to be treating phase b) as any [RFC2544] seems to treat phase b) as any type of configuration
type of configuration that cannot be configured only once (by that cannot be configured only once (by the Manager, before the
Manager, before Search starts), as some crucial SUT state could Search starts), as some crucial SUT state could time out during
time-out during the Search. It is RECOMMENDED to interpret the the Search. It is RECOMMENDED to interpret the "learning frames"
"learning frames" to be any such time-sensitive per-trial to be any such time-sensitive per-trial configuration method, with
configuration method, with bridge MAC learning being only one bridge MAC learning being only one possible example.
possible examples. Appendix C.2.4.1 of [RFC2544] lists another Appendix C.2.4.1 of [RFC2544] lists another example: ARP with a
example: ARP with wait time of 5 seconds. wait time of 5 seconds.
Some methodologies describe recurring tests. If those are based Some methodologies describe recurring tests. If those are based
on Trials, they are treated as multiple independent Trials. on Trials, they are treated as multiple independent Trials.
4.5. Trial Terms 4.5. Trial Terms
This section defines new and redefine existing terms for quantities This section defines new terms and redefines existing terms for
relevant as inputs or outputs of a Trial, as used by the Measurer quantities relevant as inputs or outputs of a Trial, as used by the
component. This includes also any derived quantities related to Measurer component. This also includes any derived quantities
results of one Trial. related to results of one Trial.
4.5.1. Trial Duration 4.5.1. Trial Duration
Definition: Definition:
Trial Duration is the intended duration of phase c) of a Trial.
Trial Duration is the intended duration of the phase c) of a
Trial.
Discussion: Discussion:
The value MUST be positive. The value MUST be positive.
While any positive real value may be provided, some Measurer While any positive real value may be provided, some Measurer
implementations MAY limit possible values, e.g., by rounding down implementations MAY limit possible values, e.g., by rounding down
to nearest integer in seconds. In that case, it is RECOMMENDED to to the nearest integer in seconds. In that case, it is
give such inputs to the Controller so that the Controller only RECOMMENDED to give such inputs to the Controller so that the
uses the accepted values. Controller only uses the accepted values.
4.5.2. Trial Load 4.5.2. Trial Load
Definition: Definition:
Trial Load is the per-interface Intended Load for a Trial. Trial Load is the per-interface Intended Load for a Trial.
Discussion: Discussion:
Trial Load is equivalent to the quantities defined as constant Trial Load is equivalent to the quantities defined as constant
load (Section 3.4 of [RFC1242]), data rate (Section 14 of load (Section 3.4 of [RFC1242]), data rate (Section 14 of
[RFC2544]), and Intended Load (Section 3.5.1 of [RFC2285]), in the [RFC2544]), and Intended Load (Section 3.5.1 of [RFC2285]), in the
sense that all three definitions specify that this value applies sense that all three definitions specify that this value applies
to one (input or output) interface. to one (input or output) interface.
For specification purposes, it is assumed that this is a constant For specification purposes, it is assumed that this is a constant
load by default, as specified in Section 3.4 of [RFC1242]). load by default, as specified in Section 3.4 of [RFC1242].
Informally, Traffic Load is a single number that can "scale" any Informally, Traffic Load is a single number that can "scale" any
traffic pattern as long as the intuition of load intended against traffic pattern as long as the intuition of load intended against
a single interface can be applied. a single interface can be applied.
It MAY be possible to use a Trial Load value to describe a non- It MAY be possible to use a Trial Load value to describe non-
constant traffic (using average load when the traffic consists of constant traffic (using average load when the traffic consists of
repeated bursts of frames e.g., as suggested in Section 21 of repeated bursts of frames, e.g., as suggested in Section 21 of
[RFC2544]). In the case of a non-constant load, the Test Report [RFC2544]). In the case of a non-constant load, the Test Report
MUST explicitly mention how exactly non-constant the traffic is MUST explicitly mention exactly how non-constant the traffic is
and how it reacts to Traffic Load value. But the rest of the and how it reacts to a Traffic Load value. But the rest of the
MLRsearch Specification assumes that is not the case, to avoid MLRsearch Specification assumes that is not the case, to avoid
discussing corner cases (e.g., which values are possible within discussing corner cases (e.g., which values are possible within
medium limitations). medium limitations).
Similarly, traffic patterns where different interfaces are subject Similarly, traffic patterns where different interfaces are subject
to different loads MAY be described by a single Trial Load value to different loads MAY be described by a single Trial Load value
(e.g. using largest load among interfaces), but again the Test (e.g., using the largest load among interfaces), but again, the
Report MUST explicitly describe how the traffic pattern reacts to Test Report MUST explicitly describe how the traffic pattern
Traffic Load value, and this specification does not discuss all reacts to the Traffic Load value, and this specification does not
the implications of that approach. discuss all the implications of that approach.
In the common case of bidirectional traffic, as described in In the common case of bidirectional traffic, as described in
Section 14. Bidirectional Traffic of [RFC2544], Trial Load is the "Bidirectional traffic" (Section 14 of [RFC2544]), Trial Load is
data rate per direction, half of aggregate data rate. the data rate per direction, half of the aggregate data rate.
Traffic patterns where a single Trial Load does not describe their Traffic patterns where a single Trial Load does not describe their
scaling cannot be used for MLRsearch benchmarks. scaling cannot be used for MLRsearch benchmarks.
Similarly to Trial Duration, some Measurers MAY limit the possible Similarly to Trial Duration, some Measurers MAY limit the possible
values of Trial Load. Contrary to Trial Duration, documenting values of Trial Load. Contrary to Trial Duration, documenting
such behavior in the test report is OPTIONAL. This is because the such behavior in the test report is OPTIONAL. This is because the
load differences are negligible (and frequently undocumented) in load differences are negligible (and frequently undocumented) in
practice. practice.
The Controller MAY select Trial Load and Trial Duration values in The Controller MAY select the Trial Load and Trial Duration values
a way that would not be possible to achieve using any integer in a way that would not be possible to achieve using any integer
number of data frames. number of data frames.
If a particular Trial Load value is not tied to a single Trial, If a particular Trial Load value is not tied to a single Trial,
e.g., if there are no Trials yet or if there are multiple Trials, e.g., if there are no Trials yet or if there are multiple Trials,
this document uses a shorthand *Load*. this document uses a shorthand "Load".
The test report MAY present the aggregate load across multiple The test report MAY present the aggregate load across multiple
interfaces, treating it as the same quantity expressed using interfaces, treating it as the same quantity expressed using
different units. Each reported Trial Load value MUST state different units. Each reported Trial Load value MUST
unambiguously whether it refers to (i) a single interface, (ii) a unambiguously state whether it refers to (i) a single interface,
specified subset of interfaces (such as all logical interfaces (ii) a specified subset of interfaces (such as all logical
mapped to one physical port), or (iii) the total across every interfaces mapped to one physical port), or (iii) the total across
interface. For any aggregate load value, the report MUST also every interface. For any aggregate load value, the report MUST
give the fixed conversion factor that links the per-interface and also give the fixed conversion factor that links the per-interface
multi-interface load values. and multi-interface load values.
The per-interface value remains the primary unit, consistent with The per-interface value remains the primary unit, consistent with
prevailing practice in [RFC1242], [RFC2544], and [RFC2285]. the prevailing practice described in [RFC1242], [RFC2544], and
[RFC2285].
The last paragraph also applies to other terms related to Load.
For example, tests with symmetric bidirectional traffic can report The previous paragraph also applies to other terms related to
load-related values as "bidirectional load" (double of Load. For example, tests with symmetric bidirectional traffic can
report load-related values as "bidirectional load" (double of
"unidirectional load"). "unidirectional load").
4.5.3. Trial Input 4.5.3. Trial Input
Definition: Definition:
Trial Input is a composite quantity, consisting of two attributes: Trial Input is a composite quantity, consisting of two attributes:
Trial Duration and Trial Load. Trial Duration and Trial Load.
Discussion: Discussion:
When talking about multiple Trials, it is common to say "Trial When talking about multiple Trials, it is common to say "Trial
Inputs" to denote all corresponding Trial Input instances. Inputs" to denote all corresponding Trial Input instances.
A Trial Input instance acts as the input for one call of the A Trial Input instance acts as the input for one call of the
Measurer component. Measurer component.
Contrary to other composite quantities, MLRsearch implementations Contrary to other composite quantities, MLRsearch implementations
MUST NOT add optional attributes into Trial Input. This improves MUST NOT add optional attributes into Trial Input. This improves
interoperability between various implementations of a Controller interoperability between various implementations of a Controller
and a Measurer. and a Measurer.
Note that both attributes are *intended* quantities, as only those Note that both attributes are *intended* quantities, as only those
can be fully controlled by the Controller. The actual offered can be fully controlled by the Controller. The actual offered
quantities, as realized by the Measurer, can be different (and quantities, as realized by the Measurer, can be different (and
must be different if not multiplying into integer number of must be different if not multiplying into an integer number of
frames), but questions around those offered quantities are frames), but questions around those offered quantities are
generally outside of the scope of this document. generally outside of the scope of this document.
4.5.4. Traffic Profile 4.5.4. Traffic Profile
Definition: Definition:
Traffic Profile is a composite quantity containing all attributes Traffic Profile is a composite quantity containing all attributes
other than Trial Load and Trial Duration, that are needed for other than Trial Load and Trial Duration that are needed for the
unique determination of the Trial to be performed. unique determination of the Trial to be performed.
Discussion: Discussion:
All the attributes are assumed to be constant during the Search, All the attributes are assumed to be constant during the Search,
and the composite is configured on the Measurer by the Manager and the composite is configured on the Measurer by the Manager
before the Search starts. This is why the traffic profile is not before the Search starts. This is why the traffic profile is not
part of the Trial Input. part of the Trial Input.
Specification of traffic properties included in the Traffic Specification of traffic properties included in the Traffic
Profile is the responsibility of the Manager, but the specific Profile is the responsibility of the Manager, but the specific
configuration mechanisms are outside of the scope of this configuration mechanisms are outside of the scope of this
document. document.
Informally, implementations of the Manager and the Measurer must Informally, implementations of the Manager and the Measurer must
be aware of their common set of capabilities, so that Traffic be aware of their common set of capabilities, so that the Traffic
Profile instance uniquely defines the traffic during the Search. Profile instance uniquely defines the traffic during the Search.
Typically, Manager and Measurer implementations are tightly Typically, Manager and Measurer implementations are tightly
integrated. integrated.
Integration efforts between independent Manager and Measurer Integration efforts between independent Manager and Measurer
implementations are outside of the scope of this document. An implementations are outside of the scope of this document. An
example standardization effort is [Vassilev]. example standardization effort is described in [Vassilev].
Examples of traffic properties include: Examples of traffic properties include:
Data link frame size: * Data link frame size:
* Fixed sizes as listed in Section 3.5 of [RFC1242] and in - Fixed sizes as listed in Section 3.5 of [RFC1242] and in
Section 9 of [RFC2544] Section 9 of [RFC2544]
* IMIX mixed sizes as defined in [RFC6985] - Internet Mix (IMIX) mixed sizes as defined in [RFC6985]
Frame formats and protocol addresses: * Frame formats and protocol addresses:
* Sections 8, 12 and Appendix C of [RFC2544] - Sections 8 and 12 of [RFC2544] and Appendix C of [RFC2544]
Symmetric bidirectional traffic: * Symmetric bidirectional traffic:
* Section 14 of [RFC2544]. - Section 14 of [RFC2544]
Other traffic properties that need to be somehow specified in Other traffic properties that need to somehow be specified in
Traffic Profile, and MUST be mentioned in Test Report if they Traffic Profile, and MUST be mentioned in Test Report if they
apply to the benchmark, include: apply to the benchmark, include:
* bidirectional traffic from Section 14 of [RFC2544], * bidirectional traffic from Section 14 of [RFC2544],
* fully meshed traffic from Section 3.3.3 of [RFC2285], * fully meshed traffic from Section 3.3.3 of [RFC2285],
* modifiers from Section 11 of [RFC2544]. * modifiers from Section 11 of [RFC2544], and
* IP version mixing from Section 5.3 of [RFC8219]. * IP version mixing from Section 5.3 of [RFC8219].
4.5.5. Trial Forwarding Ratio 4.5.5. Trial Forwarding Ratio
Definition: Definition:
The Trial Forwarding Ratio is a dimensionless floating point The Trial Forwarding Ratio is a dimensionless floating point
value. It MUST range between 0.0 and 1.0, both inclusive. It is value. It MUST range between 0.0 and 1.0, both inclusive. It is
calculated by dividing the number of frames successfully forwarded calculated by dividing the number of frames successfully forwarded
by the SUT by the total number of frames expected to be forwarded by the SUT by the total number of frames expected to be forwarded
during the trial. during the trial.
Discussion: Discussion:
For most Traffic Profiles, "expected to be forwarded" means For most Traffic Profiles, "expected to be forwarded" means
"intended to get received by SUT from tester". This SHOULD be the "intended to get received by a SUT from the tester". This SHOULD
default interpretation. Only if this is not the case, the test be the default interpretation. However, if this is not the case,
report MUST describe the Traffic Profile in a detail sufficient to the test report MUST describe the Traffic Profile in a detail
imply how Trial Forwarding Ratio should be calculated. sufficient to imply how the Trial Forwarding Ratio should be
calculated.
Trial Forwarding Ratio MAY be expressed in other units (e.g., as a The Trial Forwarding Ratio MAY be expressed in other units (e.g.,
percentage) in the test report. as a percentage) in the test report.
Note that, contrary to Load terms, frame counts used to compute Note that, contrary to Load terms, frame counts used to compute
Trial Forwarding Ratio are generally aggregates over all SUT the Trial Forwarding Ratio are generally aggregates over all SUT
output interfaces, as most test procedures verify all outgoing output interfaces, as most test procedures verify all outgoing
frames. The procedure for [RFC2544] Throughput counts received frames. The procedure for [RFC2544] Throughput counts received
frames, so implicitly it implies bidirectional counts for frames, so it implies bidirectional counts for bidirectional
bidirectional traffic, even though the final value is "rate" that traffic, even though the final value is the "rate" that is still
is still per-interface. per-interface. For example, in a test with symmetric
bidirectional traffic, if one direction is forwarded without
For example, in a test with symmetric bidirectional traffic, if losses, but the opposite direction does not forward at all, the
one direction is forwarded without losses, but the opposite Trial Forwarding Ratio would be 0.5 (50%).
direction does not forward at all, the Trial Forwarding Ratio
would be 0.5 (50%).
In future extensions, more general ways to compute Trial In future extensions, more general ways to compute the Trial
Forwarding Ratio may be allowed, but the current MLRsearch Forwarding Ratio may be allowed, but the current MLRsearch
Specification relies on this specific averaged counters approach. Specification relies on this specific averaged counters approach.
4.5.6. Trial Loss Ratio 4.5.6. Trial Loss Ratio
Definition: Definition:
The Trial Loss Ratio is equal to one minus the Trial Forwarding The Trial Loss Ratio is equal to one minus the Trial Forwarding
Ratio. Ratio.
Discussion: Discussion:
100% minus the Trial Forwarding Ratio, when expressed as a 100% minus the Trial Forwarding Ratio, when expressed as a
percentage. percentage.
This is almost identical to Frame Loss Rate of Section 3.6 of This is almost identical to Frame Loss Rate in Section 3.6 of
[RFC1242]. The only minor differences are that Trial Loss Ratio [RFC1242]. The only minor differences are that Trial Loss Ratio
does not need to be expressed as a percentage, and Trial Loss does not need to be expressed as a percentage, and Trial Loss
Ratio is explicitly based on averaged frame counts when more than Ratio is explicitly based on averaged frame counts when more than
one data stream is present. one data stream is present.
4.5.7. Trial Forwarding Rate 4.5.7. Trial Forwarding Rate
Definition: Definition:
The Trial Forwarding Rate is a derived quantity, calculated by The Trial Forwarding Rate is a derived quantity, calculated by
multiplying the Trial Load by the Trial Forwarding Ratio. multiplying the Trial Load by the Trial Forwarding Ratio.
Discussion: Discussion:
This quantity differs from the forwarding rate described in
This quantity differs from the Forwarding Rate described in Section 3.6.1 of [RFC2285]. Under the method described in
Section 3.6.1 of [RFC2285]. Under the RFC 2285 method, each [RFC2285], each output interface is measured separately, so every
output interface is measured separately, so every interface may interface may report a distinct rate. The Trial Forwarding Rate,
report a distinct rate. The Trial Forwarding Rate, by contrast, by contrast, uses a single set of frame counts and therefore
uses a single set of frame counts and therefore yields one value yields one value that represents the whole system while still
that represents the whole system, while still preserving the preserving the direct link to the per-interface load.
direct link to the per-interface load.
When the Traffic Profile is symmetric and bidirectional, as When the Traffic Profile is symmetric and bidirectional, as
defined in Section 14 of [RFC2544], the Trial Forwarding Rate is defined in Section 14 of [RFC2544], the Trial Forwarding Rate is
numerically equal to the arithmetic average of the individual per- numerically equal to the arithmetic average of the individual per-
interface forwarding rates that would be produced by the RFC 2285 interface forwarding rates that would be produced by the procedure
procedure. described in [RFC2285].
For more complex traffic patterns, such as many-to-one as For more complex traffic patterns, such as many-to-one, as
mentioned in Section 3.3.2 Partially Meshed Traffic of [RFC2285], mentioned in "Partially meshed traffic" (Section 3.3.2 of
the meaning of Trial Forwarding Rate is less straightforward. For [RFC2285]), the meaning of Trial Forwarding Rate is less
example, if two input interfaces receive one million frames per straightforward. For example, if two input interfaces receive one
second each, and a single interface outputs 1.4 million frames per million frames per second (fps) each and a single interface
second (fps), Trial Load is 1 million fps, Trial Loss Ratio is outputs 1.4 million frames per second (fps), the Trial Load is 1
30%, and Trial Forwarding Rate is 0.7 million fps. million fps, the Trial Loss Ratio is 30%, and the Trial Forwarding
Rate is 0.7 million fps.
Because this rate is anchored to the Load defined for one Because this rate is anchored to the Load defined for one
interface, a test report MAY show it either as the single averaged interface, a test report MAY show it either as the single averaged
figure just described, or as the sum of the separate per-interface figure just described or as the sum of the separate per-interface
forwarding rates. For the example above, the aggregate trial forwarding rates. For the example above, the aggregate Trial
forwarding rate is 1.4 million fps. Forwarding Rate is 1.4 million fps.
4.5.8. Trial Effective Duration 4.5.8. Trial Effective Duration
Definition: Definition:
The Trial Effective Duration is a time quantity related to a
Trial Effective Duration is a time quantity related to a Trial, by Trial. By default, it is equal to the Trial Duration.
default equal to the Trial Duration.
Discussion: Discussion:
This is an optional feature. If the Measurer does not return any This is an optional feature. If the Measurer does not return any
Trial Effective Duration value, the Controller MUST use the Trial Trial Effective Duration value, the Controller MUST use the Trial
Duration value instead. Duration value instead.
Trial Effective Duration may be any positive time quantity chosen The Trial Effective Duration may be any positive time quantity
by the Measurer to be used for time-based decisions in the chosen by the Measurer to be used for time-based decisions in the
Controller. Controller.
The test report MUST explain how the Measurer computes the The test report MUST explain how the Measurer computes the
returned Trial Effective Duration values, if they are not always returned Trial Effective Duration values if they are not always
equal to the Trial Duration. equal to the Trial Duration.
This feature can be beneficial for time-critical benchmarks This feature can be beneficial for time-critical benchmarks
designed to manage the overall search duration, rather than solely designed to manage the overall search duration, rather than solely
the traffic portion of it. An approach is to measure the duration the traffic portion of it. An approach is to measure the duration
of the whole trial (including all wait times) and use that as the of the whole trial (including all wait times) and use that as the
Trial Effective Duration. Trial Effective Duration.
This is also a way for the Measurer to inform the Controller about This is also a way for the Measurer to inform the Controller about
its surprising behavior, for example, when rounding the Trial its surprising behavior, for example, when rounding the Trial
skipping to change at page 28, line 45 skipping to change at line 1299
of the whole trial (including all wait times) and use that as the of the whole trial (including all wait times) and use that as the
Trial Effective Duration. Trial Effective Duration.
This is also a way for the Measurer to inform the Controller about This is also a way for the Measurer to inform the Controller about
its surprising behavior, for example, when rounding the Trial its surprising behavior, for example, when rounding the Trial
Duration value. Duration value.
4.5.9. Trial Output 4.5.9. Trial Output
Definition: Definition:
Trial Output is a composite quantity consisting of several Trial Output is a composite quantity consisting of several
attributes. Required attributes are: Trial Loss Ratio, Trial attributes. The required attributes are Trial Loss Ratio, Trial
Effective Duration and Trial Forwarding Rate. Effective Duration, and Trial Forwarding Rate.
Discussion: Discussion:
When referring to more than one trial, the plural term "Trial
When referring to more than one trial, plural term "Trial Outputs" Outputs" is used to collectively describe multiple Trial Output
is used to collectively describe multiple Trial Output instances. instances.
Measurer implementations may provide additional optional Measurer implementations may provide additional optional
attributes. The Controller implementations SHOULD ignore values attributes. The Controller implementations SHOULD ignore values
of any optional attribute they are not familiar with, except when of any optional attribute they are not familiar with, except when
passing Trial Output instances to the Manager. passing Trial Output instances to the Manager.
Example of an optional attribute: The aggregate number of frames An example of an optional attribute is the aggregate number of
expected to be forwarded during the trial, especially if it is not frames expected to be forwarded during the trial, especially if it
(a rounded-down value) implied by Trial Load and Trial Duration. is not (a rounded-down value) implied by Trial Load and Trial
Duration.
While Section 3.5.2 of [RFC2285] requires the Offered Load value While Section 3.5.2 of [RFC2285] requires the Offered Load value
to be reported for forwarding rate measurements, it is not to be reported for forwarding rate measurements, it is not
required in MLRsearch Specification, as search results do not required in the MLRsearch Specification, as search results do not
depend on it. depend on it.
4.5.10. Trial Result 4.5.10. Trial Result
Definition: Definition:
Trial Result is a composite quantity, consisting of the Trial Trial Result is a composite quantity, consisting of the Trial
Input and the Trial Output. Input and the Trial Output.
Discussion: Discussion:
When referring to more than one trial, the plural term "Trial
When referring to more than one trial, plural term "Trial Results" Results" is used to collectively describe multiple Trial Result
is used to collectively describe multiple Trial Result instances. instances.
4.6. Goal Terms 4.6. Goal Terms
This section defines new terms for quantities relevant (directly or This section defines new terms for quantities relevant (directly or
indirectly) for inputs and outputs of the Controller component. indirectly) for inputs and outputs of the Controller component.
Several goal attributes are defined before introducing the main Several goal attributes are defined before introducing the main
composite quantity: the Search Goal. composite quantity: the Search Goal.
Contrary to other sections, definitions in subsections of this Contrary to other sections, definitions in subsections of this
section are necessarily vague, as their fundamental meaning is to act section are necessarily vague, as their fundamental meaning is to act
as coefficients in formulas for Controller Output, which are not as coefficients in formulas for Controller Output, which are not
defined yet. defined yet.
The discussions in this section relate the attributes to concepts The discussions in this section relate the attributes to concepts
mentioned in Overview of RFC 2544 Problems (Section 2), but even mentioned in "Overview of RFC 2544 Problems" (Section 2), but these
these discussion paragraphs are short, informal, and mostly discussion paragraphs are short and informal, and they mostly
referencing later sections, where the impact on search results is reference later sections, where the impact on search results is
discussed after introducing the complete set of auxiliary terms. discussed after introducing the complete set of auxiliary terms.
4.6.1. Goal Final Trial Duration 4.6.1. Goal Final Trial Duration
Definition: Definition:
This is the minimal value for Trial Duration that must be reached.
Minimal value for Trial Duration that must be reached. The value The value MUST be positive.
MUST be positive.
Discussion: Discussion:
Certain trials must reach this minimum duration before a load can Certain trials must reach this minimum duration before a load can
be classified as a lower bound. be classified as a lower bound.
The Controller may choose shorter durations, results of those may The Controller may choose shorter durations; results of those may
be enough for classification as an Upper Bound. be enough for classification as an Upper Bound.
It is RECOMMENDED for all search goals to share the same Goal It is RECOMMENDED for all search goals to share the same Goal
Final Trial Duration value. Otherwise, Trial Duration values Final Trial Duration value. Otherwise, Trial Duration values
larger than the Goal Final Trial Duration may occur, weakening the larger than the Goal Final Trial Duration may occur, weakening the
assumptions the Load Classification Logic (Section 6.1) is based assumptions the Load Classification Logic (Section 6.1) is based
on. on.
4.6.2. Goal Duration Sum 4.6.2. Goal Duration Sum
Definition: Definition:
This is a threshold value for a particular sum of Trial Effective
A threshold value for a particular sum of Trial Effective Duration Duration values. The value MUST be positive.
values. The value MUST be positive.
Discussion: Discussion:
Informally, this prescribes the sufficient number of trials Informally, this prescribes the sufficient number of trials
performed at a specific Trial Load and Goal Final Trial Duration performed at a specific Trial Load and Goal Final Trial Duration
during the search. during the search.
If the Goal Duration Sum is larger than the Goal Final Trial If the Goal Duration Sum is larger than the Goal Final Trial
Duration, multiple trials may be needed to be performed at the Duration, multiple trials may be needed to be performed at the
same load. same load.
Refer to MLRsearch Compliant with TST009 (Section 4.10.3) for an Refer to "MLRsearch Compliant with TST009" (Section 4.10.3) for an
example where the possibility of multiple trials at the same load example where the possibility of multiple trials at the same load
is intended. is intended.
A Goal Duration Sum value shorter than the Goal Final Trial A Goal Duration Sum value shorter than the Goal Final Trial
Duration (of the same goal) could save some search time, but is Duration (of the same goal) could save some search time but is NOT
NOT RECOMMENDED, as the time savings come at the cost of decreased RECOMMENDED, as the time savings come at the cost of decreased
repeatability. repeatability.
In practice, the Search can spend less than Goal Duration Sum In practice, the Search can spend less than the Goal Duration Sum
measuring a Load value when the results are particularly one- measuring a Load value when the results are particularly one-
sided, but also, the Search can spend more than Goal Duration Sum sided, but also, the Search can spend more than the Goal Duration
measuring a Load when the results are balanced and include trials Sum measuring a Load when the results are balanced and include
shorter than Goal Final Trial Duration. trials shorter than the Goal Final Trial Duration.
4.6.3. Goal Loss Ratio 4.6.3. Goal Loss Ratio
Definition: Definition:
This is a threshold value for Trial Loss Ratio values. The value
A threshold value for Trial Loss Ratio values. The value MUST be MUST be non-negative and smaller than one.
non-negative and smaller than one.
Discussion: Discussion:
A trial with the Trial Loss Ratio larger than this value signals
the SUT may be unable to process this Trial Load well enough.
A trial with Trial Loss Ratio larger than this value signals the See "Throughput with Non-Zero Loss" (Section 2.5) for reasons why
SUT may be unable to process this Trial Load well enough.
See Throughput with Non-Zero Loss (Section 2.5) for reasons why
users may want to set this value above zero. users may want to set this value above zero.
Since multiple trials may be needed for one Load value, the Load Since multiple trials may be needed for one Load value, the Load
Classification may be more complicated than mere comparison of Classification may be more complicated than the mere comparison of
Trial Loss Ratio to Goal Loss Ratio. the Trial Loss Ratio to the Goal Loss Ratio.
4.6.4. Goal Exceed Ratio 4.6.4. Goal Exceed Ratio
Definition: Definition:
This is a threshold value for a particular ratio of sums of Trial
A threshold value for a particular ratio of sums of Trial
Effective Duration values. The value MUST be non-negative and Effective Duration values. The value MUST be non-negative and
smaller than one. smaller than one.
Discussion: Discussion:
Informally, up to this proportion of the Trial Results with the
Trial Loss Ratio above the Goal Loss Ratio is tolerated at a Lower
Bound. This is the full impact if every Trial was measured at
Goal Final Trial Duration. The actual full logic is more
complicated, as shorter Trials are allowed.
Informally, up to this proportion of Trial Results with Trial Loss For explainability reasons, the RECOMMENDED value for the exceed
Ratio above Goal Loss Ratio is tolerated at a Lower Bound. This ratio is 0.5 (50%), as in practice that value leads to the
is the full impact if every Trial was measured at Goal Final Trial smallest variation in overall Search Duration.
Duration. The actual full logic is more complicated, as shorter
Trials are allowed.
For explainability reasons, the RECOMMENDED value for exceed ratio
is 0.5 (50%), as in practice that value leads to the smallest
variation in overall Search Duration.
Refer to Exceed Ratio and Multiple Trials (Section 5.4) for more Refer to "Exceed Ratio and Multiple Trials" (Section 5.4) for more
details. details.
4.6.5. Goal Width 4.6.5. Goal Width
Definition: Definition:
This is a threshold value for deciding whether two Trial Load
A threshold value for deciding whether two Trial Load values are values are close enough. This is an OPTIONAL attribute. If
close enough. This is an OPTIONAL attribute. If present, the present, the value MUST be positive.
value MUST be positive.
Discussion: Discussion:
Informally, this acts as a stopping condition, controlling the Informally, this acts as a stopping condition, controlling the
precision of the search result. The search stops if every goal precision of the search result. The search stops if every goal
has reached its precision. has reached its precision.
Implementations without this attribute MUST provide the Controller Implementations without this attribute MUST provide the Controller
with other means to control the search stopping conditions. with other means to control the search stopping conditions.
Absolute load difference and relative load difference are two Absolute load difference and relative load difference are two
popular choices, but implementations may choose a different way to popular choices, but implementations may choose a different way to
specify width. specify width.
The test report MUST make it clear what specific quantity is used The test report MUST make it clear what specific quantity is used
as Goal Width. as the Goal Width.
It is RECOMMENDED to express Goal Width as a relative difference It is RECOMMENDED to express the Goal Width as a relative
and setting it to a value not lower than the Goal Loss Ratio. difference and set it to a value not lower than the Goal Loss
Ratio.
Refer to Generalized Throughput (Section 5.6) for more elaboration Refer to "Generalized Throughput" (Section 5.6) for more
on the reasoning. elaboration on the reasoning.
4.6.6. Goal Initial Trial Duration 4.6.6. Goal Initial Trial Duration
Definition: Definition:
This is the minimal value for the Trial Duration suggested to use
Minimal value for Trial Duration suggested to use for this goal. for this goal. If present, this value MUST be positive.
If present, this value MUST be positive.
Discussion: Discussion:
This is an example of an optional Search Goal. This is an example of an optional Search Goal.
A typical default value is equal to the Goal Final Trial Duration A typical default value is equal to the Goal Final Trial Duration
value. value.
Informally, this is the shortest Trial Duration the Controller Informally, this is the shortest Trial Duration the Controller
should select when focusing on the goal. should select when focusing on the goal.
Note that shorter Trial Duration values can still be used, for Note that shorter Trial Duration values can still be used, for
example, selected while focusing on a different Search Goal. Such example, selected while focusing on a different Search Goal. Such
results MUST be still accepted by the Load Classification logic. results MUST be still accepted by the Load Classification logic.
Goal Initial Trial Duration is a mechanism for a user to The Goal Initial Trial Duration is a mechanism for a user to
discourage trials with Trial Duration values deemed as too discourage trials with Trial Duration values deemed as too
unreliable for a particular SUT and a given Search Goal. unreliable for a particular SUT and a given Search Goal.
4.6.7. Search Goal 4.6.7. Search Goal
Definition: Definition:
The Search Goal is a composite quantity consisting of several The Search Goal is a composite quantity consisting of several
attributes, some of them are required. attributes, some of which are required.
Required attributes: Goal Final Trial Duration, Goal Duration Sum, The required attributes are Goal Final Trial Duration, Goal
Goal Loss Ratio and Goal Exceed Ratio. Duration Sum, Goal Loss Ratio, and Goal Exceed Ratio.
Optional attributes: Goal Initial Trial Duration and Goal Width. The optional attributes are Goal Initial Trial Duration and Goal
Width.
Discussion: Discussion:
Implementations MAY add their own attributes. Those additional Implementations MAY add their own attributes. Those additional
attributes may be required by an implementation even if they are attributes may be required by an implementation even if they are
not required by MLRsearch Specification. However, it is not required by the MLRsearch Specification. However, it is
RECOMMENDED for those implementations to support missing RECOMMENDED for those implementations to support missing
attributes by providing typical default values. attributes by providing typical default values.
For example, implementations with Goal Initial Trial Durations may For example, implementations with Goal Initial Trial Durations may
also require users to specify "how quickly" should Trial Durations also require users to specify "how quickly" Trial Durations should
increase. increase.
Refer to Section 4.10 for important Search Goal settings. Refer to "Compliance" (Section 4.10) for important Search Goal
settings.
4.6.8. Controller Input 4.6.8. Controller Input
Definition: Definition:
Controller Input is a composite quantity required as an input for Controller Input is a composite quantity required as an input for
the Controller. The only REQUIRED attribute is a list of Search the Controller. The only REQUIRED attribute is a list of Search
Goal instances. Goal instances.
Discussion: Discussion:
MLRsearch implementations MAY use additional attributes. Those MLRsearch implementations MAY use additional attributes. Those
additional attributes may be required by an implementation even if additional attributes may be required by an implementation even if
they are not required by MLRsearch Specification. they are not required by the MLRsearch Specification.
Formally, the Manager does not apply any Controller configuration Formally, the Manager does not apply any Controller configuration
apart from one Controller Input instance. apart from one Controller Input instance.
For example, Traffic Profile is configured on the Measurer by the For example, Traffic Profile is configured on the Measurer by the
Manager, without explicit assistance of the Controller. Manager, without explicit assistance of the Controller.
The order of Search Goal instances in a list SHOULD NOT have a big The order of Search Goal instances in a list SHOULD NOT have a big
impact on Controller Output, but MLRsearch implementations MAY impact on Controller Output, but MLRsearch implementations MAY
base their behavior on the order of Search Goal instances in a base their behavior on the order of Search Goal instances in a
skipping to change at page 34, line 17 skipping to change at line 1544
Manager, without explicit assistance of the Controller. Manager, without explicit assistance of the Controller.
The order of Search Goal instances in a list SHOULD NOT have a big The order of Search Goal instances in a list SHOULD NOT have a big
impact on Controller Output, but MLRsearch implementations MAY impact on Controller Output, but MLRsearch implementations MAY
base their behavior on the order of Search Goal instances in a base their behavior on the order of Search Goal instances in a
list. list.
4.6.8.1. Max Load 4.6.8.1. Max Load
Definition: Definition:
Max Load is an optional attribute of Controller Input. It is the Max Load is an optional attribute of Controller Input. It is the
maximal value the Controller is allowed to use for Trial Load maximal value the Controller is allowed to use for Trial Load
values. values.
Discussion: Discussion:
Max Load is an example of an optional attribute (outside the list Max Load is an example of an optional attribute (outside the list
of Search Goals) required by some implementations of MLRsearch. of Search Goals) required by some implementations of MLRsearch.
If the Max Load value is provided, Controller MUST NOT select If the Max Load value is provided, the Controller MUST NOT select
Trial Load values larger than that value. Trial Load values larger than that value.
In theory, each search goal could have its own Max Load value, but In theory, each search goal could have its own Max Load value, but
as all Trial Results are possibly affecting all Search Goals, it as all Trial Results are possibly affecting all Search Goals, it
makes more sense for a single Max Load value to apply to all makes more sense for a single Max Load value to apply to all
Search Goal instances. Search Goal instances.
While Max Load is a frequently used configuration parameter, While Max Load is a frequently used configuration parameter,
already governed (as maximum frame rate) by Section 20 of already governed (as maximum frame rate) by Section 20 of
[RFC2544] and (as maximum offered load) by Section 3.5.3 of [RFC2544] and (as maximum offered load) by Section 3.5.3 of
[RFC2285], some implementations may detect or discover it (instead [RFC2285], some implementations may detect or discover it (instead
of requiring a user-supplied value). of requiring a user-supplied value).
In MLRsearch Specification, one reason for listing the Relevant In the MLRsearch Specification, one reason for listing the
Upper Bound (Section 4.8.1) as a required attribute is that it Relevant Upper Bound (Section 4.8.1) as a required attribute is
makes the search result independent of Max Load value. that it makes the search result independent of the Max Load value.
Given that Max Load is a quantity based on Load, Test Report MAY Given that Max Load is a quantity based on Load, Test Report MAY
express this quantity using multi-interface values, as sum of per- express this quantity using multi-interface values, as the sum of
interface maximal loads. per-interface maximal loads.
4.6.8.2. Min Load 4.6.8.2. Min Load
Definition: Definition:
Min Load is an optional attribute of Controller Input. It is the Min Load is an optional attribute of Controller Input. It is the
minimal value the Controller is allowed to use for Trial Load minimal value the Controller is allowed to use for Trial Load
values. values.
Discussion: Discussion:
Min Load is another example of an optional attribute required by Min Load is another example of an optional attribute required by
some implementations of MLRsearch. Similarly to Max Load, it some implementations of MLRsearch. Similarly to Max Load, it
makes more sense to prescribe one common value, as opposed to makes more sense to prescribe one common value, as opposed to
using a different value for each Search Goal. using a different value for each Search Goal.
If the Min Load value is provided, Controller MUST NOT select If the Min Load value is provided, the Controller MUST NOT select
Trial Load values smaller than that value. Trial Load values smaller than that value.
Min Load is mainly useful for saving time by failing early, Min Load is mainly useful for saving time by failing early,
arriving at an Irregular Goal Result when Min Load gets classified arriving at an Irregular Goal Result when Min Load gets classified
as an Upper Bound. as an Upper Bound.
For implementations, it is RECOMMENDED to require Min Load to be For implementations, it is RECOMMENDED to require Min Load to be
non-zero and large enough to result in at least one frame being non-zero and large enough to result in at least one frame being
forwarded even at shortest allowed Trial Duration, so that Trial forwarded even at the shortest allowed Trial Duration, so that the
Loss Ratio is always well-defined, and the implementation can Trial Loss Ratio is always well-defined and the implementation can
apply relative Goal Width safely. apply the relative Goal Width safely.
Given that Min Load is a quantity based on Load, Test Report MAY Given that Min Load is a quantity based on Load, Test Report MAY
express this quantity using multi-interface values, as sum of per- express this quantity using multi-interface values, as the sum of
interface minimal loads. per-interface minimal loads.
4.7. Auxiliary Terms 4.7. Auxiliary Terms
While the terms defined in this section are not strictly needed when While the terms defined in this section are not strictly needed when
formulating MLRsearch requirements, they simplify the language used formulating MLRsearch requirements, they simplify the language used
in discussion paragraphs and explanation sections. in "Discussion" paragraphs and explanation sections.
4.7.1. Trial Classification 4.7.1. Trial Classification
When one Trial Result instance is compared to one Search Goal When one Trial Result instance is compared to one Search Goal
instance, several relations can be named using short adjectives. instance, several relations can be named using short adjectives.
As trial results do not affect each other, this *Trial As trial results do not affect each other, this *Trial
Classification* does not change during a Search. Classification* does not change during a Search.
4.7.1.1. High-Loss Trial 4.7.1.1. High-Loss Trial
A trial with Trial Loss Ratio larger than a Goal Loss Ratio value is A trial with a Trial Loss Ratio larger than a Goal Loss Ratio value
called a *high-loss trial*, with respect to given Search Goal (or is called a *high-loss trial*, with respect to the given Search Goal
lossy trial, if Goal Loss Ratio is zero). (or lossy trial, if the Goal Loss Ratio is zero).
4.7.1.2. Low-Loss Trial 4.7.1.2. Low-Loss Trial
If a trial is not high-loss, it is called a *low-loss trial* (or If a trial is not high-loss, it is called a *low-loss trial* (or
zero-loss trial, if Goal Loss Ratio is zero). zero-loss trial, if the Goal Loss Ratio is zero).
4.7.1.3. Short Trial 4.7.1.3. Short Trial
A trial with Trial Duration shorter than the Goal Final Trial A trial with a Trial Duration shorter than the Goal Final Trial
Duration is called a *short trial* (with respect to the given Search Duration is called a *short trial* (with respect to the given Search
Goal). Goal).
4.7.1.4. Full-Length Trial 4.7.1.4. Full-Length Trial
A trial that is not short is called a *full-length* trial. A trial that is not short is called a *full-length* trial.
Note that this includes Trial Durations larger than Goal Final Trial Note that this includes a Trial Durations larger than the Goal Final
Duration. Trial Duration.
4.7.1.5. Long Trial 4.7.1.5. Long Trial
A trial with Trial Duration longer than the Goal Final Trial Duration A trial with a Trial Duration longer than the Goal Final Trial
is called a *long trial*. Duration is called a *long trial*.
4.7.2. Load Classification 4.7.2. Load Classification
When a set of all Trial Result instances, performed so far at one When a set of all Trial Result instances, performed so far at one
Trial Load, is compared to one Search Goal instance, their relation Trial Load, is compared to one Search Goal instance, their relation
can be named using the concept of a bound. can be named using the concept of a bound.
In general, such bounds are a current quantity, even though cases of In general, such bounds are a current quantity, even though cases of
a Load changing its classification more than once during the Search a Load changing its classification more than once during the Search
is rare in practice. is rare in practice.
skipping to change at page 36, line 47 skipping to change at line 1660
Trial Load, is compared to one Search Goal instance, their relation Trial Load, is compared to one Search Goal instance, their relation
can be named using the concept of a bound. can be named using the concept of a bound.
In general, such bounds are a current quantity, even though cases of In general, such bounds are a current quantity, even though cases of
a Load changing its classification more than once during the Search a Load changing its classification more than once during the Search
is rare in practice. is rare in practice.
4.7.2.1. Upper Bound 4.7.2.1. Upper Bound
Definition: Definition:
A Load value is called an Upper Bound if and only if it is A Load value is called an Upper Bound if and only if it is
classified as such by load classification code (Appendix A) classified as such by a load classification code (Appendix A)
algorithm for the given Search Goal at the current moment of the algorithm for the given Search Goal at the current moment of the
Search. Search.
Discussion: Discussion:
In more detail, the set of all Trial Result instances performed so In more detail, the set of all Trial Result instances performed so
far at the Trial Load (and any Trial Duration) is certain to fail far at the Trial Load (and any Trial Duration) is certain to fail
to uphold all the requirements of the given Search Goal, mainly to uphold all the requirements of the given Search Goal, mainly
the Goal Loss Ratio in combination with the Goal Exceed Ratio. In the Goal Loss Ratio in combination with the Goal Exceed Ratio. In
this context, "certain to fail" relates to any possible results this context, "certain to fail" relates to any possible results
within the time remaining till Goal Duration Sum. within the time remaining till the Goal Duration Sum.
One search goal can have multiple different Trial Load values One search goal can have multiple different Trial Load values
classified as its Upper Bounds. While search progresses and more classified as its Upper Bounds. While search progresses and more
trials are measured, any load value can become an Upper Bound in trials are measured, any load value can become an Upper Bound in
principle. principle.
Moreover, a Load can stop being an Upper Bound, but that can only Moreover, a Load can stop being an Upper Bound, but that can only
happen when more than Goal Duration Sum of trials are measured happen when more than a Goal Duration Sum of trials are measured
(e.g., because another Search Goal needs more trials at this (e.g., because another Search Goal needs more trials at this
load). Informally, the previous Upper Bound got invalidated. In load). Informally, the previous Upper Bound got invalidated. In
practice, the Load frequently becomes a Lower Bound practice, the Load frequently becomes a Lower Bound
(Section 4.7.2.2) instead. (Section 4.7.2.2) instead.
4.7.2.2. Lower Bound 4.7.2.2. Lower Bound
Definition: Definition:
A Load value is called a Lower Bound if and only if it is A Load value is called a Lower Bound if and only if it is
classified as such by load classification code (Appendix A) classified as such by a load classification code (Appendix A)
algorithm for the given Search Goal at the current moment of the algorithm for the given Search Goal at the current moment of the
search. search.
Discussion: Discussion:
In more detail, the set of all Trial Result instances performed so In more detail, the set of all Trial Result instances performed so
far at the Trial Load (and any Trial Duration) is certain to far at the Trial Load (and any Trial Duration) is certain to
uphold all the requirements of the given Search Goal, mainly the uphold all the requirements of the given Search Goal, mainly the
Goal Loss Ratio in combination with the Goal Exceed Ratio. Here Goal Loss Ratio in combination with the Goal Exceed Ratio. Here,
"certain to uphold" relates to any possible results within the "certain to uphold" relates to any possible results within the
time remaining till Goal Duration Sum. time remaining till the Goal Duration Sum.
One search goal can have multiple different Trial Load values One search goal can have multiple different Trial Load values
classified as its Lower Bounds. As search progresses and more classified as its Lower Bounds. As search progresses and more
trials are measured, any load value can become a Lower Bound in trials are measured, any load value can become a Lower Bound in
principle. principle.
No load can be both an Upper Bound and a Lower Bound for the same No load can be both an Upper Bound and a Lower Bound for the same
Search goal at the same time, but it is possible for a larger load Search Goal at the same time, but it is possible for a larger load
to be a Lower Bound while a smaller load is an Upper Bound. to be a Lower Bound while a smaller load is an Upper Bound.
Moreover, a Load can stop being a Lower Bound, but that can only Moreover, a Load can stop being a Lower Bound, but that can only
happen when more than Goal Duration Sum of trials are measured happen when more than a Goal Duration Sum of trials are measured
(e.g., because another Search Goal needs more trials at this (e.g., because another Search Goal needs more trials at this
load). Informally, the previous Lower Bound got invalidated. In load). Informally, the previous Lower Bound got invalidated. In
practice, the Load frequently becomes an Upper Bound practice, the Load frequently becomes an Upper Bound
(Section 4.7.2.1) instead. (Section 4.7.2.1) instead.
4.7.2.3. Undecided 4.7.2.3. Undecided
Definition: Definition:
A Load value is called Undecided if it is currently neither an A Load value is called Undecided if it is currently neither an
Upper Bound nor a Lower Bound. Upper Bound nor a Lower Bound.
Discussion: Discussion:
A Load value that has not been measured so far is Undecided. A Load value that has not been measured so far is Undecided.
It is possible for a Load to transition from an Upper Bound to It is possible for a Load to transition from an Upper Bound to
Undecided by adding Short Trials with Low-Loss results. That is Undecided by adding Short Trials with Low-Loss results. That is
yet another reason for users to avoid using Search Goal instances yet another reason for users to avoid using Search Goal instances
with different Goal Final Trial Duration values. with different Goal Final Trial Duration values.
4.8. Result Terms 4.8. Result Terms
Before defining the full structure of a Controller Output, it is Before defining the full structure of a Controller Output, it is
useful to define the composite quantity, called Goal Result. The useful to define the composite quantity, called Goal Result. The
following subsections define its attribute first, before describing following subsections define its attribute first, before describing
the Goal Result quantity. the Goal Result quantity.
There is a correspondence between Search Goals and Goal Results. There is a correspondence between Search Goals and Goal Results.
Most of the following subsections refer to a given Search Goal, when Most of the following subsections refer to a given Search Goal when
defining their terms. Conversely, at the end of the search, each defining their terms. Conversely, at the end of the search, each
Search Goal instance has its corresponding Goal Result instance. Search Goal instance has its corresponding Goal Result instance.
4.8.1. Relevant Upper Bound 4.8.1. Relevant Upper Bound
Definition: Definition:
The Relevant Upper Bound is the smallest Trial Load value The Relevant Upper Bound is the smallest Trial Load value
classified as an Upper Bound for a given Search Goal at the end of classified as an Upper Bound for a given Search Goal at the end of
the Search. the Search.
Discussion: Discussion:
If no measured load had enough High-Loss Trials, the Relevant If no measured load had enough High-Loss Trials, the Relevant
Upper Bound MAY be non-existent. For example, when Max Load is Upper Bound MAY be non-existent, for example, when Max Load is
classified as a Lower Bound. classified as a Lower Bound.
Conversely, when Relevant Upper Bound does exist, it is not Conversely, when the Relevant Upper Bound does exist, it is not
affected by Max Load value. affected by the Max Load value.
Given that Relevant Upper Bound is a quantity based on Load, Test Given that the Relevant Upper Bound is a quantity based on Load,
Report MAY express this quantity using multi-interface values, as Test Report MAY express this quantity using multi-interface
sum of per-interface loads. values, as the sum of per-interface loads.
4.8.2. Relevant Lower Bound 4.8.2. Relevant Lower Bound
Definition: Definition:
The Relevant Lower Bound is the largest Trial Load value among The Relevant Lower Bound is the largest Trial Load value among
those smaller than the Relevant Upper Bound, that got classified those smaller than the Relevant Upper Bound that got classified as
as a Lower Bound for a given Search Goal at the end of the search. a Lower Bound for a given Search Goal at the end of the search.
Discussion: Discussion:
If no load had enough Low-Loss Trials, the Relevant Lower Bound If no load had enough Low-Loss Trials, the Relevant Lower Bound
MAY be non-existent. MAY be non-existent.
Strictly speaking, if the Relevant Upper Bound does not exist, the Strictly speaking, if the Relevant Upper Bound does not exist, the
Relevant Lower Bound also does not exist. In a typical case, Max Relevant Lower Bound also does not exist. In a typical case, Max
Load is classified as a Lower Bound, making it impossible to Load is classified as a Lower Bound, making it impossible to
increase the Load to continue the search for an Upper Bound. increase the Load to continue the search for an Upper Bound.
Thus, it is not clear whether a larger value would be found for a Thus, it is not clear whether a larger value would be found for a
Relevant Lower Bound if larger Loads were possible. Relevant Lower Bound if larger Loads were possible.
Given that Relevant Lower Bound is a quantity based on Load, Test Given that the Relevant Lower Bound is a quantity based on Load,
Report MAY express this quantity using multi-interface values, as Test Report MAY express this quantity using multi-interface
sum of per-interface loads. values, as the sum of per-interface loads.
4.8.3. Conditional Throughput 4.8.3. Conditional Throughput
Definition: Definition:
Conditional Throughput is a value computed at the Relevant Lower Conditional Throughput is a value computed at the Relevant Lower
Bound according to algorithm defined in conditional throughput Bound according to the algorithm defined in "Conditional
code (Appendix B). Throughput Code" (Appendix B).
Discussion: Discussion:
The Relevant Lower Bound is defined only at the end of the Search, The Relevant Lower Bound is defined only at the end of the Search,
and so is the Conditional Throughput. But the algorithm can be and so is the Conditional Throughput. But the algorithm can be
applied at any time on any Lower Bound load, so the final applied at any time on any Lower Bound load, so the final
Conditional Throughput value may appear sooner than at the end of Conditional Throughput value may appear sooner than at the end of
a Search. a Search.
Informally, the Conditional Throughput should be a typical Trial Informally, the Conditional Throughput should be a typical Trial
Forwarding Rate, expected to be seen at the Relevant Lower Bound Forwarding Rate, expected to be seen at the Relevant Lower Bound
of a given Search Goal. of a given Search Goal.
But frequently it is only a conservative estimate thereof, as But frequently, it is only a conservative estimate thereof, as
MLRsearch implementations tend to stop measuring more Trials as MLRsearch implementations tend to stop measuring more Trials as
soon as they confirm the value cannot get worse than this estimate soon as they confirm the value cannot get worse than this estimate
within the Goal Duration Sum. within the Goal Duration Sum.
This value is RECOMMENDED to be used when evaluating repeatability This value is RECOMMENDED to be used when evaluating repeatability
and comparability of different MLRsearch implementations. and comparability of different MLRsearch implementations.
Refer to Generalized Throughput (Section 5.6) for more details. Refer to "Generalized Throughput" (Section 5.6) for more details.
Given that Conditional Throughput is a quantity based on Load, Given that Conditional Throughput is a quantity based on Load,
Test Report MAY express this quantity using multi-interface Test Report MAY express this quantity using multi-interface
values, as sum of per-interface forwarding rates. values, as the sum of per-interface forwarding rates.
4.8.4. Goal Results 4.8.4. Goal Results
MLRsearch Specification is based on a set of requirements for a The MLRsearch Specification is based on a set of requirements for a
"regular" result. But in practice, it is not always possible for "regular" result. But in practice, it is not always possible for
such result instance to exist, so also "irregular" results need to be such a result instance to exist, so "irregular" results also need to
supported. be supported.
4.8.4.1. Regular Goal Result 4.8.4.1. Regular Goal Result
Definition: Definition:
Regular Goal Result is a composite quantity consisting of several Regular Goal Result is a composite quantity consisting of several
attributes. Relevant Upper Bound and Relevant Lower Bound are attributes. Relevant Upper Bound and Relevant Lower Bound are
REQUIRED attributes. Conditional Throughput is a RECOMMENDED REQUIRED attributes. Conditional Throughput is a RECOMMENDED
attribute. attribute.
Discussion: Discussion:
Implementations MAY add their own attributes. Implementations MAY add their own attributes.
Test report MUST display Relevant Lower Bound. Displaying Test report MUST display the Relevant Lower Bound. Displaying the
Relevant Upper Bound is RECOMMENDED, especially if the Relevant Upper Bound is RECOMMENDED, especially if the
implementation does not use Goal Width. implementation does not use Goal Width.
In general, stopping conditions for the corresponding Search Goal In general, stopping conditions for the corresponding Search Goal
MUST be satisfied to produce a Regular Goal Result. Specifically, MUST be satisfied to produce a Regular Goal Result. Specifically,
if an implementation offers Goal Width as a Search Goal attribute, if an implementation offers Goal Width as a Search Goal attribute,
the distance between the Relevant Lower Bound and the Relevant the distance between the Relevant Lower Bound and the Relevant
Upper Bound MUST NOT be larger than the Goal Width. Upper Bound MUST NOT be larger than the Goal Width.
For stopping conditions refer to Goal Width (Section 4.6.5) and For stopping conditions, refer to "Goal Width" (Section 4.6.5) and
Stopping Conditions and Precision (Section 5.2). "Stopping Conditions and Precision" (Section 5.2).
4.8.4.2. Irregular Goal Result 4.8.4.2. Irregular Goal Result
Definition: Definition:
Irregular Goal Result is a composite quantity. No attributes are Irregular Goal Result is a composite quantity. No attributes are
required. required.
Discussion: Discussion:
It is RECOMMENDED to report any useful quantity even if it does It is RECOMMENDED to report any useful quantity even if it does
not satisfy all the requirements. For example, if Max Load is not satisfy all the requirements. For example, if Max Load is
classified as a Lower Bound, it is fine to report it as an classified as a Lower Bound, it is fine to report it as an
"effective" Relevant Lower Bound (although not a real one, as that "effective" Relevant Lower Bound (although not a real one, as that
requires Relevant Upper Bound which does not exist in this case), requires a Relevant Upper Bound, which does not exist in this
and compute Conditional Throughput for it. In this case, only the case) and compute Conditional Throughput for it. In this case,
missing Relevant Upper Bound signals this result instance is only the missing Relevant Upper Bound signals this result instance
irregular. is irregular.
Similarly, if both relevant bounds exist, it is RECOMMENDED to Similarly, if both relevant bounds exist, it is RECOMMENDED to
include them as Irregular Goal Result attributes, and let the include them as Irregular Goal Result attributes and let the
Manager decide if their distance is too far for Test Report Manager decide if their distance is too far for Test Report
purposes. purposes.
If Test Report displays some Irregular Goal Result attribute If Test Report displays some Irregular Goal Result attribute
values, they MUST be clearly marked as coming from irregular values, they MUST be clearly marked as coming from irregular
results. results.
The implementation MAY define additional attributes, for example The implementation MAY define additional attributes, for example,
explicit flags for expected situations, so the Manager logic can explicit flags for expected situations, so the Manager logic can
be simpler. be simpler.
4.8.4.3. Goal Result 4.8.4.3. Goal Result
Definition: Definition:
Goal Result is a composite quantity. Each instance is either a Goal Result is a composite quantity. Each instance is either a
Regular Goal Result or an Irregular Goal Result. Regular Goal Result or an Irregular Goal Result.
Discussion: Discussion:
The Manager MUST be able of distinguishing whether the instance is The Manager MUST be able of distinguishing whether the instance is
regular or not. regular or not.
4.8.5. Search Result 4.8.5. Search Result
Definition: Definition:
The Search Result is a single composite object that maps each The Search Result is a single composite object that maps each
Search Goal instance to a corresponding Goal Result instance. Search Goal instance to a corresponding Goal Result instance.
Discussion: Discussion:
As an alternative to mapping, the Search Result may be represented As an alternative to mapping, the Search Result may be represented
as an ordered list of Goal Result instances that appears in the as an ordered list of Goal Result instances that appears in the
exact sequence of their corresponding Search Goal instances. exact sequence of their corresponding Search Goal instances.
When the Search Result is expressed as a mapping, it MUST contain When the Search Result is expressed as a mapping, it MUST contain
an entry for every Search Goal instance supplied in the Controller an entry for every Search Goal instance supplied in the Controller
Input. Input.
Identical Goal Result instances MAY be listed for different Search Identical Goal Result instances MAY be listed for different Search
Goals, but their status as regular or irregular may be different. Goals, but their status as regular or irregular may be different,
For example, if two goals differ only in Goal Width value, and the for example, if two goals differ only in the Goal Width value, and
relevant bound values are close enough according to only one of the relevant bound values are close enough according to only one
them. of them.
4.8.6. Controller Output 4.8.6. Controller Output
Definition: Definition:
The Controller Output is a composite quantity returned from the The Controller Output is a composite quantity returned from the
Controller to the Manager at the end of the search. The Search Controller to the Manager at the end of the search. The Search
Result instance is its only required attribute. Result instance is its only required attribute.
Discussion: Discussion:
The MLRsearch implementation MAY return additional data in the
MLRsearch implementation MAY return additional data in the Controller Output, e.g., the number of trials performed and the
Controller Output, e.g., number of trials performed and the total total Search Duration.
Search Duration.
4.9. Architecture Terms 4.9. Architecture Terms
MLRsearch architecture consists of three main system components: the The MLRsearch architecture consists of three main system components:
Manager, the Controller, and the Measurer. The components were the Manager, the Controller, and the Measurer. The components were
introduced in Architecture Overview (Section 4.2), and the following introduced in "Architecture Overview" (Section 4.2), and the
subsections finalize their definitions using terms from previous following subsections finalize their definitions using terms from
sections. previous sections.
Note that the architecture also implies the presence of other Note that the architecture also implies the presence of other
components, such as the SUT and the tester (as a sub-component of the components, such as the SUT and the tester (as a sub-component of the
Measurer). Measurer).
Communication protocols and interfaces between components are left Communication protocols and interfaces between components are left
unspecified. For example, when MLRsearch Specification mentions unspecified. For example, when the MLRsearch Specification mentions
"Controller calls Measurer", it is possible that the Controller "Controller calls Measurer", it is possible that the Controller
notifies the Manager to call the Measurer indirectly instead. In notifies the Manager to call the Measurer indirectly instead. In
doing so, the Measurer implementations can be fully independent from doing so, the Measurer implementations can be fully independent from
the Controller implementations, e.g., developed in different the Controller implementations, e.g., developed in different
programming languages. programming languages.
4.9.1. Measurer 4.9.1. Measurer
Definition: Definition:
The Measurer is a functional element that, when called with a
The Measurer is a functional element that when called with a Trial Trial Input (Section 4.5.3) instance, performs one Trial
Input (Section 4.5.3) instance, performs one Trial (Section 4.4.3) (Section 4.4.3) and returns a Trial Output (Section 4.5.9)
and returns a Trial Output (Section 4.5.9) instance. instance.
Discussion: Discussion:
This definition assumes the Measurer is already initialized. In This definition assumes the Measurer is already initialized. In
practice, there may be additional steps before the Search, e.g., practice, there may be additional steps before the Search, e.g.,
when the Manager configures the traffic profile (either on the when the Manager configures the traffic profile (either on the
Measurer or on its tester sub-component directly) and performs a Measurer or on its tester sub-component directly) and performs a
warm-up (if the tester or the test procedure requires one). warm-up (if the tester or the test procedure requires one).
It is the responsibility of the Measurer implementation to uphold It is the responsibility of the Measurer implementation to uphold
any requirements and assumptions present in MLRsearch any requirements and assumptions present in the MLRsearch
Specification, e.g., Trial Forwarding Ratio not being larger than Specification, e.g., the Trial Forwarding Ratio not being larger
one. than one.
Implementers have some freedom. For example, Section 10 of Implementers have some freedom. For example, Section 10 of
[RFC2544] gives some suggestions (but not requirements) related to [RFC2544] gives some suggestions (but not requirements) related to
duplicated or reordered frames. Implementations are RECOMMENDED duplicated or reordered frames. Implementations are RECOMMENDED
to document their behavior related to such freedoms in as detailed to document their behavior related to such freedoms in as detailed
a way as possible. a way as possible.
It is RECOMMENDED to benchmark the test equipment first, e.g., It is RECOMMENDED to benchmark the test equipment first, e.g.,
connect sender and receiver directly (without any SUT in the connect the sender and receiver directly (without any SUT in the
path), find a load value that guarantees the Offered Load is not path), find a load value that guarantees the Offered Load is not
too far from the Intended Load and use that value as the Max Load too far from the Intended Load, and use that value as the Max Load
value. When testing the real SUT, it is RECOMMENDED to turn any value. When testing the real SUT, it is RECOMMENDED to turn any
severe deviation between the Intended Load and the Offered Load severe deviation between the Intended Load and the Offered Load
into increased Trial Loss Ratio. into the increased Trial Loss Ratio.
Neither of the two recommendations are made into mandatory Neither of these two recommendations are made into mandatory
requirements, because it is not easy to provide guidance about requirements, because it is not easy to provide guidance about
when the difference is severe enough, in a way that would be when the difference is severe enough in a way that would be
disentangled from other Measurer freedoms. disentangled from other Measurer freedoms.
For a sample situation where the Offered Load cannot keep up with For a sample situation where the Offered Load cannot keep up with
the Intended Load, and the consequences on MLRsearch result, refer the Intended Load, and the consequences on the MLRsearch result,
to Hard Performance Limit (Section 5.6.1). refer to "Hard Performance Limit" (Section 5.6.1).
4.9.2. Controller 4.9.2. Controller
Definition: Definition:
The Controller is a functional element that, upon receiving a The Controller is a functional element that, upon receiving a
Controller Input instance, repeatedly generates Trial Input Controller Input instance, repeatedly generates Trial Input
instances for the Measurer and collects the corresponding Trial instances for the Measurer and collects the corresponding Trial
Output instances. This cycle continues until the stopping Output instances. This cycle continues until the stopping
conditions are met, at which point the Controller produces a final conditions are met, at which point, the Controller produces a
Controller Output instance and terminates. final Controller Output instance and terminates.
Discussion: Discussion:
Informally, the Controller has big freedom in selection of Trial Informally, the Controller has big freedom in selection of Trial
Inputs, and the implementations want to achieve all the Search Inputs, and the implementations want to achieve all the Search
Goals in the shortest average time. Goals in the shortest average time.
The Controller's role in optimizing the overall Search Duration The Controller's role in optimizing the overall Search Duration
distinguishes MLRsearch algorithms from simpler search procedures. distinguishes MLRsearch algorithms from simpler search procedures.
Informally, each implementation can have different stopping Informally, each implementation can have different stopping
conditions. Goal Width is only one example. In practice, conditions. Goal Width is only one example. In practice,
implementation details do not matter, as long as Goal Result implementation details do not matter, as long as Goal Result
skipping to change at page 44, line 37 skipping to change at line 2008
distinguishes MLRsearch algorithms from simpler search procedures. distinguishes MLRsearch algorithms from simpler search procedures.
Informally, each implementation can have different stopping Informally, each implementation can have different stopping
conditions. Goal Width is only one example. In practice, conditions. Goal Width is only one example. In practice,
implementation details do not matter, as long as Goal Result implementation details do not matter, as long as Goal Result
instances are regular. instances are regular.
4.9.3. Manager 4.9.3. Manager
Definition: Definition:
The Manager is a functional element that is responsible for The Manager is a functional element that is responsible for
provisioning other components, calling a Controller component provisioning other components, calling a Controller component
once, and for creating the test report following the reporting once, and for creating the test report following the reporting
format as defined in Section 26 of [RFC2544]. format as defined in Section 26 of [RFC2544].
Discussion: Discussion:
The Manager initializes the SUT, the Measurer (and the tester if The Manager initializes the SUT, the Measurer (and the tester if
independent from Measurer) with their intended configurations independent from Measurer) with their intended configurations
before calling the Controller. before calling the Controller.
Note that Section 7 of [RFC2544] already puts requirements on SUT Note that Section 7 of [RFC2544] already puts requirements on SUT
setups: setups:
"It is expected that all of the tests will be run without changing | It is expected that all of the tests will be run without
the configuration or setup of the DUT in any way other than that | changing the configuration or setup of the DUT in any way other
required to do the specific test. For example, it is not | than that required to do the specific test. For example, it is
acceptable to change the size of frame handling buffers between | not acceptable to change the size of frame handling buffers
tests of frame handling rates or to disable all but one transport | between tests of frame handling rates or to disable all but one
protocol when testing the throughput of that protocol." | transport protocol when testing the throughput of that
| protocol.
It is REQUIRED for the test report to encompass all the SUT It is REQUIRED for the test report to encompass all the SUT
configuration details, including description of a "default" configuration details, including the description of a "default"
configuration common for most tests and configuration changes if configuration common for most tests and configuration changes if
required by a specific test. required by a specific test.
For example, Section 5.1.1 of [RFC5180] recommends testing jumbo For example, Section 5.1.1 of [RFC5180] recommends testing jumbo
frames if SUT can forward them, even though they are outside the frames if SUT can forward them, even though they are outside the
scope of the 802.3 IEEE standard. In this case, it is acceptable scope of the 802.3 IEEE standard. In this case, it is acceptable
for the SUT default configuration to not support jumbo frames, and for the SUT default configuration to not support jumbo frames and
only enable this support when testing jumbo traffic profiles, as only enable this support when testing jumbo traffic profiles, as
the handling of jumbo frames typically has different packet buffer the handling of jumbo frames typically has different packet buffer
requirements and potentially higher processing overhead. Non- requirements and potentially higher processing overhead. Non-
jumbo frame sizes should also be tested on the jumbo-enabled jumbo frame sizes should also be tested on the jumbo-enabled
setup. setup.
The Manager does not need to be able to tweak any Search Goal The Manager does not need to be able to tweak any Search Goal
attributes, but it MUST report all applied attribute values even attributes, but it MUST report all applied attribute values even
if not tweaked. if not tweaked.
A "user" - human or automated - invokes the Manager once to launch A "user" -- human or automated -- invokes the Manager once to
a single Search and receive its report. Every new invocation is launch a single Search and receive its report. Every new
treated as a fresh, independent Search; how the system behaves invocation is treated as a fresh, independent Search; how the
across multiple calls (for example, combining or comparing their system behaves across multiple calls (for example, combining or
results) is explicitly out of scope for this document. comparing their results) is explicitly out of scope for this
document.
4.10. Compliance 4.10. Compliance
This section discusses compliance relations between MLRsearch and This section discusses compliance relations between MLRsearch and
other test procedures. other test procedures.
4.10.1. Test Procedure Compliant with MLRsearch 4.10.1. Test Procedure Compliant with MLRsearch
Any networking measurement setup that could be understood as Any networking measurement setup that could be understood as
consisting of functional elements satisfying requirements for the consisting of functional elements satisfying requirements for the
Measurer, the Controller and the Manager, is compliant with MLRsearch Measurer, the Controller, and the Manager is compliant with the
Specification. MLRsearch Specification.
These components can be seen as abstractions present in any testing These components can be seen as abstractions present in any testing
procedure. For example, there can be a single component acting both procedure. For example, there can be a single component acting as
as the Manager and the Controller, but if values of required both the Manager and the Controller, but if values of required
attributes of Search Goals and Goal Results are visible in the test attributes of Search Goals and Goal Results are visible in the test
report, the Controller Input instance and Controller Output instance report, the Controller Input instance and Controller Output instance
are implied. are implied.
For example, any setup for conditionally (or unconditionally) For example, any setup for conditionally (or unconditionally)
compliant [RFC2544] throughput testing can be understood as a compliant [RFC2544] throughput testing can be understood as an
MLRsearch architecture, if there is enough data to reconstruct the MLRsearch architecture if there is enough data to reconstruct the
Relevant Upper Bound. Relevant Upper Bound.
Refer to MLRsearch Compliant with RFC 2544 (Section 4.10.2) for an Refer to "MLRsearch Compliant with RFC 2544" (Section 4.10.2) for an
equivalent Search Goal. equivalent Search Goal.
Any test procedure that can be understood as one call to the Manager Any test procedure that can be understood as one call to the Manager
of MLRsearch architecture is said to be compliant with MLRsearch of the MLRsearch architecture is said to be compliant with the
Specification. MLRsearch Specification.
4.10.2. MLRsearch Compliant with RFC 2544 4.10.2. MLRsearch Compliant with RFC 2544
The following Search Goal instance makes the corresponding Search The following Search Goal instance makes the corresponding Search
Result unconditionally compliant with Section 24 of [RFC2544]. Result unconditionally compliant with Section 24 of [RFC2544].
* Goal Final Trial Duration = 60 seconds * Goal Final Trial Duration = 60 seconds
* Goal Duration Sum = 60 seconds * Goal Duration Sum = 60 seconds
* Goal Loss Ratio = 0% * Goal Loss Ratio = 0%
* Goal Exceed Ratio = 0% * Goal Exceed Ratio = 0%
Goal Loss Ratio and Goal Exceed Ratio attributes, are enough to make The Goal Loss Ratio and Goal Exceed Ratio attributes are enough to
the Search Goal conditionally compliant. Adding Goal Final Trial make the Search Goal conditionally compliant. Adding Goal Final
Duration makes the Search Goal unconditionally compliant. Trial Duration makes the Search Goal unconditionally compliant.
Goal Duration Sum prevents MLRsearch from repeating zero-loss Full- The Goal Duration Sum prevents MLRsearch from repeating zero-loss
Length Trials. Full-Length Trials.
The presence of other Search Goals does not affect the compliance of The presence of other Search Goals does not affect the compliance of
this Goal Result. The Relevant Lower Bound and the Conditional this Goal Result. In this case, the Relevant Lower Bound and the
Throughput are in this case equal to each other, and the value is the Conditional Throughput are equal to each other, and the value is the
[RFC2544] throughput. [RFC2544] throughput.
Non-zero exceed ratio is not strictly disallowed, but it could A non-zero exceed ratio is not strictly disallowed, but it could
needlessly prolong the search when Low-Loss short trials are present. needlessly prolong the search when Low-Loss short trials are present.
4.10.3. MLRsearch Compliant with TST009 4.10.3. MLRsearch Compliant with TST009
One of the alternatives to [RFC2544] is Binary search with loss One of the alternatives to [RFC2544] is binary search with loss
verification as described in Section 12.3.3 of [TST009]. verification, as described in Section 12.3.3 of [TST009].
The rationale of such search is to repeat high-loss trials, hoping The rationale of such search is to repeat high-loss trials, hoping
for zero loss on second try, so the results are closer to the for zero loss on the second try, so the results are closer to the
noiseless end of performance spectrum, thus more repeatable and noiseless end of the performance spectrum, thus more repeatable and
comparable. comparable.
Only the variant with "z = infinity" is achievable with MLRsearch. Only the variant with "z = infinity" is achievable with MLRsearch.
For example, for "max(r) = 2" variant, the following Search Goal For example, for the "max(r) = 2" variant, the following Search Goal
instance should be used to get a compatible Search Result: instance should be used to get a compatible Search Result:
* Goal Final Trial Duration = 60 seconds * Goal Final Trial Duration = 60 seconds
* Goal Duration Sum = 120 seconds * Goal Duration Sum = 120 seconds
* Goal Loss Ratio = 0% * Goal Loss Ratio = 0%
* Goal Exceed Ratio = 50% * Goal Exceed Ratio = 50%
If the first 60-second trial has zero loss, it is enough for If the first 60-second trial has zero loss, it is enough for
MLRsearch to stop measuring at that load, as even a second high-loss MLRsearch to stop measuring at that load, as even a second high-loss
trial would still fit within the exceed ratio. trial would still fit within the exceed ratio.
But if the first trial is high-loss, MLRsearch needs to perform also But if the first trial is high-loss, MLRsearch also needs to perform
the second trial to classify that load. Goal Duration Sum is twice the second trial to classify that load. The Goal Duration Sum is
as long as Goal Final Trial Duration, so third full-length trial is twice as long as the Goal Final Trial Duration, so a third full-
never needed. length trial is never needed.
5. Methodology Rationale and Design Considerations 5. Methodology Rationale and Design Considerations
This section explains the Why behind MLRsearch. Building on the This section explains the "why" behind MLRsearch. Building on the
normative specification in MLRsearch Specification (Section 4), it normative specification in "MLRsearch Specification" (Section 4), it
contrasts MLRsearch with the classic [RFC2544] single-ratio Binary contrasts MLRsearch with the classic single-ratio Binary Search
Search (Section 2.1) and walks through the key design choices: search (Section 2.1) in [RFC2544] and walks through the key design choices:
mechanics, stopping-rule precision, loss-inversion for multiple search mechanics, stopping-rule precision, loss inversion for
goals, exceed-ratio handling, short-trial strategies, and the multiple goals, exceed-ratio handling, short-trial strategies, and
generalised throughput concept. Together, these considerations show the generalized throughput concept. Together, these considerations
how the methodology reduces test time, supports multiple loss ratios, show how the methodology reduces test time, supports multiple loss
and improves repeatability. ratios, and improves repeatability.
5.1. Binary Search Commonalities 5.1. Binary Search Commonalities
A typical search implementation for [RFC2544] such as Binary Search A typical search implementation for [RFC2544], such as Binary Search
(Section 2.1) tracks only the two tightest bounds (in variables (Section 2.1), tracks only the two tightest bounds (in variables
"lower-bound" and "upper-bound"). To start, the search needs both "lower-bound" and "upper-bound"). To start, the search needs both
Max Load and Min Load values. Then, one trial is used to confirm Max Max Load and Min Load values. Then, one trial is used to confirm Max
Load is an Upper Bound, and one trial to confirm Min Load is a Lower Load is an Upper Bound, and one trial is used to confirm Min Load is
Bound. a Lower Bound.
Then, next Trial Load is chosen as the mean of the current tightest Then, Trial Load is chosen as the mean of the current tightest upper
upper bound and the current tightest lower bound, and becomes a new bound and the current tightest lower bound and becomes a new tightest
tightest bound depending on the Trial Loss Ratio. bound depending on the Trial Loss Ratio.
After some number of trials, the tightest lower bound becomes the After some number of trials, the tightest lower bound becomes the
throughput, but [RFC2544] does not specify when, if ever, the search throughput, but [RFC2544] does not specify when, if ever, the search
should stop. In practice, the search stops either at some distance should stop. In practice, the search stops either at some distance
between the tightest upper bound and the tightest lower bound, or between the tightest upper bound and the tightest lower bound or
after some number of Trials. after some number of Trials.
For a given pair of Max Load and Min Load values, there is one-to-one For a given pair of Max Load and Min Load values, there is a one-to-
correspondence between number of Trials and final distance between one correspondence between the number of Trials and the final
the tightest bounds. Thus, the search always takes the same time, distance between the tightest bounds. Thus, the search always takes
assuming initial bounds are confirmed. the same time, assuming initial bounds are confirmed.
5.2. Stopping Conditions and Precision 5.2. Stopping Conditions and Precision
MLRsearch Specification requires listing both Relevant Bounds for The MLRsearch Specification requires listing both Relevant Bounds for
each Search Goal, and the difference between the bounds implies each Search Goal, and the difference between the bounds implies
whether the result precision is achieved. Therefore, it is not whether the result precision is achieved. Therefore, it is not
necessary to report the specific stopping condition used. necessary to report the specific stopping condition used.
MLRsearch implementations may use Goal Width to allow direct control MLRsearch implementations may use Goal Width to allow direct control
of result precision and indirect control of the Search Duration. of result precision and indirect control of the Search Duration.
Other MLRsearch implementations may use different stopping Other MLRsearch implementations may use different stopping
conditions: for example based on the Search Duration, trading off conditions, for example, based on the Search Duration, trading off
precision control for duration control. precision control for duration control.
Due to various possible time optimizations, there is no strict Due to various possible time optimizations, there is no strict
correspondence between the Search Duration and Goal Width values. In correspondence between the Search Duration and Goal Width values. In
practice, noisy SUT performance increases both average search time practice, noisy SUT performance increases both average search time
and its variance. and its variance.
5.3. Loss Ratios and Loss Inversion 5.3. Loss Ratios and Loss Inversion
The biggest difference between MLRsearch and Binary Search The biggest difference between MLRsearch and Binary Search
(Section 2.1) is in the goals of the search. [RFC2544] has a single (Section 2.1) is in the goals of the search. [RFC2544] has a single
goal, based on classifying a single full-length trial as either zero- goal, based on classifying a single full-length trial as either zero
loss or non-zero-loss. MLRsearch supports searching for multiple loss or non-zero loss. MLRsearch supports searching for multiple
Search Goals at once, usually differing in their Goal Loss Ratio Search Goals at once, usually differing in their Goal Loss Ratio
values. values.
5.3.1. Single Goal and Hard Bounds 5.3.1. Single Goal and Hard Bounds
Each bound in Binary Search (Section 2.1) is "hard", in the sense Each bound in Binary Search (Section 2.1) is "hard", in the sense
that all further Trial Load values are smaller than any current upper that all further Trial Load values are smaller than any current upper
bound and larger than any current lower bound. bound and larger than any current lower bound.
This is also possible for MLRsearch implementations, when the search This is also possible for MLRsearch implementations when the search
is started with only one Search Goal instance. is started with only one Search Goal instance.
5.3.2. Multiple Goals and Loss Inversion 5.3.2. Multiple Goals and Loss Inversion
MLRsearch Specification supports multiple Search Goals, making the The MLRsearch Specification supports multiple Search Goals, making
search procedure more complicated compared to Binary Search the search procedure more complicated compared to Binary Search
(Section 2.1) with single goal, but most of the complications do not (Section 2.1) with a single goal, but most of the complications do
affect the final results much. Except for one phenomenon: Loss not affect the final results much, except for one phenomenon: Loss
Inversion. Inversion.
Depending on Search Goal attributes, Load Classification results may Depending on Search Goal attributes, Load Classification results may
be resistant to small amounts of Inconsistent Trial Results be resistant to small amounts of Inconsistent Trial Results
(Section 2.6). However, for larger amounts, a Load that is (Section 2.6). However, for larger amounts, a Load that is
classified as an Upper Bound for one Search Goal may still be a Lower classified as an Upper Bound for one Search Goal may still be a Lower
Bound for another Search Goal. Due to this other goal, MLRsearch Bound for another Search Goal. Due to this other goal, MLRsearch
will probably perform subsequent Trials at Trial Loads even larger will probably perform subsequent Trials at Trial Loads even larger
than the original value. than the original value.
This introduces questions any many-goals search algorithm has to This introduces questions any many-goals search algorithm has to
address. For example: What to do when all such larger load trials address, such as: What to do when all such larger load trials happen
happen to have zero loss? Does it mean the earlier upper bound was to have zero loss? Does it mean the earlier upper bound was not
not real? Does it mean the later Low-Loss trials are not considered real? Does it mean the later Low-Loss trials are not considered a
a lower bound? lower bound?
The situation where a smaller Load is classified as an Upper Bound, The situation where a smaller Load is classified as an Upper Bound,
while a larger Load is classified as a Lower Bound (for the same while a larger Load is classified as a Lower Bound (for the same
search goal), is called Loss Inversion. search goal), is called Loss Inversion.
Conversely, only single-goal search algorithms can have hard bounds Conversely, only single-goal search algorithms can have hard bounds
that shield them from Loss Inversion. that shield them from Loss Inversion.
5.3.3. Conservativeness and Relevant Bounds 5.3.3. Conservativeness and Relevant Bounds
MLRsearch is conservative when dealing with Loss Inversion: the Upper MLRsearch is conservative when dealing with Loss Inversion: The Upper
Bound is considered real, and the Lower Bound is considered to be a Bound is considered real, and the Lower Bound is considered to be a
fluke, at least when computing the final result. fluke, at least when computing the final result.
This is formalized using definitions of Relevant Upper Bound This is formalized using the definitions of Relevant Upper Bound
(Section 4.8.1) and Relevant Lower Bound (Section 4.8.2). (Section 4.8.1) and Relevant Lower Bound (Section 4.8.2).
The Relevant Upper Bound (for specific goal) is the smallest Load The Relevant Upper Bound (for a specific goal) is the smallest Load
classified as an Upper Bound. But the Relevant Lower Bound is not classified as an Upper Bound. But the Relevant Lower Bound is not
simply the largest among Lower Bounds. It is the largest Load among simply the largest among Lower Bounds. It is the largest Load among
Loads that are Lower Bounds while also being smaller than the Loads that are Lower Bounds while also being smaller than the
Relevant Upper Bound. Relevant Upper Bound.
With these definitions, the Relevant Lower Bound is always smaller With these definitions, the Relevant Lower Bound is always smaller
than the Relevant Upper Bound (if both exist), and the two relevant than the Relevant Upper Bound (if both exist), and the two relevant
bounds are used analogously as the two tightest bounds in the Binary bounds are used analogously as the two tightest bounds in the Binary
Search (Section 2.1). When they meet the stopping conditions, the Search (Section 2.1). When they meet the stopping conditions, the
Relevant Bounds are used in the output. Relevant Bounds are used in the output.
skipping to change at page 50, line 37 skipping to change at line 2280
The consequence of the way the Relevant Bounds are defined is that The consequence of the way the Relevant Bounds are defined is that
every Trial Result can have an impact on any current Relevant Bound every Trial Result can have an impact on any current Relevant Bound
larger than that Trial Load, namely by becoming a new Upper Bound. larger than that Trial Load, namely by becoming a new Upper Bound.
This also applies when that Load is measured before another Load gets This also applies when that Load is measured before another Load gets
enough measurements to become a current Relevant Bound. enough measurements to become a current Relevant Bound.
This also implies that if the SUT tested (or the Traffic Generator This also implies that if the SUT tested (or the Traffic Generator
used) needs a warm-up, it should be warmed up before starting the used) needs a warm-up, it should be warmed up before starting the
Search, otherwise the first few measurements could become unjustly Search; otherwise, the first few measurements could become unjustly
limiting. limiting.
For MLRsearch implementations, it means it is better to measure at For MLRsearch implementations, this means that it is better to
smaller Loads first, so bounds found earlier are less likely to get measure at smaller Loads first, so bounds found earlier are less
invalidated later. likely to get invalidated later.
5.4. Exceed Ratio and Multiple Trials 5.4. Exceed Ratio and Multiple Trials
The idea of performing multiple Trials at the same Trial Load comes The idea of performing multiple Trials at the same Trial Load comes
from a model where some Trial Results (those with high Trial Loss from a model where some Trial Results (those with a high Trial Loss
Ratio) are affected by infrequent effects, causing unsatisfactory Ratio) are affected by infrequent effects, causing unsatisfactory
repeatability repeatability of [RFC2544] Throughput results. Refer to "DUT in SUT"
of [RFC2544] Throughput results. Refer to DUT in SUT (Section 2.3) (Section 2.3) for a discussion about noiseful and noiseless ends of
for a discussion about noiseful and noiseless ends of the SUT the SUT performance spectrum. Stable results are closer to the
performance spectrum. Stable results are closer to the noiseless end noiseless end of the SUT performance spectrum, so MLRsearch may need
of the SUT performance spectrum, so MLRsearch may need to allow some to allow some frequency of high-loss trials to ignore the rare but
frequency of high-loss trials to ignore the rare but big effects near big effects near the noiseful end.
the noiseful end.
For MLRsearch to perform such Trial Result filtering, it needs a For MLRsearch to perform such Trial Result filtering, it needs a
configuration option to tell how frequent the "infrequent" big loss configuration option to tell how frequent the "infrequent" big loss
can be. This option is called the Goal Exceed Ratio (Section 4.6.4). can be. This option is called the Goal Exceed Ratio (Section 4.6.4).
It tells MLRsearch what ratio of trials (more specifically, what It tells MLRsearch what ratio of trials (more specifically, what
ratio of Trial Effective Duration seconds) can have a Trial Loss ratio of Trial Effective Duration seconds) can have a Trial Loss
Ratio (Section 4.5.6) larger than the Goal Loss Ratio (Section 4.6.3) Ratio (Section 4.5.6) larger than the Goal Loss Ratio (Section 4.6.3)
and still be classified as a Lower Bound (Section 4.7.2.2). and still be classified as a Lower Bound (Section 4.7.2.2).
Zero exceed ratio means all Trials must have a Trial Loss Ratio equal A zero exceed ratio means all Trials must have a Trial Loss Ratio
to or lower than the Goal Loss Ratio. equal to or lower than the Goal Loss Ratio.
When more than one Trial is intended to classify a Load, MLRsearch When more than one Trial is intended to classify a Load, MLRsearch
also needs something that controls the number of trials needed. also needs something that controls the number of trials needed.
Therefore, each goal also has an attribute called Goal Duration Sum. Therefore, each goal also has an attribute called Goal Duration Sum.
The meaning of a Goal Duration Sum (Section 4.6.2) is that when a The meaning of a Goal Duration Sum (Section 4.6.2) is that when a
Load has (Full-Length) Trials whose Trial Effective Durations when Load has (Full-Length) Trials whose Trial Effective Durations when
summed up give a value at least as big as the Goal Duration Sum summed up give a value at least as big as the Goal Duration Sum
value, the Load is guaranteed to be classified either as an Upper value, the Load is guaranteed to be classified as either an Upper
Bound or a Lower Bound for that Search Goal instance. Bound or a Lower Bound for that Search Goal instance.
5.5. Short Trials and Duration Selection 5.5. Short Trials and Duration Selection
MLRsearch requires each Search Goal to specify its Goal Final Trial MLRsearch requires each Search Goal to specify its Goal Final Trial
Duration. Duration.
Section 24 of [RFC2544] already anticipates possible time savings Section 24 of [RFC2544] already anticipates possible time savings
when Short Trials are used. when Short Trials are used.
skipping to change at page 52, line 6 skipping to change at line 2339
decide whether, when, and how short trial durations are used. The decide whether, when, and how short trial durations are used. The
exact heuristics and controls are left to the discretion of the exact heuristics and controls are left to the discretion of the
implementer. implementer.
While MLRsearch implementations are free to use any logic to select While MLRsearch implementations are free to use any logic to select
Trial Input values, comparability between MLRsearch implementations Trial Input values, comparability between MLRsearch implementations
is only assured when the Load Classification logic handles any is only assured when the Load Classification logic handles any
possible set of Trial Results in the same way. possible set of Trial Results in the same way.
The presence of Short Trial Results complicates the Load The presence of Short Trial Results complicates the Load
Classification logic, see more details in Load Classification Logic Classification logic; see more details in "Load Classification Logic"
(Section 6.1). (Section 6.1).
While the Load Classification algorithm is designed to avoid any While the Load Classification algorithm is designed to avoid any
unneeded Trials, for explainability reasons it is recommended for unneeded Trials, for explainability reasons, it is recommended for
users to use such Controller Input instances that lead to all Trial users to use such Controller Input instances that lead to all Trial
Duration values selected by Controller to be the same, e.g., by Duration values selected by the Controller to be the same, e.g., by
setting any Goal Initial Trial Duration to be a single value also setting any Goal Initial Trial Duration to be a single value also
used in all Goal Final Trial Duration attributes. used in all Goal Final Trial Duration attributes.
5.6. Generalized Throughput 5.6. Generalized Throughput
Because testing equipment takes the Intended Load as an input Because testing equipment takes the Intended Load as an input
parameter for a Trial measurement, any load search algorithm needs to parameter for a Trial measurement, any load search algorithm needs to
deal with Intended Load values internally. deal with Intended Load values internally.
But in the presence of Search Goals with a non-zero Goal Loss Ratio But in the presence of Search Goals with a non-zero Goal Loss Ratio
(Section 4.6.3), the Load usually does not match the user's intuition (Section 4.6.3), the Load usually does not match the user's intuition
of what a throughput is. The forwarding rate as defined in of what a throughput is. The forwarding rate as defined in
Section 3.6.1 of [RFC2285] is better, but it is not obvious how to Section 3.6.1 of [RFC2285] is better, but it is not obvious how to
generalize it for Loads with multiple Trials and a non-zero Goal Loss generalize it for Loads with multiple Trials and a non-zero Goal Loss
Ratio. Ratio.
The clearest illustration - and the chief reason for adopting a The clearest illustration -- and the chief reason for adopting a
generalized throughput definition - is the presence of a hard generalized throughput definition -- is the presence of a hard
performance limit. performance limit.
5.6.1. Hard Performance Limit 5.6.1. Hard Performance Limit
Even if bandwidth of a medium allows higher traffic forwarding Even if the bandwidth of a medium allows higher traffic forwarding
performance, the SUT interfaces may have their additional own performance, the SUT interfaces may have their own additional
limitations, e.g., a specific frames-per-second limit on the NIC (a limitations, e.g., a specific frames-per-second limit on the NIC (a
common occurrence). common occurrence).
Those limitations should be known and provided as Max Load Those limitations should be known and provided as Max Load
(Section 4.6.8.1). (Section 4.6.8.1).
But if Max Load is set larger than what the interface can receive or But if Max Load is set larger than what the interface can receive or
transmit, there will be a "hard limit" behavior observed in Trial transmit, there will be a "hard limit" behavior observed in Trial
Results. Results.
Consider that the hard limit is at hundred million frames per second Consider that the hard limit is at one hundred million frames per
(100 Mfps), Max Load is larger, and the Goal Loss Ratio is 0.5%. If second (100 Mfps), Max Load is larger, and the Goal Loss Ratio is
DUT has no additional losses, 0.5% Trial Loss Ratio will be achieved 0.5%. If DUT has no additional losses, 0.5% Trial Loss Ratio will be
at Relevant Lower Bound of 100.5025 Mfps. achieved at the Relevant Lower Bound of 100.5025 Mfps.
Reporting a throughput that exceeds the SUT's verified hard limit is Reporting a throughput that exceeds the SUT's verified hard limit is
counter-intuitive. Accordingly, the [RFC2544] Throughput metric counterintuitive. Accordingly, the [RFC2544] Throughput metric
should be generalized - rather than relying solely on the Relevant should be generalized -- rather than relying solely on the Relevant
Lower Bound - to reflect realistic, limit-aware performance. Lower Bound -- to reflect realistic, limit-aware performance.
MLRsearch defines one such generalization, the Conditional Throughput MLRsearch defines one such generalization, the Conditional Throughput
(Section 4.8.3). It is the Trial Forwarding Rate from one of the (Section 4.8.3). It is the Trial Forwarding Rate from one of the
Full-Length Trials performed at the Relevant Lower Bound. The Full-Length Trials performed at the Relevant Lower Bound. For the
algorithm to determine which trial exactly is in conditional algorithm to determine exactly which trial, see "Conditional
throughput code (Appendix B). Throughput Code" (Appendix B).
In the hard limit example, 100.5025 Mfps Load will still have only In the hard limit example, a 100.5025 Mfps Load will still have only
100.0 Mfps forwarding rate, nicely confirming the known limitation. a 100.0 Mfps forwarding rate, nicely confirming the known limitation.
5.6.2. Performance Variability 5.6.2. Performance Variability
With non-zero Goal Loss Ratio, and without hard performance limits, With a non-zero Goal Loss Ratio, and without hard performance limits,
Low-Loss trials at the same Load may achieve different Trial Low-Loss trials at the same Load may achieve different Trial
Forwarding Rate values just due to DUT performance variability. Forwarding Rate values simply due to DUT performance variability.
By comparing the best case (all Relevant Lower Bound trials have zero By comparing the best case (all Relevant Lower Bound trials have zero
loss) and the worst case (all Trial Loss Ratios at Relevant Lower loss) and the worst case (all Trial Loss Ratios at the Relevant Lower
Bound are equal to the Goal Loss Ratio), one can prove that Bound are equal to the Goal Loss Ratio), one can prove that
Conditional Throughput values may have up to the Goal Loss Ratio Conditional Throughput values may have a relative difference up to
relative difference. the Goal Loss Ratio.
Setting the Goal Width below the Goal Loss Ratio may cause the Setting the Goal Width below the Goal Loss Ratio may cause the
Conditional Throughput for a larger Goal Loss Ratio to become smaller Conditional Throughput for a larger Goal Loss Ratio to become smaller
than a Conditional Throughput for a goal with a lower Goal Loss than a Conditional Throughput for a goal with a lower Goal Loss
Ratio, which is counter-intuitive, considering they come from the Ratio, which is counterintuitive, considering they come from the same
same Search. Therefore, it is RECOMMENDED to set the Goal Width to a Search. Therefore, it is RECOMMENDED to set the Goal Width to a
value no lower than the Goal Loss Ratio of the higher-loss Search value no lower than the Goal Loss Ratio of the higher-loss Search
Goal. Goal.
Although Conditional Throughput can fluctuate from one run to the Although Conditional Throughput can fluctuate from one run to the
next, it still offers a more discriminating basis for comparison than next, it still offers a more discriminating basis for comparison than
the Relevant Lower Bound - particularly when deterministic load the Relevant Lower Bound -- particularly when deterministic load
selection yields the same Lower Bound value across multiple runs. selection yields the same Lower Bound value across multiple runs.
6. MLRsearch Logic and Example 6. MLRsearch Logic and Example
This section uses informal language to describe two aspects of This section uses informal language to describe two aspects of
MLRsearch logic: Load Classification and Conditional Throughput, MLRsearch logic: Load Classification and Conditional Throughput,
reflecting formal pseudocode representation provided in load reflecting formal pseudocode representation provided in "Load
classification code (Appendix A) and conditional throughput code Classification Code" (Appendix A) and "Conditional Throughput Code"
(Appendix B). This is followed by example search. (Appendix B). This is followed by an example search.
The logic is equivalent but not identical to the pseudocode on The logic is equivalent but not identical to the pseudocode in the
appendices. The pseudocode is designed to be short and frequently appendices. The pseudocode is designed to be short and frequently
combines multiple operations into one expression. The logic as combines multiple operations into one expression. The logic, as
described in this section lists each operation separately and uses described in this section, lists each operation separately and uses
more intuitive names for the intermediate values. more intuitive names for the intermediate values.
6.1. Load Classification Logic 6.1. Load Classification Logic
Note: For clarity of explanation, variables are tagged as (I)nput, Note: For clarity of explanation, variables are tagged as (I)nput,
(T)emporary, (O)utput. (T)emporary, and (O)utput.
* Collect Trial Results: * Collect Trial Results:
- Take all Trial Result instances (I) measured at a given load. - Take all Trial Result instances (I) measured at a given load.
* Aggregate Trial Durations: * Aggregate Trial Durations:
- Full-length high-loss sum (T) is the sum of Trial Effective - Full-length high-loss sum (T) is the sum of Trial Effective
Duration values of all full-length high-loss trials (I). Duration values of all full-length high-loss trials (I).
skipping to change at page 54, line 36 skipping to change at line 2463
Duration values of all full-length low-loss trials (I). Duration values of all full-length low-loss trials (I).
- Short high-loss sum is the sum (T) of Trial Effective Duration - Short high-loss sum is the sum (T) of Trial Effective Duration
values of all short high-loss trials (I). values of all short high-loss trials (I).
- Short low-loss sum is the sum (T) of Trial Effective Duration - Short low-loss sum is the sum (T) of Trial Effective Duration
values of all short low-loss trials (I). values of all short low-loss trials (I).
* Derive goal-based ratios: * Derive goal-based ratios:
- Subceed ratio (T) is One minus the Goal Exceed Ratio (I). - Subceed ratio (T) is one minus the Goal Exceed Ratio (I).
- Exceed coefficient (T) is the Goal Exceed Ratio divided by the - Exceed coefficient (T) is the Goal Exceed Ratio divided by the
subceed ratio. subceed ratio.
* Balance short-trial effects: * Balance short-trial effects:
- Balancing sum (T) is the short low-loss sum multiplied by the - Balancing sum (T) is the short low-loss sum multiplied by the
exceed coefficient. exceed coefficient.
- Excess sum (T) is the short high-loss sum minus the balancing - Excess sum (T) is the short high-loss sum minus the balancing
sum. sum.
- Positive excess sum (T) is the maximum of zero and excess sum. - Positive excess sum (T) is the maximum of zero and the excess
sum.
* Compute effective duration totals:
* Compute effective duration totals
- Effective high-loss sum (T) is the full-length high-loss sum - Effective high-loss sum (T) is the full-length high-loss sum
plus the positive excess sum. plus the positive excess sum.
- Effective full sum (T) is the effective high-loss sum plus the - Effective full sum (T) is the effective high-loss sum plus the
full-length low-loss sum. full-length low-loss sum.
- Effective whole sum (T) is the larger of the effective full sum - Effective whole sum (T) is the larger of the effective full sum
and the Goal Duration Sum. and the Goal Duration Sum.
- Missing sum (T) is the effective whole sum minus the effective - Missing sum (T) is the effective whole sum minus the effective
skipping to change at page 55, line 35 skipping to change at line 2512
divided by the effective whole sum. divided by the effective whole sum.
* Classify the Load: * Classify the Load:
- The load is classified as an Upper Bound (O) if the optimistic - The load is classified as an Upper Bound (O) if the optimistic
exceed ratio is larger than the Goal Exceed Ratio. exceed ratio is larger than the Goal Exceed Ratio.
- The load is classified as a Lower Bound (O) if the pessimistic - The load is classified as a Lower Bound (O) if the pessimistic
exceed ratio is not larger than the Goal Exceed Ratio. exceed ratio is not larger than the Goal Exceed Ratio.
- The load is classified as undecided (O) otherwise. - The load is classified as Undecided (O) otherwise.
6.2. Conditional Throughput Logic 6.2. Conditional Throughput Logic
* Collect Trial Results * Collect Trial Results:
- Take all Trial Result instances (I) measured at a given Load. - Take all Trial Result instances (I) measured at a given Load.
* Sum Full-Length Durations: * Sum of Full-Length Durations:
- Full-length high-loss sum (T) is the sum of Trial Effective - Full-length high-loss sum (T) is the sum of Trial Effective
Duration values of all full-length high-loss trials (I). Duration values of all full-length high-loss trials (I).
- Full-length low-loss sum (T) is the sum of Trial Effective - Full-length low-loss sum (T) is the sum of Trial Effective
Duration values of all full-length low-loss trials (I). Duration values of all full-length low-loss trials (I).
- Full-length sum (T) is the full-length high-loss sum (I) plus - Full-length sum (T) is the full-length high-loss sum (I) plus
the full-length low-loss sum (I). the full-length low-loss sum (I).
* Derive initial thresholds: * Derive initial thresholds:
- Subceed ratio (T) is One minus the Goal Exceed Ratio (I) is - Subceed ratio (T) is one minus the Goal Exceed Ratio (I) is
called. called.
- Remaining sum (T) initially is full-lengths sum multiplied by - Remaining sum (T) is initially the full-length sum multiplied
subceed ratio. by the subceed ratio.
- Current loss ratio (T) initially is 100%. - Current loss ratio (T) is initially 100%.
* Iterate through ordered trials * Iterate through ordered trials:
- For each full-length trial result, sorted in increasing order - For each full-length trial result, sorted in increasing order
by Trial Loss Ratio: by Trial Loss Ratio:
o If remaining sum is not larger than zero, exit the loop. o If the remaining sum is not larger than zero, exit the loop.
o Set current loss ratio to this trial's Trial Loss Ratio (I). o Set the current loss ratio to this trial's Trial Loss Ratio
(I).
o Decrease the remaining sum by this trial's Trial Effective o Decrease the remaining sum by this trial's Trial Effective
Duration (I). Duration (I).
* Compute Conditional Throughput * Compute Conditional Throughput:
- Current forwarding ratio (T) is One minus the current loss - Current forwarding ratio (T) is one minus the current loss
ratio. ratio.
- Conditional Throughput (T) is the current forwarding ratio - Conditional Throughput (T) is the current forwarding ratio
multiplied by the Load value. multiplied by the Load value.
6.2.1. Conditional Throughput and Load Classification 6.2.1. Conditional Throughput and Load Classification
Conditional Throughput and results of Load Classification overlap but Conditional Throughput and results of Load Classification overlap but
are not identical. are not identical.
* When a load is marked as a Relevant Lower Bound, its Conditional * When a load is marked as a Relevant Lower Bound, its Conditional
Throughput is taken from a trial whose loss ratio never exceeds Throughput is taken from a trial whose loss ratio never exceeds
the Goal Loss Ratio. the Goal Loss Ratio.
* The reverse is not guaranteed: if the Goal Width is narrower than * The reverse is not guaranteed: If the Goal Width is narrower than
the Goal Loss Ratio, Conditional Throughput can still end up the Goal Loss Ratio, Conditional Throughput can still end up
higher than the Relevant Upper Bound. higher than the Relevant Upper Bound.
6.3. SUT Behaviors 6.3. SUT Behaviors
In DUT in SUT (Section 2.3), the notion of noise has been introduced. In "DUT in SUT" (Section 2.3), the notion of noise is introduced.
This section uses new terms to describe possible SUT behaviors more This section uses new terms to describe possible SUT behaviors more
precisely. precisely.
From measurement point of view, noise is visible as inconsistent From a measurement point of view, noise is visible as inconsistent
trial results. See Inconsistent Trial Results (Section 2.6) for trial results. See "Inconsistent Trial Results" (Section 2.6) for
general points and Loss Ratios and Loss Inversion (Section 5.3) for general points and "Loss Ratios and Loss Inversion" (Section 5.3) for
specifics when comparing different Load values. specifics when comparing different Load values.
Load Classification and Conditional Throughput apply to a single Load Load Classification and Conditional Throughput apply to a single Load
value, but even the set of Trial Results measured at that Trial Load value, but even the set of Trial Results measured at that Trial Load
value may appear inconsistent. value may appear inconsistent.
As MLRsearch aims to save time, it executes only a small number of As MLRsearch aims to save time, it executes only a small number of
Trials, getting only a limited amount of information about SUT Trials, getting only a limited amount of information about SUT
behavior. It is useful to introduce an "SUT expert" point of view to behavior. It is useful to introduce a "SUT expert" point of view to
contrast with that limited information. contrast with that limited information.
6.3.1. Expert Predictions 6.3.1. Expert Predictions
Imagine that before the Search starts, a human expert had unlimited Imagine that before the Search starts, a human expert had unlimited
time to measure SUT and obtain all reliable information about it. time to measure SUT and obtain all reliable information about it.
The information is not perfect, as there is still random noise The information is not perfect, as there is still random noise
influencing SUT. But the expert is familiar with possible noise influencing SUT. But the expert is familiar with possible noise
events, even the rare ones, and thus the expert can do probabilistic events, even the rare ones, and thus, the expert can do probabilistic
predictions about future Trial Outputs. predictions about future Trial Outputs.
When several outcomes are possible, the expert can assess probability When several outcomes are possible, the expert can assess the
of each outcome. probability of each outcome.
6.3.2. Exceed Probability 6.3.2. Exceed Probability
When the Controller selects new Trial Duration and Trial Load, and When the Controller selects a new Trial Duration and Trial Load, and
just before the Measurer starts performing the Trial, the SUT expert just before the Measurer starts performing the Trial, the SUT expert
can envision possible Trial Results. can envision possible Trial Results.
With respect to a particular Search Goal instance, the possibilities With respect to a particular Search Goal instance, the possibilities
can be summarized into a single number: Exceed Probability. It is can be summarized into a single number: Exceed Probability. It is
the probability (according to the expert) that the measured Trial the probability (according to the expert) that the measured Trial
Loss Ratio will be higher than the Goal Loss Ratio. Loss Ratio will be higher than the Goal Loss Ratio.
6.3.3. Trial Duration Dependence 6.3.3. Trial Duration Dependence
When comparing Exceed Probability values for the same Trial Load When comparing Exceed Probability values for the same Trial Load
value but different Trial Duration values, there are several patterns value but different Trial Duration values, there are several patterns
that commonly occur in practice. that commonly occur in practice.
6.3.3.1. Strong Increase 6.3.3.1. Strong Increase
Exceed Probability is very low at short durations but very high at Exceed Probability is very low at short durations but very high at
full-length. This SUT behavior is undesirable, and may hint at full-length. This SUT behavior is undesirable and may hint at a
faulty SUT, e.g., SUT leaks resources and is unable to sustain the faulty SUT, e.g., SUT leaks resources and is unable to sustain the
desired performance. desired performance.
But this behavior is also seen when SUT uses large amount of buffers. But this behavior is also seen when SUT uses large amount of buffers.
This is the main reasons users may want to set large Goal Final Trial This is the main reason users may want to set a large Goal Final
Duration. Trial Duration.
6.3.3.2. Mild Increase 6.3.3.2. Mild Increase
Short trials are slightly less likely to exceed the loss-ratio limit, Short trials are slightly less likely to exceed the loss-ratio limit,
but the improvement is modest. This mild benefit is typical when but the improvement is modest. This mild benefit is typical when
noise is dominated by rare, large loss spikes: during a full-length noise is dominated by rare, large loss spikes: During a full-length
trial, the good-performing periods cannot fully offset the heavy trial, the good-performing periods cannot fully offset the heavy
frame loss that occurs in the brief low-performing bursts. frame loss that occurs in the brief low-performing bursts.
6.3.3.3. Independence 6.3.3.3. Independence
Short trials have basically the same Exceed Probability as full- Short trials have basically the same Exceed Probability as full-
length trials. This is possible only if loss spikes are small (so length trials. This is possible only if loss spikes are small (so
other parts can compensate) and if Goal Loss Ratio is more than zero other parts can compensate) and if the Goal Loss Ratio is more than
(otherwise, other parts cannot compensate at all). zero (otherwise, other parts cannot compensate at all).
6.3.3.4. Decrease 6.3.3.4. Decrease
Short trials have larger Exceed Probability than full-length trials. Short trials have a larger Exceed Probability than full-length
This can be possible only for non-zero Goal Loss Ratio, for example trials. This can only be possible for a non-zero Goal Loss Ratio,
if SUT needs to "warm up" to best performance within each trial. Not for example, if SUT needs to "warm up" to the best performance within
commonly seen in practice. each trial, which is not commonly seen in practice.
7. IANA Considerations 7. IANA Considerations
This document does not make any request to IANA. This document has no IANA actions.
8. Security Considerations 8. Security Considerations
Benchmarking activities as described in this memo are limited to Benchmarking activities as described in this document are limited to
technology characterization of a DUT/SUT using controlled stimuli in the technology characterization of a DUT/SUT using controlled stimuli
a laboratory environment, with dedicated address space and the in a laboratory environment, with dedicated address space and the
constraints specified in the sections above. constraints specified in the sections above.
The benchmarking network topology will be an independent test setup The benchmarking network topology will be an independent test setup
and MUST NOT be connected to devices that may forward the test and MUST NOT be connected to devices that may forward the test
traffic into a production network or misroute traffic to the test traffic into a production network or misroute traffic to the test
management network. management network.
Further, benchmarking is performed on an "opaque" basis, relying Further, benchmarking is performed on an "opaque" basis, relying
solely on measurements observable external to the DUT/SUT. solely on measurements observable external to the DUT/SUT.
The DUT/SUT SHOULD NOT include features that serve only to boost The DUT/SUT SHOULD NOT include features that serve only to boost
benchmark scores - such as a dedicated "fast-track" test mode that is benchmark scores -- such as a dedicated "fast-track" test mode that
never used in normal operation. is never used in normal operation.
Any implications for network security arising from the DUT/SUT SHOULD Any implications for network security arising from the DUT/SUT SHOULD
be identical in the lab and in production networks. be identical in the lab and in production networks.
9. Acknowledgements 9. Acknowledgements
Special wholehearted gratitude and thanks to the late Al Morton for Special wholehearted gratitude and thanks to the late Al Morton for
his thorough reviews filled with very specific feedback and his thorough reviews filled with very specific feedback and
constructive guidelines. Thank You Al for the close collaboration constructive guidelines. Thank you Al for the close collaboration
over the years, Your Mentorship, Your continuous unwavering over the years, your mentorship, and your continuous unwavering
encouragement full of empathy and energizing positive attitude. Al, encouragement full of empathy and an energizing positive attitude.
You are dearly missed. Al, you are dearly missed.
Thanks to Gabor Lencse, Giuseppe Fioccola, Carsten Rossenhoevel and Thanks to Gábor Lencse, Giuseppe Fioccola, Carsten Rossenhövel, and
BMWG contributors for good discussions and thorough reviews, guiding BMWG contributors for good discussions and thorough reviews, guiding
and helping us to improve the clarity and formality of this document. and helping us to improve the clarity and formality of this document.
Many thanks to Alec Hothan of the OPNFV NFVbench project for a Many thanks to the Linux Foundation FastData I/O (FD.io) project,
thorough review and numerous useful comments and suggestions in the specifically committers and contributors to the two core projects of
earlier versions of this document. FD.io: VPP and CSIT. It was there where the need for MLRsearch
originally came up, and it was in FD.io CSIT labs where the first
MLRsearch open-source code got prototyped and then over the years
productized. Thanks also goes to Alec Hothan of the OPNFV NFVbench
project for a thorough review and numerous useful comments and
suggestions in the earlier draft versions of this document.
We are equally indebted to Mohamed Boucadair for a very thorough and We are equally indebted to Mohamed Boucadair for a very thorough and
detailed AD review and providing many good comments and suggestions, detailed AD review, for providing many good comments and suggestions,
helping us make this document complete. for helping us make this document complete.
Our appreciation is also extended to Shawn Emery, Yoshifumi Nishida, Our appreciation is also extended to Shawn Emery, Yoshifumi Nishida,
David Dong, Nabeel Cocker, Lars Eggert, Jen Linkova, Mike Bishop and David Dong, Nabeel Cocker, Lars Eggert, Jen Linkova, Mike Bishop, and
Eric Vyncke for their reviews and valuable comments. Éric Vyncke for their reviews and valuable comments.
10. References 10. References
10.1. Normative References 10.1. Normative References
[RFC1242] Bradner, S., "Benchmarking Terminology for Network [RFC1242] Bradner, S., "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242,
July 1991, <https://www.rfc-editor.org/info/rfc1242>. July 1991, <https://www.rfc-editor.org/info/rfc1242>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
skipping to change at page 60, line 26 skipping to change at line 2747
May 2017, <https://www.rfc-editor.org/info/rfc8174>. May 2017, <https://www.rfc-editor.org/info/rfc8174>.
10.2. Informative References 10.2. Informative References
[FDio-CSIT-MLRsearch] [FDio-CSIT-MLRsearch]
"FD.io CSIT Test Methodology - MLRsearch", October 2023, "FD.io CSIT Test Methodology - MLRsearch", October 2023,
<https://csit.fd.io/cdocs/methodology/measurements/ <https://csit.fd.io/cdocs/methodology/measurements/
data_plane_throughput/mlr_search/>. data_plane_throughput/mlr_search/>.
[Lencze-Kovacs-Shima] [Lencze-Kovacs-Shima]
"Gaming with the Throughput and the Latency Benchmarking Lencse, G., Kovács, Á., and K. Shima, "Gaming with the
Measurement Procedures of RFC 2544", n.d., Throughput and the Latency Benchmarking Measurement
<http://dx.doi.org/10.11601/ijates.v9i2.288>. Procedures of RFC 2544", International Journal of Advances
in Telecommunications, Electrotechnics, Signals and
Systems, vol. 9, no. 2, pp. 10-17,
DOI 10.11601/ijates.v9i2.288, June 2020,
<https://doi.org/10.11601/ijates.v9i2.288>.
[Lencze-Shima] [Lencze-Shima]
"An Upgrade to Benchmarking Methodology for Network Lencse, G. and K. Shima, "An Upgrade to Benchmarking
Interconnect Devices - expired", n.d., Methodology for Network Interconnect Devices", Work in
<https://datatracker.ietf.org/doc/html/draft-lencse-bmwg- Progress, Internet-Draft, draft-lencse-bmwg-rfc2544-bis-
rfc2544-bis-00>. 00, 20 May 2020, <https://datatracker.ietf.org/doc/html/
draft-lencse-bmwg-rfc2544-bis-00>.
[Ott-Mathis-Semke-Mahdavi] [Ott-Mathis-Semke-Mahdavi]
"The Macroscopic Behavior of the TCP Congestion Avoidance Mathis, M., Semke, J., Mahdavi, J., and T. Ott, "The
Algorithm", n.d., Macroscopic Behavior of the TCP Congestion Avoidance
Algorithm", ACM SIGCOMM Computer Communication Review,
vol. 27, no. 3, pp. 67-82, July 1997,
<https://www.cs.cornell.edu/people/egs/cornellonly/ <https://www.cs.cornell.edu/people/egs/cornellonly/
syslunch/fall02/ott.pdf>. syslunch/fall02/ott.pdf>.
[PyPI-MLRsearch] [PyPI-MLRsearch]
"MLRsearch 1.2.1, Python Package Index", October 2023, Python Package Index, "MLRsearch 1.2.1", October 2023,
<https://pypi.org/project/MLRsearch/1.2.1/>. <https://pypi.org/project/MLRsearch/1.2.1/>.
[RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D.
Dugatkin, "IPv6 Benchmarking Methodology for Network Dugatkin, "IPv6 Benchmarking Methodology for Network
Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May
2008, <https://www.rfc-editor.org/info/rfc5180>. 2008, <https://www.rfc-editor.org/info/rfc5180>.
[RFC6349] Constantine, B., Forget, G., Geib, R., and R. Schrage, [RFC6349] Constantine, B., Forget, G., Geib, R., and R. Schrage,
"Framework for TCP Throughput Testing", RFC 6349, "Framework for TCP Throughput Testing", RFC 6349,
DOI 10.17487/RFC6349, August 2011, DOI 10.17487/RFC6349, August 2011,
skipping to change at page 61, line 20 skipping to change at line 2794
[RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet [RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet
Sizes for Additional Testing", RFC 6985, Sizes for Additional Testing", RFC 6985,
DOI 10.17487/RFC6985, July 2013, DOI 10.17487/RFC6985, July 2013,
<https://www.rfc-editor.org/info/rfc6985>. <https://www.rfc-editor.org/info/rfc6985>.
[RFC8219] Georgescu, M., Pislaru, L., and G. Lencse, "Benchmarking [RFC8219] Georgescu, M., Pislaru, L., and G. Lencse, "Benchmarking
Methodology for IPv6 Transition Technologies", RFC 8219, Methodology for IPv6 Transition Technologies", RFC 8219,
DOI 10.17487/RFC8219, August 2017, DOI 10.17487/RFC8219, August 2017,
<https://www.rfc-editor.org/info/rfc8219>. <https://www.rfc-editor.org/info/rfc8219>.
[TST009] "TST 009", n.d., <https://www.etsi.org/deliver/etsi_gs/ [TST009] ETSI, "Network Functions Virtualisation (NFV) Release 3;
NFV-TST/001_099/009/03.04.01_60/gs_NFV- Testing; Specification of Networking Benchmarks and
TST009v030401p.pdf>. Measurement Methods for NFVI", ETSI GS NFV-TST 009 V3.4.1,
December 2020, <https://www.etsi.org/deliver/etsi_gs/NFV-
TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf>.
[Vassilev] "A YANG Data Model for Network Tester Management", n.d., [Vassilev] Vassilev, V., "A YANG Data Model for Network Tester
<https://datatracker.ietf.org/doc/draft-ietf-bmwg-network- Management", Work in Progress, Internet-Draft, draft-ietf-
tester-cfg/06>. bmwg-network-tester-cfg-10, 7 April 2026,
<https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-
network-tester-cfg-10>.
[Y.1564] "Y.1564", n.d., <https://www.itu.int/rec/ [Y.1564] ITU-T, "Ethernet service activation test methodology",
dologin_pub.asp?lang=e&id=T-REC-Y.1564-201602-I!!PDF- ITU-T Recommendation Y.1564, February 2016,
E&type=items>. <https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-
Y.1564-201602-I!!PDF-E&type=items>.
Appendix A. Load Classification Code Appendix A. Load Classification Code
This appendix specifies how to perform the Load Classification. This appendix specifies how to perform the Load Classification.
Any Trial Load value can be classified, according to a given Search Any Trial Load value can be classified, according to a given Search
Goal (Section 4.6.7) instance. Goal (Section 4.6.7) instance.
The algorithm uses (some subsets of) the set of all available Trial The algorithm uses (some subsets of) the set of all available Trial
Results from Trials measured at a given Load at the end of the Results from Trials measured at a given Load at the end of the
Search. Search.
The block at the end of this appendix holds pseudocode which computes The block at the end of this appendix holds pseudocode that computes
two values, stored in variables named optimistic_is_lower and two values, stored in variables named optimistic_is_lower and
pessimistic_is_lower. pessimistic_is_lower.
Although presented as pseudocode, the listing is syntactically valid Although presented as pseudocode, the listing is syntactically valid
Python and can be executed without modification. Python and can be executed without modification.
If values of both variables are computed to be true, the Load in If values of both variables are computed to be true, the Load in
question is classified as a Lower Bound according to the given Search question is classified as a Lower Bound according to the given Search
Goal instance. If values of both variables are false, the Load is Goal instance. If values of both variables are false, the Load is
classified as an Upper Bound. Otherwise, the load is classified as classified as an Upper Bound. Otherwise, the load is classified as
Undecided. Undecided.
Some variable names are shortened to fit expressions in one line. Some variable names are shortened to fit expressions in one line.
Namely, variables holding sum quantities end in _s instead of _sum, Namely, variables holding sum quantities end in "_s" instead of
and variables holding effective quantities start in effect_ instead "_sum", and variables holding effective quantities start with
of effective_. "effect_" instead of "effective_".
The pseudocode expects the following variables to hold the following The pseudocode expects the following variables to hold the following
values: values:
* goal_duration_s: The Goal Duration Sum value of the given Search * goal_duration_s: The Goal Duration Sum value of the given Search
Goal. Goal.
* goal_exceed_ratio: The Goal Exceed Ratio value of the given Search * goal_exceed_ratio: The Goal Exceed Ratio value of the given Search
Goal. Goal.
* full_length_low_loss_s: Sum of Trial Effective Durations across * full_length_low_loss_s: Sum of Trial Effective Durations across
Trials with Trial Duration at least equal to the Goal Final Trial Trials with the Trial Duration at least equal to the Goal Final
Duration and with Trial Loss Ratio not higher than the Goal Loss Trial Duration and with the Trial Loss Ratio not higher than the
Ratio (across Full-Length Low-Loss Trials). Goal Loss Ratio (across Full-Length Low-Loss Trials).
* full_length_high_loss_s: Sum of Trial Effective Durations across * full_length_high_loss_s: Sum of Trial Effective Durations across
Trials with Trial Duration at least equal to the Goal Final Trial Trials with the Trial Duration at least equal to the Goal Final
Duration and with Trial Loss Ratio higher than the Goal Loss Ratio Trial Duration and with the Trial Loss Ratio higher than the Goal
(across Full-Length High-Loss Trials). Loss Ratio (across Full-Length High-Loss Trials).
* short_low_loss_s: Sum of Trial Effective Durations across Trials * short_low_loss_s: Sum of Trial Effective Durations across Trials
with Trial Duration shorter than the Goal Final Trial Duration and with the Trial Duration shorter than the Goal Final Trial Duration
with Trial Loss Ratio not higher than the Goal Loss Ratio (across and with the Trial Loss Ratio not higher than the Goal Loss Ratio
Short Low-Loss Trials). (across Short Low-Loss Trials).
* short_high_loss_s: Sum of Trial Effective Durations across Trials * short_high_loss_s: Sum of Trial Effective Durations across Trials
with Trial Duration shorter than the Goal Final Trial Duration and with the Trial Duration shorter than the Goal Final Trial Duration
with Trial Loss Ratio higher than the Goal Loss Ratio (across and with the Trial Loss Ratio higher than the Goal Loss Ratio
Short High-Loss Trials). (across Short High-Loss Trials).
The code works correctly also when there are no Trial Results at a The code also works correctly when there are no Trial Results at a
given Load. given Load.
<CODE BEGINS> <CODE BEGINS>
exceed_coefficient = goal_exceed_ratio / (1.0 - goal_exceed_ratio) exceed_coefficient = goal_exceed_ratio / (1.0 - goal_exceed_ratio)
balancing_s = short_low_loss_s * exceed_coefficient balancing_s = short_low_loss_s * exceed_coefficient
positive_excess_s = max(0.0, short_high_loss_s - balancing_s) positive_excess_s = max(0.0, short_high_loss_s - balancing_s)
effect_high_loss_s = full_length_high_loss_s + positive_excess_s effect_high_loss_s = full_length_high_loss_s + positive_excess_s
effect_full_length_s = full_length_low_loss_s + effect_high_loss_s effect_full_length_s = full_length_low_loss_s + effect_high_loss_s
effect_whole_s = max(effect_full_length_s, goal_duration_s) effect_whole_s = max(effect_full_length_s, goal_duration_s)
quantile_duration_s = effect_whole_s * goal_exceed_ratio quantile_duration_s = effect_whole_s * goal_exceed_ratio
pessimistic_high_loss_s = effect_whole_s - full_length_low_loss_s pessimistic_high_loss_s = effect_whole_s - full_length_low_loss_s
pessimistic_is_lower = pessimistic_high_loss_s <= quantile_duration_s pessimistic_is_lower = pessimistic_high_loss_s <= quantile_duration_s
optimistic_is_lower = effect_high_loss_s <= quantile_duration_s optimistic_is_lower = effect_high_loss_s <= quantile_duration_s
<CODE ENDS> <CODE ENDS>
Appendix B. Conditional Throughput Code Appendix B. Conditional Throughput Code
This section specifies an example of how to compute Conditional This section specifies an example of how to compute Conditional
Throughput, as referred to in Conditional Throughput (Section 4.8.3). Throughput, as referred to in "Conditional Throughput"
(Section 4.8.3).
Any Load value can be used as the basis for the following Any Load value can be used as the basis for the following
computation, but only the Relevant Lower Bound (at the end of the computation, but only the Relevant Lower Bound (at the end of the
Search) leads to the value called the Conditional Throughput for a Search) leads to the value called the Conditional Throughput for a
given Search Goal. given Search Goal.
The algorithm uses (some subsets of) the set of all available Trial The algorithm uses (some subsets of) the set of all available Trial
Results from Trials measured at a given Load at the end of the Results from Trials measured at a given Load at the end of the
Search. Search.
The block at the end of this appendix holds pseudocode which computes The block at the end of this appendix holds pseudocode that computes
a value stored as variable conditional_throughput. a value stored as the conditional_throughput variable.
Although presented as pseudocode, the listing is syntactically valid Although presented as pseudocode, the listing is syntactically valid
Python and can be executed without modification. Python and can be executed without modification.
Some variable names are shortened in order to fit expressions in one Some variable names are shortened in order to fit expressions in one
line. Namely, variables holding sum quantities end in _s instead of line. Namely, variables holding sum quantities end in "_s" instead
_sum, and variables holding effective quantities start in effect_ of "_sum", and variables holding effective quantities start with
instead of effective_. "effect_" instead of "effective_".
The pseudocode expects the following variables to hold the following The pseudocode expects the following variables to hold the following
values: values:
* goal_duration_s: The Goal Duration Sum value of the given Search * goal_duration_s: The Goal Duration Sum value of the given Search
Goal. Goal.
* goal_exceed_ratio: The Goal Exceed Ratio value of the given Search * goal_exceed_ratio: The Goal Exceed Ratio value of the given Search
Goal. Goal.
* full_length_low_loss_s: Sum of Trial Effective Durations across * full_length_low_loss_s: Sum of Trial Effective Durations across
Trials with Trial Duration at least equal to the Goal Final Trial Trials with the Trial Duration at least equal to the Goal Final
Duration and with Trial Loss Ratio not higher than the Goal Loss Trial Duration and with the Trial Loss Ratio not higher than the
Ratio (across Full-Length Low-Loss Trials). Goal Loss Ratio (across Full-Length Low-Loss Trials).
* full_length_high_loss_s: Sum of Trial Effective Durations across * full_length_high_loss_s: Sum of Trial Effective Durations across
Trials with Trial Duration at least equal to the Goal Final Trial Trials with the Trial Duration at least equal to the Goal Final
Duration and with Trial Loss Ratio higher than the Goal Loss Ratio Trial Duration and with the Trial Loss Ratio higher than the Goal
(across Full-Length High-Loss Trials). Loss Ratio (across Full-Length High-Loss Trials).
* full_length_trials: An iterable of all Trial Results from Trials * full_length_trials: An iterable of all Trial Results from Trials
with Trial Duration at least equal to the Goal Final Trial with the Trial Duration at least equal to the Goal Final Trial
Duration (all Full-Length Trials), sorted by increasing Trial Loss Duration (all Full-Length Trials), sorted by increasing the Trial
Ratio. One item trial is a composite with the following two Loss Ratio. One item trial is a composite with the following two
attributes available: attributes available:
- trial.loss_ratio: The Trial Loss Ratio as measured for this - trial.loss_ratio: The Trial Loss Ratio as measured for this
Trial. Trial.
- trial.effect_duration: The Trial Effective Duration of this - trial.effect_duration: The Trial Effective Duration of this
Trial. Trial.
The code works correctly only when there is at least one Trial Result The code works correctly only when there is at least one Trial Result
measured at a given Load. measured at a given Load.
skipping to change at page 65, line 14 skipping to change at line 2971
Appendix C. Example Search Appendix C. Example Search
The following example Search is related to one hypothetical run of a The following example Search is related to one hypothetical run of a
Search test procedure that has been started with multiple Search Search test procedure that has been started with multiple Search
Goals. Several points in time are chosen, to show how the logic Goals. Several points in time are chosen, to show how the logic
works, with specific sets of Trial Result available. The trial works, with specific sets of Trial Result available. The trial
results themselves are not very realistic, as the intention is to results themselves are not very realistic, as the intention is to
show several corner cases of the logic. show several corner cases of the logic.
In all Trials, the Effective Trial Duration is equal to Trial In all Trials, the Effective Trial Duration is equal to the Trial
Duration. Duration.
Only one Trial Load is in focus, its value is one million frames per Only one Trial Load is in focus; its value is one million frames per
second. Trial Results at other Trial Loads are not mentioned, as the second. Trial Results at other Trial Loads are not mentioned, as the
parts of logic present here do not depend on those. In practice, parts of logic present here do not depend on those. In practice,
Trial Results at other Load values would be present, e.g., MLRsearch Trial Results at other Load values would be present, e.g., MLRsearch
will look for a Lower Bound smaller than any Upper Bound found. will look for a Lower Bound smaller than any Upper Bound found.
At any given moment, exactly one Search Goal is designated as in At any given moment, exactly one Search Goal is designated as in
focus. This designation affects only the Trial Duration chosen for focus. This designation affects only the Trial Duration chosen for
new trials; it does not alter the rest of the decision logic. new trials; it does not alter the rest of the decision logic.
An MLRsearch implementation is free to evaluate several goals An MLRsearch implementation is free to evaluate several goals
simultaneously - the "focus" mechanism is optional and appears here simultaneously -- the "focus" mechanism is optional and appears here
only to show that a load can still be classified against goals that only to show that a load can still be classified against goals that
are not currently in focus. are not currently in focus.
C.1. Example Goals C.1. Example Goals
The following four Search Goal instances are selected for the example The following four Search Goal instances are selected for the example
Search. Each goal has a readable name and dense code, the code is Search. Each goal has a readable name and dense code; the code is
useful to show Search Goal attribute values. useful to show Search Goal attribute values.
As the variable "exceed coefficient" does not depend on trial As the variable "exceed coefficient" does not depend on trial
results, it is also precomputed here. results, it is also precomputed here.
Goal 1: Goal 1:
name: RFC2544 name: RFC2544
Goal Final Trial Duration: 60s Goal Final Trial Duration: 60s
Goal Duration Sum: 60s Goal Duration Sum: 60s
skipping to change at page 66, line 33 skipping to change at line 3038
Goal 4: Goal 4:
name: 20% exceed name: 20% exceed
Goal Final Trial Duration: 60s Goal Final Trial Duration: 60s
Goal Duration Sum: 60s Goal Duration Sum: 60s
Goal Loss Ratio: 0.5% Goal Loss Ratio: 0.5%
Goal Exceed Ratio: 20% Goal Exceed Ratio: 20%
exceed coefficient: 20% / (100% - 20%) = 0.25 exceed coefficient: 20% / (100% - 20%) = 0.25
code: 60f60d0.5l20e code: 60f60d0.5l20e
The first two goals are important for compliance reasons, the other The first two goals are important for compliance reasons; the other
two cover less frequent cases. two cover less frequent cases.
C.2. Example Trial Results C.2. Example Trial Results
The following six sets of trial results are selected for the example The following six sets of trial results are selected for the example
Search. The sets are defined as points in time, describing which Search. The sets are defined as points in time, describing which
Trial Results were added since the previous point. Trial Results were added since the previous point.
Each point has a readable name and dense code, the code is useful to Each point has a readable name and dense code; the code is useful to
show Trial Output attribute values and number of times identical show Trial Output attribute values and the number of times identical
results were added. results were added.
Point 1: Point 1:
name: first short good name: first short good
goal in focus: 1s final (1f120d.5l50e) goal in focus: 1s final (1f120d.5l50e)
added Trial Results: 59 trials, each 1 second and 0% loss added Trial Results: 59 trials, each 1 second and 0% loss
code: 59x1s0l code: 59x1s0l
Point 2: Point 2:
name: first short bad name: first short bad
goal in focus: 1s final (1f120d.5l50e) goal in focus: 1s final (1f120d.5l50e)
added Trial Result: one trial, 1 second, 1% loss added Trial Result: one trial, 1 second, 1% loss
code: 59x1s0l+1x1s1l code: 59x1s0l+1x1s1l
Point 3: Point 3:
name: last short bad name: last short bad
skipping to change at page 67, line 41 skipping to change at line 3095
Point 6: Point 6:
name: first long good name: first long good
goal in focus: TST009 (60f120d0l50e) goal in focus: TST009 (60f120d0l50e)
added Trial Results: one trial, 60 seconds, 0% loss added Trial Results: one trial, 60 seconds, 0% loss
code: 60x1s0l+60x1s1l+1x60s.1l+1x60s0l code: 60x1s0l+60x1s1l+1x60s.1l+1x60s0l
Comments on point in time naming: Comments on point in time naming:
* When a name contains "short", it means the added trial had Trial * When a name contains "short", it means the added trial had a Trial
Duration of 1 second, which is Short Trial for 3 of the Search Duration of 1 second, which is a Short Trial for 3 of the Search
Goals, but it is a Full-Length Trial for the "1s final" goal. Goals, but it is a Full-Length Trial for the "1s final" goal.
* Similarly, "long" in name means the added trial had Trial Duration * Similarly, when a name contains "long", it means the added trial
of 60 seconds, which is Full-Length Trial for 3 goals but Long had a Trial Duration of 60 seconds, which is a Full-Length Trial
Trial for the "1s final" goal. for 3 goals but a Long Trial for the "1s final" goal.
* When a name contains "good" it means the added trial is Low-Loss
Trial for all the goals.
* When a name contains "short bad" it means the added trial is High- * When a name contains "good", it means the added trial is a Low-
Loss Trial for all the goals. Loss Trial for all the goals.
* When a name contains "short bad", it means the added trial is a
High-Loss Trial for all the goals.
* When a name contains "long bad", it means the added trial is a * When a name contains "long bad", it means the added trial is a
High-Loss Trial for goals "RFC2544" and "TST009", but it is a Low- High-Loss Trial for goals "RFC2544" and "TST009", but it is a Low-
Loss Trial for the two other goals. Loss Trial for the two other goals.
C.3. Load Classification Computations C.3. Load Classification Computations
This section shows how Load Classification logic is applied by This section shows how Load Classification logic is applied by
listing all temporary values at the specific time point. listing all temporary values at the specific time point.
C.3.1. Point 1 C.3.1. Point 1
This is the "first short good" point. Code for available results is: This is the "first short good" point. The code for available results
59x1s0l is: 59x1s0l.
+==============+==========+============+============+=============+ +==============+==========+============+============+=============+
|Goal name |RFC2544 |TST009 |1s final |20% exceed | |Goal name |RFC2544 |TST009 |1s final |20% exceed |
+==============+==========+============+============+=============+ +==============+==========+============+============+=============+
|Goal code |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e| |Goal code |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e|
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Full-length |0s |0s |0s |0s | |Full-length |0s |0s |0s |0s |
|high-loss sum | | | | | |high-loss sum | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Full-length |0s |0s |59s |0s | |Full-length |0s |0s |59s |0s |
skipping to change at page 69, line 28 skipping to change at line 3178
|Result | | | | | |Result | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
Table 1 Table 1
This is the last point in time where all goals have this load as This is the last point in time where all goals have this load as
Undecided. Undecided.
C.3.2. Point 2 C.3.2. Point 2
This is the "first short bad" point. Code for available results is: This is the "first short bad" point. The code for available results
59x1s0l+1x1s1l is: 59x1s0l+1x1s1l.
+==============+==========+============+============+=============+ +==============+==========+============+============+=============+
|Goal name |RFC2544 |TST009 |1s final |20% exceed | |Goal name |RFC2544 |TST009 |1s final |20% exceed |
+==============+==========+============+============+=============+ +==============+==========+============+============+=============+
|Goal code |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e| |Goal code |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e|
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Full-length |0s |0s |1s |0s | |Full-length |0s |0s |1s |0s |
|high-loss sum | | | | | |high-loss sum | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Full-length |0s |0s |59s |0s | |Full-length |0s |0s |59s |0s |
skipping to change at page 70, line 33 skipping to change at line 3231
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Pessimistic |100% |100% |50.833% |100% | |Pessimistic |100% |100% |50.833% |100% |
|exceed ratio | | | | | |exceed ratio | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Classification|Upper |Undecided |Undecided |Undecided | |Classification|Upper |Undecided |Undecided |Undecided |
|Result |Bound | | | | |Result |Bound | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
Table 2 Table 2
Due to zero Goal Loss Ratio, RFC2544 goal must have mild or strong Due to zero Goal Loss Ratio, the "RFC2544" goal must have a mild or
increase of exceed probability, so the one lossy trial would be lossy strong increase of exceed probability, so the one lossy trial would
even if measured at 60 second duration. Due to zero exceed ratio, be lossy even if measured at a 60-second duration. Due to zero
one High-Loss Trial is enough to preclude this Load from becoming a exceed ratio, one High-Loss Trial is enough to preclude this Load
Lower Bound for RFC2544. That is why this Load is classified as an from becoming a Lower Bound for the "RFC2544" goal. That is why this
Upper Bound for RFC2544 this early. Load is classified as an Upper Bound for the "RFC2544" goal this
early.
This is an example how significant time can be saved, compared to This is an example of how significant time can be saved, compared to
60-second trials. 60-second trials.
C.3.3. Point 3 C.3.3. Point 3
This is the "last short bad" point. Code for available trial results This is the "last short bad" point. The code for available trial
is: 59x1s0l+60x1s1l results is: 59x1s0l+60x1s1l.
+==============+==========+============+============+=============+ +==============+==========+============+============+=============+
|Goal name |RFC2544 |TST009 |1s final |20% exceed | |Goal name |RFC2544 |TST009 |1s final |20% exceed |
+==============+==========+============+============+=============+ +==============+==========+============+============+=============+
|Goal code |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e| |Goal code |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e|
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Full-length |0s |0s |60s |0s | |Full-length |0s |0s |60s |0s |
|high-loss sum | | | | | |high-loss sum | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Full-length |0s |0s |59s |0s | |Full-length |0s |0s |59s |0s |
|low-loss sum | | | | | |low-loss sum | | | | |
skipping to change at page 72, line 4 skipping to change at line 3294
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Optimistic |100% |0.833% |50% |75.417% | |Optimistic |100% |0.833% |50% |75.417% |
|exceed ratio | | | | | |exceed ratio | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Pessimistic |100% |100% |50.833% |100% | |Pessimistic |100% |100% |50.833% |100% |
|exceed ratio | | | | | |exceed ratio | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Classification|Upper |Undecided |Undecided |Upper Bound | |Classification|Upper |Undecided |Undecided |Upper Bound |
|Result |Bound | | | | |Result |Bound | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
Table 3 Table 3
This is the last point for "1s final" goal to have this Load still This is the last point for the "1s final" goal to have this Load
Undecided. Only one 1-second trial is missing within the 120-second still Undecided. Only one 1-second trial is missing within the
Goal Duration Sum, but its result will decide the classification 120-second Goal Duration Sum, but its result will decide the
result. classification result.
The "20% exceed" started to classify this load as an Upper Bound The "20% exceed" goal started to classify this load as an Upper Bound
somewhere between points 2 and 3. somewhere between points 2 and 3.
C.3.4. Point 4 C.3.4. Point 4
This is the "last short good" point. Code for available trial This is the "last short good" point. The code for available trial
results is: 60x1s0l+60x1s1l results is: 60x1s0l+60x1s1l.
+==============+==========+============+============+=============+ +==============+==========+============+============+=============+
|Goal name |RFC2544 |TST009 |1s final |20% exceed | |Goal name |RFC2544 |TST009 |1s final |20% exceed |
+==============+==========+============+============+=============+ +==============+==========+============+============+=============+
|Goal code |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e| |Goal code |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e|
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Full-length |0s |0s |60s |0s | |Full-length |0s |0s |60s |0s |
|high-loss sum | | | | | |high-loss sum | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Full-length |0s |0s |60s |0s | |Full-length |0s |0s |60s |0s |
skipping to change at page 73, line 21 skipping to change at line 3360
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Pessimistic |100% |100% |50% |100% | |Pessimistic |100% |100% |50% |100% |
|exceed ratio | | | | | |exceed ratio | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Classification|Upper |Undecided |Lower Bound |Upper Bound | |Classification|Upper |Undecided |Lower Bound |Upper Bound |
|Result |Bound | | | | |Result |Bound | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
Table 4 Table 4
The one missing trial for "1s final" was Low-Loss, half of trial The one missing trial for "1s final" was Low-Loss; half of trial
results are Low-Loss which exactly matches 50% exceed ratio. This results are Low-Loss, which exactly matches a 50% exceed ratio. This
shows time savings are not guaranteed. shows time savings are not guaranteed.
C.3.5. Point 5 C.3.5. Point 5
This is the "first long bad" point. Code for available trial results This is the "first long bad" point. The code for available trial
is: 60x1s0l+60x1s1l+1x60s.1l results is: 60x1s0l+60x1s1l+1x60s.1l.
+==============+==========+============+============+=============+ +==============+==========+============+============+=============+
|Goal name |RFC2544 |TST009 |1s final |20% exceed | |Goal name |RFC2544 |TST009 |1s final |20% exceed |
+==============+==========+============+============+=============+ +==============+==========+============+============+=============+
|Goal code |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e| |Goal code |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e|
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Full-length |60s |60s |60s |0s | |Full-length |60s |60s |60s |0s |
|high-loss sum | | | | | |high-loss sum | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Full-length |0s |0s |120s |60s | |Full-length |0s |0s |120s |60s |
skipping to change at page 74, line 32 skipping to change at line 3419
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Pessimistic |100% |100% |33.333% |42.857% | |Pessimistic |100% |100% |33.333% |42.857% |
|exceed ratio | | | | | |exceed ratio | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Classification|Upper |Undecided |Lower Bound |Lower Bound | |Classification|Upper |Undecided |Lower Bound |Lower Bound |
|Result |Bound | | | | |Result |Bound | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
Table 5 Table 5
As designed for TST009 goal, one Full-Length High-Loss Trial can be As designed for the "TST009" goal, one Full-Length High-Loss Trial
tolerated. 120s worth of 1-second trials is not useful, as this is can be tolerated. 120s worth of 1-second trials is not useful, as
allowed when Exceed Probability does not depend on Trial Duration. this is allowed when Exceed Probability does not depend on Trial
As Goal Loss Ratio is zero, it is not possible for 60-second trials Duration. As the Goal Loss Ratio is zero, it is not possible for
to compensate for losses seen in 1-second results. But Load 60-second trials to compensate for losses seen in 1-second results.
Classification logic does not have that knowledge hardcoded, so But Load Classification logic does not have that knowledge hardcoded,
optimistic exceed ratio is still only 50%. so the optimistic exceed ratio is still only 50%.
But the 0.1% Trial Loss Ratio is lower than "20% exceed" Goal Loss But the 0.1% Trial Loss Ratio is lower than the "20% exceed" Goal
Ratio, so this unexpected Full-Length Low-Loss trial changed the Loss Ratio, so this unexpected Full-Length Low-Loss Trial changed the
classification result of this Load to Lower Bound. classification result of this Load to Lower Bound.
C.3.6. Point 6 C.3.6. Point 6
This is the "first long good" point. Code for available trial This is the "first long good" point. The code for available trial
results is: 60x1s0l+60x1s1l+1x60s.1l+1x60s0l results is: 60x1s0l+60x1s1l+1x60s.1l+1x60s0l.
+==============+==========+============+============+=============+ +==============+==========+============+============+=============+
|Goal name |RFC2544 |TST009 |1s final |20% exceed | |Goal name |RFC2544 |TST009 |1s final |20% exceed |
+==============+==========+============+============+=============+ +==============+==========+============+============+=============+
|Goal code |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e| |Goal code |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e|
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Full-length |60s |60s |60s |0s | |Full-length |60s |60s |60s |0s |
|high-loss sum | | | | | |high-loss sum | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Full-length |60s |60s |180s |120s | |Full-length |60s |60s |180s |120s |
|low-loss sum | | | | | |low-loss sum | | | | |
skipping to change at page 76, line 4 skipping to change at line 3483
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Optimistic |66.667% |50% |25% |27.273% | |Optimistic |66.667% |50% |25% |27.273% |
|exceed ratio | | | | | |exceed ratio | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Pessimistic |66.667% |50% |25% |27.273% | |Pessimistic |66.667% |50% |25% |27.273% |
|exceed ratio | | | | | |exceed ratio | | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
|Classification|Upper |Lower Bound |Lower Bound |Lower Bound | |Classification|Upper |Lower Bound |Lower Bound |Lower Bound |
|Result |Bound | | | | |Result |Bound | | | |
+--------------+----------+------------+------------+-------------+ +--------------+----------+------------+------------+-------------+
Table 6 Table 6
This is the Low-Loss Trial the "TST009" goal was waiting for. This This is the Low-Loss Trial the "TST009" goal was waiting for. This
Load is now classified for all goals; the search may end. Or, more Load is now classified for all goals; the search may end. Or more
realistically, it can focus on larger load only, as the three goals realistically, it can focus on larger loads only, as the three goals
will want an Upper Bound (unless this Load is Max Load). will want an Upper Bound (unless this Load is Max Load).
C.4. Conditional Throughput Computations C.4. Conditional Throughput Computations
At the end of this hypothetical search, the "RFC2544" goal labels the At the end of this hypothetical search, the "RFC2544" goal labels the
load as an Upper Bound, making it ineligible for Conditional- load as an Upper Bound, making it ineligible for Conditional
Throughput calculations. By contrast, the other three goals treat Throughput calculations. By contrast, the other three goals treat
the same load as a Lower Bound; if it is also accepted as their the same load as a Lower Bound; if it is also accepted as their
Relevant Lower Bound, Conditional Throughput values can be computed Relevant Lower Bound, Conditional Throughput values can be computed
for each of them. for each of them.
(The load under discussion is 1 000 000 frames per second.) (The load under discussion is one million frames per second.)
C.4.1. Goal 2 C.4.1. Goal 2
The Conditional Throughput is computed from sorted list of Full- The Conditional Throughput is computed from a sorted list of Full-
Length Trial results. As TST009 Goal Final Trial Duration is 60 Length Trial results. As the "TST009" Goal Final Trial Duration is
seconds, only two of 122 Trials are considered Full-Length Trials. 60 seconds, only two of 122 Trials are considered Full-Length Trials.
One has Trial Loss Ratio of 0%, the other of 0.1%. One has a Trial Loss Ratio of 0%, the other of 0.1%.
* Full-length high-loss sum is 60 seconds. * Full-length high-loss sum is 60 seconds.
* Full-length low-loss sum is 60 seconds. * Full-length low-loss sum is 60 seconds.
* Full-length is 120 seconds. * Full-length is 120 seconds.
* Subceed ratio is 50%. * Subceed ratio is 50%.
* Remaining sum initially is 0.5x12s = 60 seconds. * Remaining sum is initially 0.5x12s = 60 seconds.
* Current loss ratio initially is 100%. * Current loss ratio is initially 100%.
* For first result (duration 60s, loss 0%): * For first result (duration 60s, loss 0%):
- Remaining sum is larger than zero, not exiting the loop. - Remaining sum is larger than zero, not exiting the loop.
- Set current loss ratio to this trial's Trial Loss Ratio which - Set current loss ratio to this trial's Trial Loss Ratio, which
is 0%. is 0%.
- Decrease the remaining sum by this trial's Trial Effective - Decrease the remaining sum by this trial's Trial Effective
Duration. Duration.
- New remaining sum is 60s - 60s = 0s. - New remaining sum is 60s - 60s = 0s.
* For second result (duration 60s, loss 0.1%): * For second result (duration 60s, loss 0.1%):
* Remaining sum is not larger than zero, exiting the loop. - Remaining sum is not larger than zero, exiting the loop.
* Current loss ratio was most recently set to 0%. * Current loss ratio was most recently set to 0%.
* Current forwarding ratio is one minus the current loss ratio, so * Current forwarding ratio is one minus the current loss ratio, so
100%. 100%.
* Conditional Throughput is the current forwarding ratio multiplied * Conditional Throughput is the current forwarding ratio multiplied
by the Load value. by the Load value.
* Conditional Throughput is one million frames per second. * Conditional Throughput is one million frames per second.
C.4.2. Goal 3 C.4.2. Goal 3
The "1s final" has Goal Final Trial Duration of 1 second, so all 122 The "1s final" has a Goal Final Trial Duration of 1 second, so all
Trial Results are considered Full-Length Trials. They are ordered 122 Trial Results are considered Full-Length Trials. They are
like this: ordered like this:
60 1-second 0% loss trials, * 60 1-second 0% loss trials,
1 60-second 0% loss trial,
1 60-second 0.1% loss trial, * 1 60-second 0% loss trial,
60 1-second 1% loss trials.
* 1 60-second 0.1% loss trial, and
* 60 1-second 1% loss trials.
The result does not depend on the order of 0% loss trials. The result does not depend on the order of 0% loss trials.
* Full-length high-loss sum is 60 seconds. * Full-length high-loss sum is 60 seconds.
* Full-length low-loss sum is 180 seconds. * Full-length low-loss sum is 180 seconds.
* Full-length is 240 seconds. * Full-length is 240 seconds.
* Subceed ratio is 50%. * Subceed ratio is 50%.
* Remaining sum initially is 0.5x240s = 120 seconds. * Remaining sum is initially 0.5x240s = 120 seconds.
* Current loss ratio initially is 100%. * Current loss ratio is initially 100%.
* For first 61 results (duration varies, loss 0%): * For first 61 results (duration varies, loss 0%):
- Remaining sum is larger than zero, not exiting the loop. - Remaining sum is larger than zero, not exiting the loop.
- Set current loss ratio to this trial's Trial Loss Ratio which - Set current loss ratio to this trial's Trial Loss Ratio, which
is 0%. is 0%.
- Decrease the remaining sum by this trial's Trial Effective - Decrease the remaining sum by this trial's Trial Effective
Duration. Duration.
- New remaining sum varies. - New remaining sum varies.
* After 61 trials, duration of 60x1s + 1x60s has been subtracted * After 61 trials, duration of 60x1s + 1x60s has been subtracted
from 120s, leaving 0s. from 120s, leaving 0s.
skipping to change at page 78, line 26 skipping to change at line 3606
* Current forwarding ratio is one minus the current loss ratio, so * Current forwarding ratio is one minus the current loss ratio, so
100%. 100%.
* Conditional Throughput is the current forwarding ratio multiplied * Conditional Throughput is the current forwarding ratio multiplied
by the Load value. by the Load value.
* Conditional Throughput is one million frames per second. * Conditional Throughput is one million frames per second.
C.4.3. Goal 4 C.4.3. Goal 4
The Conditional Throughput is computed from sorted list of Full- The Conditional Throughput is computed from a sorted list of Full-
Length Trial results. As "20% exceed" Goal Final Trial Duration is Length Trial results. As "20% exceed" Goal Final Trial Duration is
60 seconds, only two of 122 Trials are considered Full-Length Trials. 60 seconds, only two of 122 Trials are considered Full-Length Trials.
One has Trial Loss Ratio of 0%, the other of 0.1%. One has a Trial Loss Ratio of 0%, the other of 0.1%.
* Full-length high-loss sum is 60 seconds. * Full-length high-loss sum is 60 seconds.
* Full-length low-loss sum is 60 seconds. * Full-length low-loss sum is 60 seconds.
* Full-length is 120 seconds. * Full-length is 120 seconds.
* Subceed ratio is 80%. * Subceed ratio is 80%.
* Remaining sum initially is 0.8x120s = 96 seconds. * Remaining sum is initially 0.8x120s = 96 seconds.
* Current loss ratio initially is 100%. * Current loss ratio is initially 100%.
* For first result (duration 60s, loss 0%): * For first result (duration 60s, loss 0%):
- Remaining sum is larger than zero, not exiting the loop. - Remaining sum is larger than zero, not exiting the loop.
- Set current loss ratio to this trial's Trial Loss Ratio which - Set current loss ratio to this trial's Trial Loss Ratio, which
is 0%. is 0%.
- Decrease the remaining sum by this trial's Trial Effective - Decrease the remaining sum by this trial's Trial Effective
Duration. Duration.
- New remaining sum is 96s - 60s = 36s. - New remaining sum is 96s - 60s = 36s.
* For second result (duration 60s, loss 0.1%): * For second result (duration 60s, loss 0.1%):
- Remaining sum is larger than zero, not exiting the loop. - Remaining sum is larger than zero, not exiting the loop.
- Set current loss ratio to this trial's Trial Loss Ratio which - Set current loss ratio to this trial's Trial Loss Ratio, which
is 0.1%. is 0.1%.
- Decrease the remaining sum by this trial's Trial Effective - Decrease the remaining sum by this trial's Trial Effective
Duration. Duration.
- New remaining sum is 36s - 60s = -24s. - New remaining sum is 36s - 60s = -24s.
* No more trials (and remaining sum is not larger than zero), * No more trials (and remaining sum is not larger than zero),
exiting loop. exiting loop.
* Current loss ratio was most recently set to 0.1%. * Current loss ratio was most recently set to 0.1%.
* Current forwarding ratio is one minus the current loss ratio, so * Current forwarding ratio is one minus the current loss ratio, so
99.9%. 99.9%.
* Conditional Throughput is the current forwarding ratio multiplied * Conditional Throughput is the current forwarding ratio multiplied
by the Load value. by the Load value.
* Conditional Throughput is 999 thousand frames per second. * Conditional Throughput is 999 thousand frames per second.
Due to stricter Goal Exceed Ratio, this Conditional Throughput is Due to a stricter Goal Exceed Ratio, this Conditional Throughput is
smaller than Conditional Throughput of the other two goals. smaller than Conditional Throughput of the other two goals.
Authors' Addresses Authors' Addresses
Maciek Konstantynowicz Maciek Konstantynowicz
Cisco Systems Cisco Systems
Email: mkonstan@cisco.com Email: mkonstan@cisco.com
Vratko Polak Vratko Polak
Cisco Systems Cisco Systems
 End of changes. 456 change blocks. 
978 lines changed or deleted 944 lines changed or added

This html diff was produced by rfcdiff 1.48.