]>
Entropy ValuesJuniper Networkstony.li@tony.liMPLS Working GroupEqual Cost Multi-Path (ECMP) forwarding is an essential function in
distributing traffic across parallel paths. Packets within a flow must
be kept on a single path to avoid reordering, while different flows
must be distributed across paths to achieve parallelism.Previously, MPLS has addressed this through the use of an entropy
label, providing up to 20 bits of entropy that can be added to the
label stack to distinguish different flows. With the
interest in MPLS Network Actions, there are proposals to
embedding entropy into alternate structures, so it is an appropriate
time to consider how many bits should be used for entropy in the
future. In this document, we examine the question of how to provide adequate
entropy through a simple stochastic simulation. This is not intended
to be a comprehensive and extensive treatise, but rather a simple
investigation to build intuition into the issues.IntroductionEqual Cost Multi-Path (ECMP) forwarding is an essential function in
distributing traffic across parallel paths. Packets within a flow must
be kept on a single path to avoid reordering, while different flows
must be distributed across paths to achieve parallelism.Previously, MPLS has addressed this through the use of an entropy
label, providing up to 20 bits of entropy that can be added to the
label stack to distinguish different flows. With the
interest in MPLS Network Actions, there are proposals to
embedding entropy into alternate structures, so it is an appropriate
time to consider how many bits should be used for entropy in the
future. In this document, we examine the question of how to provide adequate
entropy through a simple stochastic simulation. This is not intended
to be a comprehensive and extensive treatise, but rather a simple
investigation to build intuition into the issues.ModelIn a typical ECMP situation, a Label Edge Router (LER)
would examine incoming traffic and use values in the incoming packet
to compute an Entropy Label (EL). This would be placed in the label
stack. Subsequent routers on the path could use this EL as input to a
hashing function and the resulting hash value would then be used to
select one of the possible output paths. If the entropy label and the
hashing function are implemented correctly, packets would be uniformly
distributed across the possible paths. If packet sizes are uniformly
distributed, this would also imply that bandwidth would be uniformly
distributed across paths.Moving forward, we expect that entropy will continue to be carried in
the label stack, but may not occupy a Label field inside of a Label
Stack Entry (LSE) . We generalize this to an Entropy
Value (EV) and seek to understand the necessary size of this field.MethodologyTo study this, we employ a pseudo-random number generator to create
Entropy Values and then we apply a hashing function to them. Ideally,
the results of the hash will be uniformly distributed. Error metrics
are used to characterize how far results are from this ideal.RandomnessReal hardware does not use a random number generator to create
entropy. Doing so would require mapping each flow to an entropy value
and this would be painful to implement due to scale. Rather,
implementations that we are familiar with use a hashing function on
the packet header to create an entropy value. Ideally, the results of
this would be uniformly distributed. However, we cannot assume that
all traffic will be quite so cooperative.Conversely, we also cannot assume that the LER will generate no
entropy. If an LER does nothing except place a constant in the
entropy value, no downstream hashing function will be able to extract
any entropy.To model this, we intentionally choose an imperfect random number
generator, specifically, we choose one that has a normal
distribution (i.e., Gaussian or bell curve). This should produce many
different entropy values and if the hashing function is doing its job,
the resulting entropy should result in a uniform distribution of hash
values.HashingSimilarly, there is no standardized way of implementing the hash
function that downstream routers will use to convert an entropy value
into a hash value. In cases where the entropy value has more bits than
the hash value can support, it is up to the hashing function to fold
the entropy from all of the entropy value bits into the hash value.Thus, we consider a number of hashing functions:Addition: High order bits are added to the low order bits to produce
the hash value.Masking: High order bits are simply discarded.Mixing and Masking: Bits are shifted, added, and then masked.Xor: High order bits are XOR'ed into the low order bits.CRC-CCITT: Compute a CRC over the entropy value and XOR the
resulting bits together.MD4: Compute the MD4 digest over the entropy value and XOR the
resulting bits together.MD4: Compute the MD5 digest over the entropy value and XOR the
resulting bits together.SHA1: Compute the SHA1 digest over the entropy value and XOR the
resulting bits together.SHA256: Compute the SHA256 digest over the entropy value and XOR the
resulting bits together.Some of these hash functions work better than others. Additional hash
functions could easily be added.To simplify our simulation, we have primarily focused on hashes that
have a range that is a power of two. However, in a practical
application, that is not appropriate and the range must be an
arbitrary integer, so that we can support n-way ECMP. This is
discussed further below.Future Hardware ConsiderationsWhen ECMP was first implemented, 4-way ECMP was the norm. In the
early days of sparse networks, that was frequently adequate. As the
number of interfaces on a router has grown, ECMP has followed suit,
and 128-way and 256-way ECMP is now in production.While hardware has provided an impetetus to grow the breadth of ECMP,
it is also suggesting that there is an upper bound to what needs to be
supported. It is fairly clear that with current networking trends, we
will never need more paths from a single router than it has physical
interfaces. While logically, more would be possible, the growth in
bandwidth has driven the need for ECMP and that bandwidth is most
easily provided through the use of non-fractional interfaces.Concurrently, we note a trend away from building multi-chassis routers
with an arbitrary number of physical interfaces, so this implies a
bound of around 1000 physical interface per router for the foreseable
future.Hardware is also most conveniently impelemented on power of two
boundaries. Thus, we investigate ECMP from 4-way to 4096-way, in
powers of two.ResultsFor hash values from 8 to 12 bits, we consider entropy values with 0
to 5 additional bits (i.e., 8 to 17 bits). We generate enough random
samples so that each hash bucket should ideally have 1000 entries.The error for each has bucket is then difference between the number of
hash results and 1000. We then compute the following error metrics
for each combination of algorithm, hash value bits, and additional
bits:Root Mean Square Error (RMSE): The Root Mean Square Error is the
square root of the sum of the squares of the errors for each bucket,
divided by the number of buckets.Mean Average Error (MAE): The Mean Average Error is the arithmetic
mean of the absolute value of the error for each bucket.Mean Average Percentage Error (MAPE): The Mean Average Percentage
Error is the arithmetic mean of the percentage of error for each
bucket.We then summarize the results based on the number of additional bits
and on algorithm, computing the mean of each of the above metrics.ConclusionThe arguments in suggest that 4096-way ECMP will be
sufficient for some time to come. This suggests that a base of 12 bits
would be a good start.We arbitrarily select a MAPE of 5.0 as a threshold of acceptability.
This represents a distribution that is within 5% of our goal of a
uniform distribution.A solution to the issue would be a number of bits entropy value and a
set of hashing algorithms that would provide acceptable performance
over the full range of hash values, from 2 to 12 bits.We observe that for some entropy values, we get unacceptable solutions
across all algorithms:Clearly, a 2 bit entropy value is not acceptable.We also observe that some algorithms perform better than others:Understanding why some algorithms perform poorly is not understood at
this time and a subject for future research.Once we reach about 10 bits of entropy value, most algorithms work
well, most of the time:If we look at summaries by additional bits and by algorithm we see:This leads us to propose that use at least 4 additional bits for our
entropy value, resulting in total of 16 (12 + 4). Please note that
this is a minimum. Most proposals are proposing higher numbers of
bits, thus exceeding this minimum.ValidationTo further validate that 16 bits of entropy is sufficient, we also
simulated hashing with a non-power of two result to support n-way
ECMP. To map the hash to a range of n possible results, we performed
a 'mod n' operation after each of the above hashes. We then selected
10 random n values that are not powers of two and repeated the above
experiment.In this step, we generated 16 bits of entropy, hashed that and then
folded it to down to 16 bits. We then applied the 'mod n' operation to
get down to a range of n results. In these circumstances, the 'add',
'mask', and 'xor' hashes are all identical and result in no changes,
as they simply are the identity function on 16 bits. The 'add' and
'xor' functions were removed for efficiency.The results demonstrate that the 'mask' (identity) function performs
adequately in this circumstance, again supporting the claim that 16
bits is sufficient entropy.Security ConsiderationsNo security issues are discussed in this document.IANA ConsiderationsThis document makes no requests of IANA.
&RFC6790;
&RFC4221;
&RFC3032;
&I-D.bocci-mpls-miad-adi-requirements;
&I-D.andersson-mpls-mna-fwk;
Source CodeFull results