Business Premium Access and InternetConnect performance metrics test

The professional Business Premium Access internet connection can be used as a basis for telephony, internet and cloud applications. With bandwidths of up to 10 Gbit/s, it meets the highest standards of stability, flexibility and security.
Separate services for voice and Internet (e.g. operation of web and mail servers; site networking) can be flexibly combined and scaled, allowing our customers to decide which services they need at any time.

The BPA provides its customers with the InternetConnect service, enabling them to access Telekom's internet backbone through Telkom-installed devices at the customer locations, offering a range of speeds and traffic class profiles for internet connectivity.
The customer is allocated both version 4 (IPv4) and version 6 (IPv6) IP addresses, employing a dual-stack approach. This dual-stack configuration facilitates communication between the Internet and the customer's local area network (LAN) using IPv4 and IPv6 addresses.

Telekom Deutschland GmbH commissioned EANTC to evaluating Telekom’s Business Premium Access Portfolio.

A series of tests were conducted across internet link speeds from 50 Mbits to 10 Gbits as specified in the Test Results Annex, focusing on key performance indicators such as link bandwidth, latency, jitter, and packet loss.

These KPIs were selected due to their significant influence on network functionality and user experience, monitoring link bandwidth is essential to understand the data transmission capacity of the links.

• Latency was tracked as it affects the data transmission speed, which is crucial for time-sensitive applications.
• Jitter, representing the variability in latency, was also tracked, as it can impact the quality of real-time communication services like VoIP.
• Packet loss was another focus, given its impact on network reliability; high levels of packet loss lead to communication disruptions and reduced data integrity.

Various parameters, including different frame sizes and service classes, were applied to comprehensively assess the internet links' performance and verify whether it aligns with the performance criteria outlined in its BPA and IP service InternetConnect SLAs.

Telekom deployed a remote device (EANTC RD) at the EANTC office premises, equipped with a 10 Gbit/s interface, and established a permanent connection to the Telekom network via Business Premium Access. Telkom followed the same installation procedure and configuration settings for the Remote Device (RD) at EANTC's lab as they typically apply to their regular customers.

The same setup was executed at Bielefeld, with two remote devices - one with a 10 Gbit interface (Bielefeld RD-1) and the other with a 1 Gbit/s interface (Bielefeld RD-2), to provide the ability to test both bandwidth options as shown in Figure1.

In the test bed, the remote devices were connected to the Veex testers at both sites. At the Bielefeld location, the tester was linked via a pre-programmed Next Unit of Computing (NUC), enabling the EANTC team to remotely access the tester through various services, including SSH, VNC, HTTP, and HTTPS.

Figure1: Test Setup

Tables 1 and 2 detail the remote devices utilized at each site, including the traffic generators.

     Table 1: Remote Devices Hardware and Software

The information regarding the software versions is confidential. Therefore, EANTC agreed with the Telekom team to identify them using the test dates, and Telekom will internally correlate these dates with the actual software versions.

Testing Equipment: VeEX

Table 2: Traffic Generators Details

EANTC set the testers in loopback mode to ensure the bidirectional traffic flow. We initiated the traffic from the Berlin-based tester, while the second tester in Bielefeld received the traffic and reflected it to the traffic initiator tester at the Berlin site. In this setup, we ensured that both directions were tested .

        Table 3: Test Traffic parameters

EANTC conducted thorough testing on the Telekom BPA connection to ensure its throughput and performance aligned with the bandwidths promised to Telekom's customers. In our test, we followed the standards of  RFC2544. This involved two rounds of testing, the first with a frame size of 128 bytes and the second with a frame size of 1518 bytes. We verified that the connection offered the average data throughput of the Telekom stated in the InternetConnect performance parameters (Leistungsbeschreibung und Preise Business Premium Access 01.02.2022 Version 1).

We also evaluated the BPA connection's latency, jitter, and packet loss in separate tests using Telekom Class of Service Profiles. Due to constraints in the testing setup, round-trip delays were measured without GPS antenna synchronization. These tests also included an assessment of jitter simultaneously with latency. Furthermore, EANTC examined packet loss in correlation with latency and jitter.

Per the Telekom Service Level Agreement  (SLA), they have six profiles encompassing three quality classes. This includes a "Best Effort" class for standard service, "Low Delay" classes aimed at reducing transmission times, and "Low Loss" classes that concentrate on minimizing data loss.

These offered classes can enable businesses and individuals to optimize their network resources effectively, ensuring that critical applications receive the bandwidth and quality of service they require.

Our evaluation focused on Profile 1, detailed in Table 4. This profile allocates up to 90% of the bandwidth for the Low Delay and Low Loss classes between the Customer Premises Equipments (CPE) over the Telekom IP backbone network. The unused bandwidth in these two categories becomes accessible for data packets classified under the Best Effort class. Additionally, Table 4 includes the SLA requirements, such as one-way delay, jitter, and loss, for the three quality classes within Profile 1.

Table 4: Telekom Class of Service Profile

After completing the throughput test following the RFC2544, we aimed to conduct the remaining tests for latency, jitter, and packet loss under a realistic load while not utilizing 100% of the bandwidth capacity. Thus, we generated three simultaneous streams during the testing phase, ensuring that the traffic load amounted to 85% of the maximum throughput capacity listed in Telekom SLA. So a significant portion, amounting to 90%, was allocated to streams with low delay (DSCP AF41) and low loss (DSCP AF31) marking. The remaining 10% of the traffic comprised the best effort stream, identified by DSCP CS0.

EANTC conducted the testing at specific link speeds (100 Mbps, 600Mbps, and 1 Gbps) twice, once in the middle of the day and again in the evening. This approach helps evaluate the connection as it allows for assessing network performance under varying network traffic conditions. Testing at different times of the day provides insights into how the link handles peak and off-peak loads. Since the Deutsche Telekom core network serves both business and home internet connections, it's reasonable to anticipate increased network activity in the evenings. This could result in elevated latency levels during these times. This variation helps in understanding the consistency and reliability of the connection across different times, ensuring that the link maintains optimal performance regardless of the time or network load.

The following streams shown in Table 5 are used  for the  BPA 10 G (XL) and BPA 1 G (L) connections in the throughput test:

Table 5: Traffic streams of the throughput tests

While in the latency, jitter, and packet loss tests, we used three streams, as shown in Table 6. These streams were configured with identical source and destination IP addresses, differentiated only by the Differentiated Services Code Point (DSCP) value. This setup guaranteed that all three streams followed the same path through Telekom's backbone network.

Table 6: Test traffic streams for BPA L connection

EANTC verified that the average data rate was consistently met for all the available L and XL connection speeds. The testing results indicated no substantial differences in performance between the midday and evening tests conducted starting from 7:30 PM. Additionally, it was confirmed that both the maximum Round Trip Delay (RTD) and Frame Delay Variation (FDV) complied with the specifications for the 1 Gbit/s and 10 Gbit/s connections, utilizing a frame size of 128 bytes.

BPA 1 G (L) Results:

All test runs were considered successful, with each meeting the expected standards. The average bandwidth requirements were satisfied, and the measurements for delay, jitter, and loss all stayed below the maximum thresholds set by Telekom. The following graphs, Graph 1 and 2, illustrate the delay and jitter values, showing a consistent range across the board. However, there is a notable exception: during one midday run, there was a peak of 37 ms in delay for all three streams (Best Effort, Low Loss, and Low Delay).

Although the maximum round-trip delay value is notably high, it's important to recognize that it represents a peak occurrence. The average and minimum round-trip delays for this test were closely aligned at 7,19 ms and 7,18 ms respectively(check the result appendix for the complete values), indicating that the network typically operates with much lower latency. This suggests that the spike to 37 ms may have been an anomaly, likely caused by a temporary network issue.

Graph 3 displays the frame loss recorded in each test run. Notably, packet loss occurred in two runs at 100 M during midday and evening tests; however, these values are within the acceptable limits defined by the SLA for the respective class of service profile.

Graph 1: Delay Measurement (Max RTD) for BPA L

Graph 2:Max Jitter for BPA L

Graph 3: Frame loss for BPA L

When the data rate increased for the 1 G connection, EANTC noticed packet loss for the 128-frame streams. Telekom explained that for L connection, the installed  RD (Adtran NV4.660)  requires creating a forwarding entry for each flow in Layer-3. The time it takes to create these entries depends on the number of flows being sent. During this creation period, traffic will be routed via the Central Processing Unit (CPU) instead of the more efficient hardware path. The CPU path has limited bandwidth, so packet loss will occur if higher bandwidth is sent before the remote device has learned the flows.

To address this issue, EANTC generated traffic streams for a short period to ensure the forwarding entry was created. Then, we generated the actual test traffic to avoid packet loss.

BPA 10 G (XL) Results:

We achieved similarly successful results for the 10 G connection, where there was no packet loss, and neither delay nor jitter exceeded the permitted limits. Also, all speeds were achieved with the expected average throughput. A notable observation was that the maximum RTD  values for the XL connection were lower (less than 7,35 ms), compared to those of the L connection, which is 35 ms. This difference could stem from the different RD units used for each connection, and potentially, it indicates a variation in their capacity for rapid traffic processing.

Graph 4: Delay Measurement (Max RTD) for BPA XL

Graph 5: Max Jitter BPA  XL

While testing the 7.500 Mbit/s speed, EANTC observed that the speed rate reached 8.300 Mbit/s. This was particularly noticeable when evaluating 128-byte frame streams using a binary search method as per RFC 2.455. The data rate exceeded the preset bandwidth limit.

With further examination in collaboration with the Telekom team, it was clarified that the root of this anomaly was the configuration of the shaper on the Broadband Network Gateway (BNG) on the Bielefeld side. The shaper was set to operate at Layer 2 (L2)  rather than Layer 1 (L1).

For the Layer 3 (L3) frame size of 128 bytes, an additional 18 bytes are accounted for at L2 due to the Ethernet header and trailer, which translates to a 14% overhead. Moreover, at the L1, the Inter Frame Gap (IFG) and the Preamble contribute an extra 20 bytes, constituting a 13,6% overhead.

Thus, when the shaper was configured for a specific data rate limit (x Mbit/s) at the L2, the actual data rate at the L1  will be higher, precisely calculated as (x Mbit/s + 14%  of the original x Mbit/s speed).

EANTC successfully tested the BPA L and XL services, focusing on critical performance metrics like bandwidth, latency, jitter, and packet loss across the three distinct quality classes of traffic flows.

The approach used in our test, which involved measuring the round-trip delay, aligns well with the Telekom specifications that focus on one-way delay values. Although there is potential for asymmetry in delay contributions from each direction, the maximum round-trip delay falling below the anticipated threshold confirmed the validity of this testing. Furthermore, this suggests that the quality of BPA connections not only meets but potentially exceeds the standards outlined in the Service Level Agreement (SLA).

The following table demonstrates that with two key observations.

Table 8: Test Results Summary

*We encountered a 1 G remote device performance issue during these tests. This issue manifested as packet loss at the start of flow generation, particularly noticeable at high frame rates per second. However, from Telekom's point of view, this degree of packet loss is deemed acceptable for real-world applications.

**Moreover, our testing revealed that the shaper's design, which is based on L2, allows for a data rate that exceeds the levels promised to customers. This finding is particularly beneficial as it suggests the network can handle higher data rates as a buffer, which is advantageous in case of unexpected traffic increases or congestion.