Wrong Traffic Shaping in Secondary NIC over L2 networks

[luganofer]

ISSUE TYPE

• Bug Report

COMPONENT NAME

L2 Networks

CLOUDSTACK VERSION

4.19.1

CONFIGURATION

Advance networking, VLANs and VMware ESXi 8.0.0 hypervisor

SUMMARY

When the same VM is connected to two or more L2 networks, the secondary NIC gets a different bandwidth (vmware traffic shapping) than the bandwidth of the default NIC, even if all the L2 networks use the same network offer.

STEPS TO REPRODUCE

1) Have an L2 network offer configured with a network rate (Mb/s) different from the global setting “Vm network throttling rate”( vm.network.throttling.rate). 

2) Create two or more L2 networks from the same network offer.

3) Connect the same VM to the two o more L2 networks.

EXPECTED RESULTS

All the NICs of the VM in question should have the same bandwidth allocated (VMware Traffic Shapping).

ACTUAL RESULTS

The default NIC takes a different bandwidth value (VMware Traffic Shapping) than the secondary NICs.
The default NIC gets a Traffic Shapping in VMware according to the value of the global setting vm.network.throttling.rate while the secondary NIC(s) takes the value of the network offer.

[iishitahere]

Hi @luganofer , I’d like to work on this issue. From my understanding, the problem involves incorrect traffic shaping in secondary NICs over L2 networks. Could you provide more details on the expected traffic shaping behavior or any specific test cases to validate the fix? Once assigned, I’ll dive in and start working on this. Thank you!

[luganofer]

Hi @iishitahere . Thank you for your comment. Yes, the use case is relatively simple.
A user creates two L2 type networks (from the same network offering for example) and the bandwidth that cloudstack applies to the default NIC (Traffic Shapping in VMware) differs from the one configured in the secondary NIC, when it should be the same bandwidth since both networks were created from the same network offering.
From what I could find out, CloudStack would NOT be using the value configured in the vm.network.throttling.rate parameter of the global settings for the secondary NICs as it does for the default NIC. Let me know if this helps in understanding the case and thanks for taking an interest in it.

[iishitahere]

Hi @iishitahere . Thank you for your comment. Yes, the use case is relatively simple. A user creates two L2 type networks (from the same network offering for example) and the bandwidth that cloudstack applies to the default NIC (Traffic Shapping in VMware) differs from the one configured in the secondary NIC, when it should be the same bandwidth since both networks were created from the same network offering. From what I could find out, CloudStack would NOT be using the value configured in the vm.network.throttling.rate parameter of the global settings for the secondary NICs as it does for the default NIC. Let me know if this helps in understanding the case and thanks for taking an interest in it.

Hi @luganofer , thank you for the detailed explanation! I understand now that the issue arises because CloudStack does not apply the vm.network.throttling.rate parameter to secondary NICs, even though both NICs are created from the same network offering. The expected behavior is that the bandwidth should be consistent across all NICs.

I’ll investigate this further and look into how the vm.network.throttling.rate parameter is being applied. If there are any specific areas of the codebase or logs that you recommend focusing on, please let me know. I'll keep you updated on my progress. Thanks again for your guidance!

[BryanMLima]

Hello, @luganofer and @iishitahere

From what I remember of the code, the throttling is applied differently if the NIC of the VM is the default one or not.

cloudstack/server/src/main/java/com/cloud/network/NetworkModelImpl.java

Line 1037 in db2e89a

public Integer getNetworkRate(long networkId, Long vmId) {

For the default NIC, the network throttling is applied based on the compute offering network bandwidth of the VM. If there is no limit set in the compute offering, the value of the configuration vm.network.throttling.rate will be used.

For additional NICs, the network throttling is applied based on the network offering bandwidth. If there is no limit set in the network offering, then the value of the configuration network.throttling.rate will be used.

To be fair, this behavior is kind of confusing, and maybe a discussion should be done before any actual implementation.

What do you guys think? cc @DaanHoogland @JoaoJandre @GutoVeronezi @weizhouapache

[iishitahere]

Hello, @luganofer and @iishitahere

From what I remember of the code, the throttling is applied differently if the NIC of the VM is the default one or not.

cloudstack/server/src/main/java/com/cloud/network/NetworkModelImpl.java

Line 1037 in db2e89a

public Integer getNetworkRate(long networkId, Long vmId) {

For the default NIC, the network throttling is applied based on the compute offering network bandwidth of the VM. If there is no limit set in the compute offering, the value of the configuration vm.network.throttling.rate will be used.

For additional NICs, the network throttling is applied based on the network offering bandwidth. If there is no limit set in the network offering, then the value of the configuration network.throttling.rate will be used.

To be fair, this behavior is kind of confusing, and maybe a discussion should be done before any actual implementation.

What do you guys think? cc @DaanHoogland @JoaoJandre @GutoVeronezi @weizhouapache

Hello @BryanMLima , @luganofer
Thank you for pointing out the behavior of network throttling for default and additional NICs. I agree that the current approach can be somewhat confusing. I’d like to explore this issue further to clarify and potentially improve the implementation.

Here’s how I plan to proceed:

Analyze the existing logic in NetworkModelImpl.java to ensure a comprehensive understanding of the throttling rules for both NIC types.
Propose a consistent or more intuitive mechanism for throttling, balancing functionality and ease of configuration.
Incorporate input from the team to ensure alignment with the broader system design.
Could you assign this issue to me so I can take the lead on the discussion and possible implementation? I’m excited to contribute!

Looking forward to your thoughts.

Best regards,
Ishita

[iishitahere]

Thank you for assigning me to this issue! I’ll start working on it right away and keep you updated on my progress. Please let me know if there’s anything specific you’d like me to focus on. Thanks again for the opportunity!

[JoaoJandre]

Hello, @luganofer and @iishitahere

From what I remember of the code, the throttling is applied differently if the NIC of the VM is the default one or not.

cloudstack/server/src/main/java/com/cloud/network/NetworkModelImpl.java

Line 1037 in db2e89a

public Integer getNetworkRate(long networkId, Long vmId) {

For the default NIC, the network throttling is applied based on the compute offering network bandwidth of the VM. If there is no limit set in the compute offering, the value of the configuration vm.network.throttling.rate will be used.

For additional NICs, the network throttling is applied based on the network offering bandwidth. If there is no limit set in the network offering, then the value of the configuration network.throttling.rate will be used.

To be fair, this behavior is kind of confusing, and maybe a discussion should be done before any actual implementation.

What do you guys think? cc @DaanHoogland @JoaoJandre @GutoVeronezi @weizhouapache

You're correct on the throttling behavior @BryanMLima.

I also think that this behavior is very unintuitive for users, the throttling should happen either only on the network offering or on the compute offering. Personally I think that only having the setting in the network offering makes more sense. However, we could also have a setting to determine if the throttle (for all NICs of a VM) will be based on the compute or the network offering.

@iishitahere I would wait for some other opinions before implementing anything, if you're not familiar with ACS and its code, I would start by getting used to it. Once the way forward is established, I would start the actual code changes (this is just my opinion, feel free to ignore it). In any case, I think that the easiest way forward would be creating that configuration that I mentioned.

Furthermore, for maximum configurability, we could eventually add a feature to allow configuring the throttling of each NIC of a given VM separately, regardless of offerings. I know that @hsato03 was working on this specific feature.

[DaanHoogland]

@iishitahere , very welcome to the community (again). I have seen your name on several issues. Can you (introduce yourself and) tell us which ones you are taking on first, on [email protected]?

I hope you are successful in submitting code, soon.

[DaanHoogland]

@iishitahere , very welcome to the community (again). I have seen your name on several issues. Can you (introduce yourself and) tell us which ones you are taking on first, on [email protected]?

I hope you are successful in submitting code, soon.

oh and if it is this one, do you plan to submit your code for 4.19.2 or later (just an administrative question, you can always change your mind later)

[iishitahere]

@iishitahere , very welcome to the community (again). I have seen your name on several issues. Can you (introduce yourself and) tell us which ones you are taking on first, on [email protected]?
I hope you are successful in submitting code, soon.

oh and if it is this one, do you plan to submit your code for 4.19.2 or later (just an administrative question, you can always change your mind later)

Hi @DaanHoogland,

Thank you for reaching out and for your warm welcome to the community!

I am currently working on Issue #10007 - Wrong Traffic Shaping in Secondary NIC over L2 Networks. My primary focus is on analyzing the disparity between the bandwidth allocation for the default NIC and the secondary NICs in L2 networks, as described in the issue. I am addressing this by reviewing the configuration of VMware Traffic Shaping in relation to the global setting (vm.network.throttling.rate) and the network offer parameters.

I plan to submit my code for this fix by December 6th, targeting version 4.19.2 for now. Please let me know if there are any specific guidelines or suggestions for this fix.

Looking forward to contributing and receiving feedback from the community!

[iishitahere]

@iishitahere , very welcome to the community (again). I have seen your name on several issues. Can you (introduce yourself and) tell us which ones you are taking on first, on [email protected]?

I hope you are successful in submitting code, soon.

Hi @DaanHoogland ,

Thank you so much for the warm welcome to the community! I am thrilled to be a part of this amazing group and contribute to the Apache CloudStack project.

A bit about me: I am Ishita Jaiswal, a first-year B.Tech student with a strong interest in open-source projects. I specialize in Java development and am particularly enthusiastic about debugging and resolving issues in complex systems like CloudStack.
I’m looking forward to contributing and learning more from this incredible community. Please feel free to share any feedback or suggestions as I work on this!

Best regards,
Ishita Jaiswal

[iishitahere]

Hi @DaanHoogland ,

I hope this message finds you well. I apologize for the delay in submitting this fix; I was one day late due to unforeseen circumstances. Thank you for your patience.

Regarding the issue, I’ve made the following changes in the VmwareTrafficLabel class to resolve the inconsistent bandwidth allocation for secondary NICs in VMware L2 networks:

Traffic Label Parsing Enhancement: Updated the _parseLabel method to ensure consistent interpretation of the traffic label, irrespective of NIC type.
Default Values Adjustment: Ensured that both the default NIC and secondary NICs adhere to the same bandwidth specifications derived from the network offering or global throttling settings.
Error Handling Improvement: Added checks to validate and handle unexpected or invalid traffic label fields gracefully, reducing potential misconfigurations.
Code Cleanup: Simplified logic and ensured compliance with the project's coding standards for better readability and maintainability.
These changes aim to align the bandwidth configuration across all NICs of a VM, as per the expected behavior described in the issue report. I have tested the changes locally to verify their correctness.

Please let me know if there’s anything else I need to address or refine. I look forward to your feedback and further guidance.

Thank you again for your time and support!

Best regards,
Ishita Jaiswal

[DaanHoogland]

Hi @DaanHoogland ,

I hope this message finds you well. I apologize for the delay in submitting this fix; I was one day late due to unforeseen circumstances. Thank you for your patience.

Regarding the issue, I’ve made the following changes in the VmwareTrafficLabel class to resolve the inconsistent bandwidth allocation for secondary NICs in VMware L2 networks:

Traffic Label Parsing Enhancement: Updated the _parseLabel method to ensure consistent interpretation of the traffic label, irrespective of NIC type. Default Values Adjustment: Ensured that both the default NIC and secondary NICs adhere to the same bandwidth specifications derived from the network offering or global throttling settings. Error Handling Improvement: Added checks to validate and handle unexpected or invalid traffic label fields gracefully, reducing potential misconfigurations. Code Cleanup: Simplified logic and ensured compliance with the project's coding standards for better readability and maintainability. These changes aim to align the bandwidth configuration across all NICs of a VM, as per the expected behavior described in the issue report. I have tested the changes locally to verify their correctness.

Please let me know if there’s anything else I need to address or refine. I look forward to your feedback and further guidance.

Thank you again for your time and support!

Best regards, Ishita Jaiswal

hi @iishitahere (Ishita),

this works sounds awesome, but we deal (communicate largely through code and/or illustrated by code. Can I ask you to submit a PR and discuss the changes there?

You do not have to apologise for anything. We do not expect things and everything is a bonus. We are very glad that you are willing to work on the codebase.
regards,

[iishitahere]

Hi @DaanHoogland ,
I hope this message finds you well. I apologize for the delay in submitting this fix; I was one day late due to unforeseen circumstances. Thank you for your patience.
Regarding the issue, I’ve made the following changes in the VmwareTrafficLabel class to resolve the inconsistent bandwidth allocation for secondary NICs in VMware L2 networks:
Traffic Label Parsing Enhancement: Updated the _parseLabel method to ensure consistent interpretation of the traffic label, irrespective of NIC type. Default Values Adjustment: Ensured that both the default NIC and secondary NICs adhere to the same bandwidth specifications derived from the network offering or global throttling settings. Error Handling Improvement: Added checks to validate and handle unexpected or invalid traffic label fields gracefully, reducing potential misconfigurations. Code Cleanup: Simplified logic and ensured compliance with the project's coding standards for better readability and maintainability. These changes aim to align the bandwidth configuration across all NICs of a VM, as per the expected behavior described in the issue report. I have tested the changes locally to verify their correctness.
Please let me know if there’s anything else I need to address or refine. I look forward to your feedback and further guidance.
Thank you again for your time and support!
Best regards, Ishita Jaiswal

hi @iishitahere (Ishita),

this works sounds awesome, but we deal (communicate largely through code and/or illustrated by code. Can I ask you to submit a PR and discuss the changes there?

You do not have to apologise for anything. We do not expect things and everything is a bonus. We are very glad that you are willing to work on the codebase. regards,

Hi @DaanHoogland,

Thank you for your kind words and encouragement! I truly appreciate your approach and the welcoming environment you've created.

I’ve already submitted the PR for the changes. Please feel free to review it, and let me know if there’s anything I can improve or clarify. I’m happy to make adjustments based on your feedback.

Looking forward to collaborating further!

Best regards,
Ishita

[luganofer]

I add a comment, I observed the same behaviour with secondary NICs connected to an isolated network.
Example: A VM is connected to 2 networks; NIC1 takes the traffic shapping value of the ‘vm.network.throttling.rate’ parameter while NIC2 takes the value of ‘the network offer’. In this example the value of vm.network.throttling.rate is 10,000 (10 Gbps) while the bandwidth value of ‘the network offer’ used by NIC 2 is 1,000 (1 Gbps). In NIC 1, VMware uses the value 10,000 for traffic shaping, while in NIC2 it takes the value of ‘the network offer’ which is 1,000. This does not seem to be consistent.

[DaanHoogland]

@luganofer (cc @BryanMLima @JoaoJandre ) I moved to discussions , maybe it should be under “ideas” . It seems no work is going on and we need to reconsider/redefine how to implement.