Add more configuration flexibility to Egress for user

[jainpulkit22]

Added an immutable FailurePolicy field to the Egress spec. If this field is set to "NodeSNAT" then if the egress node
is not available due to any reason then the packet transfer or we can say the traffic will go via normal Node SNAT, while in the second case where this field is set to "Drop", and if egress node is not available packet won't go out and traffic will be stuck because of unavailability of egress node.

Fixes #6988.

[antrea-bot]

Can one of the admins verify this patch?

[jainpulkit22]

Points to note:

1. In case of dynamic egress, if there is no node available then egress IP is also not assigned to the egress, and so no flows are installed and traffic goes through normal Node SNAT, so its not possible to add configurable parameter for dynamic egress because in case of no node available we don't have any egress, and we don't have any deciding condition on basis of which we can drop the traffic if we want to enable strict enforcement for dynamic egress.
2. In case of static egress what happens is the Egress has an IP, but node is not guaranteed so we have two choices now that when egress node is not available for static egress we can either allow the traffic to go through normal Node SNAT(just similar to the working of dynamic egress), or we can drop the traffic.

My current implementation gives user the flexibility to decide whether to allow the traffic(best effort), or not(strict enforcement of egress policy).

But In ideal case won't this be too confusing for the end user that static egress is configurable and dynamic is not, so should we keep the behaviour of both the egress consistent and make sure that when there is no node available for static egress then instead of dropping the traffic/packet we should allow the traffic to pass through the normal Node SNAT, same as in the case of dynamic egress.

what's your opinion on this @tnqn @antoninbas

[jainpulkit22]

Points to note:

1. In case of dynamic egress, if there is no node available then egress IP is also not assigned to the egress, and so no flows are installed and traffic goes through normal Node SNAT, so its not possible to add configurable parameter for dynamic egress because in case of no node available we don't have any egress, and we don't have any deciding condition on basis of which we can drop the traffic if we want to enable strict enforcement for dynamic egress.
2. In case of static egress what happens is the Egress has an IP, but node is not guaranteed so we have two choices now that when egress node is not available for static egress we can either allow the traffic to go through normal Node SNAT(just similar to the working of dynamic egress), or we can drop the traffic.

My current implementation gives user the flexibility to decide whether to allow the traffic(best effort), or not(strict enforcement of egress policy).

But In ideal case won't this be too confusing for the end user that static egress is configurable and dynamic is not, so should we keep the behaviour of both the egress consistent and make sure that when there is no node available for static egress then instead of dropping the traffic/packet we should allow the traffic to pass through the normal Node SNAT, same as in the case of dynamic egress.

what's your opinion on this @tnqn @antoninbas

[jainpulkit22]

@antoninbas @tnqn I have updated the code making both dynamic and static egress configurable, based on the set failurePolicy which is an enum field in Egress Spec and supports two values 1) "Drop" 2) "NodeSNAT", if the field is set to 1 then if the egress node is not available the packet will be dropped in both the kind of egresses, while if the field is set to 2 then if the egress node is not available the packet will be forwarded via local NodeSNAT.
The above field is currently set to be immutable, also if the egress node is available then the above field will not take any effect and the packet will go through the egress node.