https://storyboard.openstack.org/#!/story/1657091
Currently, the default driver of Octavia (haproxy) only supports TCP, HTTP, HTTPS, and TERMINATED_HTTPS. We need support for load balancing UDP.
For some use-cases, UDP load balancing support is useful. One such case are real-time media streaming applications which are based on RTSP [1].
For the Internet of Things (IoT) [2], there are many services or applications that use UDP as their transmission protocol. For example: CoAP [3] (Constrained Application Protocol), DDS [4] (Data Distribution Service) for Real-Time systems, and the introduction protocol Thread [5].
Applications with high demand for real-time (like video chatting) run on RDUP [6] (Reliable User Datagram Protocol), RTP [7] (RealTime Protocol) and UDT [8] (UDP-based Data Transfer Protocol). These protocols also are based on UDP.
There isn’t any option in the API for these protocols, which Layer 4 UDP would provide. This means that customers lack a way to support these services which may be running on VM instances in an OpenStack environment.
This spec extends the LBaaSv2 API to support UDP as a protocol in Listener and Pool resource requests.
It will require a new load balancing engine to support this feature, as the
current haproxy engine only supports TCP based protocols. If users want a load
balancer which supports both TCP and UDP, this need cannot be met by launching
haproxy-based amphora instances. It’s the good time to extend octavia to
support more load balancing scenarios. This spec will introduce how
LVS [9] can work with haproxy for UDP loadbalancing. The reason for
choosing LVS is that we can easily integrate it with the existing
keepalived service. That means we can configure LVS via keepalived, and
check member health as well.
For the current service VM driver implementation, haproxy runs in the
amphora-haproxy namespace in an amphora instance. So we also need to configure
keeplived in the same namespace for UDP cases even in SINGLE topology.
For ACTIVE_STANDBY, keepalived will serve two purposes: UDP and VRRP.
So, one instance of keepalived must be bound in the namespace, along with
the LVS instance it configures.
The main idea is to use keepalived to configure and manage LVS [10]
and its configuration. We also need to check the members’ statuses with
keepalived instead of haproxy, so there must be a different workflow
in Octavia resources and deployment topologies. The simplest implementation is
LVS within NAT mode, so we will only support this mode to start. If possible
we will add other modes in the future.
Currently, a single keepalived instance can support multiple virtual server
configurations, but for minimal impact of reconfiguration to the existing
listeners, we’d better not to refresh all the keepalived configuration
files and restart the instances, because that would cause all listeners traffic
to be blocked if the LVS configuration maintained by keepalived is removed.
This spec proposes that each listener will have its own keepalived process,
but that process won’t contain a VRRP instance, just the configuration
of virtual server and real servers. That means if the Loadbalancer service is
running with ACTIVE-STANDBY topology, each amphora instance will run multiple
keepalived instances, the count being N+1 (where N is the UDP Listener
count, and +1 is the VRRP instance for HA). The existing keepalived
will be used, but each “UDP Listener keepalived process” will need to be
controlled by health check of the Main VRRP keepalived process. Then the VIP
could be moved to the BACKUP amphorae instance in ACTIVE/STANDBY topology if
there is any issue with these UDP keepalived processes. The health check will
simply reflect whether the keepalived processes are alive.
The workflow for this feature contains:
Add a new keepalived jinja template to support LVS configuration.
Add netcat into dib-elements for supporting all platforms.
Extend the ability of amphora agent to run keepalived with LVS
configuration in amphora instances, including the init configuration, such
as systemd, sysvinit and upstart.
Enhance the session persistence to work with UDP and enable/disable the “One-Packet-Scheduling” option.
Update the database to allow listeners to support both tcp and udp
on the same port, add udp as a valid protocol and
ONE_PACKET_SCHEDULING as a valid session_persistence_type in the
database.
Setup validation code for supported features of UDP load balancing (such as session persistence, types of health monitors, load balancing algorithms, number of L7 policies allowed, etc).
Extend the existing LBaaSv2 API in Octavia to allow udp parameters in
the Listener resource.
Extend the Loadbalancer/Listener flows to support udp loadbalancer in the particular topologies.
Introduce a new UDP driver based on LVS or other Loadbalancer engines. Then find a way to fix the gap of the current Octavia data models which have a strong relationship with HTTP which based on TCP.
Provide a new driver provider framework to change the amphorae backend from haproxy to some other load balancer engines, for example, if we introduce LVS driver, we may just support the simple L7 functions with LVS, as it’s a risk to change provider from existing haproxy-based amphora instances to LVS ones. If possible, we need to limit the API to not support fields/resources if the backend driver is LVS, such as “insert_headers” in Listener, L7Policies, L7Rules and etc, a series fields/resources that related to L7 layer. The all things are to match the real ability of backend. That means all the configuration of L7 resources will be ignored or translate to LVS configuration if the backend is LVS. For other load balancer engines which support UDP, such as f5/nginx, we may also need to do this.
Combining the 2 load balancer engines for a simple reference implementation, LVS would only support the L4 layer LB, and haproxy would provide the L7 LB functionality which is more specific and detailed. For other engines like f5/nginx, Octavia can directly pass the UDP parameters to backend. This is very good for the community environment. Then Octavia may support more powerful and complex LoadBalancing solutions.
There may not be any data model changes, this spec just allows a user to
input the udp protocol to create/update the Listener and Pool
resources. So here, just extend the SUPPORTED_PROTOCOLS to add the value
PROTOCOL_UDP.
SUPPORTED_PROTOCOLS = (PROTOCOL_TCP, PROTOCOL_HTTPS, PROTOCOL_HTTP,
PROTOCOL_TERMINATED_HTTPS, PROTOCOL_PROXY,
PROTOCOL_UDP)
Also add a record into the table protocol for PROTOCOL_UDP.
As LVS only operates in Layer 4, there are some conflicts with current Octavia data models. There are some limitation below:
No L7 policies allowed.
For session persistence, this spec will intro persistence_timeout (sec)
and persistence_granularity (subnet mask) [11] in the virtual
server configuration. The function will be based on the LVS. With no session
persistence specified, LVS will be configured with a persistence_timeout
of 0. There are two valid session persistence options for UDP (if session
persistence is specified), SOURCE_IP and ONE_PACKET_SCHEDULING.
Intro a ‘UDP_CONNECT’ type for UDP in healthmonitor, for the simple,
only check the UDP port is open by nc command. And for current API of
healthmonitor, we need to make clear the meaning of LVS with the current
healthmonitor API like the mapping below
Option Mapping Healthmonitor->LVS |
Healthmonitor Description |
Keepalived LVS Description |
|---|---|---|
delay -> delay_loop |
Set the time in seconds, between sending probes to members. |
Delay timer for service polling. |
max_retires_down -> retry |
Set the number of allowed check failure before changing the operating status of the member to ERROR. |
Number of retries before fail. |
timeout -> delay_before_retry |
Set the maximum time, in seconds, that a monitor waits to connect before it times out. This value must be less than the delay value. |
delay before retry (default 1 unless otherwise specified) |
For UDP load balancing, we can support the same algorithms at first. Such as SOURCE_IP(sh), ROUND_ROBIN(rr) and LEAST_CONNECTIONS(lc).
Allow the protocol fields to accept udp.
Allow the healthmonitor.type field to accept UDP type values.
Add some fields to session_persistence that are specific to UDP though
SOURCE_IP type and a new type ONE_PACKET_SCHEDULING.
Create/Update Listener Request:
POST/PUT /v2.0/lbaas/listeners
{
"listener": {
"admin_state_up": true,
"connection_limit": 100,
"description": "listener one",
"loadbalancer_id": "a36c20d0-18e9-42ce-88fd-82a35977ee8c",
"name": "listener1",
"protocol": "UDP",
"protocol_port": "18000"
}
}
Note
It is the same as the current relationships, where one listener
will have only one default pool for UDP. A loadbalancer can
have multiple listeners for UDP loadbalancing on different ports.
Create/Update Pool Request
SOURCE_IP type case:
POST/PUT /v2.0/lbaas/pools
{
"pool": {
"admin_state_up": true,
"description": "simple pool",
"lb_algorithm": "ROUND_ROBIN",
"name": "my-pool",
"protocol": "UDP",
"session_persistence": {
"type": "SOURCE_IP",
"persistence_timeout": 60,
"persistence_granularity": "255.255.0.0",
}
"listener_id": "39de4d56-d663-46e5-85a1-5b9d5fa17829",
}
}
ONE_PACKET_SCHEDULING type case:
POST/PUT /v2.0/lbaas/pools
{
"pool": {
"admin_state_up": true,
"description": "simple pool",
"lb_algorithm": "ROUND_ROBIN",
"name": "my-pool",
"protocol": "UDP",
"session_persistence": {
"type": "ONE_PACKET_SCHEDULING"
}
"listener_id": "39de4d56-d663-46e5-85a1-5b9d5fa17829",
}
}
Note
The validation part for UDP will just allow to set the specific
fields which associated with UDP. For example, user can not set the
protocol with “udp” and insert_headers in the same request.
Create/Update Health Monitor Request:
POST/PUT /v2.0/lbaas/healthmonitors
{
"healthmonitor": {
"name": "Good health monitor"
"admin_state_up": true,
"pool_id": "c5e9e801-0473-463b-a017-90c8e5237bb3",
"delay": 10,
"max_retries": 4,
"max_retries_down": 4,
"timeout": 5,
"type": "UDP_CONNECT"
}
}
Note
We don’t allow to create a healthmonitor with any other L7
parameters, like “http_method”, “url_path” and “expected_code” if
the associated pool support UDP. But for the positional option
“max_retries”, it’s different from API description in keepalived/LVS,
so the default value is the same as the value of “max_retires_down”
if user specified. In general, “max_retires_down” should be
overridden by “max_retries”.
The security should be affected by the UDP server, we need to add another neutron security group rule to the existing security group to support UDP. Security impact is minimal as the keepalived/LVS will be running in the tenant traffic network namespace.
No expected change.
Users will be able to pass “UDP” to create/update Listener/Pool resources for UDP load balancer.
If enabled driver is LVS, it will have a good performance for L4 load balancing, but lack the any functionality in L7.
As this spec introduces LVS and Haproxy working together, if users update the
Listener or Pool resources in a LoadBalancer instance frequently,
the loadbalancer functionality may be delayed for a while as the refresh of
UDP related LVS configuration.
As we need to add keepalived monitoring process for each UDP listeners, it is necessary to consider RAM about amphora VM instances.
No expected change.
No expected change.
zhaobo
Add/extend startup script templates for keepalived processes, including configuration.
Extend the ability of existing amphora agent and driver to generate and
control LVS by keepalived in amphora instances.
Extend the exist Octavia V2 API to access udp parameter in Listener
and pools resources.
Extend the Loadbalancer/Listener flows to support udp loadbalancer in the particular topologies.
Extend Octavia V2 API to accept UDP fields.
Add the specified logic which involved into haproxy agent and the affected resource workflow in Octavia.
Add API validation code to validate the fields of UDP cases.
Add Unit Tests to Octavia.
Add API functional tests.
Add scenario tests into octavia tempest plugin.
Update CLI and Octavia-dashboard to support UDP fields input.
Documentation work.
None
Unit tests, Functional tests, API tests and Scenario tests are necessary.
The description of Octavia API reference will need to be updated.
The load balancing cookbook should be also updated.
Make it clear the difference of healthmonitor behaviors in UDP cases.
Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.