(19)
(11)EP 3 526 939 B1

(12)EUROPEAN PATENT SPECIFICATION

(45)Mention of the grant of the patent:
18.11.2020 Bulletin 2020/47

(21)Application number: 17787706.5

(22)Date of filing:  28.09.2017
(51)Int. Cl.: 
H04L 12/851  (2013.01)
H04L 12/721  (2013.01)
H04L 12/26  (2006.01)
(86)International application number:
PCT/US2017/053977
(87)International publication number:
WO 2018/071188 (19.04.2018 Gazette  2018/16)

(54)

VIRTUAL ROUTER WITH DYNAMIC FLOW OFFLOAD CAPABILITY

VIRTUELLER ROUTER MIT DYNAMISCHER FLUSSOFFLOADKAPAZITÄT

ROUTEUR VIRTUEL À CAPACITÉ DE DÉLESTAGE DYNAMIQUE DE FLUX


(84)Designated Contracting States:
AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

(30)Priority: 14.10.2016 US 201615294494

(43)Date of publication of application:
21.08.2019 Bulletin 2019/34

(73)Proprietor: Google LLC
Mountain View, CA 94043 (US)

(72)Inventors:
  • ALPERT, Jesse Louis
    Mountain View, CA 94043 (US)
  • NAIK, Uday
    Mountain View, CA 94043 (US)
  • AREFIN, Md Ahsan
    Mountain View, CA 94043 (US)
  • SCHULTZ, David Andrew
    Mountain View, CA 94043 (US)
  • RICHTER, Yossi
    Mountain View, CA 94043 (US)

(74)Representative: Betten & Resch 
Patent- und Rechtsanwälte PartGmbB Maximiliansplatz 14
80333 München
80333 München (DE)


(56)References cited: : 
  
  • Tom Tofigh ET AL: "The Need for Complex Analytics from Forwarding Pipelines", Open Networking Summit 2016, 17 March 2016 (2016-03-17), pages 1-17, XP055438400, Santa Clara, California, USA Retrieved from the Internet: URL:https://events.static.linuxfound.org/s ites/events/files/slides/ONS_Complex_Analy tics_Final.pdf [retrieved on 2018-01-05]
  • Nick Tausanovitch: "What Makes a NIC a SmartNIC, and Why is it Needed?", , 13 September 2016 (2016-09-13), pages 1-7, XP055438423, Retrieved from the Internet: URL:https://www.netronome.com/blog/what-ma kes-a-nic-a-smartnic-and-why-is-it-needed/ [retrieved on 2018-01-05]
  • HAMADI SALAHEDDINE ET AL: "Fast path acceleration for open vSwitch in overlay networks", 2014 GLOBAL INFORMATION INFRASTRUCTURE AND NETWORKING SYMPOSIUM (GIIS), IEEE, 15 September 2014 (2014-09-15), pages 1-5, XP032668742, DOI: 10.1109/GIIS.2014.6934286 [retrieved on 2014-10-22]
  • Nick Tausanovitch: "What Makes a NIC a SmartNIC, and Why is it Needed?", , 13 September 2016 (2016-09-13), pages 1-7, XP055438423, Retrieved from the Internet: URL:https://www.netronome.com/blog/what-ma kes-a-nic-a-smartnic-and-why-is-it-needed/ [retrieved on 2018-01-05]
  
Note: Within nine months from the publication of the mention of the grant of the European patent, any person may give notice to the European Patent Office of opposition to the European patent granted. Notice of opposition shall be filed in a written reasoned statement. It shall not be deemed to have been filed until the opposition fee has been paid. (Art. 99(1) European Patent Convention).


Description

BACKGROUND



[0001] Computing devices in a software-defined network may not have physical connections, but may be virtually linked to one another. Software-defined networking allows networks to be managed through abstraction of lower level functionality. A control plane in a virtual network usually makes decisions about how to route data packets of network traffic from a source virtual machine in the network to a destination virtual machine. A data plane forwards network traffic to selected destinations. In a software-defined network, methods of network virtualization decouple the control plane of a network from the data plane. Therefore, virtual networks typically have address spaces that bear little resemblance to the topology of the underlying physical network which means that traditional techniques for making networks scale do not work for virtual networks. As a result, routing network traffic through large virtual networks can be problematic using conventional network traffic routing models.
Document "Tom Tofigh ct al: The Need for Complex Analytics from Forwarding Pipelines". Open Networking Summit 2016. 17 March 2016. describes inter alia how to achieve autonomous control through programmable data plane analytics.

SUMMARY



[0002] The present invention is defined in the independent claims. Preferred embodiments are defined in the dependent claims. This specification describes technologies relating to routing virtual network traffic in general, and specifically to systems and methods for scaling virtual networks by using virtual routers and flow offloading to route network traffic.

[0003] An example implementation uses virtual routers and dynamic offloading to route packets through networks. The example implementation is advantageous because it improves the programming latency, which is the amount of time it takes to program changes in a virtual network, and scalability of virtual networks. The efficiency, latency and/or capacity of the virtual network may thereby be improved.

[0004] The details of one or more embodiments of the invention are set forth in the accompanying drawings which are given by way of illustration only, and the description below. Other features, aspects, and advantages of the invention will become apparent from the description, the drawings, and the claims. Like reference numbers and designations in the various drawings indicate like elements.

BRIEF DESCRIPTION OF THE DRAWINGS



[0005] 

FIG. 1 illustrates one configuration of a virtual network.

FIG. 2 illustrates a configuration of an example virtual network.

FIG. 3 illustrates a configuration of an example virtual network in multiple clusters.

FIG. 4 is a flow diagram of an example method for dynamically offloading network traffic flows in a virtual network.


DETAILED DESCRIPTION



[0006] FIG. 1 illustrates one configuration of a virtual network. A virtual network is an abstraction on top of a physical network made up of multiple virtual machines running on multiple host machines. Host machines are connected via a physical network, which consists of top-of-rack switches, routers, and other network entities.

[0007] In FIG. 1, virtual machines A and D are in different host machines (Host_1 and Host_3). Virtual machines can exist in different host machines in one or more clusters. FIG. 1 illustrates direct routes between host machines that are point-to-point tunnels. As used in this description, a direct route is a route between two host machines that does not include routing through one or more virtual routers. These tunnels may use a tunnel protocol such as generic routing encapsulation (GRE) which is a tunneling protocol that can encapsulate a wide variety of network layer protocols inside virtual point-to-point links over an internet protocol network. Point-to-point tunnels between hosts machines allow virtual machines in the same network to communicate with one another. For example, direct route 101 connects Host_1 and Host_5. In this example, direct route 101 allows virtual machines A and B to communicate with virtual machine F. Generally, there is not a direct physical link between two hosts so packets between the two hosts will traverse at least one intermediate switch.

[0008] Each machine that hosts a virtual machine in the network has a programmable software switch 110, 112, 113, 114, and 115. A control plane (102) in the network can install routing flows on these switches 110, 112, 113, 114, and 115. Each routing flow defines how to route network traffic between two virtual machines in the network by providing a specific path for network packets to traverse in order for data packets to be sent from a source virtual machine to a destination virtual machine.

[0009] For example, in FIG. 1, Host_2 has an IP address 10.0.0.2 in the physical network and Host_3 has a IP address 10.0.0.3. Virtual machine C has a virtual IP address 10.240.3.3 and virtual machine D has a virtual IP address 10.240.4.4 within the same virtual network.

[0010] If virtual machine C needs to send a packet to virtual machine D, the most direct way for the switch (112) to forward the packet is for virtual machine C to encapsulate the packet using generic routing encapsulation and send the packet to Host_3's physical IP address, 10.0.0.3. The path from Host_2 to Host_3 may traverse multiple physical network links, but have a direct virtual path from Host_2 to Host_3.

[0011] However, if Host_2, hosting virtual machine C, does not know the host hosting virtual machine D, Host_2 may instead send the packet to a virtual router, over potentially multiple physical links. The virtual router would be responsible for further forwarding. This traversal through a virtual router is not a direct virtual path.

[0012] FIG. 2 illustrates a configuration of an example network. In an example virtual network model, at least one virtual router (201) is installed to enable virtual networks to route network traffic packets through the networks. Virtual routers run on dedicated hardware and act much like routers for physical networks. Virtual machines do not need to be directly connected to a virtual router, but can forward packets to a virtual router using a default routing flow provided by the control plane.

[0013] This virtual network model supports multi-tenancy which means that virtual routers and hosts are shared by multiple customers running virtual machines in different virtual networks. For example, there may be ten virtual routers in a physical cluster and that cluster may have thousands of hosts and thousands of virtual networks.

[0014] Routing decisions for routing flows are made by the on-host switch. If the switch knows the physical IP address that hosts the destination virtual machine for a particular packet, the switch sends the packet over a GRE tunnel to the destination host. Otherwise, the switch sends the packet over a GRE tunnel to a virtual router. By sending network traffic through a virtual router, each virtual machine in the network does not need a complete routing table or a direct route to every other virtual machine in the network.

[0015] On startup, each virtual router communicates with the control plane (200) of the virtual network to let the control plane know that the router is ready to serve network traffic. The control plane keeps a list of virtual routers in the virtual network and information about each router including the location and the status of the virtual router. Additionally, the control plane can send a full configuration of the virtual network to each virtual router. Each virtual router may be able to forward network traffic packets from any source virtual machine to any destination virtual machine in the network.

[0016] In the example virtual network, each virtual machine A-E has a default routing flow (202) through at least one virtual router (201). This default routing flow may be provided to each host machine hosting a virtual machine in the network by the control plane. Additionally or alternatively, each host can choose a virtual router (201) to which the host routes packets from the list of virtual routers maintained by the control plane (200) based on characteristics of the packets needing to be routed and the flow key or keys of the virtual router defining network traffic the virtual router is programmed to handle.

[0017] FIG. 3 illustrates a configuration of an example network with multiple clusters. A cluster is a collection of physical machines connected by a high-speed network, typically within a single building. Network latencies within a cluster are much better than across clusters. In an example network, a source virtual machine can send network traffic to destination virtual machines in the network that exist in different clusters from the source virtual machine. For example, source virtual machine A may have network packets destined for virtual machine D, which is in a different cluster from virtual machine A. To transmit network traffic from virtual machine A to virtual machine D, virtual machine A first sends network traffic to a virtual router (301) in the local cluster (303) of the source virtual machine. The source cluster virtual router (303) then sends the network traffic to a virtual router (302) in the destination cluster (304). The destination cluster virtual router (302) will then send the network traffic to the destination virtual machine D.

[0018] In an example network, load through virtual routers may be load-balanced by the hosts. For example, if there are three virtual routers, a host might try to split traffic evenly across them. To split the traffic, the host might hash an n-tuple flow key that defines a flow and use the hash to choose a router load balancing on virtual routers is accomplished by equal-weight, 5-tuple, stateless load-balancing. Each n-tuple flow key may be in the form of, for example, a 3-tuple flow key defined by <sourceIP address, destination IP address, IP protocol>, or a 5-tuple flow key defined by <source IP address, destination IP address, IP protocol, source transport port, destination transport port>. In order to determine load-balancing of a flow, a host may hash of a packet's flow key. If the hash is within the first one third of the hash range, the packet may be sent to the first router. If the hash falls within the second one third of the hash range, the packet may be sent to the second router and so on.

[0019] In some implementations, the control plane may program the switch on each host machine to send only certain types of network traffic to a specific virtual router. For example, packets to a specific destination network prefix may be directed to a specific router. In a multitenant system, traffic from certain networks may be directed to specific routers. In these cases, a virtual router only needs to handle certain network traffic. In some implementations, packets to a specific destination IP range may use an advanced routing feature that the virtual router does not support and therefore the packets will not be routed through the virtual router.

[0020] Load-balancing across multiple clusters may be handled by dividing the network traffic across virtual machines in each cluster. For example, a route may have five destinations: two virtual machines in cluster 1 and three virtual machines in cluster 2. If a source virtual machine in cluster 1 sends network traffic that matches the route, the virtual routers in cluster 1 must distribute 2/5 of the traffic to each of the virtual machines in cluster 1 and 3/5 virtual routers for cluster 2. Upon receiving the flows from the virtual routers in cluster 1, the virtual routers in cluster 2 divide the flows equally among the three virtual machines in cluster 2.

[0021] In some implementations, virtual machine hosts, virtual routers, or other network entities monitor network traffic flows and reports flow statistics to the control plane. These entities can collect flow usage information for each virtual machine to virtual router flow. In each reporting interval, entities monitoring network traffic flows can collect other information including: the source switch port, the destination virtual IP address, and the number of bytes and packets in the reporting interval. Each flow can be defined by source virtual machine/virtual router virtual IP pair. In a host, the programmable software switch may collect flow usage information. A network entity, such as a virtual router or virtual host, may be programmed with a configurable bandwidth threshold, i.e., 20 kbps or 100 kbps.

[0022] Additionally or alternatively, flow usage information may be derived from sampling (e.g., netflow or sflow) rather than by collecting exact packet or byte counts. In a sampling implementation, a host or a router reports a certain fraction of packets, e.g., 1 in 10,000, and the entity collecting the sample can estimate flow bandwidth for each flow based on the number of samples received to determine a configurable bandwidth threshold.

[0023] As described, the configurable bandwidth threshold may be programmed by a network administrator or learned in the virtual network. In one implementation, the configurable threshold is statically configured. In another implementation, the control plane can adaptively set the threshold based on a number of factors such as the amount of bandwidth going through the virtual routers and the current number of offload flows.

[0024] Each network entity, such as a virtual machine host or virtual router, can monitor flow usage information to determine network traffic flows that exceed this configurable bandwidth threshold. If a network traffic flow does exceed the configurable bandwidth threshold, the network traffic flow is considered a high-bandwidth flow and the entity that made the determination reports the flow to the control plane.

[0025] The control plane receives flow statistics from network entities and can offload flows from virtual routers that meet offload criteria. When a network flow meets offload criteria (defining an offload rule), the control plane programs direct routes between the sending host and the destination host of the network flow.

[0026] As described above, FIG. 3 illustrates a configuration of an example network with multiple clusters. This figure shows an example virtual network with flows through a virtual router (301 and 302) and flows through a direct route (310). As described above, if virtual machine A has network flow traffic destined for virtual machine D, virtual machine A will send the network flow traffic to virtual router 301. Virtual router 301 will send the network flow traffic to virtual router 302, which will in turn send the traffic to virtual machine D. If virtual machine A has network flow traffic destined for virtual machine C, there are 2 routes that could be taken, a first route from virtual machine A to virtual router 301 to virtual router 302 and then to virtual machine C and a second route directly from Host_1 hosting virtual machine A to Host_3 hosting virtual machine C. The control plane contains the routing table for the network, which defines the rules for routing a packet. The rules for sending network traffic from a virtual machine to a virtual router have a lower priority than virtual machine to virtual machine forwarding rules, which is the on-demand learned rules for direct flow. Therefore, network flow traffic from virtual machine A destined to virtual machine C will use the offloaded virtual machine-to-virtual machine direct route.

[0027] Flows that meet certain offload criteria will be offloaded from virtual routers. Most often, these flows will be high-bandwidth flows as determined by flows that meet or exceed the configurable bandwidth threshold. However, the control plane can dynamically make the decision to offload a network flow based on other flow statistics (as discussed below).

[0028] FIG. 4 is a flow diagram of an example method for dynamically offloading network traffic flows in a virtual network. A source virtual machine may have data to send to a destination virtual machine. In an example virtual network, a control plane identifies a virtual router through which to route network traffic of a specific network flow from the source virtual machine to the destination virtual machine (410). The host machine then transmits the specific network flow traffic from the source virtual machine to the identified virtual router, which forwards the specific network flow traffic to the destination virtual machine. The host machine monitors flow statistics of the specific network flow from the source virtual machine to the identified virtual router to determine whether a flow exceeds the configurable bandwidth threshold. If the flow exceeds the configurable bandwidth threshold, the host reports the flow to the control plane. The control plane receives flow statistics about the virtual network including statistics about the specific network flow (425).

[0029] In order to determine whether a flow should be offloaded, the control plane determines whether the network flow meets an offload rule (430). The control plane may only consider offloading network flows that have already been determined to be high-bandwidth flows by exceeding a configurable bandwidth threshold as described above. The control plane also considers several flow statistics in an offload rule determination including: the bandwidth of each flow, the aggregate bandwidth going through each virtual router, and the number of flows already offloaded. Information about flow bandwidth can come from a variety of sources including virtual machines, hosts, from a router on the physical host where the destination virtual machine is located, from a switch, from network monitoring systems, or from the virtual routers themselves.

[0030] A configurable offload rule may be set by a network administrator or dynamically learned by the control plane based on the flow statistics. A control plane will offload flows that meet the criteria of a given offload rule. For example, an offload rule may be to offload any flow that exceed 20 kbps. Another offload rule may be to offload a flow that exceed 20 kbps unless more than 20 flows have already been offloaded for the source host of the flow. An additional offload rule may be to offload a high-bandwidth flow when the aggregate bandwidth going through the virtual router that the high-bandwidth flow is using exceeds a configurable bandwidth threshold for the virtual router.

[0031] An offload rule may be determined by the control plane dynamically based on factors such as: the aggregate bandwidth currently traveling through a virtual router, the number of flows already offloaded to the given source host, the total number of offloaded flows in the control plane, and other flow statistics.

[0032] In some implementations, a control plane can preprogram direct routes. For example, the control plane may program direct routes for all virtual machines in the same subnetwork if the subnetwork is sufficiently small. In another example, the control plane may program direct routes for all virtual machines in the same network if the network is sufficiently small.

[0033] As described above, the control plane offloads flows that meet offload rules or are preprogrammed by installing direct routes between sending hosts and destination hosts. Once a network flow is offloaded, subsequent traffic of the network flow routes directly from host of the source virtual machine to the host of the destination virtual machine without going through any virtual routers.

[0034] The control plane can expire and remove offloaded flows for various reasons including determining that the offloaded flow is no longer needed. An example system removes offloaded flows when the source or destination virtual machine of the flow has been deleted. In some networks, an offload flow can become invalid when the network configuration changes. Flow usage statistics can also be used to determine whether to expire or remove an offloaded flow. As described above, flow usage statistics can be collected or sampled in order to determine the usage of each flow. An offloaded flow can be expired/removed if it has been idle a predetermined amount of time or if it has been used below a threshold amount for a predetermined amount of time.

[0035] Host machines need to quickly detect and react to virtual router failures in order to continue routing network traffic. To determine virtual router failures, the programmable software switch on each host can establish a connection to each virtual router in its local cluster and use inline health checks to detect failures. Cross-cluster flows require sending network traffic to remote-cluster virtual routers. Therefore, each virtual router needs to health-check all remote-cluster virtual routers. To check on the health of a virtual router, each host sends a message to the router periodically. The virtual router replies with an affirmative or negative response or no response if the virtual router has failed completely. The control plane provides a complete list of available virtual routers in the cluster. Hosts load-balance packets across routers that they determine to be healthy according to the results of the health checks.

[0036] Although aspects of the invention have been described in terms of source virtual machines and destination virtual machines, the invention can be used with any virtual network or overlay network, regardless of whether the endpoints of a flow in the network are virtual machines. For example, the endpoints may be processes, not running in virtual machines, that communicate with each other by means of virtual internet protocol addresses.

[0037] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0038] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0039] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network. Such a computer program configured to perform any method as herein described when operated by a processor is further provided.

[0040] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on instance data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0041] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0042] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0043] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone, running a messaging application, and receiving responsive messages from the user in return.

[0044] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0045] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0046] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions.

[0047] Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0048] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0049] Particular embodiments of the subject matter have been described. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.


Claims

1. A computer-implemented method for dynamically offloading network traffic flows in one or more virtual networks, the computer-implemented method comprising:

identifying (410) a virtual router (201) through which to route network traffic of a specific network flow, wherein the specific network flow has a source endpoint existing on a source host machine and a destination endpoint existing on a destination host machine;

receiving (425) flow statistics of network traffic transiting the virtual router (201) including statistics about the specific network flow;

determining (430) whether current network traffic of the specific network flow meets an offload rule using the received flow statistics by:

determining the number of network flows that have already been offloaded for the source host machine of the specific network flow; and

determining whether the number of offloaded network flows exceeds a threshold amount for the source host machine; and

in response to determining that the number of offloaded network flows does not exceed a threshold amount for the source host machine, dynamically offloading (435) the specific network flow from the virtual router (201), wherein subsequent traffic of the network flow routes directly from the source host machine to the destination host machine without going through any virtual routers.


 
2. The computer-implemented method of claim 1, wherein offloading (435) the specific network flow from the virtual router (201) further comprises:

creating a direct route between the source host machine and the destination host machine through which to route the specific network flow; and

routing subsequent specific network flow traffic through the direct route instead of the virtual router (201).


 
3. The computer-implemented method of claim 1 or claim 2, wherein the virtual router (201) is identified based on characteristics of the specific network flow and at least one flow key of the virtual router (201) defining network traffic that the virtual router (201) is programmed to handle.
 
4. The computer-implemented method of any preceding claim, wherein determining (430) whether the specific network flow meets an offload rule includes:

comparing the specific network flow bandwidth with a predefined bandwidth; and

determining that the specific network flow exceeds a bandwidth network flow when the specific network flow bandwidth is greater than the predefined bandwidth.


 
5. The computer-implemented method of any preceding claim, wherein receiving (425) flow statistics includes receiving statistics about the bandwidth of the specific network flow.
 
6. The computer-implemented method of any preceding claim, wherein determining (430) whether the specific network flow meets an offload rule includes:

determining an aggregate bandwidth flowing through the virtual router (201); and

determining whether the aggregate bandwidth exceeds a threshold bandwidth for the virtual router (201).


 
7. The computer-implemented method of any preceding claim, wherein flow statistics are received from at least one source, wherein the at least one source is: the source or destination virtual machines, the source or destination hosts, a router on the destination host, a programmable software switch on the source or destination host, a network monitoring system, or the virtual router (201).
 
8. The computer-implemented method of any preceding claim, wherein the source host machine is in a source cluster and the destination host machine is in a destination cluster that is different from the source cluster.
 
9. The computer-implemented method of claim 8, wherein identifying a virtual router (201) through which to route network traffic of a specific network flow includes identifying a virtual router (201) in the source cluster and a second virtual router (201) in the destination cluster.
 
10. The computer-implemented method of any preceding claim, wherein the specific network flow has multiple destination virtual machines existing on multiple destination host machines in multiple clusters.
 
11. The computer-implemented method of any preceding claim, further comprising:
expiring or removing the specific flow in response to determining that the specific flow has been idle for a predetermined amount of time.
 
12. The computer-implemented method of any preceding claim, further comprising:
expiring or removing the specific flow in response to determining that the specific flow has been used below a threshold amount for a predetermined amount of time.
 
13. A system comprising:

one or more computers; and

one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of method claims 1 to 12.


 
14. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform the operations of method claims 1 to 12.
 


Ansprüche

1. Ein computer-implementiertes Verfahren zum dynamischen Auslagern von Netzwerk-Verkehrsflüssen in einem oder mehreren virtuellen Netzwerken, wobei das computer-implementierte Verfahren umfasst:

Identifizieren (410) eines virtuellen Routers (201), durch den Netzwerkverkehr eines spezifischen Netzwerkflusses geleitet werden soll, wobei der spezifische Netzwerkfluss einen auf einem Quell-Host-Rechner existierenden Quell-Endpunkt und einen auf einem Ziel-Host-Rechner existierenden Ziel-Endpunkt aufweist;

Empfangen (425) von Flussstatistiken des Netzwerkverkehrs, der den virtuellen Router (201) durchläuft, einschließlich Statistiken über den spezifischen Netzwerkfluss;

Bestimmen (430), ob der aktuelle Netzwerkverkehr des spezifischen Netzwerkflusses einer Auslagerungsregel entspricht, unter Verwendung der empfangenen Flussstatistiken durch:

Bestimmen der Anzahl von Netzwerkflüssen, die bereits für den Quell-Host-Rechner des spezifischen Netzwerkflusses ausgeladen wurden; und

Bestimmen, ob die Anzahl der ausgelagerten Netzwerkflüsse einen Schwellenwert für den Quell-Host-Rechner überschreitet; und

als Reaktion auf die Bestimmung, dass die Anzahl der ausgelagerten Netzwerkflüsse einen Schwellenwert für den Quell-Host-Rechner nicht überschreitet, dynamisches Auslagern (435) des spezifischen Netzwerkflusses von dem virtuellen Router (201), wobei der nachfolgende Verkehr des Netzwerkflusses direkt von dem Quell-Host-Rechner zu dem Ziel-Host-Rechner gerouted wird, ohne über irgendwelche virtuellen Router zu gehen.


 
2. Das computer-implementierte Verfahren nach Anspruch 1, wobei das Auslagern (435) des spezifischen Netzwerkflusses von dem virtuellen Router (201) weiterhin umfasst:

Erzeugen einer direkten Route zwischen dem Quell-Host-Rechner und dem Ziel-Host-Rechner, durch die der spezifische Netzwerkfluss geleitet werden soll; und

Routing des nachfolgenden spezifischen Netzwerk-Flussverkehrs über die direkte Route anstelle des virtuellen Routers (201).


 
3. Das computer-implementierte Verfahren nach Anspruch 1 oder Anspruch 2, wobei der virtuelle Router (201) auf der Grundlage von Merkmalen des spezifischen Netzwerkflusses und mindestens einem Flussschlüssel des virtuellen Routers (201) identifiziert wird, der den Netzwerkverkehr definiert, für dessen Handhabung der virtuelle Router (201) programmiert ist.
 
4. Das computer-implementierte Verfahren nach einem der vorhergehenden Ansprüche, wobei die Bestimmung (430), ob der spezifische Netzwerkfluss eine Auslagerungsregel erfüllt, umfasst:

Vergleichen der Bandbreite des spezifischen Netzwerkflusses mit einer vordefinierten Bandbreite; und

Bestimmen, dass der spezifische Netzwerkfluss eine Netzwerkbandbreite überschreitet, wenn die Bandbreite des spezifischen Netzwerkflusses größer als die vordefinierte Bandbreite ist.


 
5. Das computer-implementierte Verfahren nach einem der vorhergehenden Ansprüche, wobei das Empfangen (425) von Flussstatistiken das Empfangen von Statistiken über die Bandbreite des spezifischen Netzwerkflusses einschließt.
 
6. Das computer-implementierte Verfahren nach einem der vorhergehenden Ansprüche, wobei das Bestimmen (430), ob der spezifische Netzwerkfluss eine Auslagerungsregel erfüllt, umfasst:

das Bestimmen einer aggregierten Bandbreite, die durch den virtuellen Router fließt (201); und

Bestimmen, ob die Gesamtbandbreite eine Schwellenbandbreite für den virtuellen Router überschreitet (201).


 
7. Das computer-implementierte Verfahren nach einem der vorhergehenden Ansprüche, wobei Flussstatistiken von mindestens einer Quelle empfangen werden, wobei die mindestens eine Quelle eines des Folgenden ist: die virtuellen Quell- oder Ziel-Maschinen, die Quell- oder Ziel-Hosts, ein Router auf dem Ziel-Host, ein programmierbarer Software-Switch auf dem Quell- oder Ziel-Host, ein Netzwerküberwachungssystem oder der virtuelle Router (201).
 
8. Das computer-implementierte Verfahren nach einem der vorhergehenden Ansprüche, wobei sich der Quell-Host-Rechner in einem Quell-Cluster und der Ziel-Host-Rechner in einem Ziel-Cluster befindet, der sich von dem Quell-Cluster unterscheidet.
 
9. Das computer-implementierte Verfahren nach Anspruch 8, wobei das Identifizieren eines virtuellen Routers (201), durch den Netzwerkverkehr eines spezifischen Netzwerkflusses zu leiten ist, das Identifizieren eines virtuellen Routers (201) in dem Quell-Cluster und eines zweiten virtuellen Routers (201) in dem Ziel-Cluster umfasst.
 
10. Das computer-implementierte Verfahren nach einem der vorhergehenden Ansprüche, wobei der spezifische Netzwerkfluss mehrere virtuelle Zielmaschinen aufweist, die auf mehreren Ziel-Host-Maschinen in mehreren Clustern vorhanden sind.
 
11. Das computer-implementierte Verfahren nach einem der vorhergehenden Ansprüche, ferner umfassend:
das Erlöschen oder Entfernen des spezifischen Flusses als Reaktion auf die Feststellung, dass der spezifische Fluss für eine vorbestimmte Zeitdauer untätig gewesen ist.
 
12. Das computer-implementierte Verfahren nach einem der vorhergehenden Ansprüche, das ferner Folgendes umfasst
Erlöschen oder Entfernen des spezifischen Flusses als Reaktion auf die Feststellung, dass der spezifische Fluss für eine vorbestimmte Zeitdauer unterhalb eines Schwellenbetrags verwendet worden ist.
 
13. Ein System, bestehend aus:

einem oder mehreren Computern; und

einem oder mehreren Speichergeräten zur Speicherung von Befehlen, die, wenn sie von dem einen oder den mehreren Computern ausgeführt werden, bewirken, dass der eine oder die mehreren Computer die Schritte der Verfahrensansprüche 1 bis 12 ausführen.


 
14. Nicht-flüchtiges computerlesbares Medium zum Speichern von Software mit Befehlen, die von einem oder mehreren Computern ausführbar sind und bei einer solchen Ausführung bewirken, dass der eine oder die mehreren Computer die Operationen nach den Ansprüchen 1 bis 12 ausführen.
 


Revendications

1. Un procédé mis en œuvre par ordinateur pour délester de façon dynamique des flux de trafic réseau dans un ou plusieurs réseaux virtuels, le procédé mis en œuvre par ordinateur comprenant :

le fait (410) d'identifier un routeur virtuel (201) à travers lequel acheminer du trafic réseau d'un flux réseau spécifique, le flux réseau spécifique ayant un point d'extrémité source existant sur une machine hôte source et un point d'extrémité de destination existant sur une machine hôte de destination ;

le fait (425) de recevoir des statistiques de flux de trafic réseau transitant par le routeur virtuel (201) comprenant des statistiques relatives au flux réseau spécifique ;

le fait (430) de déterminer si le trafic réseau actuel du flux réseau spécifique répond à une règle de délestage en utilisant les statistiques de flux reçues en :

déterminant le nombre de flux réseau qui ont déjà été délestés pour la machine hôte source du flux réseau spécifique ; et

déterminant si le nombre de flux réseau délestés dépasse une quantité seuil pour la machine hôte source ; et

en réponse à la détermination que le nombre de flux réseau délestés ne dépasse pas une quantité seuil pour la machine hôte source, le fait (435) de délester de façon dynamique le routeur virtuel (201) du flux réseau spécifique, le trafic ultérieur du flux réseau étant acheminé directement de la machine hôte source à la machine hôte de destination sans passer par aucun routeur virtuel.


 
2. Le procédé mis en œuvre par ordinateur selon la revendication 1, dans lequel le fait (435) de délester le routeur virtuel (201) du flux réseau spécifique comprend en outre :

le fait de créer une route directe entre la machine hôte source et la machine hôte de destination à travers laquelle acheminer le flux réseau spécifique ; et

le fait d'acheminer le trafic de flux réseau spécifique ultérieur via la route directe au lieu du routeur virtuel (201).


 
3. Le procédé mis en œuvre par ordinateur selon la revendication 1 ou la revendication 2, dans lequel le routeur virtuel (201) est identifié sur la base de caractéristiques du flux réseau spécifique et d'au moins une clé de flux du routeur virtuel (201) définissant le trafic réseau que le routeur (201) est programmé pour gérer.
 
4. Le procédé mis en œuvre par ordinateur selon l'une quelconque des revendications précédentes, dans lequel le fait (430) de déterminer si le flux réseau spécifique répond à une règle de délestage comprend :

le fait de comparer la bande passante spécifique du flux réseau avec une bande passante prédéfinie ; et

le fait de déterminer que le flux réseau spécifique dépasse une bande passante de flux réseau lorsque la bande passante du flux réseau spécifique est supérieure à la bande passante prédéfinie.


 
5. Le procédé mis en œuvre par ordinateur selon l'une quelconque des revendications précédentes, dans lequel le fait (425) de recevoir des statistiques de flux inclut le fait de recevoir des statistiques sur la bande passante du flux réseau spécifique.
 
6. Le procédé mis en œuvre par ordinateur selon l'une quelconque des revendications précédentes, dans lequel le fait (430) de déterminer si le flux réseau spécifique répond à une règle de délestage comprend :

le fait de déterminer une bande passante agrégée traversant le routeur virtuel (201) ; et

le fait de déterminer si la bande passante agrégée dépasse un seuil de bande passante pour le routeur virtuel (201).


 
7. Le procédé mis en œuvre par ordinateur selon l'une quelconque des revendications précédentes, dans lequel des statistiques de flux sont reçues en provenance d'au moins une source, ladite au moins une source étant : les machines virtuelles source ou de destination, les hôtes source ou de destination, un routeur sur l'hôte de destination, un commutateur logiciel programmable sur l'hôte source ou de destination, un système de surveillance de réseau ou le routeur virtuel (201).
 
8. Le procédé mis en œuvre par ordinateur selon l'une quelconque des revendications précédentes, dans lequel la machine hôte source est dans un cluster source et la machine hôte de destination est dans un cluster de destination qui est différent du cluster source.
 
9. Le procédé mis en œuvre par ordinateur selon la revendication 8, dans lequel l'identification d'un routeur virtuel (201) à travers lequel acheminer le trafic réseau d'un flux réseau spécifique comprend le fait d'identifier un routeur virtuel (201) dans le cluster source et un deuxième routeur virtuel. (201) dans le cluster de destination.
 
10. Le procédé mis en œuvre par ordinateur selon l'une quelconque des revendications précédentes, dans lequel le flux réseau spécifique a plusieurs machines virtuelles de destination existant sur plusieurs machines hôtes de destination dans plusieurs clusters.
 
11. Le procédé mis en œuvre par ordinateur selon l'une quelconque des revendications précédentes, comprenant en outre :
le fait d'expirer ou de supprimer le flux spécifique en réponse à la détermination que le flux spécifique est resté inactif pendant une durée prédéterminée.
 
12. Le procédé mis en œuvre par ordinateur selon l'une quelconque des revendications précédentes, comprenant en outre :
le fait d'expirer ou de supprimer le flux spécifique en réponse à la détermination du fait que le flux spécifique a été utilisé en dessous d'une quantité seuil pendant une durée prédéterminée.
 
13. Un système comprenant :

un ou plusieurs ordinateurs ; et

un ou plusieurs dispositifs de stockage stockant des instructions qui sont utilisables, lorsqu'elles sont exécutées par lesdits un ou plusieurs ordinateurs, pour amener lesdits un ou plusieurs à mettre en œuvre les opérations des revendications de procédé 1 à 12.


 
14. Un support non transitoire de lisible par ordinateur stockant un logiciel comprenant des instructions exécutables par un ou plusieurs ordinateurs qui, lors d'une telle exécution, amènent lesdits un ou plusieurs ordinateurs à mettre en œuvre les opérations des revendications de procédé 1 à 12.
 




Drawing















REFERENCES CITED IN THE DESCRIPTION



This list of references cited by the applicant is for the reader's convenience only. It does not form part of the European patent document. Even though great care has been taken in compiling the references, errors or omissions cannot be excluded and the EPO disclaims all liability in this regard.

Non-patent literature cited in the description