Dynamic Split Tunneling – a COVID-19 Best Practice

*** updated April 15, 2020 ***

I’m sure it comes as no surprise that with the current (March 2020) response to the COVID-19 pandemic, the world is faced with many challenges that we have never been faced with before.

Companies, schools, organizations; they all have pandemic & business continuity planning. Prior to COVID-19, it seems no one planned to have the entire world “working from home”. Our children have been moved to virtual learning environments, we are doing video conferences for all of our meetings, even with family members.

I’ve personally spoken with many customers who had plans to support 60 or 70 percent of their global workforce. It has been very rare to see a VPN deployment with enough capacity (and licensing) to support 100% of their workforce being remote. (It’s actually been more common to see businesses with deployed capability to support only about 25% of their workforce).

Hopefully, it is also not surprising to find out that world’s largest and leading network company, Cisco, also has the world’s largest footprint for Virtual Private Network (VPN) clients. Cisco’s AnyConnect Secure Mobility Client is loaded on ~200 million endpoints world-wide. Before COVID-19 changed our world, you can bet Cisco was very proud of that number of endpoints, but never quite expected all 200 Million or so to be connected to VPN head-end’s at the same time.

I’m posting this blog with intentions of helping you with some best practices around your Cisco AnyConnect Remote-Access VPN (aka: RA-VPN) configuration. With these best practices, I will try to include the different thought-patterns around “why” a company might choose to deploy 1 way or another, but my recommendations will still stand as MY best practice, which also matches what the AnyConnect business unit at Cisco recommends, as well.

Let’s get down to it.

Historically, Cisco has always been known as having a lot of “Nerd Knobs”. It isn’t usually a question of “can” a Cisco product do something, it is normally a question of “in which way would you like to accomplish that goal“?

AnyConnect is no exception to that rule. While the end-user experience is usually very smooth and simple, the power comes with all the complexity and configuration options that exist behind the scenes, with the AnyConnect administrator.

When most people think of a VPN, a TRUE VPN (not the malicious actors that try to get you to send all your traffic through their “anonymizer” to provide you with more “privacy”), most people think of a tunnel created between their endpoint and their company network; and with that connection, they think that all their traffic must be sent back through the company gateway. This is simply not true. 

Most true VPNs have a way of filtering which destinations should be sent through the tunnel & which ones should not be sent through the tunnel, and naturally AnyConnect not only supports this capability, it leads the industry with its capabilities.

Enough background already, what are you recommending, Aaron?

First off. Don’t tunnel all of your encrypted and compressed real-time traffic, such as WebEx, through your on-Premise VPN head-end. What benefit would it provide to force that traffic to hair-pin through your company’s firewall? You won’t be cracking open the TLS and inspecting that traffic. Of course not, that traffic is all video and audio conferencing traffic & is VERY latency sensitive.

But Aaron, Cisco AnyConnect uses TLS and DTLS, so that such latency sensitive traffic will perform well. That is a GREAT point. In times past (i.e.: before a global pandemic) there was not much of a downside to sending this traffic through the head-end. I work for Cisco & our Infosec team had a rigid policy to require all traffic to come through the VPN (this is known as “full tunnel”). However, I’ve been deploying VPNs for well over 20 years and I find it to be about an even mix of customers who want split-tunnel vs. full-tunnel.

Surely the rigidity of a full-tunnel model made more sense when HSTS and certificate pinning were not so common, and we could do more “SSL decryption”. But in the modern world with security principals like Zero Trust in our bag, and so much of our apps and data moving to the cloud with SaaS models, the full-tunnel makes much less sense.

But my company loses visibility if we split-tunnel, doesn’t it?

If your organization is tightly regulated / controlled and you need visibility in all the traffic-flows from your manged endpoints, then you should really be looking into the use of AnyConnect’s Network Visibility Module (NVM) with Stealthwatch to provide the analytics & tracking of the endpoints on & offPremise. Or, alternatively NVM is also part of the Cisco Endpoint Security Analytics (CESA) solution with Splunk.

Next, don’t exclude the WebEx traffic by trying to maintain some sort of rigid access-list where you try to keep up with all the IP addresses (IPv4 & IPv6) that WebEx might be using. Leverage the AnyConnect feature known as Dynamic Split Tunneling (DST).

Traditional Split Tunneling relies on Access Control Lists (ACLs) to choose which traffic to include or exclude. It is always up to you to determine which model works best for your needs. Is it best to include only the traffic destined for your company’s internal network, or is it best to exclude specific specific networks so they don’t pass through the tunnel

DST was originally released with AnyConnect 4.5 and enhanced In AnyConnect 4.6. Note: ASA v9.0 or newer is required.

Let’s configure this, shall we?

I’m going to do all my configuration from ASDM, but you could also do this from Cisco Defense Orchestrator (CDO).

In order to use Dynamic Split Tunneling, you must first have some basic or standard Split Tunneling configured. Now, my actual personal preference is to use the “Split Include” method. However, based on the IM’s, Text Messages, Emails & Phone calls (yes, people still pick up a phone) that I’ve been getting, I see many of you are doing full-tunnel VPNs. So We shall start there:

Configure Dynamic Split Exclude (DSE)

You most likely already have a VPN setup, but you will want to test this with a specific group of users first (or so I assume). So we’ll create a new AnyConnect Connection Profile for the Split Exclude.

Step 1 – From ASDM, Navigate to: Configuration > Remote Access VPN > Network (Client) Access > AnyConnect Connection Profiles

Step 2 – Add a new Connection Profile and provide a name that makes sense to you.

I happen to use SAML with Duo and ISE in my lab environment. You do not have to configure all that. Use whatever Authentication method you already have in place. I will write another blog on doing SAML with AnyConnect, Duo Trusted Endpoint and ISE for posture and authorization in the near(ish) future.

Step 3 – Edit the new Group Policy (GroupPolicy5 in my example), by first clicking Manage on the Connection Profile Screen, the Edit on the Group Policy Pop Up screen.

Step 4 – Since this is a test group, I will add a banner to easily identify which Group Policy I get when I connect.

Keep in mind, you might be using ISE to assign group policy through the authorization process, which will override this setting in the connection profile. Take note in the below figure, there are a lot of settings in the Group Policy and I am inheriting most settings from the default group policy. 

Step 5 – Go to the Advanced > Split Tunneling page

Step 6 – Leave DNS Names as Inherit

Step 7 – Select YES for “Send All DNS Lookups Through Tunnel“

Step 8 – Uncheck Inherit for Policy and IPv6 Policy, and choose Exclude Network List Below

Step 9 – Uncheck Inherit for Network List

Step 10 – Cick Manage to launch the ACL Manager

Step 11 – Create an Access-List. You must have an ACL for basic split-tunneling, in order to use DST. So simply pick an external address to explicitly exclude (like the Cisco Umbrella DNS resolvers 208.67.222.222, 208.67.220.220 & 2620:119:35::35, 2620:119:53::53). 

***update 03/31/2020 03:30pm EDT After publishing the blog post, the developer who actually wrote the feature corrected me. You do not need an ACL for Dynamic Split Exclude (DSE), only for Dynamic Split Include (DSI). Therefore you can skip Steps 11 and 12, unless you want to do them.

Step 12 – Click OK and select the new SplitExclude ACL from the drop down list.

Now for the Dynamic Split Exclude (DSE) part of this

Step 13 – Click on Advanced > AnyConnect Client > Custom Attributes

Step 14 – Add a new Custom Attribute

Step 15 – Click Manage to Create a new Attribute Type

Step 16 – Add a new Custom Attribute type, it must be named “dynamic-split-exclude-domains“

Step 17 – Click OK, select the new entry & click OK

Step 18 – Click Select Value, and then Manage to create a new Custom Attribute Value

Step 19 – Add a new Custom Attribute Name, Select dynamic-split-exclude-domains from the Type drop down

Step 20 – Name the new Value ExcludeWebEx (can be any name you want)

Step 21 – Add webex.com and ciscospark.com to the value. Use a comma to separate the two domains.

*** Update 04/13/2020:
List of domains to exclude for webex are: webex.com, ciscospark.com, wbx2.com
List of domains to exclude for Office365 are: office.com, office365.com, outlook.com, sharepoint.com, live.com (source: https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges)

Step 22 – Keep clicking OK to get back to your Edit Internal Group Policy screen, where your new values should be displayed.

Step 21 – Keep clicking OK to get back to your AnyConnect Connection Profiles Screen, and Click Apply and Save.

Configuration Complete!  Let’s test this thing…

Step 1 – From a host with AnyConnect, enter the VPN headend FQDN

Step 2 – Login

Since I have multiple AnyConnect Connection Profiles in my lab environment, I am prompted to select the connection profile from a drop-down. When I select SplitExclude from the list, AnyConnect built-in user agent (browser) pops up for my SAML login with Duo. You may not have SAML configured, and therefore this screen coudl look different to you.

Step 3 – Accept the banner. We configured the banner to show users who get assigned GroupPolicy5.

Step 4 – Click on the gear icon at the bottom left of the AnyConnect client, then the Statistics tab. We can see the Dynamic Tunnel Exclusion, where all other traffic flows through the tunnel.

**** Updated 4/15/2020 – Figures below show the new Dynamic Tunnel Exclusion with the new domains and show that the route details continue to build with each communication that occurs to those excluded domains.

Success! Configuration complete.

I hope this was helpful! Keep in mind, this is only ONE method showing Dynamic Split Tunneling, excluding a few domains for Cisco Webex and Microsoft office 365 domains.

If you want more details on the different Split Tunneling options, take a look at Paul Carco’s post here: https://community.cisco.com/t5/security-documents/anyconnect-split-tunneling-local-lan-access-split-tunneling/ta-p/4050866

Visibility When Users are Remote

Earlier in the blog post, I talked about how to maintain visibility when the users are all remote now. Visibility in enterprise networks is often performed by analyzing NetFlow with tools like Cisco’s Stealthwatch (Lancope acquisition) performing the analytics. Stealthwatch has an “endpoint license” that is used to collect flows from AnyConnect’s Network Visibility Module (NVM) and stitch that flow data into the network flows for much greater visibility.

That same flow data (referred to as NVZFlow) can be leveraged by Splunk and Tetration. In fact, Cisco has special packaging available for AnyConnect NVM + Splunk licenses at huge discount off normal Splunk fees. That solution is called Cisco Endpoint Security Analytics (CESA) with Splunk, and I included a link at the bottom of this blog for more information.

You can send the flows to multiple collectors by leveraging some open source tools, or by using the Stealthwatch UDP Director (as I do). In fact, I use the UDP Director to send my NVZFlow data to three different telemetry collecting and analytics tools: Stealthwatch, Splunk and Tetration. My telemetry design looks something like this:

The cool thing about using CESA with Splunk is that it will record and analyze the endpoint telemetry for both On and OffPrem use-cases. I.e.: Remote Workers during the COVID-19 pandemic. Here is a screen shot of the dashboard, where CESA is now tracking the “trusted” versus “untrusted” traffic. Trusted = from within the tunnel. Untrusted = outside the tunnel.