Separate environments and controlling VPN access to them for remote users

VPN
1

We’re planning for isolated development, testing and production environments in an all-AWS solution. We could have 30-40 people connecting remotely to those environments. Our ideal outcomes are:

  1. All users connect to a single VPN to access the AWS resources.
  2. Each user might be allowed to see one/all/combination of those environments.
  3. Only traffic to AWS resources should go to the VPN - their regular internet traffic shouldn’t.
  4. We like OpenVPN for wide support among OS’s.

The last time I worked in a similar environment, VPCs were relatively new (our admin managed access with security groups)… so – what are the current recommended solutions for separating Dev/Test/Prod environnments (within a single AWS account), and allowing multiple remote users to securely connect to them? We obviously want good controls over which users see which environments - preferably a web UI.

​

Being a greenfield situation, we’re open to just about any approach. Things we’ve considered:

  • A single VPC segregated by Security Group (limited to 100 security groups - maybe an issue?).
  • A single VPC segregated by subnet (currently researching).
  • Multiple VPCs, each has its own VPN server and clients have a VPN profile for each (burdensome on client devices).
  • Multiple VPCs, each has its own VPN server with “mesh tunnels” between them (seems unwieldy, even with four VPCs).

We’re not opposed to a clean commercial solution, versus setting it all up on our own with OpenVPN and the various AWS tools. We’d hoped the AWS “Client CPN” solution could work, but it only allows full tunnel traffic.

​

(that’s the question but here’s more on what we’re trying now…)

​

To control access at the network level we’ve been trying a separate VPC for each environment, and a fourth VPC with an OpenVPN server. There are peering relationships between the VPCs.

​

In testing, we deployed an EC2 instance into the Development VPC. A local VPN client connects to the VPN and can see/ping the EC2 box in the Development VPC. But, that EC2 instance can’t see/ping the local client at its VPN client address - it seems to not get back through the VPN. Seems related to this URL, so maybe this isn’t supported:

https://docs.aws.amazon.com/vpc/latest/peering/invalid-peering-configurations.html#transitive-peering

​

​

2

The way we did this years ago was using Cisco ASAs with Anyconnect VPN profiles - based on AD group membership, we dropped users into different profiles on the ASA, and that had appropriate ACLs that allowed allowed access to resources at a profile level.

For our use case, it was more to break out vendor access into our environment. Vendor A supported system X, so he would get AD group “VendorA-VPN”, that would match a profile on the ASA, and that would have an ACL that allows traffic to system X IPs only. Same for VendorB, VendorC and so on. You could apply these same principles to Devs, Admins, Super Devs, whatever you wanted to define.

From there, all the peering can be done as normal for the VPCs and nothing special is required, since, in this scenario, the ASA will be dictating who can go where.

One caveat is “jumping” servers. If a Dev can access Server A, you need to be careful what server A can access. If that can access Server B, C and D, but the the Dev shouldn’t have access to those, then you need to think carefully about your security group access among servers themselves. This can/should be layered with authentication on the individual servers themselves as well, so if a Dev shouldn’t get access to server B, he simply wouldn’t have an account there or permissions to login to the device.

Cisco ASAs are one way to slice this cat - Fortigate, Palo Alto, etc. etc. I’m not sure if OpenVPN can something similar or not, haven’t used it enough. Also, the term you’re after with the split internet / network usage is “split tunneling”, where only the necessary traffic goes down the VPN tunnel, and all other traffic (e.g. internet usage) goes out the normal users network interface.

Edit: Aviatrix SSL VPN seems to leverage OpenVPN (can use those clients I believe), but it adds all a layer on top that does all of the extra management / functionality that I mentioned above, probably at a much cheaper cost than the big boys.

3

To control access at the network level we’ve been trying a separate VPC for each environment, and a fourth VPC with an OpenVPN server. There are peering relationships between the VPCs.

You have it figured out. You don’t want a hard exterior and soft interior, so you’ll want auth on every thing inside the VPCs too (ssh allow groups, http auth, etc). I think trying to do RBAC at the VPN level would be cool, but not really necessary. Don’t think there’s an easy way to do that with OpenVPN… all of the plugins I’ve seen only allow you to control auth.

4

We have multiple VPCs (on multiple accounts) and a single VPN where clients have routes assigned with the subnets each one is allowed to access.

We’ve open-sourced our VPN solution last week, it might be useful for you. It provides a web UI with SSO for clients to manage their configuration files and centralized user management on GSuite where administrators can assign permissions.

https://www.reddit.com/r/aws/comments/aybdsu/build_a_cheaper_more_flexible_vpn_solution_on_aws/

A local VPN client connects to the VPN and can see/ping the EC2 box in the Development VPC. But, that EC2 instance can’t see/ping the local client at its VPN client address - it seems to not get back through the VPN.

OpenVPN works like a NAT by default: you can connect from the VPN client to other EC2 boxes (through the OVPN server’s IP address), but the other boxes cannot initiate a connection to the Client because they don’t know how to route the packets to the OpenVPN Client “internal” IP address.

You can work around this by disabling the “source/destination check” on the OpenVPN instance, configuring a route to the internal OpenVPN network on the VPC route tables, and configuring iptables on the instance not to do NAT and send/receive packets using the internal VPN network addresses.

But this adds more complications and limitations (for example you can only have a single VPN server for the route tables to work) and is usually not worth it.

5

Thanks for the reply. Aviatrix seems worth looking at as a turnkey approach would be better for us than something that requires more in-house effort to design/deploy (and manage down the road).

​

The concern we had is that it didn’t seem that that we could get from an in-VPC resource back out to a VPN client connection if it meant going through another VPC (but not sure if that transitive peering URL I included above is a direct equivalent).

6

Many of the end users just access a web app that currently sits behind the VPN. Something simple that lets them authenticate to a VPN, and then after connecting they can brows to theapp.example.com, is preferred. At some point, we plan to move the apps out fro behind the VPN, but that’s not in the near term.

7

You have it figured out.

Hah, I wish! For some reason, we can’t get it to work… assuming VPC-A, -B, and -C… and VPN-O as the home of the OpenVPN, we have all four peered. A client connected to VPC-O via VPN can ping and see a resource in VPC-A. However, that same resource in VPC-A can’t ping or see the VPN-connected client at its VPN-issued IP address.

​

It seems like we’re close, but missing some critical piece. This thread has a few things we’ll take a look at though.

8

Thanks for pointing me to that reddit thread and congratulations on releasing that project. We’ll certainly check it out.

​

Also thanks for the info on the “source/destination check” in the OpenVPN instance - I’m not sure that’s something we’d tried yet.

9

If you do end up implemented Aviatrix, I’d love to hear you feedback on it down the track. I like to have “default” solutions like this in my back pocket when I have customers interested in a VPN solution other than an EC2 instance running OpenVPN. I haven’t had the opportunity to test it out on my own, so I’d love your take on it :slight_smile:

Assuming this works similar to OpenVPN, you should be able to access peered VPCs without a problem if you architect it right - check out page 5 of their quick start guide - https://aws-quickstart.s3.amazonaws.com/quickstart-aviatrix-controller/doc/aviatrix-user-vpn-on-the-aws-cloud.pdf

The design assumes you have a central VPC in which you deploy Aviatrix, and then peer your other VPCs to that. It won’t matter that VPCs aren’t transitive, as you’ll be using the direct peering from the VPN VPC to get to the rest (which it sounds like you may already have that design).