Configuring containers with Proxmox on OVH Kimsufi server behind a single public IP with NAT

OVH offers some interesting deals with its series of Kimsufi server, however you are limited by contract to one single public IP address which might be a problem if you want to host multiple VMs on a Proxmox hypervisor. Fortunately we can overcome these limitations by using a Nginx reverse proxy and a few iptables rules.

There are already a couple of articles out there detailing this procedure however I wanted to write a more in-depth explanation covering more aspects of this setup in a series of articles, this is the first one.

In this series we will learn how to set-up multiple containers within a local network inside the Proxmox hypervisor.

Network infrastructure overview

  • Proxmox node – 192.168.0.1
  • Container1 – Nginx reverse proxy – 192.168.0.2
  • Container2 – MySQL – 192.168.0.3
  • Container3 – Apache2 – 192.168.0.4

Our Proxmox node will be the gateway for all containers to be able to access the public internet through NAT. We will then use some port-forwarding rules to access the containers from the public internet. Ultimately we will send all http(s) requests to the nginx reverse proxy which will forward the requests to one server or the other according to the domain name in use.

This way you will be able to have many different application servers running on he same server and serving multiple domain names. You will also be able to easily secure the connections by terminating SSL on the Nginx server.

Configuring the containers

Once your Proxmox installation is complete you will receive your credentials by email, use them to login in the GUI on port 8006 of your server.

First we will configure the dummy network interface which will be used for communication on the private network. Click on the node name, go in the network panel under system, you should have by default two Linux bridge configured, vmbr0 and vmbr1 as seen on the screenshot below. Vmbr0 is the interface with your public IP configured, configure vmbr1 by double clicking on it. Enter the same parameters shown below, you will need to restart the server for the parameters to be activated.
IP address : 192.168.0.1 / Subnet mask : 255.255.255.0

screen-shot-2017-04-06-at-3-46-06-pm

Go to the local storage of your Proxmox node, as you can see in the screenshot below, in the content section you have a template button which will open a pop-up letting you choose which distribution to download and start your containers from. I will use ubuntu-16.04-standard in this guide.

Proxmox container templates
Proxmox container templates

Once you downloaded the template you can hit ‘create CT’ in the top right corner of the screen and go through the configuration, once you reach the network part,select vmbr1 and give the next free IP in the range, use the IP of the Proxmox node, 192.168.0.1, as gateway see screenshot below.

screen-shot-2017-04-06-at-3-55-20-pm

You can repeat the last step with all the containers you will need in your own setup.

Start your containers, you should be able to ping them from the Proxmox node.

If you are using Ubuntu 16.04 you need to change the default SSH configuration to permit root login, in the file : /etc/ssh/sshd_config
Look for the line PermitRootLogin and change it to PermitRootLogin yes then restart the SSH service ssh restart

Alternatively you can use the command pct enter <vmid> to log in any of the containers from the Proxmox node.

Configuring NAT on the Proxmox node

Now that our containers are ready we need to allow them access to the public internet. A simple NAT rule on the Proxmox node will allow us to do so, all traffic coming from the internal network 192.168.0.0/24 will be NATed to the internet behind the public ip of our server.

iptables -t nat -A POSTROUTING -s '192.168.0.0/24' -o vmbr0 -j MASQUERADE

Please make sure that vmbr0 is actually your public interface, and activate routing using the following command.

echo 1 > /proc/sys/net/ipv4/ip_forward

You can check that this is working by trying to ping anything on the internet from your containers, google DNS for example.

ping 8.8.8.8

Next we want to forward traffic coming on port 80 (HTTP) and 443 (HTTPS) to our NGINX reverse proxy – 192.168.0.2 – which will proxy HTTP traffic to the containers on the private network.

iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to 192.168.0.2:80

iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 443 -j DNAT --to 192.168.0.2:443

You can then add all the rules you need to access your other containers – MySQL for example, I will go into the details of the containers setup in another post.

If you need to remove a rule you can use iptables -t nat --line-numbers -L to list all rules with a line number in front.

Then use iptables -t nat -D PREROUTING 2 to remove line 2 in PREROUTING for instance.

Configuring NGINX as a reverse proxy

Start by installing Nginx on your container.

apt-get install nginx

Most of the configuration take place in the following file/folder :
/etc/nginx/nginx.conf
/etc/nginx/sites-enabled/*

Start by editing /etc/nginx/nginx.conf and paste the following config.

# Two processes work well for a single CPU
user www-data;
worker_processes 2;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
use epoll;
}

http {
include /etc/nginx/mime.types;

# Nginx does the logging
access_log /var/log/nginx/access.log;

# TCP
sendfile on;
keepalive_timeout 65;
tcp_nodelay on;
server_names_hash_bucket_size 64;
server_tokens off;

# Have nginx do the compression, turn off Apache's mod_deflate
gzip on;
gzip_vary on;
gzip_http_version 1.1;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";
gzip_comp_level 6;
gzip_min_length 1400;
gzip_proxied any;
# text/html mime type is automatically included for gzip, have to add the rest
gzip_types text/plain text/css application/x-javascript text/xml application/xml application/rss+xml text/javascript;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:ECDHE-RSA-AES128-GCM-SHA256:AES256+EECDH:DHE-RSA-AES128-GCM-SHA256:AES256+EDH:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';
ssl_session_cache shared:SSL:10m;

proxy_cache_path /var/cache/nginx levels=2:2:2 keys_zone=STATIC:1000m inactive=24h max_size=1g;
proxy_temp_path /var/lib/nginx/proxy;

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

You can also create separate config files and include them in your main config, this will help you keep your main config files tidy. Let’s create /etc/nginx/proxy.conf and paste the following config in it.

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 900s;

We will now edit /etc/nginx/sites-available/default to add our domains configuration. Each server block is a new domain, note that we include proxy.conf in the location part.

server {
listen 80;
listen [::]:80;
server_name website1.mydomain.com;

access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

location / {
include proxy.conf;
proxy_pass http://192.168.0.4:80/;
}
}

server {
listen 80;
listen [::]:80;
server_name website2.mydomain.com;

access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

location / {
include proxy.conf;
proxy_pass http://192.168.0.4:80/;
}
}

The important parts here are server_name which is your domain name, and proxy_pass, which triggers the reverse proxy to the URL next to it, note also the location block which is in this example the root of your website but you can add multiple location blocks and assign them a different behaviour we will see that in a future article on SSL configuration.

Restart Nginx to activate your configuration.
service nginx restart

After configuring Apache, typing your domain in a browser should redirect you to your website folder on your apache server.

Apache domain configuration

If you have a look at the network infrastructure you can see that we are currently redirecting everything from Nginx to our Apache server on 192.168.0.4. This means the we will need to configure it as well in order to handle the domain names.

First install Apache2 with apt-get install apache2

Go to /etc/apache2/sites-available/ and create a new configuration file with the name of your website. In our case we will use website1.conf

Then define a basic virtualhost configuration.

<VirtualHost *:80>
ServerName website1.mydomain.com

ServerAdmin webmaster@localhost
DocumentRoot /var/www/html/website1

<Directory /var/www/html/website1>
AllowOverride All
</Directory>

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Save the file and activate the new Virtualhost by typing a2ensite website1 finally the domain will now be redirected to your /var/www/website1 folder.

Conclusion

That’s it for this guide, you now have a working Nginx reverse proxy serving requests to an Apache server behind it. You can use this configuration to serve HTTP requests to any server you want to add behind your reverse proxy. This setup can be easily moved from one server to another as long as it running Proxmox.

In future articles we will see how to configure Nginx to cache the content of Apache websites to lessen the load on it and improve Apache performance. I will also walk you through the installation of letsencrypt SSL certificates on this kind of setup and how to setup your own analytics using Piwik.

If you have anything to add/correct or have any trouble following this guide please feel free the use the comments. Thank you for reading.

 

6 thoughts on “Configuring containers with Proxmox on OVH Kimsufi server behind a single public IP with NAT”

    1. Indeed, I was not really looking to use IPv6 here, but will try to add it as well in a future article ! Thanks for your comment.

      1. Thanks, according to some you have a subnet of /64 on Kimsufi instead of /128. So you can theoretically get as many public ips are you need. However, someone made an interesting comment about being on the explorer level, I am slightly below that level so I must tell you that my question was not hypothetical.

        At any rate, I have used http://www.microsofttranslator.com/bv.aspx?from=&to=en&a=http%3A%2F%2Fsngr.org%2F2017%2F02%2F%E5%8D%95ipv4%E6%9C%8D%E5%8A%A1%E5%99%A8%E5%BB%BA%E7%BA%AFipv6%E7%9A%84vps%E7%94%A8%E4%BA%8Eweb%E7%AE%A1%E7%90%86pt%E6%8C%82%E6%9C%BA.html and http://wiki.x8e.net/doku.php?id=proxmox_ipv6 in order to set it up but being Ipv6 it seems that you need a tunnel or some other mechanism to configure the setting. I may have to take another look your article to see about installing a nat but as I read article online I was persuaded against it. Looking forward to your response. God bless you!

  1. Nice post
    I finally got a KS-1 😀
    and now it’s running PROXMOX 4.4, but after I set everything up it works till the next reboot.

    for some reason after I reboot the server, all iptables config will disappear, there is no entry at all….

  2. Thanks for sharing the knowledge.
    it seems in proxmox 5 I can’t use the dummy0 in bridge port. It says “bridge ‘vmbr1’ – unable to find bridge port ‘dummy0’ (500)

Leave a Reply

Your email address will not be published. Required fields are marked *