Hi, gals and guys

As I've mentioned before, I'm late to the container party
and starting to play around with LXC containers. This time,
my goal is to model a container setup for my internet-facing
server, to provide better security for its systems. More
specifically, I am trying to model a layout which puts
both my internet connection, and various internet-facing
services into containers such that the arrangement provides
a "bastion host" fronting a set of services running in a
"DMZ" network, sandwitched between firewalls.

Graphically, the concept looks like:

Internet
|
=====(firewall)
Bastion Host
|
-----+----- ...
| |
=== === .....
me server

To this end, I've put together a couple of LXC containers,
one ("sysdev1") representing the "bastion host", and the
other ("sysdev2") representing a service within the "DMZ"

sysdev1 resides on two networks: my internal network (which
represents the big bad internet), and a "private" network
(implemented as veth devices talking through a bridge
on the hosting system).

sysdev2 resides on one network: the "private" network
that sysdev1 resides on.

I have proven that sysdev1 has a route to the "internet"
(actually the Internet, but for now we can ignore the
part outside of my network), so I've managed to prove
that
1) I have correctly configured ip_forward on my host
2) I have correctly implemented routing on my host
3) I have correctly implemented routing (outbound)
in the sysdev1 container, including setting the
default gateway.
I have also tested the route from sysdev1 to sysdev2,
so I can also say that
4) I have correctly implemented the "private network"
routing in the sysdev1 container.

From sysdev2, I can ping & traceroute to sysdev1, so
5) I have correctly implemented the "private network"
routing in the sysdev2 container.
But, I cannot ping or traceroute beyond that private
network.

In sysdev1, I have the "gateway" set up to point
to my host system's address, and ip_forward is
properly set (inherited from the host system).

In sysdev2, I have the "gateway" set up to point
to sysdev1, and (again) ip_forward is properly
set (again, inherited from the host system).

As far as I can tell, I've got all the proper
routing rules in place, but I still cannot
route from sysdev2 to any system past sysdev1.

I've attached the technical details (routing table,
interface list, tests) below.

So, to the question: What have I done wrong, or
not done yet? Any advice you can offer would be
appreciated.

Thanks. Lew

==== sysdev2 ON 192.168.55.2 ====
Note: sysdev2 is the "services" standin within the "DMZ"

Script started on Wed 01 May 2024 10:06:47 AM EDT
root@sysdev2:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.55.1 0.0.0.0 UG 1 0 0 eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
192.168.55.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
root@sysdev2:~# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.55.2 netmask 255.255.255.0 broadcast 192.168.55.255
inet6 fe80::d076:b7ff:fe55:8543 prefixlen 64 scopeid 0x20<link>
ether d2:76:b7:55:85:43 txqueuelen 1000 (Ethernet)
RX packets 25 bytes 2855 (2.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 35 bytes 3766 (3.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

root@sysdev2:~# cat /proc/sys/net/ipv4/ip_forward
1 root@sysdev2:~# traceroute -n 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 192.168.55.1 0.086 ms 0.017 ms 0.014 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 *^C
root@sysdev2:~# exit
exit

Script done on Wed 01 May 2024 10:07:21 AM EDT

==== sysdev1 ON 192.168.55.2 / 192.168.99.32 ====
Note: sysdev1 is the "bastion host" standin that fronts the "DMZ"

Script started on Wed 01 May 2024 10:08:17 AM EDT
root@sysdev1:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.99.3 0.0.0.0 UG 0 0 0 eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.55.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
root@sysdev1:~# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.99.32 netmask 255.255.255.0 broadcast 192.168.99.255
inet6 fe80::90da:85ff:fecf:24cc prefixlen 64 scopeid 0x20<link>
ether 92:da:85:cf:24:cc txqueuelen 1000 (Ethernet)
RX packets 266 bytes 28495 (27.8 KiB)
RX errors 0 dropped 1 overruns 0 frame 0
TX packets 277 bytes 28000 (27.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.55.1 netmask 255.255.255.0 broadcast 192.168.55.255
inet6 fe80::c4a:71ff:fe6c:323a prefixlen 64 scopeid 0x20<link>
ether 0e:4a:71:6c:32:3a txqueuelen 1000 (Ethernet)
RX packets 88 bytes 8308 (8.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 43 bytes 3795 (3.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 37 bytes 3328 (3.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 37 bytes 3328 (3.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

root@sysdev1:~# cat /proc/sys/net/ipv4/ip_forward
1 root@sysdev1:~# traceroute -n 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 192.168.99.3 0.061 ms 0.022 ms 0.018 ms
2 192.168.99.1 0.150 ms 0.127 ms 0.127 ms
3 my.isp.net.addr 12.614 ms 13.526 ms 16.187 ms
... elided for privacy ...
8 8.8.8.8 37.242 ms 26.502 ms 142.251.70.11 27.062 ms
root@sysdev1:~# exit
exit

Script done on Wed 01 May 2024 10:09:02 AM EDT

==== wordsworth ON 192.168.99.3 ====
Note: wordsworth is the "internet" standin that converses
with the "bastion host". It also is the host of the
"sysdev1" and "sysdev2" containers, and the bridge
that implements the DMZ "private network"

Script started on Wed 01 May 2024 10:10:55 AM EDT
root@wordsworth:~/debug# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.99.1 0.0.0.0 UG 1 0 0 br0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 br0
root@wordsworth:~/debug# ifconfig
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.99.3 netmask 255.255.255.0 broadcast 192.168.99.255
inet6 fe80::aaa1:59ff:fe26:9fbf prefixlen 64 scopeid 0x20<link>
ether a8:a1:59:26:9f:bf txqueuelen 1000 (Ethernet)
RX packets 115186 bytes 145420873 (138.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 74760 bytes 7391322 (7.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
inet6 fe80::aaa1:59ff:fe26:9fbf prefixlen 64 scopeid 0x20<link>
ether a8:a1:59:26:9f:bf txqueuelen 1000 (Ethernet)
RX packets 115132 bytes 147024486 (140.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 74911 bytes 7394972 (7.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 20 bytes 1258 (1.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 20 bytes 1258 (1.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

sysdev: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::bc75:edff:fe8b:eac3 prefixlen 64 scopeid 0x20<link>
ether fe:39:85:3c:c0:12 txqueuelen 1000 (Ethernet)
RX packets 37 bytes 2228 (2.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 10 bytes 1168 (1.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

On Wed, 01 May 2024 15:11:45 +0000, Lew Pitcher wrote:

> Hi, gals and guys
>
> As I've mentioned before, I'm late to the container party
> and starting to play around with LXC containers. This time,
> my goal is to model a container setup for my internet-facing
> server, to provide better security for its systems. More
> specifically, I am trying to model a layout which puts
> both my internet connection, and various internet-facing
> services into containers such that the arrangement provides
> a "bastion host" fronting a set of services running in a
> "DMZ" network, sandwitched between firewalls.
>
> Graphically, the concept looks like:
>
> Internet
> |
> =====(firewall)
> Bastion Host
> |
> -----+----- ...
> | |
> === === .....
> me server
>
>
> To this end, I've put together a couple of LXC containers,
> one ("sysdev1") representing the "bastion host", and the
> other ("sysdev2") representing a service within the "DMZ"

I should mention that, in this configuration, I /have not/
implemented any firewall rules. I want to get basic routing
working before I start complicating the data flow with a
firewall.

[snip]

--
Lew Pitcher
"In Skills We Trust"

On Wed, 01 May 2024 17:12:52 +0000, Lew Pitcher wrote:
> I should mention that, in this configuration, I /have not/ implemented
> any firewall rules. I want to get basic routing working before I start
> complicating the data flow with a firewall.

Maybe those firewall rules are exactly what you need. My guess is that
you are lacking NAT (IP masqueradning). When sending ping or any other
packet out from sysdev2 to internet those packages come from your private
IP address 192.168.55.2 but only the sysdev1 machine knows how to reach
back to that IP address. Your machine wordsworth with IP address
192.168.99.3 does not have a route back to 192.168.55.2. Either you will
need to implement NAT on sysdev1 or add a route on wordsworth (and
probably also implement a more cumbersome NAT in your real internet
facing router).

regards Henrik

Hi, Henrik

On Thu, 02 May 2024 05:15:19 +0000, Henrik Carlqvist wrote:

> On Wed, 01 May 2024 17:12:52 +0000, Lew Pitcher wrote:
>> I should mention that, in this configuration, I /have not/ implemented
>> any firewall rules. I want to get basic routing working before I start
>> complicating the data flow with a firewall.
>
> Maybe those firewall rules are exactly what you need. My guess is that
> you are lacking NAT (IP masqueradning). When sending ping or any other
> packet out from sysdev2 to internet those packages come from your private
> IP address 192.168.55.2 but only the sysdev1 machine knows how to reach
> back to that IP address. Your machine wordsworth with IP address
> 192.168.99.3 does not have a route back to 192.168.55.2.

And, with that simple statement, you have solved my problem.
/Of course/, wordsworth needs a route back to sysdev2. And, in the
environment I'm trying to model, that route would be supplied by
a NAT rule on sysdev1.

> Either you will
> need to implement NAT on sysdev1 or add a route on wordsworth (and
> probably also implement a more cumbersome NAT in your real internet
> facing router).

Thanks, Henrik. You've hit the nail on the head.
And now, I'm off to make some firewall rules.

--
Lew Pitcher
"In Skills We Trust"

On Thu, 02 May 2024 13:21:26 +0000, Lew Pitcher wrote:

> Hi, Henrik
>
>
> On Thu, 02 May 2024 05:15:19 +0000, Henrik Carlqvist wrote:
>
>> On Wed, 01 May 2024 17:12:52 +0000, Lew Pitcher wrote:
>>> I should mention that, in this configuration, I /have not/ implemented
>>> any firewall rules. I want to get basic routing working before I start
>>> complicating the data flow with a firewall.
[snip]
>> Either you will
>> need to implement NAT on sysdev1 or add a route on wordsworth (and
>> probably also implement a more cumbersome NAT in your real internet
>> facing router).
>
> Thanks, Henrik. You've hit the nail on the head.
> And now, I'm off to make some firewall rules.

Yup. That fixed it.

Thanks again, Henrik

--
Lew Pitcher
"In Skills We Trust"