Reference:
- etcd-io#17737

…io#17737

[Problem statement]

The blackhole() leverages proxy to drop all the incoming and outgoing
traffic passing through it, but the proxy doesn't properly block out all
the communication channels for a given member, due to the fact that the 
member initiates connections from itself to others.

[Overview of the proposed solutions]

The proposed implementation here performs traffic blocking at L7 
(application layer). There is another idea from fuweid performs blocking
at L4, as mentioned in reference [1].

[Root cause]
(Diagrams credit: fuweid)

Let's assuming the following:
- Current member ID (the member we would like to perform blackhole on): A
- Other member ID: B, C


In the e2e test set up, let's breakdown how A is able to communicate
with its peers. 

For the case of incoming connections from B and C to A, the connections 
from B and C will be established through A-proxy.

B -----> A-Proxy -----> A
           ^ 
           |
           C 

For the case of establishing outgoing connections from A to B and C, 
the connections from A will be established through B-Proxy and C-Proxy, 
before reaching B and C, respectively.

A -----> B-Proxy ----> B
|
+--------> C-Proxy ---> C

As you can see, currently the `BlackholeTx` and `BlackholeRx` only blocks 
traffic between `X-Proxy <---> X`. Thus, only the externally-established
incoming traffic will be block (B and C to A).

[Implementation]

For each member, it has 2 types of communication channels, namely stream
and pipeline. Connections made by pipeline is not persisted, but for 
stream the connection is continuously used by the member for long-poling.

In order to block the outgoing connections, we can hijack and drop the 
data from `RoundTrip` and `ServeHTTP`. 

When establishing a new connection, `RoundTrip` will be called. By using
a failpoint, we can drop the data carried by `Body` in RequestBody 
on-demand. 

When accepting a new connection, `ServeHTTP` will be called. By using a 
failpoint, we can drop the data carried by `Body` in RequestBody, and 
drop the data written to `ResponseWriter` on-demand.

[Discussion]

The downside of this approach is that we are introducing a custom 
implementation of `RoundTrip`, as we are hijacking the connection.

[Testing]

make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1

make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1

References:
[1] etcd-io#17737 (comment)
[2] https://github.com/etcd-io/etcd/pull/17790/files#diff-f01210a3082e25ff00682648f32122941a0c275b3926a8da37447589fe2ede1aR109

…io#17737

[Problem statement]

The blackhole() leverages proxy to drop all the incoming and outgoing
traffic passing through it, but the proxy doesn't properly block out all
the communication channels for a given member, due to the fact that the 
member initiates connections from itself to others.

[Overview of the proposed solutions]

The proposed implementation here performs traffic blocking at L7 
(application layer). There is another idea from fuweid performs blocking
at L4, as mentioned in reference [1].

[Root cause]
(Diagrams credit: fuweid)

Let's assuming the following:
- Current member ID (the member we would like to perform blackhole on): A
- Other member ID: B, C


In the e2e test set up, let's breakdown how A is able to communicate
with its peers. 

For the case of incoming connections from B and C to A, the connections 
from B and C will be established through A-proxy.

B -----> A-Proxy -----> A
           ^ 
           |
           C 

For the case of establishing outgoing connections from A to B and C, 
the connections from A will be established through B-Proxy and C-Proxy, 
before reaching B and C, respectively.

A -----> B-Proxy ----> B
|
+--------> C-Proxy ---> C

As you can see, currently the `BlackholeTx` and `BlackholeRx` only blocks 
traffic between `X-Proxy <---> X`. Thus, only the externally-established
incoming traffic will be block (B and C to A).

[Implementation]

For each member, it has 2 types of communication channels, namely stream
and pipeline. Connections made by pipeline is not persisted, but for 
stream the connection is continuously used by the member for long-poling.

In order to block the outgoing connections, we can hijack and drop the 
data from `RoundTrip` and `ServeHTTP`. 

When establishing a new connection, `RoundTrip` will be called. By using
a failpoint, we can drop the data carried by `Body` in RequestBody 
on-demand. 

When accepting a new connection, `ServeHTTP` will be called. By using a 
failpoint, we can drop the data carried by `Body` in RequestBody, and 
drop the data written to `ResponseWriter` on-demand.

If a connection is already opened (like a stream), because we hijacked
the `Reader` and `Writer` interface, we can still drop traffic as 
desired.

[Discussion]

The downside of this approach is that we are introducing a custom 
implementation of `RoundTrip`, as we are hijacking the connection.

[Testing]

make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1

make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1

References:
[1] etcd-io#17737 (comment)
[2] https://github.com/etcd-io/etcd/pull/17790/files#diff-f01210a3082e25ff00682648f32122941a0c275b3926a8da37447589fe2ede1aR109

…io#17737

[Problem statement]

The blackhole() leverages proxy to drop all the incoming and outgoing
traffic passing through it, but the proxy doesn't properly block out all
the communication channels for a given member, due to the fact that the
member initiates connections from itself to others.

[Overview of the proposed solutions]

The proposed implementation here performs traffic blocking at L7
(application layer). There is another idea from fuweid performs blocking
at L4, as mentioned in reference [1].

[Root cause]
(Diagrams credit: fuweid)

Let's assuming the following:
- Current member ID (the member we would like to perform blackhole on): A
- Other member ID: B, C

In the e2e test set up, let's breakdown how A is able to communicate
with its peers.

For the case of incoming connections from B and C to A, the connections
from B and C will be established through A-proxy.

B -----> A-Proxy -----> A
           ^
           |
           C

For the case of establishing outgoing connections from A to B and C,
the connections from A will be established through B-Proxy and C-Proxy,
before reaching B and C, respectively.

A -----> B-Proxy ----> B
|
+--------> C-Proxy ---> C

As you can see, currently the `BlackholeTx` and `BlackholeRx` only blocks
traffic between `X-Proxy <---> X`. Thus, only the externally-established
incoming traffic will be block (B and C to A).

[Implementation]

For each member, it has 2 types of communication channels, namely stream
and pipeline. Connections made by pipeline is not persisted, but for
stream the connection is continuously used by the member for long-poling.

In order to block the outgoing connections, we can hijack and drop the
data from `RoundTrip` and `ServeHTTP`.

When establishing a new connection, `RoundTrip` will be called. By using
a failpoint, we can drop the data carried by `Body` in RequestBody
on-demand.

When accepting a new connection, `ServeHTTP` will be called. By using a
failpoint, we can drop the data carried by `Body` in RequestBody, and
drop the data written to `ResponseWriter` on-demand.

If a connection is already opened (like a stream), because we hijacked
the `Reader` and `Writer` interface, we can still drop traffic as
desired.

[Discussion]

The downside of this approach is that we are introducing a custom
implementation of `RoundTrip`, as we are hijacking the connection.

[Testing]

make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1

make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1

References:
[1] etcd-io#17737 (comment)
[2] https://github.com/etcd-io/etcd/pull/17790/files#diff-f01210a3082e25ff00682648f32122941a0c275b3926a8da37447589fe2ede1aR109

Signed-off-by: Chun-Hung Tseng <[email protected]>

…io#17737

[Problem statement]

The blackhole() leverages proxy to drop all the incoming and outgoing
traffic passing through it, but the proxy doesn't properly block out all
the communication channels for a given member, due to the fact that the
member initiates connections from itself to others.

[Overview of the proposed solutions]

The proposed implementation here performs traffic blocking at L7
(application layer). There is another idea from fuweid performs blocking
at L4, as mentioned in reference [1].

[Root cause]
(Diagrams credit: fuweid)

Let's assuming the following:
- Current member ID (the member we would like to perform blackhole on): A
- Other member ID: B, C

In the e2e test set up, let's breakdown how A is able to communicate
with its peers.

For the case of incoming connections from B and C to A, the connections
from B and C will be established through A-proxy.

B -----> A-Proxy -----> A
           ^
           |
           C

For the case of establishing outgoing connections from A to B and C,
the connections from A will be established through B-Proxy and C-Proxy,
before reaching B and C, respectively.

A -----> B-Proxy ----> B
|
+--------> C-Proxy ---> C

As you can see, currently the `BlackholeTx` and `BlackholeRx` only blocks
traffic between `X-Proxy <---> X`. Thus, only the externally-established
incoming traffic will be block (B and C to A).

[Implementation]

For each member, it has 2 types of communication channels, namely stream
and pipeline. Connections made by pipeline is not persisted, but for
stream the connection is continuously used by the member for long-poling.

In order to block the outgoing connections, we can hijack and drop the
data from `RoundTrip` and `ServeHTTP`.

When establishing a new connection, `RoundTrip` will be called. By using
a failpoint, we can drop the data carried by `Body` in RequestBody
on-demand.

When accepting a new connection, `ServeHTTP` will be called. By using a
failpoint, we can drop the data carried by `Body` in RequestBody, and
drop the data written to `ResponseWriter` on-demand.

If a connection is already opened (like a stream), because we hijacked
the `Reader` and `Writer` interface, we can still drop traffic as
desired.

[Discussion]

The downside of this approach is that we are introducing a custom
implementation of `RoundTrip`, as we are hijacking the connection.

[Testing]

make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1

make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1

References:
[1] etcd-io#17737 (comment)
[2] https://github.com/etcd-io/etcd/pull/17790/files#diff-f01210a3082e25ff00682648f32122941a0c275b3926a8da37447589fe2ede1aR109

Signed-off-by: Chun-Hung Tseng <[email protected]>

Reference:
- etcd-io#17737

…io#17737

[Problem statement]

The blackhole() leverages proxy to drop all the incoming and outgoing
traffic passing through it, but the proxy doesn't properly block out all
the communication channels for a given member, due to the fact that the
member initiates connections from itself to others.

[Overview of the proposed solutions]

The proposed implementation here performs traffic blocking at L7
(application layer). There is another idea from fuweid performs blocking
at L4, as mentioned in reference [1].

[Root cause]
(Diagrams credit: fuweid)

Let's assuming the following:
- Current member ID (the member we would like to perform blackhole on): A
- Other member ID: B, C

In the e2e test set up, let's breakdown how A is able to communicate
with its peers.

For the case of incoming connections from B and C to A, the connections
from B and C will be established through A-proxy.

B -----> A-Proxy -----> A
           ^
           |
           C

For the case of establishing outgoing connections from A to B and C,
the connections from A will be established through B-Proxy and C-Proxy,
before reaching B and C, respectively.

A -----> B-Proxy ----> B
|
+--------> C-Proxy ---> C

As you can see, currently the `BlackholeTx` and `BlackholeRx` only blocks
traffic between `X-Proxy <---> X`. Thus, only the externally-established
incoming traffic will be block (B and C to A).

[Implementation]

For each member, it has 2 types of communication channels, namely stream
and pipeline. Connections made by pipeline is not persisted, but for
stream the connection is continuously used by the member for long-poling.

In order to block the outgoing connections, we can hijack and drop the
data from `RoundTrip` and `ServeHTTP`.

When establishing a new connection, `RoundTrip` will be called. By using
a failpoint, we can drop the data carried by `Body` in RequestBody
on-demand.

When accepting a new connection, `ServeHTTP` will be called. By using a
failpoint, we can drop the data carried by `Body` in RequestBody, and
drop the data written to `ResponseWriter` on-demand.

If a connection is already opened (like a stream), because we hijacked
the `Reader` and `Writer` interface, we can still drop traffic as
desired.

[Discussion]

The downside of this approach is that we are introducing a custom
implementation of `RoundTrip`, as we are hijacking the connection.

[Testing]

make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1

make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1

References:
[1] etcd-io#17737 (comment)
[2] https://github.com/etcd-io/etcd/pull/17790/files#diff-f01210a3082e25ff00682648f32122941a0c275b3926a8da37447589fe2ede1aR109

Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.

[Problem]

A peer will receive traffic from its peers and initiate connections to
its peers (via stream and pipeline), thus, the current mechanism of
only blocking traffic via proxy is incomplete, since the traffic
initiated from the peers to others will be "leaking", since `blackholeTX`
and `blackholeRX` only drop traffic coming in and out of the peer's
proxy.

[Solution - main idea]

Let's first agree on the naming of the existing proxy as a
"reverse proxy".

We will introduce a "forward proxy", which will be proxying all the
connection initiated from a peer to its peers.

The modified architecture will look something like this:
```
A -- B's SSL termination proxy - B's transparent proxy - B
     ^ newly introduced          ^ in the original codebase
```

By adding this forward proxy, we can block all traffic coming in and out
of a peer, without having to resort to external tools, such as iptables,
and the blocking of traffic is complete.

[Implementation]

The main subtasks are:
- Set up forward proxy
- Enable/disable forward proxy

[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1

make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1

[Reference]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891

Based on Fu Wei's idea discussed in the [issue](etcd-io#17737),
we employ the blocking on L7 but without using external tools.

[Background]

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.

[Proposed solution]

We introduce an forward proxy for each peer, which will be proxying all
the connections initiated from a peer to its peers.

The modified architecture will look something like this:
```
A -- A's forward proxy ----- B's reverse proxy - B
     ^ newly introduced      ^ in the original codebase (renamed)
```

By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables.

[Testing]
- `make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`

[Issues]
- I run into `context deadline exceeded` sometimes
```
    etcd_mix_versions_test.go:175:
                Error Trace:    /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175
                                                        /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75
                                                        /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31
                Error:          Received unexpected error:
                                [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found.  Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines:
                                {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
                                Error: context deadline exceeded
                                 (expected "OK", got []). Try EXPECT_DEBUG=TRUE
                Test:           TestBlackholeByMockingPartitionLeader
                Messages:       failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found.  Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines:
                                {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
                                Error: context deadline exceeded
                                 (expected "OK", got []). Try EXPECT_DEBUG=TRUE
```

[References]
[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891

Signed-off-by: Chun-Hung Tseng <[email protected]>

It's verified that the blocking of traffic is complete, compared to
previous solutions [2][3].

[Implementation]

The main subtasks are
- set up an environment variable `E2E_TEST_FORWARD_PROXY_IP`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test

The result is that for every peer, we will have the arch like this
```
A -- A's forward proxy
     (connections initiated from A will be forwarded from this proxy)
 |   ^ covers case (b)
 |
 --- A's (currently existing) reverse proxy
     (advertised to other peers where the connection should come in from)
     ^ covers case (a)
```

Based on Fu Wei's idea discussed in the [issue](etcd-io#17737),
we employ the blocking on L7 but without using external tools.

[Background]

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.

[Proposed solution]

We introduce an forward proxy for each peer, which will be proxying all
the connections initiated from a peer to its peers.

The modified architecture will look something like this:
```
A -- A's forward proxy ----- B's reverse proxy - B
     ^ newly introduced      ^ in the original codebase (renamed)
```

By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables.

[Testing]
- `make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`

[Issues]
- I run into `context deadline exceeded` sometimes
```
    etcd_mix_versions_test.go:175:
                Error Trace:    /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175
                                                        /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75
                                                        /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31
                Error:          Received unexpected error:
                                [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found.  Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines:
                                {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
                                Error: context deadline exceeded
                                 (expected "OK", got []). Try EXPECT_DEBUG=TRUE
                Test:           TestBlackholeByMockingPartitionLeader
                Messages:       failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found.  Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines:
                                {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
                                Error: context deadline exceeded
                                 (expected "OK", got []). Try EXPECT_DEBUG=TRUE
```

[References]
[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891

Signed-off-by: Chun-Hung Tseng <[email protected]>

It's verified that the blocking of traffic is complete, compared to
previous solutions [2][3].

[Implementation]

The main subtasks are
- set up an environment variable `E2E_TEST_FORWARD_PROXY_IP`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test

The result is that for every peer, we will have the arch like this
```
A -- A's forward proxy
     (connections initiated from A will be forwarded from this proxy)
 |   ^ covers case (b)
 |
 --- A's (currently existing) reverse proxy
     (advertised to other peers where the connection should come in from)
     ^ covers case (a)
```

Based on Fu Wei's idea discussed in the [issue](etcd-io#17737),
we employ the blocking on L7 but without using external tools.

[Background]

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.

[Proposed solution]

We introduce an forward proxy for each peer, which will be proxying all
the connections initiated from a peer to its peers.

The modified architecture will look something like this:
```
A -- A's forward proxy ----- B's reverse proxy - B
     ^ newly introduced      ^ in the original codebase (renamed)
```

By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables.

[Testing]
- `make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`

[Issues]
- I run into `context deadline exceeded` sometimes
```
    etcd_mix_versions_test.go:175:
                Error Trace:    /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175
                                                        /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75
                                                        /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31
                Error:          Received unexpected error:
                                [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found.  Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines:
                                {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
                                Error: context deadline exceeded
                                 (expected "OK", got []). Try EXPECT_DEBUG=TRUE
                Test:           TestBlackholeByMockingPartitionLeader
                Messages:       failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found.  Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines:
                                {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
                                Error: context deadline exceeded
                                 (expected "OK", got []). Try EXPECT_DEBUG=TRUE
```

[References]
[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891

Signed-off-by: Chun-Hung Tseng <[email protected]>

It's verified that the blocking of traffic is complete, compared to
previous solutions [2][3].

[Implementation]

The main subtasks are
- set up an environment variable `E2E_TEST_FORWARD_PROXY_IP`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test

The result is that for every peer, we will have the arch like this
```
A -- A's forward proxy
     (connections initiated from A will be forwarded from this proxy)
 |   ^ covers case (b)
 |
 --- A's (currently existing) reverse proxy
     (advertised to other peers where the connection should come in from)
     ^ covers case (a)
```

Based on Fu Wei's idea discussed in the [issue](etcd-io#17737),
we employ the blocking on L7 but without using external tools.

[Background]

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.

[Proposed solution]

We introduce an forward proxy for each peer, which will be proxying all
the connections initiated from a peer to its peers.

The modified architecture will look something like this:
```
A -- A's forward proxy ----- B's reverse proxy - B
     ^ newly introduced      ^ in the original codebase (renamed)
```

By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables.

[Testing]
- `make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`

[Issues]
- I run into `context deadline exceeded` sometimes
```
    etcd_mix_versions_test.go:175:
                Error Trace:    /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175
                                                        /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75
                                                        /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31
                Error:          Received unexpected error:
                                [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found.  Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines:
                                {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
                                Error: context deadline exceeded
                                 (expected "OK", got []). Try EXPECT_DEBUG=TRUE
                Test:           TestBlackholeByMockingPartitionLeader
                Messages:       failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found.  Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines:
                                {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
                                Error: context deadline exceeded
                                 (expected "OK", got []). Try EXPECT_DEBUG=TRUE
```

[References]
[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891

Signed-off-by: Chun-Hung Tseng <[email protected]>

It's verified that the blocking of traffic is complete, compared to
previous solutions [2][3].

[Implementation]

The main subtasks are
- set up an environment variable `E2E_TEST_FORWARD_PROXY_IP`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test

The result is that for every peer, we will have the arch like this
```
A -- A's forward proxy
     (connections initiated from A will be forwarded from this proxy)
 |   ^ covers case (b)
 |
 --- A's (currently existing) reverse proxy
     (advertised to other peers where the connection should come in from)
     ^ covers case (a)
```

Implement determineHTTPTransportProxyParsingFunc

Signed-off-by: Chun-Hung Tseng <[email protected]>

Tidy up net.SplitHostPort

Signed-off-by: Chun-Hung Tseng <[email protected]>

Remove reverse proxy and keep only forward proxy for all peers

There is no need of reverse proxy after the introduction of forward
proxy, as the forward proxy holds the information of the destination,
we can filter connection traffic from there, and this can reduce 1 hop
when the proxy is turned on.

Clearly separate L4 and L7 connection handling logic

Due to forward proxy's need to parse the CONNECT header, which is a
L7 layer feature, thus we are splitting the proxy into 2 types, for
better maintainability.

Use a custom Listener wrapper to implement existing L4 features but switch to L7 proxy design

If the first request is not a CONNECT, we will need to receive perfectly
parsable HTTP packets, not just random byte streams, as Serve() will
leverage ReadRequest() under the hood.

Transition to forward proxy

- Remove To and From fields, as this is only used in reverse proxy
- Add Listen field, to indicate which URL that the forward is listening
for requests
- Update the tests as required

TODO
- Figure out why DelayTx() isn't working anymore

- Doesn't support unix socket (L7 http transport proxy only support http/https/and socks5)
- It's L7 so we need to send perfectly crafted http request
- Bug fix: Doing var httpTransportProxyParsingFunc = determineHTTPTransportProxyParsingFunc() will initialize the function once only (env var read on program init)

If the traffic to the destination is HTTPS, a CONNECT request will be sent
first (containing the intended destination HOST).
If the traffic to the destination is HTTP, no CONNECT request will be sent
first, but normal HTTP request, with the HOST set to the final destination,
will be sent.

Reference:
- etcd-io#17938 (comment)
- etcd-io#17985 (comment)

Signed-off-by: Chun-Hung Tseng <[email protected]>
Signed-off-by: Siyuan Zhang <[email protected]>
Co-authored-by: Iván Valdés Castillo <[email protected]>

Based on Fu Wei's idea discussed in the issue [1], we employ the network traffic blocking on L7, using a forward proxy, without the need of using external tools.

[Background]

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing reverse proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.

[Proposed solution]

We introduce an forward proxy for each peer, which will be proxying all
the connections initiated from a peer to its peers.

We will remove the current use of the reverse proxy, as the forward proxy holds the information of the destination, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables.

The modified architecture will look something like this:
```
A --- A's forward proxy ----- B
     ^ newly introduced
```

It's verified that the blocking of traffic is complete, compared to
previous solutions attempted in PRs [2][3].

[Implementation]

The main subtasks are
- transition to L7 forward proxy
- set up an environment variable `E2E_TEST_FORWARD_PROXY_IP`
- implement L7 forward proxy by drastically reducing the the existing proxy server code and design to use blocking traffic
- implement enable/disable of the L7 forward proxy in the e2e test

Known limitations
- Doesn't support unix socket (L7 http transport proxy only support http/https/and socks5)
- It's L7 so we need to send perfectly crafted http request
-Doesn’t support reordering, dropping, etc. packets on-the-fly

[Testing]
- `make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`

[References]
[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891
[4] etcd-io#17938 (comment)
[5] etcd-io#17985 (comment)

Signed-off-by: Chun-Hung Tseng <[email protected]>
Signed-off-by: Siyuan Zhang <[email protected]>
Co-authored-by: Iván Valdés Castillo <[email protected]>

Based on Fu Wei's idea discussed in the issue [1], we employ the network traffic blocking on L7, using a forward proxy, without the need of using external tools.

[Background]

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing reverse proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.

[Proposed solution]

We introduce an forward proxy for each peer, which will be proxying all
the connections initiated from a peer to its peers.

We will remove the current use of the reverse proxy, as the forward proxy holds the information of the destination, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables.

The modified architecture will look something like this:
```
A --- A's forward proxy ----- B
   ^ newly introduced
```

It's verified that the blocking of traffic is complete, compared to
previous solutions attempted in PRs [2][3].

[Implementation]

The main subtasks are
- transition to L7 forward proxy
- set up an environment variable `E2E_TEST_FORWARD_PROXY_IP`
- implement L7 forward proxy by drastically reducing the the existing proxy server code and design to use blocking traffic
- implement enable/disable of the L7 forward proxy in the e2e test

Known limitations are
- Doesn't support unix socket (L7 http transport proxy only support http/https/and socks5)
- It's L7 so we need to send perfectly crafted http request
-Doesn’t support reordering, dropping, etc. packets on-the-fly

[Testing]
- `make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`

[References]
[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891
[4] etcd-io#17938 (comment)
[5] etcd-io#17985 (comment)

Signed-off-by: Chun-Hung Tseng <[email protected]>
Signed-off-by: Siyuan Zhang <[email protected]>
Co-authored-by: Iván Valdés Castillo <[email protected]>

Based on Fu Wei's idea discussed in the issue [1], we employ the network traffic blocking on L7, using a forward proxy, without the need of using external tools.

[Background]

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing reverse proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.

[Proposed solution]

We introduce an forward proxy for each peer, which will be proxying all
the connections initiated from a peer to its peers.

We will remove the current use of the reverse proxy, as the forward proxy holds the information of the destination, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables.

The modified architecture will look something like this:
```
A --- A's forward proxy ----- B
   ^ newly introduced
```

It's verified that the blocking of traffic is complete, compared to
previous solutions attempted in PRs [2][3].

[Implementation]

The main subtasks are
- redesigned as L7 forward proxy
- unix socket support is dropped: e2e test supports unix sockets for peer communication, but only several e2e test cases use unix socket as majority of e2e test cases use http/https
- introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP`
- implement L7 forward proxy by drastically reducing the the existing proxy server code and design to use blocking traffic

Known limitations are
- Doesn't support unix socket (L7 http transport proxy only support http/https/and socks5)
- It's L7 so we need to send perfectly crafted http request
-Doesn’t support reordering, dropping, etc. packets on-the-fly

[Testing]
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast`

[References]
[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891
[4] etcd-io#17938 (comment)
[5] etcd-io#17985 (comment)

Signed-off-by: Chun-Hung Tseng <[email protected]>
Signed-off-by: Siyuan Zhang <[email protected]>
Co-authored-by: Iván Valdés Castillo <[email protected]>
Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on Fu Wei's idea discussed in the issue [1], we employ the network traffic blocking on L7, using a forward proxy, without the need of using external tools.

[Background]

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing reverse proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.

[Proposed solution]

We introduce an forward proxy for each peer, which will be proxying all
the connections initiated from a peer to its peers.

We will remove the current use of the reverse proxy, as the forward proxy holds the information of the destination, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables.

The modified architecture will look something like this:
```
A --- A's forward proxy ----- B
   ^ newly introduced
```

It's verified that the blocking of traffic is complete, compared to
previous solutions attempted in PRs [2][3].

[Implementation]

The main subtasks are
- redesigned as L7 forward proxy
- unix socket support is dropped: e2e test supports unix sockets for peer communication, but only several e2e test cases use unix socket as majority of e2e test cases use http/https
- introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP`
- implement L7 forward proxy by drastically reducing the the existing proxy server code and design to use blocking traffic

Known limitations are
- Doesn't support unix socket (L7 http transport proxy only support http/https/and socks5)
- It's L7 so we need to send perfectly crafted http request
-Doesn’t support reordering, dropping, etc. packets on-the-fly

[Testing]
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast`

[References]
[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891
[4] etcd-io#17938 (comment)
[5] etcd-io#17985 (comment)

Signed-off-by: Chun-Hung Tseng <[email protected]>
Signed-off-by: Siyuan Zhang <[email protected]>
Co-authored-by: Iván Valdés Castillo <[email protected]>
Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on Fu Wei's idea discussed in the issue [1], we employ the network traffic blocking on L7, using a forward proxy, without the need to use external tools.

[Background]

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's existing reverse proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all.

[Proposed solution]

We introduce a forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers.

We will remove the current use of the reverse proxy, as the forward proxy holds the information of the destination, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables.

The modified architecture will look something like this:
```
A --- A's forward proxy ----- B
   ^ newly introduced
```

It's verified that the blocking of traffic is complete, compared to previous solutions attempted in PRs [2][3].

[Implementation]

The main subtasks are
- redesigned as an L7 forward proxy
- Unix socket support is dropped: e2e test supports unix sockets for peer communication, but only several e2e test cases use Unix sockets as majority of e2e test cases use HTTP/HTTPS
- introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP`
- implement L7 forward proxy by drastically reducing the existing proxy server code and design to use blocking traffic

Known limitations are
- Doesn't support unix socket (L7 HTTP transport proxy only supports HTTP/HTTPS/and socks5)
- It's L7 so we need to send a perfectly crafted HTTP request
-Doesn’t support reordering, dropping, etc. packets on-the-fly

[Testing]
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast`

[References]
[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891
[4] etcd-io#17938 (comment)
[5] etcd-io#17985 (comment)

Signed-off-by: Chun-Hung Tseng <[email protected]>
Signed-off-by: Siyuan Zhang <[email protected]>
Co-authored-by: Iván Valdés Castillo <[email protected]>
Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on Fu Wei's idea discussed in the issue [1], we employ the network traffic blocking on L7, using a forward proxy, without the need to use external tools.

[Background]

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's existing reverse proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all.

[Proposed solution]

We introduce a forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers.

We will remove the current use of the reverse proxy, as the forward proxy holds the information of the destination, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables.

The modified architecture will look something like this:
```
A --- A's forward proxy ----- B
   ^ newly introduced
```

It's verified that the blocking of traffic is complete, compared to previous solutions attempted in PRs [2][3].

[Implementation]

The main subtasks are
- redesigned as an L7 forward proxy
- Unix socket support is dropped: e2e test supports unix sockets for peer communication, but only several e2e test cases use Unix sockets as majority of e2e test cases use HTTP/HTTPS
- introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP`
- implement L7 forward proxy by drastically reducing the existing proxy server code and design to use blocking traffic

Known limitations are
- Doesn't support unix socket (L7 HTTP transport proxy only supports HTTP/HTTPS/and socks5)
- It's L7 so we need to send a perfectly crafted HTTP request
-Doesn’t support reordering, dropping, etc. packets on-the-fly

[Testing]
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast`

[References]
[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891
[4] etcd-io#17938 (comment)
[5] etcd-io#17985 (comment)

Signed-off-by: Siyuan Zhang <[email protected]>
Co-authored-by: Iván Valdés Castillo <[email protected]>
Signed-off-by: Chun-Hung Tseng <[email protected]>

Part of the patches to fix etcd-io#17737

During the development of etcd-io#17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.

Part of the patches to fix etcd-io#17737

During the development of etcd-io#17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.

Signed-off-by: Chun-Hung Tseng <[email protected]>

Part of the patches to fix etcd-io#17737

During the development of etcd-io#17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.

This feature falls under the unused feature.

Signed-off-by: Chun-Hung Tseng <[email protected]>

Part of the patches to fix etcd-io#17737

During the development of etcd-io#17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.

This feature falls under the unused feature.

Signed-off-by: Chun-Hung Tseng <[email protected]>

Part of the patches to fix etcd-io#17737

During the development of etcd-io#17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.

This feature falls under the unused feature.

Signed-off-by: Chun-Hung Tseng <[email protected]>

Part of the patches to fix etcd-io#17737

During the development of etcd-io#17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.

This feature falls under the unused feature. Also, the initial
implementation has a bug: if connections are not created continuously,
the latency accept will not work. Consider the following case:
a) set latency accept
b) put latency accept into effect
c) latency accept will start idling the goroutine
d) block-wait at accept() - waiting for new connections
e) new connection comes in - establish it
f) go to c -> as we can see, if the request come every x seconds, where
x is larger than the latency accept time we set, we can see that the
latency accept has no effect.

Signed-off-by: Chun-Hung Tseng <[email protected]>

Part of the patches to fix etcd-io#17737

During the development of etcd-io#17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.

This feature falls under the features targeting L4 reverse proxy.

Signed-off-by: Chun-Hung Tseng <[email protected]>

Part of the patches to fix etcd-io#17737

During the development of etcd-io#17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.

This feature falls under the unused feature.

Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on the ideas discussed in the issues [1] and PRs [2][3][6],
we switch from using a L4 reverse proxy to a L7 forward proxy
to properly block peer network traffic, without the need to use
external tools.

The design aims to implement only the minimal required features that
satisfies blocking incoming and outgoing peer traffic. Complicated
features such as packet reordering, packet delivery delay, etc. to
a future container-based solution.

[Background]

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing reverse proxy is insufficient, since only scenario (a) is
handled, and network traffic in scenario (b) is not blocked at all.

[Proposed solution]

We introduce a L7 forward proxy for each peer, which will be proxying
all the connections initiated from a peer to its peers.

We will remove the current use of the L4 reverse proxy, as the L7
forward proxy holds the information of the destination, we can block
all incoming and outgoing traffic that is initiated from a peer to
others, without having to resort to external tools, such as iptables.

The modified architecture will look something like this:
```
A --- A's forward proxy --- B
   ^ newly introduced
```

[Implementation]

The main subtasks are
- redesigned as an L7 forward proxy
- introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` to
bypass the limitation of with http.ProxyFromEnvironment
- implement a L7 forward proxy

Known limitations are
- Doesn't support unix socket, as L7 HTTP transport proxy only support
HTTP/HTTPS/and socks5 -> although e2e test supports unix sockets for
peer communication, but only few of the e2e test cases use unix sockets
as majority of e2e test cases use HTTP/HTTPS. It's been discussed and
decided that without the unix socket support is ok for now.
- it's L7 so we need to send a perfectly crafted HTTP request
- doesn’t support reordering, dropping, manipulating packets on-the-fly

[Testing]
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast`

[References]
[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891
[4] etcd-io#17938 (comment)
[5] etcd-io#17985 (comment)
[6] etcd-io#17938

Signed-off-by: Siyuan Zhang <[email protected]>
Signed-off-by: Iván Valdés Castillo <[email protected]>
Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on the ideas discussed in the issues [1] and PRs [2][3][6], we
switch from using an L4 reverse proxy to an L7 forward proxy to properly
block peer network traffic, without the need to use external tools.

The design aims to implement only the minimal required features that
satisfy blocking incoming and outgoing peer traffic. Complicated features
such as packet reordering, packet delivery delay, etc. to a future
container-based solution.

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing reverse proxy is insufficient, since only scenario (a) is
handled, and network traffic in scenario (b) is not blocked at all.

We introduce an L7 forward proxy for each peer, which will be proxying
all the connections initiated from a peer to its peers.

We will remove the current use of the L4 reverse proxy, as the L7
forward proxy holds the information of the destination, we can block all
incoming and outgoing traffic that is initiated from a peer to others,
without having to resort to external tools, such as iptables.

The modified architecture will look something like this:
```
A --- A's forward proxy --- B
   ^ newly introduced
```

The main subtasks are
- redesigned as an L7 forward proxy
- introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` to
bypass the limitation of `http.ProxyFromEnvironment`
- implement an L7 forward proxy

Known limitations are
- Doesn't support unix socket, as L7 HTTP transport proxy only supports
HTTP/HTTPS/and socks5 -> Although the e2e test supports unix sockets for
peer communication, only a few of the e2e test cases use unix sockets as
the majority of e2e test cases use HTTP/HTTPS. It's been discussed and
decided that it is ok for now without the unix socket support.
- it's L7 so we need to send a perfectly crafted HTTP request
- doesn’t support reordering, dropping, or manipulating packets on the
fly

- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast`

[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891
[4] etcd-io#17938 (comment)
[5] etcd-io#17985 (comment)
[6] etcd-io#17938

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

Signed-off-by: Siyuan Zhang <[email protected]>
Signed-off-by: Iván Valdés Castillo <[email protected]>
Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on the ideas discussed in the issues [1] and PRs [2][3][6], it's
been decided to switch from using an L4 reverse proxy to an L7 forward
proxy to properly block peer network traffic, without the need to use
external tools.

The design aims to implement only the minimal required features that
satisfy blocking incoming and outgoing peer traffic. Complicated features
such as packet reordering, packet delivery delay, etc. to a future
container-based solution.

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing reverse proxy is insufficient, since only scenario (a) is
handled, and network traffic in scenario (b) is not blocked at all.

We introduce an L7 forward proxy for each peer, which will be proxying
all the connections initiated from a peer to its peers.

We will remove the current use of the L4 reverse proxy, as the L7
forward proxy holds the information of the destination, we can block all
incoming and outgoing traffic that is initiated from a peer to others,
without having to resort to external tools, such as iptables.

The modified architecture will look something like this:
```
A --- A's forward proxy --- B
   ^ newly introduced
```

The main subtasks are
- redesigned as an L7 forward proxy
- introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` to
bypass the limitation of `http.ProxyFromEnvironment`
- implement an L7 forward proxy

Known limitations are
- Doesn't support unix socket, as L7 HTTP transport proxy only supports
HTTP/HTTPS/and socks5 -> Although the e2e test supports unix sockets for
peer communication, only a few of the e2e test cases use unix sockets as
the majority of e2e test cases use HTTP/HTTPS. It's been discussed and
decided that it is ok for now without the unix socket support.
- it's L7 so we need to send a perfectly crafted HTTP request
- doesn’t support reordering, dropping, or manipulating packets on the
fly

- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast`

[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891
[4] etcd-io#17938 (comment)
[5] etcd-io#17985 (comment)
[6] etcd-io#17938

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

Signed-off-by: Siyuan Zhang <[email protected]>
Signed-off-by: Iván Valdés Castillo <[email protected]>
Signed-off-by: Chun-Hung Tseng <[email protected]>

Part of the patches to fix etcd-io#17737

During the development of etcd-io#17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.

This feature falls under the unused feature.

Signed-off-by: Chun-Hung Tseng <[email protected]>

Part of the patches to fix etcd-io#17737

During the development of etcd-io#17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.

This feature falls under the unused feature.

Signed-off-by: Chun-Hung Tseng <[email protected]>

Part of the patches to fix etcd-io#17737

During the development of etcd-io#17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.

This feature falls under the unused feature. Also, the initial
implementation has a bug: if connections are not created continuously,
the latency accept will not work. Consider the following case:
a) set latency accept
b) put latency accept into effect
c) latency accept will start idling the goroutine
d) block-wait at accept() - waiting for new connections
e) new connection comes in - establish it
f) go to c -> as we can see, if the request come every x seconds, where
x is larger than the latency accept time we set, we can see that the
latency accept has no effect.

Signed-off-by: Chun-Hung Tseng <[email protected]>

Part of the patches to fix etcd-io#17737

During the development of etcd-io#17938,
we agreed that during the transition to L7 forward proxy, unused
features and features targeting L4 reverse proxy will be dropped.

This feature falls under the unused feature.

Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on the ideas discussed in the issues [1] and PRs [2][3][6], it's
been decided to switch from using an L4 reverse proxy to an L7 forward
proxy to properly block peer network traffic, without the need to use
external tools.

The design aims to implement only the minimal required features that
satisfy blocking incoming and outgoing peer traffic. Complicated features
such as packet reordering, packet delivery delay, etc. to a future
container-based solution.

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing reverse proxy is insufficient, since only scenario (a) is
handled, and network traffic in scenario (b) is not blocked at all.

We introduce an L7 forward proxy for each peer, which will be proxying
all the connections initiated from a peer to its peers.

We will remove the current use of the L4 reverse proxy, as the L7
forward proxy holds the information of the destination, we can block all
incoming and outgoing traffic that is initiated from a peer to others,
without having to resort to external tools, such as iptables.

The modified architecture will look something like this:
```
A --- A's forward proxy --- B
   ^ newly introduced
```

The main subtasks are
- redesigned as an L7 forward proxy
- introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` to
bypass the limitation of `http.ProxyFromEnvironment`
- implement an L7 forward proxy

Known limitations are
- Doesn't support unix socket, as L7 HTTP transport proxy only supports
HTTP/HTTPS/and socks5 -> Although the e2e test supports unix sockets for
peer communication, only a few of the e2e test cases use unix sockets as
the majority of e2e test cases use HTTP/HTTPS. It's been discussed and
decided that it is ok for now without the unix socket support.
- it's L7 so we need to send a perfectly crafted HTTP request
- doesn’t support reordering, dropping, or manipulating packets on the
fly

- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast`

[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891
[4] etcd-io#17938 (comment)
[5] etcd-io#17985 (comment)
[6] etcd-io#17938

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

Signed-off-by: Siyuan Zhang <[email protected]>
Signed-off-by: Iván Valdés Castillo <[email protected]>
Signed-off-by: Chun-Hung Tseng <[email protected]>