Requests are not forwarded, if a similar path exists in YARP

[clausndk]

Describe the bug

We have a list of services, with a common endpoint of /api/ping, in order for us to monitor the services.
I would like the same for the proxy itself (YARP), however this causes YARP to always respond from its own controller, and not those of the cluster.

I have a controller in my "backed service", at api/ping, that looks like this:

public class PingController : ControllerBase
    {
        [HttpGet("api/ping")]
        public string Ping()
        {
            return $"Atlas @ {DateTimeOffset.UtcNow:O}";
        }
    }

However, if I place the same kind of controller, with the same path in the YARP project, I only get responses from the YARP gateway.

namespace Microsoft.ReverseProxy.Sample.Controllers
{
    public class PingController : Controller
    {
        [HttpGet("api/ping")]
        public string Ping()
        {
            return $"YARP @ {DateTimeOffset.UtcNow:O}";
        }
    }
}

In general, I would expect a gateway/proxy to forward requests, when available and possible, and then fallback to it's own resources if they match as well.

I would then expect the order of the ASP.NET (Core) Middlewares, would be responsible for this.

My startup looks like this:

public class Startup
    {
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddLettuceEncrypt().PersistCertificatesToAzureKeyVault();
            services.AddControllers();
            services.AddReverseProxy()
                .LoadFromServiceFabric(
                    options =>
                    {
                        options.DiscoveryPeriod = TimeSpan.FromSeconds(5);
                        options.DiscoverInsecureHttpDestinations = true;
                        options.AllowStartBeforeDiscovery = false;
                    });
        }

        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            //app.UseHttpsRedirection();
            app.UseRouting();
            app.UseAuthorization();
            app.UseEndpoints(
                endpoints =>
                {
                    endpoints.MapControllers();
                    endpoints.MapReverseProxy();
                });
        }
    }

Is this expected behavior? If so, why?
Can I change it?

To Reproduce

As described above

Further technical details

• Include the version of the packages you are using
  
• The platform (Linux/macOS/Windows)
  Windows - Service Fabric setup

[Tratcher]

In routing the most most specific match wins. What paths do you have set up to proxy?

Example: If you set the proxy to forward /{**catch-all} and you have the ping controller above matching api/ping in the proxy, then a request to /api/ping matches the controller as more specific than the catch-all.

[clausndk]

In routing the most most specific match wins. What paths do you have set up to proxy?

Example: If you set the proxy to forward /{**catch-all} and you have the ping controller above matching api/ping in the proxy, then a request to /api/ping matches the controller as more specific than the catch-all.

Hello Tratcher

I've only setup routing with hosts, so I would expect that to be "catch-all"?

<Extension Name="YARP-preview">
	<Labels xmlns="http://schemas.microsoft.com/2015/03/fabact-no-schema">
		<Label Key="YARP.Enable">true</Label>
		<Label Key="YARP.Routes.atlas.Hosts">atlas-local.domain.dk</Label>
		<Label Key="YARP.Backend.SessionAffinity.Enabled">false</Label>
	</Labels>
</Extension>

[Tratcher]

Yes, that's a catch-all path with a host constraint. You can manually control the priority by setting the Order property on one of the routes (lower values match first), but that only reverses the situation. Ultimately you can't serve /api/ping from both the proxy and one of the proxy destinations without a way to differentiate the requests such as the host constraint you've added on the proxy route.

[clausndk]

I'm not really sure of my options here then?

Are you telling me I can configure a priority? If yes, where? Would that cause destinations to win over local controllers?

I would expect YARP to forward the request, if a match in destination, route and any other parameter are met. I would expect it to forward requests for me, when I curl a request to http(s)://atlas-local.domain.dk/api/ping (when the service is running and configured as above), but not if I do a request to http(s)://127.0.0.1/api/ping, because it would not be able to destermine a target destination for such a request.

Does that make sense?
Would you not expect YARP to follow such principle (bug or not)?

[Tratcher]

This is by design, proxy routes are not given special priority over local routes, the standard route matching algorithm is applied equally to both and the most specific one wins. To override that you set the Order property on your routes.
In this example something like <Label Key="YARP.Routes.route1.Order">-10</Label> should work (lower numbered routes are checked first).
The alternative is to add a host constraint of "127.0.0.1" to your local ping route so that it doesn't match requests for atlas-local.domain.dk.

[samsp-msft]

The alternative is to add a host constraint of "127.0.0.1" to your local ping route so that it doesn't match requests for atlas-local.domain.dk.

From your description you need to be using host names in the routes to distinguish the routes for destinations or for YARP itself.

[clausndk]

I'm having trouble understanding this, as I'm unable to debug YARP's DestinationInitializerMiddleware Invoke method, having the setup as above.

If a local controller with a path route that matches exists, ASP.NET Core returns that controllers result, and I'm not even hitting the Invoke method of what seems to be the first Middelware of YARP?

Makes sense?

[Tratcher]

That's expected, YARP middleware doesn't come into play until after matching a proxy defined route. If a route matches an MVC controller instead then MVC executes.

I think the main point to get across here is that YARP is not the one doing the routing, YARP is registering the given proxy routes into the main ASP.NET Core route table. Those routes can be specific or general, and they interop with all of the other routes registered where the most specific match wins.

[knight1219]

So for example if I had:
Proxy api: /api/health - local route, /api/health - proxied route
Service1 api: /api/health - local route, /api/service1/health - proxied route
Service2 api: /api/health - local route, /api/service2/health - proxied route

Is a configuration like this possible? I would want to use the ASP.Net HealthCheck library to create healthcheck endpoints that could then be proxied by YARP. As long as i can call "/api/health" and it run on my proxy's controller method, everything would be good.

[Tratcher]

/api/heath, /api/service1/heath, and /api/service2/health are the routes registered in the proxy? That should work fine. You'd use a transform to strip out the service name before proxying the request.
https://microsoft.github.io/reverse-proxy/articles/transforms.html#request-transforms

[knight1219]

"ReverseProxy": {
    "Routes": [
      {
        "RouteId": "gateway",
        "ClusterId": "cluster1",
        "Match": {
          "Path": "/api/gateway/{**catch-all}"
        },
        "Transforms": [
          {
            "PathRemovePrefix": "/api/gateway"
          }
        ]
      },
      {
        "RouteId": "api1",
        "ClusterId": "cluster2",
        "Match": {
          "Path": "/api/api1/{**catch-all}"
        },
        "Transforms": [
          {
            "PathRemovePrefix": "/api/api1"
          }
        ]
      },
      {
        "RouteId": "api2",
        "ClusterId": "cluster3",
        "Match": {
          "Path": "/api/api2/{**catch-all}"
        },
        "Transforms": [
          {
            "PathRemovePrefix": "/api/api2"
          }
        ]
      }
    ],
    "Clusters": {
      "cluster1": {
        "Destinations": {
          "cluster1/destination1": {
            "Address": "https://localhost:44391"
          }
        }
      },
      "cluster2": {
        "Destinations": {
          "cluster2/destination1": {
            "Address": "https://localhost:44348/api/"
          }
        }
      },
      "cluster3": {
        "Destinations": {
          "cluster3/destination1": {
            "Address": "https://localhost:44361/api/"
          }
        }
      }
    }
  }

NOTE: localhost:44391 is the YARP proxy project.

So by using the transforms this seems to work, except for the gateway route and cluster1 cluster. i would like to put controllers behind the /gateway/ like my other routes, but i think there may be some conflicting routing happening behind the scenes somewhere now. By adding that gateway route, it completely breaks the ability to even go to https://localhost:44391/weatherforecast, i get a 404. Now, i know i can add /gateway to my controllers directly but when i bring in stuff like the HealthCheck apis OR any outside common controllers thats all of my services need... then it becomes an issue it seems.

[knight1219]

Wow. I'm dumb. I commented out the endpoints.MapControllers(); and services.AddControllers(); in the proxy project, no wonder i was getting a 404 on the controllers. The above config works amazing and is EXACTLY what i need.

All of my controllers live at api/weatherforecast on both my services AND proxy, which is how it would be for things like ping/healthcheck apis.

[clausndk]

Just to close this off from my end, I ended up using the Order: -1 approach, so that routes from cluster have higher priority then local controllers.

[AntonC9018]

I'm observing quite the opposite. I have both api controllers and razor views in the app, while using a proxy for everything else. I'm adding the proxy last, both in the services configuration and in the endpoints configuration. If the order of the proxy is not set, my controllers are never hit, with all requests going to the proxy. Even if the remote endpoint that the proxy ends up hitting gives back 404, no fallback mechanism is observed: the local controllers are never hit (is there a setting for this?). If I set the order to some arbitrarily large value, the razor views do get hit, but none of the api controllers do, however large the order of the proxy route may be. To be clear, I am using the catch-all thing in the route.
So logically it would seem that the controllers are added after the proxy, but they in fact aren't.

I'm pretty new to ASP.NET Core in general, but from my perspective I'm not missing anything. Please help.

// Add services to the container.
builder.Services.AddControllersWithViews(options =>
{
});

if (isDevelopment)
{
    var proxy = builder.Services.AddReverseProxy();
    proxy.LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"));
}
// ...
app.UseHttpsRedirection();
app.UseRouting();

app.UseEndpoints(endpoints =>
{
    endpoints.MapControllerRoute(
        name: "default",
        pattern: "{controller=Home}/{action=Index}/{id?}");

    endpoints.MapControllers();

    if (isDevelopment)
        endpoints.MapReverseProxy();
});

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning",

      // The proxy spits out too much stuff.
      "Yarp.ReverseProxy.Forwarder.HttpForwarder": "Information"
    }
  },
  "ReverseProxy": {
    "Routes": {
      "ViteAssets" : {
        "ClusterId": "vite",
        
        // Apparently the proxy always ends up first in the stack,
        // independent of when I actually add / use it.
        // Setting order to a large value doesn't help either.
         "Order": 100000000,
        
        "Match": {
          "Path": "/{**catch-all}"
        }
      }
    },
    "Clusters": {
      "vite": {
        "Destinations": {
          "vite/destination1": {
            "Address": "https://localhost:5173"
          }
        }
      }
    }
  }
}

And here's an example of what the logs say when trying to access an api controller:

info: Yarp.ReverseProxy.Forwarder.HttpForwarder[9]
      Proxying to https://localhost:5173/api/stuff HTTP/2 RequestVersionOrLower no-streaming
info: Yarp.ReverseProxy.Forwarder.HttpForwarder[56]
      Received HTTP/2.0 response 404.

Removing MapReverseProxy() makes it work as expected, that is, the api controllers now would get hit.