Discussion about Console.Write and ANSI based terminal

[Evangelink]

@thomhurst @bradwilson @OsirisTerje @rprouse @nohwnd @MarcoRossignoli @Youssef1313

I would like to initiate a discussion about what should be the platform behavior toward Console.Write(Line) calls. As of today, MTP uses a terminal reporter that is ANSI aware and will use a dynamic UX updating the text.

We have issues like #3491 and #4425 around the behavior of Console.Write(Line) calls when the ANSI terminal is active.

We basically have 2 paths forward:

1. Replace Console.Out

2. Keep current behavior

Replace Console.Out

In this scenario, MTP would replace Console.Out as early as possible and would capture all calls to the console to redistribute them into our terminal handling.

Pros:

• Users don't have to think about it.

Cons:

• In MSTest, we are replacing Console.Out and this can be causing weird/unexpected behaviors for users as there is a potential delay or change in the string output by the user.

Keep current behavior

In this scenario, MTP doesn't replace Console.Out, any call to Console.Write(Line) is printed directly with the risk of the line being overwritten. When the user wants to onboard properly to the terminal they should write using our output service (meaning the test framework should somehow expose it).

Pros:

• No modification to "normal" Console behavior

Cons:

• Users have to be aware about the difference between the APIs.

Personal opinion

I would personally favor option 2, although it requires some "teaching" to users it's properly decoupling the built-in behavior of System APIs and the behavior of the APIs provided by the platform. I think it's also easier to explain and reason about.

[Youssef1313]

I think it can be possible for things to work as user would expect without changing Console.Out. The runtime implementation of Console (at least the default Console.Out, which is what we care about most) uses SyncTextWriter which is already synchronized on Console.Out. So having proper synchronization with what we do and what user do feels possible to me (we already did it for the batching mode of terminal and are locking on Console.Out). I feel like the whole issue is just that we are missing more lock (Console.Out) in some places.

Let's discuss offline.

[nohwnd]

This should probably also mention the plan #4425 (comment) to lock on the Console.Out instance.

This would allow users to use console.writeLine, without any change, and it would synchronize with our ansi outputter, so overwriting of text would be solved, and if we do stuff right then mixing of output (e.g. part of sentence from us, and part of output from user) should also be solved.

One thing that will not be solved by that, and that is also not mentioned is associating the output to test results. In testfx we replace console.out with a async task aware outputter, so all output from tests can be captured, and associated to the correct test. Even if the tests are running in parallel. This prevents mixing of outputs between tests, and ensures that in UI you will see only output of the current test, instead of a slice of all outputs that happened during this particular test lifetime.

I don't think the solutions are exclusive to each other. Imho extensions should try to write using services we provide. We should lock on console.out, and user should have option to capture the outputs per-test (which does not prevent us from also capturing all outputs in their time-order).

[Evangelink]

I think the association to result should be handled by the test framework because we have no idea where the write is coming from.

[nohwnd]

Yes, that is true, and that is how it is currently. I mention it because you mention it above in cons of replacing console.out, and becaue from the point of view of testing framework user it does not matter much if it is replaced on testing platform or testing framework level.

I agree with youssef above that we should first fully explore locking on console.out.

[thomhurst]

I'd vote for #1. I always prefer the option of keeping things simple for the end user. So if complexities can be abstracted away from them, then I always prefer to go down that route.

I've seen lots of test suites, and I swear 99% of them use Console.WriteLine.

I know xUnit actually exposes its own output writer, but I don't like that. As you say, it's an extra complexity you have to know about, and also you have to pass it around to different places, whereas Console is easy because it's static.

In MSTest, we are replacing Console.Out and this can be causing weird/unexpected behaviors for users as there is a potential delay or change in the string output by the user.

TUnit replaces Console.Out too, for the purpose of like you said, associating the output to the specific test. I think that's fine to leave that up to the test framework. Just need to make sure that you replacing the console, and then me replacing the console, doesn't conflict and cause weird problems. I still forward my console calls to the original console, but also write it to my test context object too.

[bradwilson]

I've seen lots of test suites, and I swear 99% of them use Console.WriteLine.

My experience is the opposite: Console.WriteLine tends to be used for temporary debugging.

I know xUnit actually exposes its own output writer, but I don't like that.

v3 also offers capturing of console for people who are, as you say, 99% using Console.WriteLine. For backward compatibility reasons this is off by default.