MRG: Plot NIRS data between source and detector #7620

rob-luke · 2020-04-17T04:38:25Z

Reference issue

Discussed in #7057 and mne-tools/mne-nirs#59

What does this implement/fix?

Add option to plot line between source detector pairs in plot_alignment .

Additional details

This PR originally contained code to plot data over the head, but this has been moved to MNE-NIRS

rob-luke · 2020-04-17T07:37:47Z

I don't know why this is throwing an error. It runs on my computer.

rob-luke · 2020-04-17T08:06:48Z

Here is an example of what the code currently looks like on my computer (its a different sensor montage that is meant to cover the inferior parietal lobe and superior temporal gyrus, the data/color is currently randn). I am happy with the overall look. Now I need to make the code nice.

agramfort · 2020-04-17T08:20:15Z

@GuillaumeFavelier can you review this one ? thx

rob-luke · 2020-04-17T08:29:31Z

Thanks. Any review is appreciated. Its early days but any push in the right direction is appreciated.

agramfort · 2020-04-17T08:35:49Z

At first glance, we need to decide if this code will only work with pyvista or if we use the _Renderer abstraction to be able to work with mayavi and future notebook backend. I would prefer you use the _Renderer

…

rob-luke · 2020-04-17T08:53:18Z

great. I will look in to _Renderer then.

rob-luke · 2020-04-17T10:52:19Z

Ok using _Renderer it now looks like (again a different dataset)...

which I am ok with

codecov · 2020-04-17T13:15:03Z

Codecov Report

Merging #7620 into master will increase coverage by 0.03%.
The diff coverage is 93.93%.

@@            Coverage Diff             @@
##           master    #7620      +/-   ##
==========================================
+ Coverage   90.08%   90.11%   +0.03%     
==========================================
  Files         454      455       +1     
  Lines       83551    83902     +351     
  Branches    13228    13274      +46     
==========================================
+ Hits        75266    75611     +345     
- Misses       5420     5422       +2     
- Partials     2865     2869       +4

larsoner · 2020-04-17T13:40:06Z

Nice!

Might be worth setting sphinx_gallery_thumbnail_number = 16 in the example so that the brain is the thumbnail?

rob-luke · 2020-04-17T15:14:05Z

People do love a brain picture :)

GuillaumeFavelier · 2020-04-17T15:23:08Z

Very cool plot and usage of _Renderer here ^^

larsoner · 2020-04-17T15:48:35Z

I had another API thought. I'm not sure we even need a separate function. For MEG we have meg=('sensors', 'helmet'). We should probably just have (by default) fnirs=('sensors', 'pairs'), where 'pairs' means "draw a line between them". Then we don't even need a new keyword argument if we can live with users not getting to choose the colormap.

larsoner · 2020-04-17T15:48:54Z

(I mean this should all be done in plot_alignment)

larsoner · 2020-04-17T13:41:23Z

mne/viz/nirs.py

+        If True, plot the digitization points; 'fiducials' to plot fiducial
+        points only.
+    ecog : bool
+        If True (default), show ECoG sensors.


I would get rid of all of these other channel type options and only show NIRS channels. If people want NIRS+other channels they should just use plot_alignment and not see the source-detector pairs

Ignore this if you agree with my API proposal below about just using plot_alignment directly

larsoner · 2020-04-17T13:41:58Z

mne/viz/nirs.py

+    - Transform matrix (keywords ``trans``, ``meg`` and ``mri_fiducials``),
+    - BEM surfaces (keywords ``bem`` and ``surfaces``),
+    - sphere conductor model (keywords ``bem`` and ``surfaces``) and
+    - source space (keywords ``surfaces`` and ``src``).


I would get rid of these notes and just tell people to

For more information see :func:`mne.viz.plot_alignment`.

larsoner · 2020-04-17T13:42:37Z

mne/viz/nirs.py

+            # Otherwise use blue to red and ensure zero sits at white
+            vmin = -1. * np.max(np.abs(data))
+            vmax = np.max(np.abs(data))
+            cmap = 'bwr'


MNE standard diverging/bidirectional map where zero is meaningful is RdBu_r

rob-luke · 2020-04-17T23:00:28Z

Thanks for the feedback.

I mean this should all be done in plot_alignment

How does one set the values/colours of sensors in plot_alignment? In this PR I’m passing in a data array, I can’t see an equivalent in plot alignment?

I wanted the ability to set the line colour to represent the output of analysis. Either hrf or peak values etc

if we can live with users not getting to choose the colormap.

I’m a bit of a visualisation nerd. And would prefer being able to set colormaps. If we move to plot alignment can I add a keyword?

larsoner · 2020-04-18T01:26:03Z

How does one set the values/colours of sensors in plot_alignment? In this PR I’m passing in a data array, I can’t see an equivalent in plot alignment? ... I wanted the ability to set the line colour to represent the output of analysis. Either hrf or peak values etc

You can't currently. Indeed if you want to be able to "plot values" or something in 3D then a separate function would be better.

To me there are two separate use cases:

Showing the sensor arrangement relative to the head (i.e., geometry / configuration)
Showing some data values (i.e., some activation)

To me the sensor pairings with tubes between them falls at least partly under (1), and thus would be well suited to being included in plot_alignment at least to start.

Showing data doesn't really fit in the scope of plot_alignment, so a separate function would be better. Currently for (2) you have topo, topomap, and plot_evoked_field (though it's not used that much). It sounds like you're proposing a third function that is akin to plot_alignment in that it plots a brain/head/whatever plus sensor locations, but the sensor colors reflect some data values. It sounds like the first image in this ECoG example (though it's constructed by hacking the 3D transforms):

https://mne.tools/dev/auto_examples/visualization/plot_3d_to_2d.html#project-3d-electrodes-to-a-2d-snapshot

I could imagine the utility of making this sort of brain + electrode visualization work natively, rather than via some projection scheme. Clearly there is/was already a use case and need for it in ECoG, and now you're proposing something that seems quite similar in principle.

So my vote would be to first do this pairing scheme in plot_alignment. Then you don't need a colormap, since it's not showing data, right?

Then once we have that, we could build a generic sensor+data+brain plotting function. I don't think it matters much whether it's fNIRS or EEG/ECoG/MEG data (except that MEG data must have the coordinate frame changed). WDYT?

rob-luke · 2020-04-18T05:49:53Z

Sounds like a plan. My need is for use case 2. But lets knock over 1 and 2 in the same go.

So I will expand plot_alignment to allow for fnirs=('sensors', 'pairs'). Which will show the location of source-detector pairs, but not overlay any data. All pairs will have the same color.

I will create a function (generalise the one already in this PR) for generic sensor+data+brain plotting.

rob-luke · 2020-04-20T04:18:24Z

Ok plot_alignment with pairs seems to be working now and can be seen in the ci circle build doc. And looks as expected (see below). Now I will move on to the generic data display function (use case 2).

agramfort · 2020-04-20T06:54:26Z

mne/viz/nirs.py

+from . import plot_alignment
+
+
+def plot_nirs_source_detector(data, info=None, radius=0.001,


would you agree to kill this function and just add nirs support to plot_alignment?

Definitely agree to killing this function.

But I would like to add a function to overlay channel analysis over a brain.

Would you prefer I convert this PR to be only for the extension to plot_alignment and move discussion of a data overlay to another issue/PR? Or move to MNE-NIRS?

agramfort · 2020-04-20T07:19:04Z

what do you mean by "channel analysis"?

…

rob-luke · 2020-04-20T07:37:03Z

Sorry, I should be clearer with my messages.

In NIRS literature a channel is between a light source and detector. These have been marked as lines in the figures above. For each channel we analyse the measurement and extract a metric (peak value, hrf analysis, etc) and usually call this "channel level analysis". I guess its equivalent to scalp level analysis in EEG.

Plotting the output of the analysis for a source-detector pair is traditionally done by drawing a line between the source and detector and setting the line equal to the extracted metric (an example from another software is shown here). This is what I would like to create / that's what the plot_nirs_source_detector does.

agramfort · 2020-04-20T07:51:16Z

so the color of the tube or its size would match a scalar quantity?

…

rob-luke · 2020-04-20T11:19:38Z

so the color of the tube or its size would match a scalar quantity?

Various tools do it differently. My preferred approach is colour represents a scalar quantity. Size remains constant. And you have dashed or solid lines to represent a Boolean (usually to represent if a significant response was observed).

I think it will be difficult to make different line types with Renderer. So maybe tube size could represent significance. Small tube size for no significant response. Large tube size for significant response.

One of the reasons I like the approach of showing the source detectors as a line/tube is that you can see how far apart the optodes are. Optode separation distance is proportional to how deep in the brain the photons will reach, and what brain regions the pair is sensitive too (larger separation=deeper). So you gain additional insight to what’s available in the topomap.

Also happy to move this to MNE-NIRS if that’s more appropriate. Or happy to discuss alternative visualisation approaches. It’s always good to hear another opinion from someone outside the nirs field.

agramfort · 2020-04-20T12:27:22Z

I would start by putting this in mne-nirs for now. It will allow you to iterate faster and release at a much faster rate

…

rob-luke · 2020-04-21T05:28:19Z

The code is passing on PyVista but failing on Mayavi (CI fail). Could this be a bug?

If I change this line to below the tests pass, but its uglier.

def tube(self, origin, destination, radius=0.001, color='white',
         scalars=None, vmin=None, vmax=None, colormap='RdBu',
         normalized_colormap=False, reverse_lut=False):
    color = _check_color(color)
    origin = np.atleast_2d(origin)
    destination = np.atleast_2d(destination)
    if scalars is None:
        for idx in range(origin.shape[0]):
            surface = self.mlab.plot3d([origin[idx, 0], destination[idx, 0]],
                                       [origin[idx, 1], destination[idx, 1]],
                                       [origin[idx, 2], destination[idx, 2]],
                                       tube_radius=radius,
                                       color=color,
                                       figure=self.fig)

rob-luke · 2020-04-21T05:45:10Z

The other code has been moved to mne-tools/mne-nirs#70 as suggested by agramfort

agramfort · 2020-04-21T07:09:19Z

@GuillaumeFavelier can you have a look?

…

mne/viz/_3d.py

rob-luke · 2020-04-21T10:43:20Z

I can’t tell if this single CI failure due to segmentation fault is related to this PR. I recall seeing similar faults on other commits. But it does occur near the test I altered.

rob-luke · 2020-04-21T10:45:24Z

@larsoner @agramfort @GuillaumeFavelier could you please review. I have changed the status to mrg as I am happy with the code and scope of work now.

GuillaumeFavelier

I can’t tell if this single CI failure due to segmentation fault is related to this PR

I saw Travis failing on master for this exact same test. I don't think it's related to this PR.

LGTM @rob-luke

larsoner

Just a few minor comments.

Is it meant for there to be some tubes going not between sensors? It looks like there are a few (e.g., in the lower left)

mne/viz/_3d.py

mne/viz/backends/_pysurfer_mayavi.py

Co-Authored-By: Eric Larson <[email protected]>

…ial.

rob-luke · 2020-04-21T23:25:29Z

Is it meant for there to be some tubes going not between sensors?

Yes, these are the short channels. I have added an additional line to the tutorial to mention what they are.

tutorials/preprocessing/plot_70_fnirs_processing.py

larsoner · 2020-04-22T14:28:40Z

tutorials/preprocessing/plot_70_fnirs_processing.py

+fig = mne.viz.plot_alignment(raw_intensity.info, show_axes=True,
+                             subject='fsaverage',
+                             trans='fsaverage', surfaces=['brain'],
+                             fnirs=['channels', 'pairs'],


You changed the default True to mean ('pairs',) but then use ['channels', 'pairs'] in the example. Would ('channels', 'pairs') might actually be the best default for plot_alignment? Or do you indeed want to do something non-default in this example? Sorry for the back and forth on this one...

Also would this look better with surfaces=['brain', 'head']? Might look cooler?

I leave both channels and pairs in here as they are described above and I see this tutorial as a tool to show people the nirs processing options, then they can choose the option they prefer. I think pairs is still the best default. Also I prefer brain alone without head here.

larsoner · 2020-04-23T00:23:18Z

Thanks @rob-luke !

rob-luke force-pushed the nirs_sd_plot branch from 01474c1 to 6bce5d9 Compare April 17, 2020 04:39

rob-luke force-pushed the nirs_sd_plot branch from 8eae7c7 to 44e5c31 Compare April 17, 2020 10:21

rob-luke force-pushed the nirs_sd_plot branch from eacf389 to 7006366 Compare April 17, 2020 11:16

GuillaumeFavelier added the VIZ label Apr 17, 2020

larsoner reviewed Apr 17, 2020

View reviewed changes

rob-luke force-pushed the nirs_sd_plot branch from 5f905ef to e6947e8 Compare April 19, 2020 12:07

agramfort reviewed Apr 20, 2020

View reviewed changes

Add pairs option to plot_alignment

fdbca4a

rob-luke force-pushed the nirs_sd_plot branch from a6863f5 to fdbca4a Compare April 21, 2020 02:55

rob-luke mentioned this pull request Apr 21, 2020

Add nirs source detector data plot mne-tools/mne-nirs#70

Merged

GuillaumeFavelier reviewed Apr 21, 2020

View reviewed changes

mne/viz/_3d.py Show resolved Hide resolved

Plot mayavi tubes one by one

99495ed

GuillaumeFavelier mentioned this pull request Apr 21, 2020

Tube in _Renderer #7644

Closed

rob-luke changed the title ~~WIP: Plot NIRS data between source and detector~~ MRG: Plot NIRS data between source and detector Apr 21, 2020

Change tutorial image to brain figure

c2b364f

GuillaumeFavelier approved these changes Apr 21, 2020

View reviewed changes

larsoner reviewed Apr 21, 2020

View reviewed changes

mne/viz/_3d.py Outdated Show resolved Hide resolved

mne/viz/_3d.py Outdated Show resolved Hide resolved

mne/viz/_3d.py Show resolved Hide resolved

mne/viz/backends/_pysurfer_mayavi.py Show resolved Hide resolved

rob-luke and others added 2 commits April 22, 2020 09:15

Update mne/viz/_3d.py

ee4c7fa

Co-Authored-By: Eric Larson <[email protected]>

Fnirs plot_alignment tweaks. Use _check_option, add TODO, tweak tutor…

04c5da2

…ial.

Make pairs the default option for fnirs plot alignment

c56552f

larsoner reviewed Apr 22, 2020

View reviewed changes

Update plot_70_fnirs_processing.py

24ea3a2

larsoner merged commit 1d44c29 into mne-tools:master Apr 23, 2020

GuillaumeFavelier mentioned this pull request Apr 24, 2020

TST: Check Travis #7673

Closed

		from . import plot_alignment


		def plot_nirs_source_detector(data, info=None, radius=0.001,

MRG: Plot NIRS data between source and detector #7620

MRG: Plot NIRS data between source and detector #7620

Conversation

rob-luke commented Apr 17, 2020 • edited Loading

Reference issue

What does this implement/fix?

Additional details

rob-luke commented Apr 17, 2020

rob-luke commented Apr 17, 2020 • edited Loading

agramfort commented Apr 17, 2020

rob-luke commented Apr 17, 2020

agramfort commented Apr 17, 2020 via email

rob-luke commented Apr 17, 2020

rob-luke commented Apr 17, 2020 • edited Loading

codecov bot commented Apr 17, 2020 • edited Loading

Codecov Report

larsoner commented Apr 17, 2020

rob-luke commented Apr 17, 2020

GuillaumeFavelier commented Apr 17, 2020

larsoner commented Apr 17, 2020

larsoner commented Apr 17, 2020

larsoner Apr 17, 2020

Choose a reason for hiding this comment

larsoner Apr 17, 2020

Choose a reason for hiding this comment

larsoner Apr 17, 2020

Choose a reason for hiding this comment

larsoner Apr 17, 2020

Choose a reason for hiding this comment

rob-luke commented Apr 17, 2020

larsoner commented Apr 18, 2020

rob-luke commented Apr 18, 2020

rob-luke commented Apr 20, 2020

agramfort Apr 20, 2020

Choose a reason for hiding this comment

rob-luke Apr 20, 2020 • edited Loading

Choose a reason for hiding this comment

agramfort commented Apr 20, 2020 via email

rob-luke commented Apr 20, 2020 • edited Loading

agramfort commented Apr 20, 2020 via email

rob-luke commented Apr 20, 2020

agramfort commented Apr 20, 2020 via email

rob-luke commented Apr 21, 2020

rob-luke commented Apr 21, 2020

agramfort commented Apr 21, 2020 via email

rob-luke commented Apr 21, 2020

rob-luke commented Apr 21, 2020

GuillaumeFavelier left a comment

Choose a reason for hiding this comment

larsoner left a comment

Choose a reason for hiding this comment

rob-luke commented Apr 21, 2020

larsoner Apr 22, 2020

Choose a reason for hiding this comment

rob-luke Apr 22, 2020

Choose a reason for hiding this comment

larsoner commented Apr 23, 2020

rob-luke commented Apr 17, 2020 •

edited

Loading

rob-luke commented Apr 17, 2020 •

edited

Loading

rob-luke commented Apr 17, 2020 •

edited

Loading

codecov bot commented Apr 17, 2020 •

edited

Loading

rob-luke Apr 20, 2020 •

edited

Loading

rob-luke commented Apr 20, 2020 •

edited

Loading