Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Locked Figure Labels] Util function to generate spoken math + use it within Locked Point aria labels #1839

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

nishasy
Copy link
Contributor

@nishasy nishasy commented Nov 8, 2024

Summary:

When auto-generating the aria labels for locked figures, we want it to use
words as if they were spoken rather than math expressions that might be
read incorrectly by the screen reader.

  • Create a utility function using the MathJax speech engine that converts
    math details (TeX denoted by the $...$ blocks) to spoken words.
  • Use this utility within LockedPointSettings. (Async + useEffect because
    the speech engine is async. I had to use mocks in the tests for this.)

If this approach is given the okay, I'll go ahead and apply it to the
other locked figures as well in a following PR.

Issue: https://khanacademy.atlassian.net/browse/LEMS-2548

Test plan:

yarn jest packages/perseus-editor/src/widgets/interactive-graph-editor/locked-figures/util.test.ts
yarn jest packages/perseus-editor/src/widgets/interactive-graph-editor/locked-figures/locked-point-settings.test.tsx

Storybook

  • http://localhost:6006/iframe.html?args=&id=perseuseditor-widgets-interactive-graph--mafs-with-locked-figure-labels-all-flags&viewMode=story
  • Open the locked point settings
  • Change the visible label to have a mix of TeX (with $...$) and non-TeX
  • Press the "Auto-generate" button
  • Confirm that the input changes to include spoken math words for the Tex
  • Also try this with no labels and with multiple labels
image

Copy link
Contributor

github-actions bot commented Nov 8, 2024

npm Snapshot: Published

Good news!! We've packaged up the latest commit from this PR (096a79d) and published it to npm. You
can install it using the tag PR1839.

Example:

yarn add @khanacademy/perseus@PR1839

If you are working in Khan Academy's webapp, you can run:

./dev/tools/bump_perseus_version.sh -t PR1839

Copy link
Contributor

github-actions bot commented Nov 8, 2024

Size Change: +413 kB (+48.13%) 🚨

Total Size: 1.27 MB

Filename Size Change
packages/perseus-editor/dist/es/index.js 696 kB +413 kB (+146.21%) 🆘
ℹ️ View Unchanged
Filename Size
packages/kas/dist/es/index.js 38.8 kB
packages/keypad-context/dist/es/index.js 760 B
packages/kmath/dist/es/index.js 4.27 kB
packages/math-input/dist/es/index.js 77.8 kB
packages/math-input/dist/es/strings.js 1.79 kB
packages/perseus-core/dist/es/index.js 1.48 kB
packages/perseus-linter/dist/es/index.js 22.2 kB
packages/perseus/dist/es/index.js 409 kB
packages/perseus/dist/es/strings.js 3.54 kB
packages/pure-markdown/dist/es/index.js 3.66 kB
packages/simple-markdown/dist/es/index.js 12.4 kB

compressed-size-action

@nishasy nishasy marked this pull request as ready for review November 8, 2024 01:55
@khan-actions-bot
Copy link
Contributor

Gerald

Required Reviewers
  • @Khan/perseus for changes to .changeset/loud-emus-behave.md, packages/perseus-editor/src/widgets/interactive-graph-editor/locked-figures/locked-point-settings.test.tsx, packages/perseus-editor/src/widgets/interactive-graph-editor/locked-figures/locked-point-settings.tsx, packages/perseus-editor/src/widgets/interactive-graph-editor/locked-figures/util.test.ts, packages/perseus-editor/src/widgets/interactive-graph-editor/locked-figures/util.ts

Don't want to be involved in this pull request? Comment #removeme and we won't notify you of further changes.

Copy link
Member

@benchristel benchristel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool! I left some suggestions inline for making the TeX parsing more robust.

// of the math inside the $ signs.
// Example: "Circle with radius $\frac{1}{2}$" -> "Circle with radius one half"
const convertedSpeech = mathString.replace(
/\$(.*?)\$/g,
Copy link
Member

@benchristel benchristel Nov 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This way of identifying TeX isn't going to work in all cases, because the TeX itself can contain dollar signs (either escaped, or in subexpressions).

For example: This sandwich costs $\$50$ and $\text{$t$ seconds}$

I might recommend using @khanacademy/simple-markdown (see perseus-markdown.tsx) to parse the string. You'll probably want to configure a separate parser instance in order to get TeX to render as speech. You could also use the parsing algorithm from mathMatcher(), defined in pure-markdown/src/index.ts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants