Skip to content

Latest commit

 

History

History
133 lines (89 loc) · 9.63 KB

dev-design.md

File metadata and controls

133 lines (89 loc) · 9.63 KB
Raw

PenButton Event Detailed Design

Introduction:

Some digital pens and pencils can not only provide input through their interaction with a digitizer but can also pair with a device so that additional signals can be received when a button is pressed. In some cases, the signal may relate to the pen or pencil but come from another source, e.g. a charger may send a signal that the pen or pencil has been docked or undocked. Native applications can use these signals to customize their behavior, but no corresponding events are available to web applications. Native applications use these signals in a variety of ways; here are some inspirational use cases: Clicking the button on a Surface Pro pen can advance to the next slide in a slide show Double tapping the side of an Apple Pencil can switch drawing tools Removing a pen from the dock on a Surface Hub prompts the user to enter the whiteboard app

Events:

penbuttonclick, penbuttondblclick, penbuttonpressandhold events:

A PenButtonEvent named for the gesture the user has performed with the pen button must be fired to the active window when the pen button gesture is detected. Corresponding down/up events are not proposed because of hardware limitations, e.g. the Microsoft Surface Pen produces a signal describing the gesture, not the state transitions of the actual button. This event is derived from the UI event.

The pointerId property identifies the pen which triggered the event. If the user agent supports providing a stable pointerId for PointerEvents generated by the pen, that pointerId must also be used for PenButtonEvents. For devices that don't support stable pointerIds, the pointerId must be set to 0.

The button property indicates which button has been pressed. The numeric value of the button indicating which button triggered the event is taken from the recommendations made in the existing PointerEvent specification.

pendockchange event

A PenDockEvent must be fired to the active window when a pen is returned to, or removed from, a dock on hardware which supports detecting these transitions of the pen.

The pointerId property identifies the pen which triggered the event. If the user agent supports providing a stable pointerId for PointerEvents generated by the pen, that pointerId must also be used for PenDockChangeEvents. For devices that don't support stable pointerIds, the pointerId must be set to 0.

The docked property identifies whether the pen has transitioned to a docked state (the docked property is true) or an undocked state (the docked property is false).

PenButton Events Components:

PenButtonEvent:

This class is the native implementation of the JS PenButtonEvents.

IDL file:

pen_button_event_init.idl:

[Exposed=Window]
dictionary PenButtonEventInit : UIEventInit {
    long pointerId;
    long button;
};

pen_button_event.idl:

[Exposed=Window]
[Constructor(DOMString type, optional PenButtonEventInit eventInitDict)]
interface PenButtonEvent : UIEvent {
    readonly attribute long pointerId;
    readonly attribute long button;
};

PenButton event details:

PenButtonEvent listeners have side-effects, in that they override OS behavior in Windows and replace it with some app specific behavior. The signal for whether the web app wants to override the OS behavior is whether a PenButtonEvent listener is registered. Additionally, the registered listeners only have an impact when the document is the active document. When an active document/frame loses focus in the renderer process, this will trigger a message to browser process to unregister the pen button event in the OS if the current active Frame’s PenButtonEventManager doesn’t have any PenButtonEvent listeners registered As a result, we communicate the number of PenButtonEvent listeners (by type) when a frame receives focus, and while the frame is focused we also transmit changes from add/removeEventListener calls.

PenButtonEventProcessor:

The lifetime of this component is managed by HWNDMessageHandler (This receives the Windows specific messages from the WndProc. It then creates an ui::Event object from the wm_* messages and dispatches to the aura::WindowTreeHost to process it). This component has the following responsibilities in the browser process: Processes the focus change event from browser and renderer processes. Processes the native window hotkey message for pen and dispatches it to the delegate for further processing in the browser process.

PenButtonEventProcessor API details:

OnFocusChanged:

This API is responsible for handling focus change related messages. If there is a focus loss message and pen button keys are registered, then it will unregister them by calling the system APIs.

OnPenButtonEvent:

This API is responsible for handling WM hotkey pen events from the OS. It filters the messages (based on VKey code such as VK_F18, VK_F19, VK_F20) and dispatches an ui::PenButtonEvent message to the delegate if it satisfies the filtering criteria.

ui::PenButtonEvent: This creates a PenButtonEvent from the native windows hotkey event for pen. It derives from ui::Event which contains states of the modifier keys, type of events, timestamp etc WebInputEvent in renderer process: This is a platform agnostic input event object in blink that is also used to fire JS events. This will have a PenButtonEvent that will contain the pointerId, button, timestamp etc

PenButtonEventManager:

The lifetime of the component is managed by the Page.

This component has the following responsibilities in the renderer process:

  1. Registered as a focus change observer in FocusController that manages focus changes between different documents, elements etc.
  2. Fires PenButtonEvent to JS when there is active listener registered for it.
  3. Sends IPC message to browser process to register the hotkey when an active document registers for this event.
  4. Sends IPC message to browser process to unregister the hotkey if the active document that registered for the event loses focus or removes the event listener.

PenButtonEventManager API details:

EventListenerAdded:

This will trigger a RegisterPenButton message to browser process if the Frame’s PenButtonEventManager has a PenButtonEvent listener registered. It will also keep track of the state of the event listener when script adds any PenButtonEvent listener for the document.

EventListenerRemoved:

This will trigger an UnRegisterPenButton message to browser process if the Frame’s PenButtonEventManager doesn’t have any PenButtonEvent listener registered. It will also keep track of the state of the event listener when script removes any PenButtonEvent listener for the document.

FocusedFrameChanged:

This API is responsible for handling focus change related messages when the active document/frame loses focus in the renderer process. This will trigger an UnregisterPenButton message to browser process if the current active Frame’s PenButtonEventManager doesn’t have any PenButtonEvent listeners registered.

Event Flow:

Browser-Renderer process communication:

browser_renderer_comm

  1. HWNDMessageHandler gets the WM_HOTKEY message from the WndProc.

  2. Creates the ui::PenButtonEvent after filtering the hotkeys for PenButton.

  3. Calls PenButtonEventProcessor::OnPenButtonEvent that sends this event to EventSource::SendEventToSink to process the message in the browser process. Event is dispatched via EventDispatcher RenderWidgetHostViewAura receives this new PenButtonEvent and forwards it to the RenderWidgetHostViewEventHandler.

  4. RenderWidgetHostViewEventHandler creates a native PenButtonEvent from the ui::PenButtonEvent deriving from blink::WebPenButton. This blink::WebPenButton will derive from blink::WebInputEvent that will be used in the blink to fire JS event. RenderWidgetHostImpl receives this event and IPCs this message to renderer process via input router

Renderer-Browser process communication:

browser_renderer_comm

  1. WidgetInputHandler receives the IPC message from the browser process and posts it to the main thread to be processed by the RenderWidget.

  2. RenderWidget receives the PenButtonEvent from the MainThread queue and sends it to RenderWidgetInputHandler for further processing

  3. Then the message flows through the below route and eventually reaches the PenButtonEventManager that fires the PenButton JS event: RenderWidgetInputHandler->WebViewFrameWidget->WebViewImpl->PageWidgetDelegate->EventHandler->PenButtonEventManager.

Adding and Removing Event Listener in JS:

When an event listener is added for the PenButtonEvent, PenButtonEventManager sends an IPC message to browser process to register for the hotkey corresponding to the event. EventTarget has AddEventListenerInternal and RemoveEventListenerInternal that gets called when JS adds/removes an event listener. This then calls PenButtonEventManager to send the appropriate IPC message to browser process to register/unregister the hotkey IPC message is sent via RenderWidget. In browser process, RenderWidgetHostImpl receives the IPC message (WidgetHostMsg_PenButtonEvent*) which then goes through RenderWidgetHostViewAura and then to HWNDMessageHandler that sends it to PenButtonEventProcessor to register/unregister hotkeys by calling the system APIs.

Question:

HWNDMessageHandler is registered as an observer in InputMethodBase so we can probably notify about PenButton event through an observer API?

What to do when no frame has focus, e.g. the OmniBox is focused? Should we eat the shell’s events or should we unregister until the frame has focus again?