added pointer events to HTMLMesh

[jrjdavidson]

Related issue: #None

Description

Added pointer events to the HTMLMesh

[mrdoob]

I think we can probably just replace mousemove with pointermove and so on?

I don't remember why I didn't just use pointer events...

[jrjdavidson]

I think it’s important to trigger them all because they’re being dispatched artificially(I.e. not by an actual mouse). Mouse events won’t trigger if we only have pointers there.. might be wrong!

…

On Tue, 21 Jun 2022 at 16:00, mrdoob ***@***.***> wrote: I think we can probably just replace mousemove with pointermove and so on? I don't remember why I didn't just use pointermove... — Reply to this email directly, view it on GitHub <#24259 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALFBMN45OIOYNSYTSUQTBMDVQE437ANCNFSM5ZKUB67A> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mrdoob]

Pointer Events are triggered by mouses, fingers and pencils 🤔

Do you mind replacing them for now and see if it still works for you?

[jrjdavidson]

Pointer Events are triggered by mouses, fingers and pencils 🤔

Correct, but in this case we are triggering the pointer event with a line of code, and not a mouse, finger or pencil. Any DOM Element that has a mouse event only (and not a pointer event) will not be triggered when selected in VR if we only fire pointer event in the code.

This might actually be the behaviour we want? Do we want mouse event to be triggered as well in VR, or only pointer events? Intuitively I think I would the latter makes sense - Pointer Events are triggered by mouses, fingers, pencils, and VR selections?

Do you mind replacing them for now and see if it still works for you?

Will do, it'll work for my case but I believe that it will ignore the cases where only mouse events are attached to DOM elements.

[Mugen87]

Maybe like so?

'select': 'pointerdown',

[jrjdavidson]

I think pointerup is more in line with the select event.

[jrjdavidson]

Ok that was wrong, according to this , you are right, it should be at the start.
However, I've run into a strange issue where the event gets fired twice, but only sometimes. I've narrowed it down to the select event triggering the callback a second time.

[jrjdavidson]

Removed click and changed select XR events to correspond to pointerup DOM events

[jrjdavidson]

I think pointerup is more in line with the select event.

[mrdoob]

That's for updating the PR.

Seems like with pointer events we lose desktop events...

Before:

https://threejs.org/examples/webxr_vr_sandbox.html

Screen.Recording.2022-06-28.at.3.48.18.PM.mov

After:

https://raw.githack.com/jrjdavidson/three.js/pointerEvents/examples/webxr_vr_sandbox.html

Screen.Recording.2022-06-28.at.3.48.45.PM.mov

[LeviPesin]

And it also seems to not work on mobile devices (and the current version before this PR works with clicking but not moving)...

[mrdoob]

Ah... I remember now...

The problem is that lil-gui is still using mouse and touch events instead of pointer events.

@georgealways any plans on moving from mouse/touch events to pointer events?

[georgealways]

I did not know they brought that API back! I'll definitely look into it—I'm sure it would make the lib a bit smaller to consolidate mouse/touch listeners.

My 2 cents: it does seem a little odd that HTMLMesh wouldn't trigger listeners unless they're specifically bound to pointer events.

[LeviPesin]

HTMLMesh now produces pointer events and they do not trigger mouse events listeners.

[georgealways]

So at quick glance, moving lil-gui to pointerevents would be non-trivial.

There's a fair amount of listeners that have subtle differences between mouse and touch. For example, if the user is on touch and the GUI has a scrollbar, a slider will wait for a few moves before it actually decides to slide. If the movement is mostly vertical, it assumes the user is trying to scroll.

It looks like there's a pointerType property that lets you distinguish between touch/mouse/pen. I guess I could consolidate the events to pointer*, and then switch on pointerType inside the handlers? I'm not really sure what that accomplishes though—maybe I need a little more context on what we're trying to achieve.

[Mugen87]

I guess I could consolidate the events to pointer*, and then switch on pointerType inside the handlers?

Exactly. This is how controls like OrbitControls do it when a distinction is required:

three.js/examples/jsm/controls/OrbitControls.js

Lines 823 to 837 in a605e57

	function onPointerMove( event ) {
	if ( scope.enabled === false ) return;
	if ( event.pointerType === 'touch' ) {
	onTouchMove( event );
	} else {
	onMouseMove( event );
	}
	}

[mrdoob]

I'm not really sure what that accomplishes though—maybe I need a little more context on what we're trying to achieve.

One benefit of Pointer events is that it has sub pixel precision. Probably not that important for lil-gui though.

We transitioned all controls from mouse/touch events to point events last year. It was a bit bumpy at first but we hadn't heard of anyone having issues with it for a while.

[georgealways]

gotcha gotcha

So I took a pass at moving the logic for the sliders to pointer*: georgealways/lil-gui@dev...pointer-events — no real effect on the bundle size, but it does make the code a lot less repetitive which is great.

There's two main blockers though:

1. All the unit tests would need to be updated to broadcast pointer events instead of mouse/touch
2. e.preventDefault() doesn't seem to work the same way for pointermove as it does for touchmove? After this change, I can trigger the "pull to refresh" gesture on iOS when using the slider.

So overall I think I'd like to move to pointer events, but not before 2 gets figured out. Appreciate any insight!

edit: lil-gui aside, I still agree w @jrjdavidson—feels a bit weird to me that HTMLMesh will no longer work with mouse events after this change.

[Mugen87]

e.preventDefault() doesn't seem to work the same way for pointermove as it does for touchmove? After this change, I can trigger the "pull to refresh" gesture on iOS when using the slider.

When I remember correctly you can fix this by specifying the CSS touch-action: none for the slider component.

[georgealways]

Thanks @Mugen87—so you're right, that fixes the pull-to-refresh gesture, but it breaks scrolls that start on a slider. Seems kinda insignificant, but it's a pretty common scenario on a phone.

Right now I'm happy with the way lil-gui deals with ambiguity between touch-scrolling and sliding. I want to use pointer events, but it's actually turning out to be pretty tough to get it working the same way.

I've tried a lot: setPointerCapture, preventing default in pointercancel, and dropping my scroll detection code entirely. My best attempt is here, and I'd definitely welcome any help! Also happy to describe the specific test cases that are breaking in more detail.

That said—even if we do get it working perfectly, I really think HTMLMesh should create both pointer* and mouse* events. Especially if it's intended for use outside of this repository. It's very unlikely that you'd be listening to both of those, and also unlikely you'd be listening to touch* exclusively. It seems much more likely that third-party devs aren't listening to pointer* at all, no?

Hope that doesn't sound like I'm making excuses 😅, I really would like to get pointer events working.

[LeviPesin]

That said—even if we do get it working perfectly, I really think HTMLMesh should create both pointer* and mouse* events.

Agree.

It seems much more likely that third-party devs aren't listening to pointer* at all, no?

I think in general they are also very widely used and are the way forward.

[Mugen87]

That said—even if we do get it working perfectly, I really think HTMLMesh should create both pointer* and mouse* events. Especially if it's intended for use outside of this repository.

I tend to vote for a PointerEvents only solution. The idea is to guide users into the right direction. "Forcing" them into future-proof concepts (in this context hardware agnostic pointer input) is better than support and tolerate "outdated" approaches. Even if that means users have to migrate their code towards pointer events if they want to use HTMLMesh.

[mrdoob]

@georgealways

Also happy to describe the specific test cases that are breaking in more detail.

Do it 👍

[Mugen87]

@jrjdavidson Are you still available for updating the PR?

I think a pure pointer-events solution for HTMLMesh does indeed not work and the policy I have asked for is just too strict. It was okay to migrate the controls exclusively to pointer events but it became clear HTMLMesh is a different matter. We never know which kind of content is used in combination with HTMLMesh so for best compatibility it makes most sense to support both pointer and mouse events (e.g. latest lil-gui is still mouse/touch only, same for tweakpane afaics). Do you think you can update the PR accordingly?

[jrjdavidson]

Hey @Mugen87 , I did maintain a fork of threejs with a slightly modified version of this PR for awhile to use in a little experiment that we ran here. I vaguely remember that having both pointer and mouse events fire caused subtle bugs where events occurred twice and I wasn't able to debug it at the app level, and ended up just remove mouse events from HTMLMesh.
I have since stopped working with VR headsets and don't have a VR headset at the moment, but could try to get my hands on one to get this working properly again. I think it would be worth adding a couple of regression tests that tests what mrdoob showed in his video above, making sure that the events fire once in XR.

On a different note- again, this is a while ago, but I remember HTMLMesh feeling like a bit of a hack, surely there is a better way to render HTML pages in XR mode by now?

[Mugen87]

I vaguely remember that having both pointer and mouse events fire caused subtle bugs where events occurred twice and I wasn't able to debug it at the app level, and ended up just remove mouse events from HTMLMesh.

That is unfortunate to hear and makes me less confident to adapt a combined support for mouse and pointer events.

On a different note- again, this is a while ago, but I remember HTMLMesh feeling like a bit of a hack, surely there is a better way to render HTML pages in XR mode by now?

I'm not sure. Event the native layers demo uses HTMLMesh for GUI rendering.

New idea:

How about we add a new ctor parameter "eventType" that allows to define what type of events HTMLMesh should register? Default is "mouse" but "pointer is also supported. Then the app can decide what event type fits to the DOM that HTMLMesh carries.

[jrjdavidson]

Ok I've been thinking about this, and again this is a while ago and I might be making it up, but I think the issue I was seeing was that the two events were called separately, instead of as one native browser ui event which fires mouse and pointer events. This means that you weren't able to use of event functions such as e.preventDefault() to prevent the mouse event from firing, and the result is that if an element had both event types registered, the behaviour would be different in and out XR.

It might be possible to intercept/modify preventDefault() on the HTMLMesh event? At the time it seemed too complicated to implement, but it might be worth the effort if there is interest? I'd have to look into it and seek some input from more knowledgeable people on the matter.. I think if that could work, it would be simpler on the app side, instead of introducing an eventType.

[Mugen87]

It might be possible to intercept/modify preventDefault() on the HTMLMesh event?

How would this look like? I fear that such logic could end up in a more complex HTMLMesh implementation with potential side effects for every existing user.

The ctor parameter idea would be easy to implement. And mouse events should be safe default for most apps. If you know the component you are going to use with HTMLMesh uses pointer events, you can give HTMLMesh a hint with the new event type parameter. This approach is totally safe for existing users (so there are no potential side effects) since pointer event usage would be an opt-in.

[jrjdavidson]

Hey again,

I think the correct design decision is to keep improving HTMLMesh so it emulates an actual HMTL page. The whole thing is a shim that allows HTML to be displayed in XR (and very impressive shim I must say, I actually learned quite a bit by studying it!). We already have examples in this PR of how we don't have full control of the code that we use in HTMLMesh, such as lilgui above, and it being hard knowing what listeners are used in said code. I think it is important to keep striving to match HTML/js behaviour in and out of XR and we have a chance to do that here.

I guess we could add a flag that keeps the old behaviour to keep it safe for current users.

[Mugen87]

I guess we could add a flag that keeps the old behaviour to keep it safe for current users.

That sounds like a good compromise. If no issues are reported overt time, we can remove the fallback at a later point.