Editing Project:Village pump (proposals) (section)

=== UX research and usability testing ===
Note, I am an engineer that uses terminals a lot and I still use the MonoBook skin. But, here's a question. If moving to the new Vector skin is controversial, why not commission and publish an extensive [[user-centered design]] case study to prove that the Vector 2022 is actually better. Then the community will have to see reason. (Maybe) <span style="font-family:Roboto Mono,Droid Sans Mono,Courier New;font-size:small;">'''[[User:Andrevan|Andrevan]]'''<span style="border-radius: 50%; border: 2px solid #073642;margin-top:-16px">[[User_talk:Andrevan|@]]</span></span> 03:23, 15 July 2022 (UTC)

Hey all,

A number of you have asked us about our research and testing (both Qualitative and Quantitative), so we wanted to write a pretty detailed and long comment to address this. We wanted to confirm that not only the [[Wikipedia:Growth Team features|Growth team]] conducts complex testing :) This is more like the standard for big projects now. Each feature change has gone through the process below (which we also described in the [[Wikipedia:Wikipedia_Signpost/2022-04-24/Technology_report#Desktop Improvements project nearing an end|Signpost]] in April). This is what gives us the confidence that everything we have built so far is, in principle, an improvement. At the same time, we acknowledge that there's room for more adjustments.

# '''Problem identification research with both readers and editors''' - during this phase, in 2019, we studied the way people used the site and identified the largest usability issues as well as issues to exploring the site further, becoming more engaged with reading or editing. We did this by interviewing readers and editors across multiple countries and locations. (See the links: [[mw:Reading/Web/Desktop Improvements/Repository/Research and design: Phase 1|Research and design: Phase 1]], [[mw:Reading/Web/Desktop Improvements/Repository/Research and design: Phase 2|Research and design: Phase 2]].)
# '''Prototype development and testing''' - this is when we build out the ideas of a feature and begin showing solutions to our audiences. Each feature was tested with readers and editors through interviews and wider rounds of prototype testing. Generally, for testing with editors we used central notice across multiple language Wikipedias, {{tq|including English Wikipedia,}} so that we can get the widest audience possible. Each prototype was tested by approximately 200 editors on average. ([[mw:Reading/Web/Desktop Improvements/Prototype testing|Example]])
# '''Refining and building''' - we then take the feedback from the prototype testing and refine or change the prototype based on what needs were identified in the prototype testing. In some cases, we ask for additional feedback during this process so that we’re sure we’re making the right decisions.
# '''A/B testing and other quantitative testing on [[mw:Vector 2022#test-wikis|pilot (early adopter) wikis]]''' - we perform a quantitative test for whether the feature works as expected based on the criteria of success we have previously defined. For example, the [[mw:Reading/Web/Desktop Improvements/Features/Sticky Header|sticky header]] was designed to decrease scrolling to the top of the page. We gave the sticky header to 50% of users and compared them to the other 50% for two weeks. After two weeks we compared the results and identified that people who had the sticky header were indeed scrolling less to the top of the page in order to select any of the tools available there. If we get negative results from our test, we change the feature and test again. This is the "[[Software_release_life_cycle#Beta|beta]]" phase. {{tq|During this phase, we also monitor usage across all wikis, including English Wikipedia, where many account holders are already using the new skin.}}
# Finally, we '''deploy Vector 2022 on more wikis''' and continue monitoring the way people are using it so that we can flag any issues. In this phase, Vector 2022 isn't "beta" anymore. It's more like a [[Wikipedia:Content_assessment|B-class article]]. Different wikis have different thresholds for B-class, and we believe that in the case of English Wikipedia, we'll be there when the [[mw:Reading/Web/Desktop_Improvements/Features/Visual_Refinements|visual refinements]] and [[phab:T309972|other deployment blockers]] are ready.

We are currently working on an easy way to explore all of the above data and research (and are welcome to suggestions on the best format). For now, the best way to learn more about the testing is:
* From [[mw:Special:MyLanguage/Reading/Web/Desktop_Improvements/Features|Reading/Web/Desktop Improvements/Features]], select the feature you are most interested in
* Within that feature page, refer to the Qualitative or Quantitative testing section to see the results and our conclusions

Just so we can have a short version of this as a part of this conversation, we're posting a quick list of our learnings:

{{hidden|1=Collapsible sidebar| headerstyle = text-align:left | style = border: 1px solid #aaa;  | 2= 

The collapsible sidebar allows people to collapse the main menu in order to focus on reading - helping to find the information needed without distraction

* Qualitative testing with readers and editors on the usefulness of the sidebar and our navigation. Our conclusion here was that the number of different tools provided on the page by default was found to be overwhelming by readers and actively discouraged them from reading, but also from exploring the functionality within the page, an effect opposite of what the exposure of multiple tools aims to do. More details can be found on our [[mw:Reading/Web/Desktop_Improvements/Features/Collapsible_sidebar#User_testing|feature page for the collapsible sidebar]], as well as within the [[mw:Reading/Web/Desktop_Improvements/Repository/Hureo_User_Research_Report|original report]]
* Quantitative testing on the usage behavior of the sidebar itself, in both its open and collapsed states ([[mw:Reading/Web/Desktop_Improvements/Features/Collapsible_sidebar#Results|see the results]]). When using the sidebar, logged-out users are much more likely to collapse it and, once collapsed, to keep it collapsed. In addition, the rate of un-collapsing also indicated that users are aware that, were they to need to navigate to an item in the sidebar, that option was available to them.
}}
{{hidden|1=Maximum line width| headerstyle = text-align:left | style = border: 1px solid #aaa;  | 2= 

We have introduced a maximum line width to articles. Research has shown that limiting the width of long-form text leads to a more comfortable reading experience, and better retention of the content itself. 

* Our studies with readers showed that readability was an issue with the current interface, in particular being able to focus on the content
* Pages that are not in a long-text format {{tq|(such as diffs, special pages, page history)}} will be presented at full-width as before
* Logged-in users who wish to read articles at full width are welcomed to set up a script or gadget that will allow for this{{tq|, such as [[User:Jdlrobson/vector-max-width-toggle.js|this one]]}}
* For more details on research and motivation, see our[[mw:Reading/Web/Desktop Improvements/Features/Limiting content width#Goals%20and%20motivation|research section]]
}}
{{hidden|1=Search| headerstyle = text-align:left | style = border: 1px solid #aaa;  | 2= 

The new search widget includes important context that makes it easier for users to find the query they are looking for by adding images and descriptions for each search results

* People had difficulties finding the correct result using our previous search
* Our A/B testing showed that adding the new search can lead to a 30% increase in search sessions initiated on the wikis we tested
}}
{{hidden|1=Language switching| headerstyle = text-align:left | style = border: 1px solid #aaa;  | 2= 

The new language switching tools are more prominently-placed than before. They allow multilingual readers and editors to find their preferred language more easily. 

* Readers did not previously know they could switch languages from the page, even if they read multiple language wikis habitually. They would use external search engines to find the correct article instead.
* In our user testing, new readers were able to find the new location much quicker than the previous location
* Our qualitative testing showed that this was more difficult to find for existing users who were used to the previous location, leading us to iterate on the feature.  We have since added a note in the previous location of the language switcher and made the button itself a more prominent color
* In the future, we will continue exploration on languages, considering potentially a direct link to a person’s most frequent languages

(note to {{re|sdkb}} we know you have some questions on language links that are still open - we’ll get back to you on these in a separate message)
}}
{{hidden|1=User menu| headerstyle = text-align:left | style = border: 1px solid #aaa;  | 2= 
The new user menu provides links to all links related to the user in one place. This reduces confusion between general navigation links and specific user links

* New editors were confused between the links at the top of the page and other navigation. They didn’t know these links pertained to their personal tools
* Our user testing with readers and editors showed that people found it intuitive that all user links are in a single menu and that the menu is easy to find
* In our prototype testing, 27 out of 38 (71%) editors and other logged-in users showed strong positive experiences with the user menu
* Based on community requests and current data, we iterated on the feature and moved the watchlist link out of the user menu for easier access
}}
{{hidden|1=Sticky header| headerstyle = text-align:left | style = border: 1px solid #aaa;  | 2= 

The sticky header gives access to functionality that is used most frequently that was previously only accessible at the top of the page. The goal is for people to scroll less and thus, save time

* Our A/B test showed an average 15% decrease in scrolls to the top per session for logged-in users within the 15 pilot wikis we tested on
}}
[[user:OVasileva (WMF)|OVasileva (WMF)]], [[User:SGrabarczuk (WMF)|SGrabarczuk (WMF)]] ([[User talk:SGrabarczuk (WMF)|talk]]) 16:14, 15 July 2022 (UTC)
::Thank you for posting this; there is a lot to read through so I have only reviewed two features so far, sticky header and persistent table of contents. For the latter, it appears you have yet to conduct A/B testing but when you do I would be interested in seeing data on the percentage of page views that involve at least one click on the table of contents, and the percentage of page views that involve at least two clicks on the table of contents. <s>In addition, I would be interested in seeing separate data for mobile users and desktop users.</s> <small>Struck following clarification that this is only proposed for desktop. [[User:BilledMammal|BilledMammal]] ([[User talk:BilledMammal|talk]]) 04:43, 23 July 2022 (UTC)</small>
::For the former, I see you have already conducted A/B testing but there is some additional data that I would like to see:
::#Currently, you show the clicks per session and clicks per page only when skinversion=2; I would be interested in comparing this to the clicks per session and clicks per page for skinversion=1. My hypothesis would be that the sticky_header makes it more convenient for readers to access these links, and thus increases the number of readers using them.
::#The rate of accidental clicks. Assessing this would vary by link, but I have a few ideas and am happy to discuss further if required. My hypothesis would be that the sticky_header increases the number of accidental clicks, and we would need to consider whether this increase offsets the benefits of the sticky_header.
::#Time on page, time on page when limited to pageviews that do not involve following a header link, and time on page when limited to pageviews that involve following a header link. Clicks on pages with stickyHeaderDisabled would need to be split between those that involve a scroll back and those that do not. My hypothesis would be that it does not affect time on page for readers who do not click on a header link, and that it has a small but relatively constant absolute decrease in time on page for readers who follow links on the sticky_header compared to those who scroll back to click on a header link. The former would suggest that this does not negatively affect the reading experience, the latter would suggest that that this has a positive effect on the reading experience for readers who are wanting to navigate to one of those pages.
::Alternatively, is there raw data that we can look at from the A/B testing for the sticky header? I suspect it won't answer #2, but it may contain information on #1 and #3.
::[[User:BilledMammal|BilledMammal]] ([[User talk:BilledMammal|talk]]) 03:33, 17 July 2022 (UTC)
:::@[[User:BilledMammal|BilledMammal]] - thanks for your questions! You bring up some really good points that we considered during the design phase of the experiment. The full data analysis is available here: https://jenniferwang-wmf.github.io/Web_sticky_header/.  In terms of your questions:
:::1. Comparing overall clicks.  This is something we can look into and report back.  The reason it wasn't a main goal for the A/B test is because we wanted to focus on decreasing scrolling (i.e. making the site easier to navigate by requiring less of the user) rather than setting a goal for increased interactions. As in, we would still consider the feature a success if people used the tools as frequently as they did before but had to scroll significantly less in order to do so. That said, I agree with your hypothesis that we most likely would see a significant increase in clicks as well.  
:::2. Accidental clicks are a bit trickier to measure. Generally, with new features, we get a lot of clicks in the first day or so after deployment (which are generally more based on curiosity than accident).  This is why we run our tests for an extended period of time - 2 weeks, to make sure it's sufficient time for people to get used to the new functionality and begin using it as they would naturally
:::3. We discussed looking at time on page at the beginning of the test but decided to look at scrolling specifically instead.  While I personally believe that your hypothesis is correct, we've had some issues in the past with looking at the time on page metric and receiving conflicting data.  For example - time on page may actually increase over time with the sticky header available because people would be less frustrated with not being able to access specific tasks and thus more open to spending longer on our sites overall. The same is true of scrolling - people might scroll more overall because the site is easier to use.  This is why we specifically looked at scrolling to the top of the page as we thought it was the clearest signal that people are going there specifically to find one of the tools available in the sticky header. [[User:OVasileva (WMF)|OVasileva (WMF)]] ([[User talk:OVasileva (WMF)|talk]]) 09:06, 2 August 2022 (UTC)
::::{{ping|OVasileva (WMF)}}, thank you for your reply, both here and below.
::::To summarize my position; I see the tests you have done as testing whether the feature is used, but not anything beyond this. With this, it is only possible to come to the conclusion that this proves that each feature is an improvement if the pre-existing assumption is that the feature is an improvement, and thus the user experience is improved so long as the feature is used; I understand how you can see this differently, but I disagree.
::::I do agree that time on page isn't a perfect metric, but I believe it is a better metric than what is currently being used, and I also believe that those concerns can be partially addressed. For example, looking at the various options on the header bar the only one that I can see plausibly altering behaviour is the search bar, with readers using it more and doing a shallow dive into multiple articles rather than a deep dive into one; to address this we could separate the data into sessions that use the search bar and sessions that don't.
::::Regarding accidental clicks, that is a good point regarding the curiosity clicks; most methods to identify accidental clicks that I am aware of would likely see those as false positives. Are you able to identify which sessions belong to same logged-in user, as that might offer a way to exclude most curiosity clicks? [[User:BilledMammal|BilledMammal]] ([[User talk:BilledMammal|talk]]) 16:46, 7 August 2022 (UTC)
Has anyone actually used the link given at the very top as "[https://en.wikipedia.org/wiki/Galaxy?useskin=vector-2022 see what it looks like]" on a smartphone? For me, instead of getting [https://en.wikipedia.org/wiki/Galaxy an encyclopedia article], I get a full screen with the sidebar and ''no'' encyclopedic content until I scroll down. Can other people please test this? Because this seems like a quite major bug or worse experience than the current mobile version. [[User:Fram|Fram]] ([[User talk:Fram|talk]]) 08:46, 17 July 2022 (UTC)
:Same bug here but only if I'm using the mobile view in a browser. If I'm using the desktop view on mobile it works fine (and actually looks quite nice). With this said, it may be a non-issue as I've not read anything about mobile transitioning away from MinervaNeue. [[User:Anarchyte|<span style="color:#202122;font-family:Trebuchet MS">Anarchyte</span>]] ([[User talk:Anarchyte|<span style="color:#202122">talk</span>]]) 09:01, 17 July 2022 (UTC)
:FWIW, I see the same bug as Fram does, on both a Macbook laptop ('''not mobile''') running Safari 14.1.2, and on an Android phone running Opera 69.3.3606.65458 '''in desktop mode'''. I just see a banner at the top, a sidebar on the left, and a big blank space on the rest of the page. I have to scroll way down past the sidebar before I see any content, and the content fills the full window width (that is, the sidebar is not to the left of the content, it's above it). There's also no visible TOC on the left side or anywhere, just a hamburger icon that I have to click on to open a TOC. I do not see this bug on Firefox 102.0.1 on a Windows machine. It seems there is still some browser dependency that makes this skin very unpleasant to use in some environments, and not only mobile ones. [[User:CodeTalker|CodeTalker]] ([[User talk:CodeTalker|talk]]) 00:51, 18 July 2022 (UTC)
:Confirming I see the same thing as Fram in mobile view on Firefox 101.2.0 on Android 11. Desktop view looks intentional. [[User:Folly Mox|Folly Mox]] ([[User talk:Folly Mox|talk]]) 18:01, 18 July 2022 (UTC)
::It looks great to me (tablet, mobile & desktop) if the sidebar's collapsed.&nbsp;&#8213;&nbsp;<span id="Qwerfjkl:1658179742448:WikipediaBWLCLNVillage_pump_(proposals)" class="BawlCmt">[[User:Qwerfjkl|<span style="background:#1d9ffc; color:white; padding:5px; box-shadow:darkgray 2px 2px 2px;">Qwerfjkl</span>]][[User talk:Qwerfjkl|<span style="background:#79c0f2;color:white; padding:2px; box-shadow:darkgray 2px 2px 2px;">talk</span>]] 21:29, 18 July 2022 (UTC)</span>
::This is indeed what you see if you force the mobile website to the desktop website/skin (not something anyone but a few realistically is doing) and you open the menu (for me the menu is collapsed by default on that size). While this desktop skin is more compatible with mobile and will eventually probably be fully suitable for mobile, that has not been the primary target of these changes. I believe there is an invite on its talk page asking people for feedback and ideas on what the menu SHOULD look like on mobile. But don't forget that lots of the content is not mobile compatible (minerva on the mobile site has all kinds of hacks to make content not break out of the mobile constraints) and Vector 2022 doesn't have those hacks. So for the immediate future it is still better to use the mobile website on mobile. —[[User:TheDJ|Th<span style="color: green">e</span>DJ]] ([[User talk:TheDJ|talk]] • [[Special:Contributions/TheDJ|contribs]]) 08:49, 19 July 2022 (UTC)
:::Apparently you get the same bad results on a Macbook laptop though, see above... [[User:Fram|Fram]] ([[User talk:Fram|talk]]) 09:21, 19 July 2022 (UTC)
::::I've just tested this on a Macbook in Safari and can't reproduce the issue - the link destination looks exactly as intended. [[User:Samwalton9|'''S'''am '''W'''alton]] ([[User talk:Samwalton9|talk]]) 10:39, 19 July 2022 (UTC)
:::And of course, ''nothing'' in the opening statement of this section said anything about this not being for mobile and only for desktop, it just said that this would become ''the'' default, full stop. [[User:Fram|Fram]] ([[User talk:Fram|talk]]) 09:23, 19 July 2022 (UTC)
::::Just to confirm, this conversation concerns changing the skin on desktop only.  This will affect the desktop view on mobile as well, but not the current default mobile view. [[User:OVasileva (WMF)|OVasileva (WMF)]] ([[User talk:OVasileva (WMF)|talk]]) 13:27, 20 July 2022 (UTC)
:::::On desktop (not mobile), using Firefox on a MacBook Pro, if I click on the suggested Galaxy link and narrow my window to about half my screen width, I get a view in which only the left toolbar is visible. The rest of the window is blank. The article itself is off-screen below the toolbar. This is a normal width to narrow the window for when I want to see two apps at once, and much wider than its minimum width (which I sometimes use as a quick test of mobile views). On the other hand, when I view it in full screen width, only maybe 60% of the window width is dedicated to content, with maybe 10% sidebar and 30% unused blank space. This extreme sensitivity to window width, unusability on too-narrow windows, prioritization of sidebar over content, and inability to use much of the real estate on too-wide windows, makes this seem like a non-starter to me, but maybe these are things that are still subject to improved design before the push to roll this out to the world? —[[User:David Eppstein|David Eppstein]] ([[User talk:David Eppstein|talk]]) 01:29, 8 August 2022 (UTC)