Editing Project:Village pump (proposals) (section)

====2. Why are these changes improvements?====

Many of you were curios about the changes, and especially expressed interest in getting more details on our data and process. Below, we are outlining a bit about our process, as well as the data we have collected that proves that each feature is an improvement. Ping: @[[User:BilledMammal|BilledMammal]], @[[User:IAmChaos|IAmChaos]], [[User:Barkeep49|Barkeep49]], [[User:KevinL|KevinL]], [[User:Andrevan|Andrevan]].

TLDR: Every one of our changes goes through a rigorous process of research, development, qualitative and quantitative testing with readers and editors, prototype testing with editors (across 30+ language communities), iteration, and post-deployment monitoring. When a change does not meet the success criteria or does not perform better than the existing version of a tool, we either stop developing the change or iterate until performance is improved.

We believe that the changes we have made will be crucial to making the site more welcoming and easier to use to new readers and editors.

When compared to the older version of the Vector skin, Vector 2022 is proven to:

* Save time while reading and editing (measured by decreases in scrolling to the top of the page after the introduction of sticky header and table of contents)
* Increase exploration within a given page (measured by increased clicks to the table of contents)
* Improve readability (measured via collapsible sidebar usage)
* Improve the discovery of new content (measured by an increase in searching)

You will find more details in the section [[#UX research and usability testing]]. We have tweaked the existing comment a bit.
:Thank you for spending the time to write this out. My concern is that you are focused too much on testing for the change you intended to make and in doing so miss the broader impact. Because of this I feel your analysis only proves that you brought about your intended change, rather than proving {{tq|that each feature is an improvement}}.
:For example, consider the sticky header. Here, the goal is to (1) improve the user experience, by (2) saving reader time, by (3) reducing the amount of scrolling to the top that they need to do. 
:However, you only test for (3); you then infer (2) from the positive result for (3), and infer (1) from the positive result for (2). Testing directly for user experience is difficult, but to reduce the risk of errors the goal should be to get as close to that level as possible, and in this case it means testing for (2) rather than (3); if you look at #3 in my previous comment you will see that I am requesting data that should give us an answer to (2).
:In addition, you assume there are no negative impacts from the change. This isn't a safe assumption, and with #2 and #3 of my previous comment I request data that will allow us to consider this possibility. [[User:BilledMammal|BilledMammal]] ([[User talk:BilledMammal|talk]]) 06:56, 28 July 2022 (UTC)

:Hey @[[User:BilledMammal|BilledMammal]] - sorry for the delay in response!  I replied briefly to your initial comment above.  TLDR is that we try to design experiments using more precise metrics because more open-ended metrics (such as time spent on page) could be interpreted in multiple ways.  More time on page could be an improvement (people have a better experience and thus spend more time on the site overall).  Less time on page could also be seen as an improvement (we're saving people time in scrolling so they get to what they need to do quicker). We can keep discussing there or under this thread - whatever is easier! [[User:OVasileva (WMF)|OVasileva (WMF)]] ([[User talk:OVasileva (WMF)|talk]]) 09:11, 2 August 2022 (UTC)
::Thank you; no worries about the delay in response, and apologies for my own delay in response. I've replied above. [[User:BilledMammal|BilledMammal]] ([[User talk:BilledMammal|talk]]) 16:46, 7 August 2022 (UTC)