Announcing the new Staging Ground Reviewer Stats Widget

Please make this feature available to all reviewers, even those who haven't yet "accumulated enough activity." That'll make it much easier to notice when your reviewed posts are doing badly, which helps you become a better reviewer!

To put it differently - if someone's reviewed posts have a 1% survival rate, isn't that something they should know now rather than later?

bug status-review

I just got access to the stats widget today, but even though I have approved questions that fit the survival criteria, my survival rate currently shows as 0% (which puts me in the top 1% of reviewers! 🎉). I assume this data hasn't been filled in yet?

For many metrics, only the reviewer that "Approve and publish"es a post gets credit for it

It seems like only the reviewer that "Approve and publish"es a post gets credit for it (for metrics 4,5,6), even if someone else actually spent their time in the back-and-forth with OP. Wouldn't we want to incentivize the time-consuming edit-feedback-repeat process rather than making the stats based around approvals? – ipodtouch0218 [15 upvotes]

Staff response:

@ipodtouch0218 We apologize for not considering this as part of the first iteration of the widget. We acknowledge that this is not a perfect representation of all of the reviewers' contributions because multiple reviewers can provide feedback in a post which should be reflected in some way in the widget. We will take this into account for the next iteration of the widget to represent the effort put into a review even if the person was not the one to approve the post. – Bella_Blue [2 upvotes]

^{This post was extracted from the comments for ease of following discussions.}

feature-request (or stat-request)

If you have suggestions on additional stats to include, we would love to hear how this widget can be improved. Please let us know below or by completing this survey

Let me get an idea out here.

I have the opinion that providing good reviews/improving questions should be incentivized over robo-reviewing/providing any reviews.

With that in mind, I think it would be good to have a statistics about questions that actually improved. My concrete suggestion is to include the number of questions where the reviewer used the "Requres Major Changes" or "Conditionally approve pending minor edit" option followed by the asker editing their question which then goes on to be approved. I could also imagine including reviewer edits where the question is later approved (by the same or other reviewers). I would suggest the name "Questions improved" or similar for that.

What is the definition of "helping" an asker?

What is your definition of "helping" an asker? – dan1st [2 upvotes]

Staff response:

@dan1st Any reviewers who have applied eligible review actions (eg, major changes, minor edits, approved, edited post). Comments added were not included as part of the criteria to simplify the query. We can reconsider adding comments for the use case where a reviewer added a comment but did not take action because another reviewer had already done so. – Bella_Blue

^{This post was extracted from the comments for ease of following discussions.}

status-completed

At the bottom of the new help page for the stats there is a sentence suggesting to complete a survey:

Please complete this survey to provide feedback.

But no survey is actually linked anywhere on that page. I believe this is supposed to point to the survey linked in the question here.

What is "enough activity" to see these stats?

Yeah, could you define "enough activity"...?, what is the metric/threshold...?, all times/last 30 days...? // I don't understand the "vs" in "vs last 30 days" for a few sections, compared to "in last 30 days" for other sections...? // In 'Reviewer activity rank', "edit post" (all commas should be outside the double quotes btw) is mentioned, while it doesn't count as a Review. – chivracq [5 upvotes]

Staff response:

@chivracq We classify reviewers based on their last 30 days of activity as either "low" or "high" activity. A reviewer rank is applied (i.e. 1, 2, 3, etc). The top 150 reviewers are classified as "high activity" and anyone ranked lower than this would be considered low activity. The all time metric (the denominator value) in “total askers helped” and “total approved questions” accounts for all reviewers, only the numerator reflects your direct contributions. In the example screenshot, 2.3k helped out of 29k means you helped 2.3k askers out of 29k askers helped by all reviewers. – tanj92

^{This post was extracted from the comments for ease of following discussions.}

feature-request

Currently, the "Question survival rate" and "Question success rate" stats are hidden behind a "Show more" button:

Would it be too much trouble to show them by default?

Why was gray chosen to represent a decrease, rather than red?

Is there any reason that the gray color is picked to represent a decrease, instead of the usual choice of red? – ray [1 upvote]

I would assume because red suggests "negative" as well, @ray , however, there can be many reasons people's activity can go down (in the SG): other commitments, holidays, boredom, no questions sent to the SG, etc, etc. Red could imply you're doing a "bad job" and I kinda am on sind that I don't want to see I'm "performing badly" because no questions in the tags I follow went to the SG that day/week. – Thom A [4 upvotes]

@ThomA This is kind of subjective, but it looks like a sensible interpretation to me. Actually, I have no opinion on whichever colour is picked. I just want to understand the reason behind the choice. – ray

I also wouldn't be surprised if the colour choice is due to "accessibility" reasons, @ray . Stack Overflow are on a big hype at the moment of making things "more accessible" by making the UX different/odd/worse. – Thom A

Staff response:

@ray like other users theorized we didn't want the user experience to be too negative so chose gray to be more neutral instead of the red. – Bella_Blue [3 upvotes]

^{This post was extracted from the comments for ease of following discussions.}

Not enough content is going to SG right now

Great, and I'll never appear on it because you refuse to send content to the SG... Fix the SG first then add game features... – Thom A [8 upvotes]

Since I only review the stuff that's in my wheelhouse it's gonna be a while before I get the stats, methinks. I've yet to see 50 C++ SG questions in one day, and trust me you don't want me <expletive deleted>ing around in the python questions. – user4581301

Staff response:

@ThomA Our ideal state is for all new askers' questions to go through Staging Ground first, until they have successfully graduated. But that requires having enough reviewers to handle the load (something we are actively working towards). If we opened the floodgates tomorrow, a significant number of these questions would end up auto-graduated after 24 hours, before reviewers have a chance to tackle them. We are slowly but surely increasing the number of posts that are routed to Staging Ground, as long as we have enough reviewers to handle them. – Sasha

Further discussion, part 1:

@Sasha "a significant number of these questions would end up auto-graduated after 24 hours": Yeah, and...? The worst case is that those Askers will have to wait for 24h before their Qt gets posted on the 'Main'-Site. I'm pretty sure the worst ones will be intercepted... That's already a win-win.. – chivracq [1 upvote]

@chivracq If there aren't sufficiently many reviewers, it may not result in the worst questions not being intercepted but just a small kind of random sample being intercepted. Also, some potentially good questions being stuck in Re-Evaluate forever because no one looks at them in the Staging Ground and these can't auto-graduate. – dan1st [1 upvote]

@dan1st. For the "Stats", I'm pretty sure some Reviewers will love to do the 'Re-Evaluate' Qt's, and approve everything that is fairly OK... Maybe not best, but the overall quality will still go up... (I currently see quite many Qt's staying in 'Major Changes' or 'Re-Evaluate' while the quality is already pretty decent, for a non-SME in the main Tag...) – chivracq

@chivracq that assumes these reviewers even seeing those questions which I'm not sure about if all questions would go to the SG. – dan1st

Further discussion, part 2:

"Our ideal state is for all new askers' questions to go through Staging Ground first, until they have successfully graduated" And that's fine, but at the moment you're sending about 2% (finger in the air, randomly generated number) of users that should go to the SG to the SG; the rest skip it completely; that's significantly worse that asking 98% of those users to "wait" 24 hours. So what if, at the moment, they get released after 24 hours? That's better that sending 95% of bad material straight to the website because Stack Overflow won't commit to onboarding... – Thom A

@ThomA If you consider it worse, that may not be the case for others. While I am not sure which option would be better or worse, I can understand not wanting to add an unnecessary delay to almost all questions, especially if the current system seems to work (e.g. in the perspective of the company) and a useless delay can do active damage (as opposed to just keeping the current state which wouldn't make it actively worse). – dan1st [1 upvote]

"I can understand not wanting to add an unnecessary delay to almost all questions" But that is actually Stack Overflow's goal, @dan1st , to send all eligible questions to the SG and remain there until they are released (no auto release after 24 hours). So some will sit there until reviewed, which might be quite quick, and others could remain in "purgatory" for days (or weeks), like edit suggestions. – Thom A

Yes but the goal is for these questions to be handled, not for them to remain in the Staging Ground without anyone looking at it. – dan1st [1 upvote]

And sending everything to the SG allows for that, @dan1st . Sending barely anything to it does not. – Thom A

It allows them going into the Staging Ground without anyone looking at these questions. – dan1st

^{This post was extracted from the comments for ease of following discussions.}