Help for Judges

Judging tutorial video

If you never got around to watching this, please do. It runs just 3 minutes and it's well worth the investment: The payoff is less stress and frustration for you and a much better experience for the folks you're evaluating and the organizers of this event. 

 

Incomplete scores?? I thought I did everything I was supposed to do!

You are probably reading this because the folks running this event and telling you that you haven't completed your scoring, and you think you have.

Chances are, you are very close to the goal line. Any previous work you've done has been saved and you'll be building on that. 

Your task is simply to finish up by assigning a numerical score to all of the dimensions for the team(s) in question.

Valid Eval requires both feedback clicks and numerical scores on all dimensions. Otherwise, the evaluation is incomplete and the scores are not counted.

Please have (another) look at this video https://vimeo.com/valideval/evaluation. If you’d like, zoom ahead to the 1:30 mark to see how the numerical scoring process works.

 

 

How are we supposed to pick the winners?

Unlike other approaches to scoring complex performances, Valid Eval does not require you to make arbitrary "stack rankings," where you're asked to pick the best company from the batch you review.

On our platform, you don't pick winners. This is one of the biggest benefits of using Valid Eval as a judge. Your job is simply to provide the most accurate evidence-based assessment possible using the rubric chosen for this event. The score distributions are typically beautiful bell curves, which make it easy for the organizers and judges to agree on which companies were the strongest. 

Please be as thorough and precise as you can based on your expert opinions as to how the teams performed in each dimension of the rubric. (Remember that while all dimensions must be scored, the best judges make feedback clicks only in the relevant rows. Generally, this is 75% or so of the possible feedback clicks.)

I don't want to be mean. Am I allowed to judge nicely?

Certain teams are not a good fit based on the evaluation criteria captured in this particular rubric. You are not doing that team a favor by "being nice." If they do poorly on certain dimensions of the rubric, that's just the way it is. If they are awful, then they are awful.

The rubrics we deploy are custom tailored to measure the attributes that matter in reaching a specific goal our clients defined. Same goes for the application templates, which are custom tailored to fit the rubrics. Please know that every applicant was provided with the rubric and very clear instructions as to what we were looking for before they applied. A failure to present you with strong performance is on them. 

Entrepreneurship can be brutal. But, the cruelest cuts are from the mistakes others saw us making, but stood by silently while we wasted time and money going down the wrong path. On Valid Eval, your evaluations and comments are anonymous. The kindest and most valuable thing you can do for entrepreneurs is to provide brutally honest feedback.

Keep it constructive, civil and compassionate. But, don't pull punches!

What am I supposed to be judging? Is due diligence part of what's expected?

Your evaluations should be exclusively based on the materials the company uploaded for your review. Due diligence is beyond the scope of what we are asking you for. The organizers of this event will take appropriate steps to verify eligibility and the applicants' honesty.

This is important for the integrity of the evaluation process. It ensures that each performance for a given company is identical, which is crucial for the veracity of the data we will analyze later. 

We have taken a good deal of care to ensure that the "playing field" is level by providing all of the companies with the same templates and instructions. Please respect the process and keep knowledge you may have from activities beyond their application out of your assessment.

If you are enthused by a particular company and want to dive into their company and the competitive landscape via your own research, that's great! But, please do not incorporate those activities into the your evaluation.

Recommendations. What are those for?

Recommendations are the place for you to give actionable advice. Think of these as bits of a prioritized to-do list for the entrepreneurs.

Please do not confuse a recommendation with a comment. A comment is anonymous, a recommendation is attributed to you. A comment is a general purpose place to share your thoughts about the performance you've evaluate. A recommendation offers a hand up. 

In short, your recommendation should point to a specific action the company should take to improve its fortunes.

What is Judge Awesomeness and why are you measuring me?

Your "awesomeness" as a judge is a measurement of how you do in three key areas:

  1. Level of Effort,
  2. Inter-Rater Reliability, and
  3. Helpfulness

At Valid Eval, we believe that the judges on our platform who complete their tasks thoroughly, whose scores are in line with those of other experts, and who offer a helping hand to those they evaluate are AWESOME!

Our clients -- the folks who convened the teams and you guys as experts -- value knowing which judges are the most awesome. And, by contrast, which judges shouldn't be invited back next time. 

What exactly are we measuring? That's a secret. But, if you sincerely engage, evaluate honestly and invest a little extra time to make comments (anonymously) and recommendations (attributed to you), your impact will be awesome!

When should I recuse myself from judging?

The organizers of your event will be able to give you specific guidance on recusals. But, here is the Valid Eval party line.

Generally speaking, if you have a financial stake in a company to which you have been matched, you should recuse yourself. What does this mean? Examples of recusal-worthy financial involvement would be owning shares / options / warrants in the company, earning money from the company, having a board of directors seat, or being engaged in a business partnership of any kind with the company. With this in mind, if you find yourself wondering if you should recuse yourself, you probably should do so. 

Generally speaking, if you know the people in the company, or have seen their pitch before, or you are friends with a board member those are not grounds for recusal.

Does a recusal impact my Awesomeness?

No!

We never penalize you for being honest and bowing out of an evaluation where you have a conflict.

Recusals are cool with us!

My scores are not showing up in the “My Evaluations” list. Why?

You are reading this for one of two reasons. Either:

  1. The folks running this event and telling you that you haven't completed your scoring, and you think you have, or
  2. You've arrived here on your own because that darn "My Evaluations" list isn't showing numerical scores in the right column, and you've already clicked a bunch of boxes and have left helpful comments. 

Chances are, you are very close to the goal line. Any previous work you've done has been saved and you'll be building on that. 

Your task is simply to finish up by assigning a numerical score to all of the dimensions for the team(s) in question.

Valid Eval requires both feedback clicks and numerical scores on all dimensions. Otherwise, the evaluation is incomplete and the scores are not counted.

Please have (another) look at this video https://vimeo.com/valideval/evaluation. If you’d like, zoom ahead to the 1:30 mark to see how the numerical scoring process works.

 

 

 

Help for General Users

I forgot my password.

If you do not remember your password, you may reset it here: https://app.valideval.com/v2/reset-password-request.

The reset will only work if you enter an email address that Valid Eval knows about. For obvious reasons, our system only knows the email address(es) you've set up in your profile.

If you are like us and hate remembering passwords, the Login with LinkedIn option is a good solution.

I would like to change my password.

First, log into Valid Eval here: https://app.valideval.com/v2/users/authentication. From the “Dashboard,” click the “Account” link at the top right of the page. This will lead you to you “Global Account Profile” page. Once there, you may edit your profile, including changing your password.

If you do not remember your password, you may reset it here: https://app.valideval.com/v2/reset-password-request by entering the email address associated with your Valid Eval account. 

I cannot login with LinkedIn. How do I do that?

Sometimes, it's nice to just be able to click a button to log into a website.

In just two minutes, this video explains how to set this feature up so that you don't need to remember your Valid Eval password: https://vimeo.com/valideval/linkedin

 

I don't want to login with LinkedIn. How do I just use my email and password?

Go to the usual login page: https://app.valideval.com/v2/users/authentication. Then, Click the link directly below the "Login with LinkedIn" button, which says "Login without LinkedIn."

Once you do that, you will see the typical boxes asking for your email address and password.

I want to add / remove an email address from my account.

First, log into Valid Eval here: https://app.valideval.com/v2/users/authentication. From the “Dashboard,” click the “Account” link at the top right of the page. This will lead you to your “Global Account Profile” page. Once there, you may edit your profile, including adding additional email addresses to your profile.

Why would I do this?

You may easily login with any of the email addresses you have, all of which will be linked to your one-and-only profile. This way, you won't have to remember which email address you used. All of them will use the same password since they all point back to the same Account Profile.

Adding additional email addresses also helps you avoid future duplicate Valid Eval accounts. For example, if your account is currently setup with only your work email address, a future invitation to be a Judge to your personal email address would require you to set up a new profile. Then you'd have two Valid Eval accounts. Of course, we can fix that, but it's a pain in the butt for you and it's easily avoided. 

We think it's a good idea to populate your profile with all of the email addresses you frequently use.

I've recently had more entrepreneurial experiences. How do I change my profile?

Valid Eval captures data on your experience levels and the type of work you do. If you've gained experiences, or have switched professions, please let us know by making changes in your profile. 

First, log into Valid Eval here: https://app.valideval.com/v2/users/authentication. From the “Dashboard,” click the “Account” link at the top right of the page. This will lead you to you “Global Account Profile” page.

On that page, you may edit your entire profile, including the parts that help us anonymously give your evaluations context.

Browser Compatibility

If you the site looks broken you may be using a browser that is too old to run Valid Evaluation. 

We require the following versions (or newer) of these popular browsers:

* Internet Explorer -- Version 9

* Google Chrome -- Version 11

* Apple Safari -- Version 5

* Mozilla Firefox -- Version 4

For more details, including instructions on determining which browser version you have, please see this PDF.