Help for Organizer Users

Getting around the Valid Eval platform as an Organizer
The Settings page
Inviting Judges
Manually assigning Judges to Teams, and vice versa
How do I know how many teams to assign to each judge?

Here is a link to a spreadsheet that walks you through the process of determining how many teams should be assigned to each judge.

What's the optimal number of evaluations per team?

Valid Eval's platform allow you to measure the impact of your event holistically. Each additional judge that evaluates a team adds value. (More thorough triangulation. More comments and recommendations from different perspectives.) However, the volunteers you count on to do this work are not a free resource.

When we account for (1) the diminishing returns for adding incremental judges, and (2) the opportunity cost for the judges' participation, we find an optimum around 7 or 8 judges per team.

A little more detail. The difference between 2 judges evaluating a performance and 3 judges in the value of feedback to an entrepreneur is huge. The different between 21 and 22 judges looking at the same performance is pretty tiny. When we account for the time of the judges at $200 / hour, the value to the community, net of the cost of the judges' time, peaks around 7 to 8 judges looking at a given team.

What's the minimum number of judges that should look at each team?

If 7 to 8 judges is the ideal (as discussed in another FAQ), but I cannot ask my limited pool of judges to do that much work, what should I do?

We think it's helpful to think about 7 to 8 as hitting a "home run." Most of our clients' events are not blessed with enough volunteer-power to get coverage that deep for each and every company.

Valid Eval's experience is that you want to wind up with a minimum of 3 judges who actually followed through and completed those evaluations. Of course, every event winds up with a handful of judges who -- despite their good intentions -- flaked out and didn't complete all of their evaluations. Accounting for this attrition, we think that an average of just over 4 judge*looks per company will get you good results.

Automatically matching judges to teams.
How do I change the number of evaluations per judge for Auto-Matching?

Please email us to request the change by emailing us here.

How do I un-recuse a Judge?

In the case of a judge who has accidentally recused him or herself from a particular company, you may override that recusal by adding that company in question back to their list of teams to evaluate. 

Should I show the Judge Analysis report to my judges?

It's up to you.

Valid Eval thinks it is a best practice to openly and transparently show the judges how they all did relative to their peers. Our "Awesomeness" algorithm blends their effort (number of evaluations, % of feedback clicks), their inter-rater reliability* (measured three ways), and their helpfulness (number of comments, length of comment text, recommendations made).

If part of your mission is to build capacity within your community, we think closing the loop on the expert evaluators is a good way to do this. If the judges are warned that they'll be measured, you can use their competitive nature to help get better engagement with the companies and more thoughtful assessments.

* Inter-rater reliability is a term from the field of Assessment. It measures the distances between one judge's scores and the scores of her peers on the same companies.

When should I disable the relative performance feedback for companies?

This question refers to the "Show relative performance information" toggle setting you control for each phase of the evaluation.

Best practice here is to leave this toggle "ON" in almost all cases. It's incredibly helpful to entrepreneurs to see where they stand relative to their peers on each and every dimension of the rubric you've chosen for this event.

Many of the entrepreneur end-users we've spoken with have said the "box plot" visualizations Valid Eval providers were the most useful aspect of the feedback they got.

So, when do you turn it "OFF?" If you are doing a mentoring phase where there are no decisions being made as to who wins, who advances, etc., then it's appropriate to have the toggle set to OFF. Why? If you aren't making decisions and are using a particular phase of your event for mentoring or practice feedback, evidence of a poor showing can hurt motivation.

In other words, someone will come in last during those evaluations. The knowledge that they were last (when we aren't make a decisions anyway) can be very discouraging and keep the entrepreneur from focusing on improving for whatever comes next. In some cases, we've seen entrepreneurs drop out of programs when they learned that they ranked at the bottom of a consequence-free "mock judging" exercise. 

So, best practice boils down to this: If you're making decisions based on the results from a phase, keep the "show relative performance information" toggle ON. If you're purely in teaching more, set the toggle to OFF.

How do I change the judging windows a.k.a. the phases?

Do make those changes, you need to contact Valid Eval here: info@valideval.com .

The art of fake deadlines -- for applicants.

Everyone is incredibly busy these days. The vast majority of the applications for your event will come is the day of -- if not the last couple of hours before -- the deadline. The same of true of your volunteer experts: They all wait until the last minute.

Happily, you can use this to your advantage with a bit of prior planning. 

For applicants, plan an extra few days, if not an extra week, after your published deadline to gather late-coming applications. Here's the key, do not extend the deadline until the initial deadline has passed. This way, you'll capture the initial crop of procrastinators and get an idea of how many applications you will have. If you extend the deadline while there is still time left, a large group of the procrastinators will happily delay without tipping their hand and you won't get any indication of how you are doing in recruiting applications.

Please remember that the application window is opened and closed by you. The top toggle on your Settings page, "

Similarly, you control whether existing teams have the ability to edit there profiles once it's been established. Even when the "Team sign-up" toggle is off, teams may still edits their profiles, including uploading new documents, if the "Teams can edit profiles" toggle is set to "ON."

 

Help for General Users

I forgot my password.

If you do not remember your password, you may reset it here: https://app.valideval.com/v2/reset-password-request.

The reset will only work if you enter an email address that Valid Eval knows about. For obvious reasons, our system only knows the email address(es) you've set up in your profile.

If you are like us and hate remembering passwords, the Login with LinkedIn option is a good solution.

I would like to change my password.

First, log into Valid Eval here: https://app.valideval.com/v2/users/authentication. From the “Dashboard,” click the “Account” link at the top right of the page. This will lead you to you “Global Account Profile” page. Once there, you may edit your profile, including changing your password.

If you do not remember your password, you may reset it here: https://app.valideval.com/v2/reset-password-request by entering the email address associated with your Valid Eval account. 

I cannot login with LinkedIn. How do I do that?

Sometimes, it's nice to just be able to click a button to log into a website.

In just two minutes, this video explains how to set this feature up so that you don't need to remember your Valid Eval password: https://vimeo.com/valideval/linkedin

 

I don't want to login with LinkedIn. How do I just use my email and password?

Go to the usual login page: https://app.valideval.com/v2/users/authentication. Then, Click the link directly below the "Login with LinkedIn" button, which says "Login without LinkedIn."

Once you do that, you will see the typical boxes asking for your email address and password.

I want to add / remove an email address from my account.

First, log into Valid Eval here: https://app.valideval.com/v2/users/authentication. From the “Dashboard,” click the “Account” link at the top right of the page. This will lead you to your “Global Account Profile” page. Once there, you may edit your profile, including adding additional email addresses to your profile.

Why would I do this?

You may easily login with any of the email addresses you have, all of which will be linked to your one-and-only profile. This way, you won't have to remember which email address you used. All of them will use the same password since they all point back to the same Account Profile.

Adding additional email addresses also helps you avoid future duplicate Valid Eval accounts. For example, if your account is currently setup with only your work email address, a future invitation to be a Judge to your personal email address would require you to set up a new profile. Then you'd have two Valid Eval accounts. Of course, we can fix that, but it's a pain in the butt for you and it's easily avoided. 

We think it's a good idea to populate your profile with all of the email addresses you frequently use.

I've recently had more entrepreneurial experiences. How do I change my profile?

Valid Eval captures data on your experience levels and the type of work you do. If you've gained experiences, or have switched professions, please let us know by making changes in your profile. 

First, log into Valid Eval here: https://app.valideval.com/v2/users/authentication. From the “Dashboard,” click the “Account” link at the top right of the page. This will lead you to you “Global Account Profile” page.

On that page, you may edit your entire profile, including the parts that help us anonymously give your evaluations context.

Browser Compatibility

If you the site looks broken you may be using a browser that is too old to run Valid Evaluation. 

We require the following versions (or newer) of these popular browsers:

* Internet Explorer -- Version 9

* Google Chrome -- Version 11

* Apple Safari -- Version 5

* Mozilla Firefox -- Version 4

For more details, including instructions on determining which browser version you have, please see this PDF.

 

Help for Judges

Judging tutorial video

If you never got around to watching this, please do. It runs just 3 minutes and it's well worth the investment: The payoff is less stress and frustration for you and a much better experience for the folks you're evaluating and the organizers of this event. 

 

Incomplete scores?? I thought I did everything I was supposed to do!

You are probably reading this because the folks running this event and telling you that you haven't completed your scoring, and you think you have.

Chances are, you are very close to the goal line. Any previous work you've done has been saved and you'll be building on that. 

Your task is simply to finish up by assigning a numerical score to all of the dimensions for the team(s) in question.

Valid Eval requires both feedback clicks and numerical scores on all dimensions. Otherwise, the evaluation is incomplete and the scores are not counted.

Please have (another) look at this video https://vimeo.com/valideval/evaluation. If you’d like, zoom ahead to the 1:30 mark to see how the numerical scoring process works.

 

 

How are we supposed to pick the winners?

Unlike other approaches to scoring complex performances, Valid Eval does not require you to make arbitrary "stack rankings," where you're asked to pick the best company from the batch you review.

On our platform, you don't pick winners. This is one of the biggest benefits of using Valid Eval as a judge. Your job is simply to provide the most accurate evidence-based assessment possible using the rubric chosen for this event. The score distributions are typically beautiful bell curves, which make it easy for the organizers and judges to agree on which companies were the strongest. 

Please be as thorough and precise as you can based on your expert opinions as to how the teams performed in each dimension of the rubric. (Remember that while all dimensions must be scored, the best judges make feedback clicks only in the relevant rows. Generally, this is 75% or so of the possible feedback clicks.)

I don't want to be mean. Am I allowed to judge nicely?

Certain teams are not a good fit based on the evaluation criteria captured in this particular rubric. You are not doing that team a favor by "being nice." If they do poorly on certain dimensions of the rubric, that's just the way it is. If they are awful, then they are awful.

The rubrics we deploy are custom tailored to measure the attributes that matter in reaching a specific goal our clients defined. Same goes for the application templates, which are custom tailored to fit the rubrics. Please know that every applicant was provided with the rubric and very clear instructions as to what we were looking for before they applied. A failure to present you with strong performance is on them. 

Entrepreneurship can be brutal. But, the cruelest cuts are from the mistakes others saw us making, but stood by silently while we wasted time and money going down the wrong path. On Valid Eval, your evaluations and comments are anonymous. The kindest and most valuable thing you can do for entrepreneurs is to provide brutally honest feedback.

Keep it constructive, civil and compassionate. But, don't pull punches!

What am I supposed to be judging? Is due diligence part of what's expected?

Your evaluations should be exclusively based on the materials the company uploaded for your review. Due diligence is beyond the scope of what we are asking you for. The organizers of this event will take appropriate steps to verify eligibility and the applicants' honesty.

This is important for the integrity of the evaluation process. It ensures that each performance for a given company is identical, which is crucial for the veracity of the data we will analyze later. 

We have taken a good deal of care to ensure that the "playing field" is level by providing all of the companies with the same templates and instructions. Please respect the process and keep knowledge you may have from activities beyond their application out of your assessment.

If you are enthused by a particular company and want to dive into their company and the competitive landscape via your own research, that's great! But, please do not incorporate those activities into the your evaluation.

Recommendations. What are those for?

Recommendations are the place for you to give actionable advice. Think of these as bits of a prioritized to-do list for the entrepreneurs.

Please do not confuse a recommendation with a comment. A comment is anonymous, a recommendation is attributed to you. A comment is a general purpose place to share your thoughts about the performance you've evaluate. A recommendation offers a hand up. 

In short, your recommendation should point to a specific action the company should take to improve its fortunes.

What is Judge Awesomeness and why are you measuring me?

Your "awesomeness" as a judge is a measurement of how you do in three key areas:

  1. Level of Effort,
  2. Inter-Rater Reliability, and
  3. Helpfulness

At Valid Eval, we believe that the judges on our platform who complete their tasks thoroughly, whose scores are in line with those of other experts, and who offer a helping hand to those they evaluate are AWESOME!

Our clients -- the folks who convened the teams and you guys as experts -- value knowing which judges are the most awesome. And, by contrast, which judges shouldn't be invited back next time. 

What exactly are we measuring? That's a secret. But, if you sincerely engage, evaluate honestly and invest a little extra time to make comments (anonymously) and recommendations (attributed to you), your impact will be awesome!

When should I recuse myself from judging?

The organizers of your event will be able to give you specific guidance on recusals. But, here is the Valid Eval party line.

Generally speaking, if you have a financial stake in a company to which you have been matched, you should recuse yourself. What does this mean? Examples of recusal-worthy financial involvement would be owning shares / options / warrants in the company, earning money from the company, having a board of directors seat, or being engaged in a business partnership of any kind with the company. With this in mind, if you find yourself wondering if you should recuse yourself, you probably should do so. 

Generally speaking, if you know the people in the company, or have seen their pitch before, or you are friends with a board member those are not grounds for recusal.

Does a recusal impact my Awesomeness?

No!

We never penalize you for being honest and bowing out of an evaluation where you have a conflict.

Recusals are cool with us!

My scores are not showing up in the “My Evaluations” list. Why?

You are reading this for one of two reasons. Either:

  1. The folks running this event and telling you that you haven't completed your scoring, and you think you have, or
  2. You've arrived here on your own because that darn "My Evaluations" list isn't showing numerical scores in the right column, and you've already clicked a bunch of boxes and have left helpful comments. 

Chances are, you are very close to the goal line. Any previous work you've done has been saved and you'll be building on that. 

Your task is simply to finish up by assigning a numerical score to all of the dimensions for the team(s) in question.

Valid Eval requires both feedback clicks and numerical scores on all dimensions. Otherwise, the evaluation is incomplete and the scores are not counted.

Please have (another) look at this video https://vimeo.com/valideval/evaluation. If you’d like, zoom ahead to the 1:30 mark to see how the numerical scoring process works.

 

 

 

Help for Teams

How do I access and make sense of feedback?
How do I do a Self-Evaluation?
How do I create a Voice-Over of my slides and upload the movie?

Part 1: Creating a movie file of your voice-over and slides

Depending on the details of the your preferred presentation software, the steps will vary. If you already know how to create a simple voice-over narration movie of your slide deck, you may skip to Part 2.

Mac, Keynote: https://vimeo.com/172466216

Mac, PowerPoint 2011https://support.office.com/en-us/article/Save-a-presentation-as-a-movie-file-4e1ebcc1-f46b-47b6-922a-bac76c4a5691  Note: This WILL NOT WORK AS INTENDED. This version of PowerPoint will not export your saved audio to a movie file. If you are stuck with PPT 2011 on a Mac, we recommend purchasing Keynote or a newer version of PowerPoint.

Windows, PowerPoint 2010 https://support.office.com/en-us/article/Turn-your-presentation-into-a-video-c140551f-cb37-4818-b5d4-3e30815c3e83

Windows, PowerPoint 2013 http://www.ellenfinkelstein.com/pptblog/post-your-powerpoint-presentation-as-a-video-to-youtube/

Windows, PowerPoint 2013 & 2016 https://support.office.com/en-US/article/Save-your-presentation-as-a-video-FAFB9713-14CD-4013-BCC7-0879E6B7E6CE

 

Part 2: Uploading your newly created movie file

https://vimeo.com/172470076

 

Part 3: Adding the URL to your Valid Eval profile

Login here: https://app.valideval.com/v2/users/authentication.

From the Dashboard, please find your company’s name to the right of the logo and name for this event. Click the link with your company’s name.

Find and click the “Profile” link in the navigation area at the top right of the page.

Please find the field labeled “Presentation Video” on the right side of your profile. Simply paste the new URL into the box above the blue "Replace URL" button. Then, click that button.

We CANNOT ACCEPT PASSWORD PROTECTED VIDEOS. Please watch the tutorial in Part 2 above regarding the appropriate privacy settings to make sure your video is appropriately protected. We will only share the URL with the assigned evaluators.

If you’ve successfully replaced the URL, a modal pop-up window will say “URL Replaced.” Click “OK” to dismiss the window.

To double-confirm that you’ve successfully replaced the file, click the video icon or the blue link name of the URL, such as “Video” or similar. Doing this will open the fresh video URL in a new tab.