Betting on Results
Posted: November 12th, 2015 | Author: Michael Goldstein | | 29 Comments »
Alex pointed me to this on 538:
Today in the Proceedings of the National Academy of Sciences, Nosek and an international team of researchers present a tool for doing that — betting. They found that compared to simply asking experts to predict the likelihood that studies will be reproduced, asking them to bet money on the outcomes improved the accuracy of the guesses.
The researchers began by selecting some studies slated for replication in the Reproducibility Project: Psychology — a project that aimed to reproduce 100 studies published in three high-profile psychology journals in 2008. They then recruited psychology researchers to take part in two prediction markets. These are the same types of markets that people use to bet on who’s going to be president. In this case, though, researchers were betting on whether a study would replicate or not.
Before each prediction market began, participants (47 actively took part in the first market, 45 traded in the second) were asked two questions: How likely do you think it is that each hypothesis in this market will be replicated, and how well do you know this topic?
They were then given points worth a total of $100 to bet on whether the studies in their prediction market would replicate. A replication was considered successful if it produced a result, with a p-value of less than 0.05, in the same direction as the original result. Players entered the market with 10,000 points each and could buy and sell contracts for each hypothesis. If a replication succeeded, then its share paid 100, but if the replication failed, then it paid nothing. “If you believe the result will be replicated, you buy the contract, which increases the price,” said the study’s lead author, Anna Dreber, an economist at the Stockholm School of Economics. “If you don’t believe in a study, then you can short-sell it.”
….The prediction market correctly called nearly three-quarters (71 percent) of the attempted replications, 39 percent of which succeeded in the reproducibility project. By comparison, the survey conducted before the market began correctly predicted the result of only 58 percent of the replication studies. The prediction market anticipated a finding’s reproducibility better than asking the same bunch of experts to put their best guesses in a hat.
Our team has sometimes done that at Bridge. For example, we’ve had a pool for over a year to predict our kids’ KCPE scores. Not sure yet how we did!
“The beauty of the market is that we allow people to be Bayesian,” Dreber said. People come in with some prior belief, but they can also follow prices to see what other people believe and may update their beliefs accordingly. While the survey required everyone to provide an estimate for every study, participants in the market could focus their bets on the studies they felt most sure of, and as a result, rough guesses didn’t skew the averages as much.
Finally, putting money at stake motivated people to try harder to find the right answer and reveal what they really think. “It’s really putting your money where your mouth is,” Dreber said. “You want to see what people do, not what they say.”
I’m not sure if our “winner gets to pick restaurant, everyone else takes him/her out” is particularly motivating. But I do think we changed our picks to a degree as we noticed what others said, and were allowed to ask their rationale.
Hello Mike,
This is an exciting new school network that you are founding and expanding! I have been browsing the website to learn more. I discovered it while looking for an e-mail address for you, which I was unable to find, but I did find your blog.
I work with KIPP in St. Louis, MO, and have been reading and studying your book, Phoning Parents, to create structures for our school in its founding year.
I wanted to share an online course I created for our staff as they study the book this summer leading up to PD, in case it could be helpful to your colleagues at Match, or perhaps now at Bridge International. The link to the folder is in the “website” field.
Enter comment here.
Europe, and in Ancient Russia
Enter comment here.
new texts were rewritten
Europe, and in Ancient Russia
Enter comment here.
Проблемы и решения в сфере искусственного интеллекта
Understanding Artificial Intelligence, Machine Learning, and Deep Learning
Find the Right Products With Our Product Reviews
Tech and Artificial Intelligence: Cutting-edge Trends
Learn Quickly With Tutorials and Tips
Restore Your Nervous System and Achieve Inner Peace with Self-Hypnosis and Healing Techniques
mental resilience self-healing practices.
Все о медицинской книжке: важная информация для каждого
купить медкнижку с анализами без прохождения врачей https://medknizhki-cena.ru.
Enter comment here.
Enter comment here.
Enter comment here.
Enter comment here.
Enter comment here.
Enter comment here.
Enter comment here.
Enter comment here.
Enter comment here.
Найдите актуальное зеркало pinup казино и продолжайте играть в свои любимые слоты и азартные игры без проблем доступа на официальном сайте.
Enter comment here.
изготовление металлоконструкций по чертежам заказчика
1win – популярный онлайн-букмекер с широким выбором ставок высокими коэффициентами и удобным интерфейсом для игроков.
Enter comment here.
Enter comment here.