In recent years, entire industries have popped up that rely on the delicate interplay between human workers and automated software. Companies like Facebook work to keep hateful and violent content off their platforms usinga combination of automated filtering and human moderators. In the medical field, researchers at MIT and elsewhere have used machine learning to help radiologistsbetter detect different forms of cancer.
What can be tricky about these hybrid approaches is understanding when to rely on the expertise of people versus programs. This isnt always merely a question of who does a task better; indeed, if a person has limited bandwidth, the system may have to be trained to minimize how often it asks for help.
To tackle this complex issue, researchers from MITs Computer Science and Artificial Intelligence Lab (CSAIL) have developed a machine learning system that can either make a prediction about a task, or defer the decision to an expert. Most importantly, it can adapt when and how often it defers to its human collaborator, based on factors such as its teammates availability and level of experience.
The team trained the system on multiple tasks, including looking at chest X-rays to diagnose specific conditions such as atelectasis (lung collapse) and cardiomegaly (an enlarged heart). In the case of cardiomegaly, they found that their human-AI hybrid model performed 8 percent better than either could on their own (based on AU-ROC scores).
In medical environments where doctors dont have many extra cycles, its not the best use of their time to have them look at every single data point from a given patients file, says PhD student Hussein Mozannar, lead author with David Sontag, the Von Helmholtz Associate Professor of Medical Engineering in the Department of Electrical Engineering and Computer Science, of a new paper about the system that was recently presented at the International Conference of Machine Learning. In that sort of scenario, its important for the system to be especially sensitive to their time and only ask for their help when absolutely necessary.
The system has two parts: a classifier that can predict a certain subset of tasks, and a rejector that decides whether a given task should be handled by either its own classifier or the human expert.
Through experiments on tasks in medical diagnosis and text/image classification, the team showed that their approach not only achieves better accuracy than baselines, but does so with a lower computational cost and with far fewer training data samples.
Our algorithms allow you to optimize for whatever choice you want, whether thats the specific prediction accuracy or the cost of the experts time and effort, says Sontag, who is also a member of MITs Institute for Medical Engineering and Science. Moreover, by interpreting the learned rejector, the system provides insights into how experts make decisions, and in which settings AI may be more appropriate, or vice-versa.
The systems particular ability to help detect offensive text and images could also have interesting implications for content moderation. Mozanner suggests that it could be used at companies like Facebook in conjunction with a team of human moderators. (He is hopeful that such systems could minimize the amount of hateful or traumatic posts that human moderators have to review every day.)
Sontag clarified that the team has not yet tested the system with human experts, but instead developed a series of synthetic experts so that they could tweak parameters such as experience and availability. In order to work with a new expert its never seen before, the system would need some minimal onboarding to get trained on the persons particular strengths and weaknesses.
In future work, the team plans to test their approach with real human experts, such as radiologists for X-ray diagnosis. They will also explore how to develop systems that can learn from biased expert data, as well as systems that can work with and defer to several experts at once.For example, Sontag imagines a hospital scenario where the system could collaborate with different radiologists who are more experienced with different patient populations.
There are many obstacles that understandably prohibit full automation in clinical settings, including issues of trust and accountability, says Sontag. We hope that our method will inspire machine learning practitioners to get more creative in integrating real-time human expertise into their algorithms.
Mozanner is affiliated with both CSAIL and the MIT Institute for Data, Systems and Society (IDSS). The teams work was supported, in part, by the National Science Foundation.
More here:
An automated health care system that understands when to step in - MIT News
- How good is fish for your diet? - November 8th, 2009 [November 8th, 2009]
- What foods have both fiber and protein? - November 8th, 2009 [November 8th, 2009]
- Is There Evidence That Acupuncture Works? - November 8th, 2009 [November 8th, 2009]
- What are some good fiber foods? - November 8th, 2009 [November 8th, 2009]
- Can Green Tea fight HIV infection? - November 8th, 2009 [November 8th, 2009]
- Quality of GNC Supplements - November 8th, 2009 [November 8th, 2009]
- The Gene Smart Diet - November 8th, 2009 [November 8th, 2009]
- Tips for eating in a vegetarian lifestyle - November 8th, 2009 [November 8th, 2009]
- Regarding the Starting of a new diet regimen - November 8th, 2009 [November 8th, 2009]
- Thoughts on Dropping Weight - November 8th, 2009 [November 8th, 2009]
- What do doctors think about vitamin E supplements? - November 8th, 2009 [November 8th, 2009]
- What herbs or supplements are commonly used for depression? - November 8th, 2009 [November 8th, 2009]
- Is there a safe way to lose a lot of weight? - November 8th, 2009 [November 8th, 2009]
- Diets that promise you will lose weight - November 8th, 2009 [November 8th, 2009]
- Are Fish Toxins Linked to Diabetes? - November 8th, 2009 [November 8th, 2009]
- Can Plastic Surgery Help Migraines? - November 8th, 2009 [November 8th, 2009]
- Weight Loss Surgery Eradicates Diabetes Symptoms - November 8th, 2009 [November 8th, 2009]
- Found: A Gene That Controls Fat Cells - November 8th, 2009 [November 8th, 2009]
- Sugars and Starches, what's the difference? - November 8th, 2009 [November 8th, 2009]
- Improving your health with a serving of nuts? - November 8th, 2009 [November 8th, 2009]
- Is Your Diet Causing You To Be Depressed? - December 13th, 2009 [December 13th, 2009]
- Placebo Effect: Magnetic Bracelet Therapy - December 13th, 2009 [December 13th, 2009]
- Does serotonin promote sleep? - December 13th, 2009 [December 13th, 2009]
- Decreased energy levels - Overeating, Sleep, Nutrition - February 1st, 2010 [February 1st, 2010]
- When will the body begin to cannibalize muscle tissue? - February 7th, 2010 [February 7th, 2010]
- Foods that are high in antioxidants - February 7th, 2010 [February 7th, 2010]
- Editorial: Reduce health care costs by cutting administrative overhead - April 1st, 2012 [April 1st, 2012]
- Health Care Winners: Centene, Onyxx - April 25th, 2012 [April 25th, 2012]
- Dan Morain: Nurses union puts politics ahead of health - April 25th, 2012 [April 25th, 2012]
- What's next if Obamacare falls? - April 25th, 2012 [April 25th, 2012]
- Global Managed Health Care Services Industry - April 25th, 2012 [April 25th, 2012]
- Immigration Fight Echoes Health-Care Case at High Court - April 25th, 2012 [April 25th, 2012]
- Medical Centers Lead Workplace Wellness Effort - April 25th, 2012 [April 25th, 2012]
- The next health care overhaul? Look to employers - April 25th, 2012 [April 25th, 2012]
- Capital BlueCross Hosts Community Health Care Forums Focused on Managing Costs While Maintaining Quality - April 25th, 2012 [April 25th, 2012]
- County committee to look at area health care - April 25th, 2012 [April 25th, 2012]
- President Obama Calls on Students to Tell Congress: #DontDoubleMyRate - Video - April 25th, 2012 [April 25th, 2012]
- Sen. Coburn: Competition in Health Care to Allocate Resources - Video - April 25th, 2012 [April 25th, 2012]
- Jesse Kelly health care - Video - April 25th, 2012 [April 25th, 2012]
- PHC Vision Statement (short version) - Video - April 30th, 2012 [April 30th, 2012]
- The Bioeconomy Blueprint Panel - Video - April 30th, 2012 [April 30th, 2012]
- Providence Health Care's Vision Statement - Video - April 30th, 2012 [April 30th, 2012]
- Department of Health and Human Services: Minority Health Blogger Townhall - Video - April 30th, 2012 [April 30th, 2012]
- Health care worker accused of stealing identities of brain i - Video - April 30th, 2012 [April 30th, 2012]
- Health Care Reform, Part 1 of 3 | KYVE Insiders Roundtable - Video - April 30th, 2012 [April 30th, 2012]
- Health Care Reform: The ACA and Beyond - Video - April 30th, 2012 [April 30th, 2012]
- Is Broccoli Like Health Insurance? - Video - April 30th, 2012 [April 30th, 2012]
- Did Obama make a mistake on health care? - Video - April 30th, 2012 [April 30th, 2012]
- How Democrats Lie About Health Care - Video - April 30th, 2012 [April 30th, 2012]
- Toobin on Health Care: This was a "judicial hissy-fit" - Video - April 30th, 2012 [April 30th, 2012]
- President Obama says health care law will stand - Video - April 30th, 2012 [April 30th, 2012]
- Obama Defends Health Care Law From 'Judicial Activism' - Video - April 30th, 2012 [April 30th, 2012]
- Obama Healthcare Individual Mandate - Video - April 30th, 2012 [April 30th, 2012]
- President Obama attacks Supreme Court on health care - Video - April 30th, 2012 [April 30th, 2012]
- Raw Audio: High Court Dissects Health Care Act - Video - April 30th, 2012 [April 30th, 2012]
- Health care on trial - Video - April 30th, 2012 [April 30th, 2012]
- Final Day of Health Care Law Arguments Before Supreme Court - Video - April 30th, 2012 [April 30th, 2012]
- Feds to put up $1.9B for Oregon health overhaul - May 4th, 2012 [May 4th, 2012]
- Scuttling health care act will freeze Medicare, White House warns - May 4th, 2012 [May 4th, 2012]
- U.S. Health Care Spending High, But Quality Lags: Report - May 4th, 2012 [May 4th, 2012]
- Yes, the Health-Care Mandate Is About Liberty - May 4th, 2012 [May 4th, 2012]
- Health center gets $1M federal grant - May 4th, 2012 [May 4th, 2012]
- Health-care reform panel considers exchange options for Va. - May 4th, 2012 [May 4th, 2012]
- Hmong health care gap focus of Healthy House dinner - May 4th, 2012 [May 4th, 2012]
- Walsh-led health bill to be unveiled - May 4th, 2012 [May 4th, 2012]
- Ontario health system confusing for ailing seniors, study finds - May 4th, 2012 [May 4th, 2012]
- 'Health Care Deserts' More Common in Black Neighborhoods - May 5th, 2012 [May 5th, 2012]
- With federal money, Oregon kicks health care reform into high gear - May 5th, 2012 [May 5th, 2012]
- Nurse practitioners tackling more 'doctor' tasks - May 5th, 2012 [May 5th, 2012]
- Konza Prairie Health Center Receives $4.5 Million Grant - May 5th, 2012 [May 5th, 2012]
- Alberta wages hurting Sask. health care - May 5th, 2012 [May 5th, 2012]
- House releases plan to cut growth of Massachusetts health spending in half - May 5th, 2012 [May 5th, 2012]
- Mass. House Will Unveil Bill Seeking To Rein In Health Costs - May 5th, 2012 [May 5th, 2012]
- President Obama Welcomes the Kentucky Wildcats - Video - May 5th, 2012 [May 5th, 2012]
- President Obama Speaks on College Affordability - Video - May 5th, 2012 [May 5th, 2012]
- My First Job: Gene Sperling - Video - May 5th, 2012 [May 5th, 2012]
- Health-care costs worry near-retirees - May 7th, 2012 [May 7th, 2012]
- Massachusetts Institutes Health-Care Price Controls. Is America Next? - May 7th, 2012 [May 7th, 2012]
- Massachusetts Moves Toward Health-Care Price Controls. Is America Next? - May 7th, 2012 [May 7th, 2012]
- GOP plan boosts Pentagon, cuts social programs - May 7th, 2012 [May 7th, 2012]