Object recognition systems have made spectacular advances in recent years, but they rely on training datasets with thousands of high-quality, labelled examples per object category. Learning new objects from only a few examples could open the door to many new applications. For example, robotics manufacturing requires a system to quickly learn new parts, while assistive technologies need to be adapted to the unique needs and abilities of every individual.
Few-shot learning aims to reduce these demands by training models that canrecognizecompletely novel objects from only a fewexamples, say 1 to 10.In particular,meta-learning algorithmswhichlearn to learnusing episodic trainingareapromisingapproachto significantly reduce the number of training examplesneeded totrain a model.However, most research infew-shot learning has been driven bybenchmark datasets that lack the high variationthat applications face when deployed in therealworld.
In partnership with City, University of London, we introduce the ORBIT dataset and few-shot benchmark for learning new objects from only a few, high-variation examples to close this gap. The dataset and benchmark set a new standard for evaluating machine learning models in few-shot, high-variation learning scenarios, which will help to train models for higher performance in real-world scenarios. This work is done in collaboration with a multi-disciplinary team, including Simone Stumpf, Lida Theodorou, and Matthew Tobias Harris from City, University of London and Luisa Zintgraf from University of Oxford. The work was funded by Microsoft AI for Accessibility. You can read more about the ORBIT research project and its goal to make AI more inclusive of people with disabilities in this AI Blog post.
You can learn more aboutthe workin our research papers:ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition,published atthe International Conference of Computer Vision (ICCV2021),andDisability-first Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors, published at the 23rd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2021).
Youre also invited to join Senior Researcher DanielaMassicetifor a talk about the ORBIT benchmark dataset and harnessing few-shot learning for teachable AI at the firstMicrosoft Research Summit.Massicetiwill be presenting Bucket of me: Using few-shot learning to realize teachable AI systems as part of the Responsible AI track on October 19. To view the presentation on demand, register at the Research Summit event page.
The ORBIT benchmark dataset contains 3,822 videos of 486 objects recorded by 77 people who are blind or low vision using their mobile phonesa total of 2,687,934 frames. Code for loading the dataset, computing benchmark metrics, and running baselines is available at the ORBIT dataset GitHub page.
The ORBIT dataset and benchmark are inspired by a real-world application for the blind and low-vision community: teachable object recognizers. These allow a person to teach a system to recognize objects that may be important for them by capturing just a few short videos of those objects. These videos are then used to train an object recognizer that is personalized. This would allow a person who is blind to teach the object recognizer their house keys or favorite shirt, and then recognize them with a phone. Such objects cannot be identified by typical object recognizers as they are not included in common object recognition training datasets.
Teachable object recognition is an excellent example of a few-shot, high-variation scenario. Its few-shot because people can only capture a handful of short videos recorded to teach a new object. Most current machine learning models for object recognition require thousands of images to train. Its not feasible to have people submit videos at that scale, which is why few-shot learning is so important when people are teaching object recognizers from their own videos. Its high-variation because each person has only a few objects, and the videos they capture of these objects will vary in quality, blur, centrality of object, and other factors as shown in Figure 2.
While datasets are fundamental for driving innovation in machine learning, good metrics are just as important in helping researchers evaluate their work in realistic settings. Grounded in this challenging, real-world scenario, we propose a benchmark on the ORBIT dataset. Unlike typical computer vision benchmarks, performance on the teachable object recognition benchmark is measured based on input from each user.
This means that the trained machine learning model is given just the objects and associated videos for a single user, and it is evaluated by how well it can recognize that users objects. This process is done for each user in a set of test users. The result is a suite of metrics that more closely captures how well a teachable object recognizer would work for a single user in the real world.
Evaluations on highly cited few-shot learning models show that there is significant scope for innovation in high-variation, few-shot learning. Despite saturation of model performance on existing few-shot benchmarks, few-shot models only achieve 50-55% accuracy on the teachable object recognition benchmark. Moreover, there is a high variance between users. These results illustrate the need to make algorithms more robust to high-variation (or noisy) data.
Creating teachable object recognizers presents challenges for machine learning beyond object recognition. One example of a challenge posed by a human-centric task formulation is the need for the model to provide feedback to users about the data they provided when training in a new personal object. Is it enough data? Is it good-quality data? Uncertainty quantification is an area of machine learning that can contribute to solving this challenge.
Moreover, the challenges in building teachable object recognition systems go beyond machine learning algorithmic improvements, making it an area ripe for multi-disciplinary teams. Designing the feedback of the model to help users become better teachers requires a great deal of subtlety in user interaction. Supporting the adaptation of models to run on resource-constrained devices such as mobile phones is also a significant engineering task.
In summary, the ORBIT dataset and benchmark provide a rich playground to drive research in approaches that are more robust to few-shot, high-variation conditions, a step beyond existing curated vision datasets and benchmarks. In addition to the ORBIT benchmark, the dataset can be used to explore a wide set of other real-world recognition tasks. We hope that these contributions will not only have real-world impact by shaping the next generation of recognition tools for the blind and low-vision community, but also improve the robustness of computer vision systems across a broad range of other applications.
Read this article:
- As 2024 Travel Hits Pre-Covid Levels, Here's When To Go To Europe - Forbes - March 10th, 2024 [March 10th, 2024]
- Comparison of Impressions of COVID-19 Vaccinations Stratified by the Number of Vaccinations Among Japanese ... - Cureus - March 10th, 2024 [March 10th, 2024]
- Effect of the COVID-19 pandemic on mental health visits in primary care: an interrupted time series analysis from nine ... - The Lancet - March 10th, 2024 [March 10th, 2024]
- Faith and rehab keep Southwick woman on path to recovery from COVID - MassLive.com - March 10th, 2024 [March 10th, 2024]
- China to draw on Covid-19 experience to tackle future pandemics - theSun - March 10th, 2024 [March 10th, 2024]
- Free COVID tests through USPS are ending todayhere's where to get tests online - Reviewed - March 10th, 2024 [March 10th, 2024]
- Concern about COVID reaches record low across political spectrum: Survey - The Hill - March 10th, 2024 [March 10th, 2024]
- He Had 217 Covid Shots Without Side Effects, Study Finds - The New York Times - March 10th, 2024 [March 10th, 2024]
- A Man Got 217 COVID-19 Vaccines. Here's What Happened - TIME - March 10th, 2024 [March 10th, 2024]
- German man vaccinated 217 times against covid with no ill effects - The Washington Post - March 10th, 2024 [March 10th, 2024]
- Andrew Cuomo Faces House Subpoena Over Covid Deaths in Nursing Homes - The New York Times - March 10th, 2024 [March 10th, 2024]
- It's been 4 years since COVID hit Michigan. For Long COVID patients, the pandemic isn't over. Michigan Advance - Michigan Advance - March 10th, 2024 [March 10th, 2024]
- 4th Anniversary of the Covid-19 Pandemic - erienewsnow.com - March 10th, 2024 [March 10th, 2024]
- Reflecting on 4 years of the COVID-19 pandemic and discussing what's to come - WBUR News - March 10th, 2024 [March 10th, 2024]
- CDC shortens 5-day COVID isolation, updates guidance on masks and testing in new 2024 recommendations - CBS News - March 10th, 2024 [March 10th, 2024]
- Father, daughter convicted in fraud related to COVID-19 relief - The Cincinnati Enquirer - February 7th, 2024 [February 7th, 2024]
- Supreme Court to weigh whether Covid misinformation is protected speech - STAT - February 7th, 2024 [February 7th, 2024]
- Oklahoma leads country in long Covid - 2 News Oklahoma KJRH Tulsa - February 7th, 2024 [February 7th, 2024]
- MCFR firefighter dies after battle with COVID-19 - WCJB - February 7th, 2024 [February 7th, 2024]
- Audio-based AI classifiers show no evidence of improved COVID-19 screening over simple symptoms checkers - Nature.com - February 7th, 2024 [February 7th, 2024]
- Hidden death toll of COVID-19 pandemic revealed - Earth.com - February 7th, 2024 [February 7th, 2024]
- Switching arms improves effectiveness of two-dose vaccinations, OHSU study suggests - OHSU News - February 7th, 2024 [February 7th, 2024]
- Tributes paid to Irish health official 'central' to EU Covid-19 response - The Irish Times - February 7th, 2024 [February 7th, 2024]
- Court: Not wearing mask during COVID-19 health emergency isn't protected speech - Honolulu Star-Advertiser - February 7th, 2024 [February 7th, 2024]
- One arm or two? How you get vaccinated may make a difference - The Seattle Times - February 7th, 2024 [February 7th, 2024]
- 70% of kindergarteners didn't pass readiness test in pandemic, study estimates - University of Minnesota Twin Cities - February 7th, 2024 [February 7th, 2024]
- USS Theodore Roosevelt sailors roam free on Guam for first time since COVID-19 outbreak - Stars and Stripes - February 7th, 2024 [February 7th, 2024]
- Pandemic linked to 14% increase in underweight children in India - Medical Xpress - February 7th, 2024 [February 7th, 2024]
- COVID and travel: Should I still wear a mask on the plane? - USA TODAY - February 7th, 2024 [February 7th, 2024]
- Vaccine hesitancy and equity: lessons learned from the past and how they affect the COVID-19 countermeasure in ... - Globalization and Health - February 7th, 2024 [February 7th, 2024]
- Increase in STIs Among Adolescents Witnessed During COVID-19 Pandemic - Drug Topics - February 7th, 2024 [February 7th, 2024]
- VDH: COVID deaths not seeing decline - Vermont Biz - February 7th, 2024 [February 7th, 2024]
- US outpatient care for serious mental health issues declined during COVID-19 - University of Minnesota Twin Cities - February 7th, 2024 [February 7th, 2024]
- COVID-19 Vaccination in a Patient With Gluten Enteropathy: A Case Report - Cureus - February 7th, 2024 [February 7th, 2024]
- COVID-19 cases drop, but still lots of flu, RSV cases in Erie County - GoErie.com - February 7th, 2024 [February 7th, 2024]
- Lives versus livelihoods: The COVID-19 trade-off from an epidemiological-economic perspective - CEPR - February 7th, 2024 [February 7th, 2024]
- Weatherhead's Jonathan Ernest notes economic changes of childcare facilities following COVID-19 pandemic - The Daily | Case Western Reserve University - February 7th, 2024 [February 7th, 2024]
- U.S. adults face distress, unequal mental health care access during the COVID-19 era - News-Medical.Net - February 7th, 2024 [February 7th, 2024]
- Simnotrelvir to reduces the symptoms of mild to moderate COVID-19 - 2 Minute Medicine - February 7th, 2024 [February 7th, 2024]
- Vaccine Effectiveness: Which COVID-19 Shots Offer the Most Protection? - SciTechDaily - February 7th, 2024 [February 7th, 2024]
- New evidence confirms COVID-19 vaccination in pregnancy is safe for babies - Gavi, the Vaccine Alliance - February 7th, 2024 [February 7th, 2024]
- Audit Committee co-chairs, Evers at odds over interest from COVID-19 funds - WisPolitics.com - February 7th, 2024 [February 7th, 2024]
- Evaluation of Olfactory Dysfunction Among COVID-19 Patients in Baghdad, Iraq - Cureus - February 7th, 2024 [February 7th, 2024]
- COVID-19's impact on early education: Retrospective study shows decrease in kindergarten readiness - News-Medical.Net - February 7th, 2024 [February 7th, 2024]
- Rutgers researchers work on breakthrough COVID-19 treatment | Video - NJ Spotlight News - February 7th, 2024 [February 7th, 2024]
- Curious Iowa: Has the state spent all of its COVID-19 relief funding? - The Gazette - October 16th, 2023 [October 16th, 2023]
- Accelerating into Immunization Agenda 2030 with momentum from ... - Infectious Diseases of Poverty - BioMed Central - October 16th, 2023 [October 16th, 2023]
- Study Uncovers Why Young Children Suffer Less Severe COVID-19 - Technology Networks - October 16th, 2023 [October 16th, 2023]
- Wenstrup, Select Subcommittee Majority Members Investigate ... - House Committee on Oversight and Reform | - October 16th, 2023 [October 16th, 2023]
- With COVID-19 emergency orders lifted, employers seek guidance ... - Hartford Business Journal - October 16th, 2023 [October 16th, 2023]
- Computer-aided diagnosis of chest X-ray for COVID-19 diagnosis in ... - Nature.com - October 16th, 2023 [October 16th, 2023]
- Maternal COVID-19 Vaccination, Infection Boosts Infant Antibody ... - Contagionlive.com - October 16th, 2023 [October 16th, 2023]
- Three Middlesex County Individuals Admit COVID-19 Fraud ... - Department of Justice - October 16th, 2023 [October 16th, 2023]
- Seventeen Broward Sheriff's Office Employees Charged with COVID ... - Department of Justice - October 16th, 2023 [October 16th, 2023]
- New COVID-19 Booster and Flu Shot Available at Select Public ... - Mecklenburg County (.gov) - October 16th, 2023 [October 16th, 2023]
- Ban on COVID vaccine mandates by private businesses, including ... - The Texas Tribune - October 16th, 2023 [October 16th, 2023]
- What Are the Side Effects of the 2023 COVID Vaccine? Experts ... - Good Housekeeping - October 16th, 2023 [October 16th, 2023]
- Covid inquiry: Bereaved families relief as High Court dismisses Cabinet Office JR - openDemocracy - July 6th, 2023 [July 6th, 2023]
- UK health agency failed to account for 3.3bn of Covid inventory, say MPs - Financial Times - July 6th, 2023 [July 6th, 2023]
- COVID-19 Working Paper: Obesity Prevalence Among U.S. Adult ... - usda.gov - July 6th, 2023 [July 6th, 2023]
- California Changes Definition of COVID-19 Outbreak Easing the ... - Fisher Phillips - July 6th, 2023 [July 6th, 2023]
- New Study Shows Robust Pandemic Preparedness Strongly Linked ... - Nuclear Threat Initiative - July 6th, 2023 [July 6th, 2023]
- COVID-19 Infection Reduces the Risk of UTIs, Bacteremia, and ... - Contagionlive.com - July 6th, 2023 [July 6th, 2023]
- Europe Phases Out COVID-19 Flexible Regulations - Pharmaceutical Technology Magazine - July 6th, 2023 [July 6th, 2023]
- Travel Pains and COVID: How to Decide if You're Ready to Travel - CreakyJoints - July 6th, 2023 [July 6th, 2023]
- COVID Depression and Anxiety | Johns Hopkins Medicine - April 17th, 2023 [April 17th, 2023]
- Man declared dead due to Covid found 'shockingly' alive after 2 years in MP - Hindustan Times - April 17th, 2023 [April 17th, 2023]
- Senate votes to end Covid-19 emergency, 3 years after initial declaration - CNN - March 31st, 2023 [March 31st, 2023]
- DC COVID-19 centers closing Friday: Here's what you need to know - WJLA - March 31st, 2023 [March 31st, 2023]
- Can grade retention help with COVID-19 learning recovery in schools? - Brookings Institution - March 28th, 2023 [March 28th, 2023]
- On 3-year anniversary of COVID-19 in NC, Winston-Salem woman shares story of loss and healing after losing 2 loved ones - WXII12 Winston-Salem - March 4th, 2023 [March 4th, 2023]
- Oregon, Washington will lift mask requirements in health care settings on April 3 - KATU - March 4th, 2023 [March 4th, 2023]
- WHO says all theories for COVID origin 'remain on table' as lab leak theory gains traction - Sky News - March 4th, 2023 [March 4th, 2023]
- New COVID-19 omicron subvariant XBB.1.5 sweeping the nation, making up majority of cases - ABC Action News Tampa Bay - February 15th, 2023 [February 15th, 2023]
- Man gets 21 years in prison for stabbing wife in the Ozarks over COVID-19 stimulus check and their children - Law & Crime - February 15th, 2023 [February 15th, 2023]
- Healthline: Medical information and health advice you can trust. - January 2nd, 2023 [January 2nd, 2023]
- China can expect repeat Covid infections with new strains on the way: experts - South China Morning Post - January 2nd, 2023 [January 2nd, 2023]
- PCR tests for travellers reintroduced around the world as Covid-19 cases surge - The National - January 2nd, 2023 [January 2nd, 2023]
- Covid-19 surge after Covid-19 surge has made it impossible for US hospitals to plan for the future - Vox.com - January 2nd, 2023 [January 2nd, 2023]
- COVID-19 in China: Demand for a particular fruit rises as people seek natural remedies to fight the virus - WION - December 21st, 2022 [December 21st, 2022]