2013 TSA cohort traits
The traits scored in the cohort represent measures of confidence/fear, quality of hunting related behaviors, and dog-trainer interaction characteristics19,20. The traits Chase/Retrieve, Physical Possession, and Independent Possession were measured in both the Airport Terminal and Environmental tests whereas five and seven other traits were specific to each test, respectively (Table 1). The Airport Terminal tests include the search for a scented towel placed in a mock terminal and observation of a dogs responsiveness to the handler. This represents the actual odor detection work expected of fully trained and deployed dogs. Because the tasks were consistent between the time periods, the Airport Terminal tests demonstrate improvements of the dogs with age. All trait scores except for Physical and Independent Possession increased over time, with the largest increase between the 6- and 9-month tests (Fig.1a). This may be due to puppies having increased possessiveness and lack of training at younger ages. The general improvement over time could be due to the increased age of the dogs or to the testing experience gained. Compared to accepted dogs, those eliminated from the program for behavioral reasons had lower mean scores across all traits.
(a) Radar plots of the mean scores for each of the traits for the airport terminal tests. (b) Radar plots of the mean scores for each of the traits in the environmental tests; M03=BX (gift shop), M06=Woodshop, M09=Airport Cargo, M12=Airport Terminal.
Environmental tests involved taking dogs on a walk, a search, and playing with toys in a noisy location that changed for each time point. The traits measured a variety of dog behaviors as they moved through the locations, and their performance while engaging with toys. Accepted dogs had both higher and more consistent scores across the tests (Fig.1b). The largest separation of scores between accepted dogs and those eliminated for behavior occurred at 6-months, at the Woodshop. That suggests this test and environment combination might best predict which dogs will be accepted into the training program. Among the traits that showed the greatest separation between the two outcomes were Physical and Independent Possession, and Confidence.
Three different classification Machine Learning algorithms were employed to predict acceptance based on their ability to handle binary classifiers: Logistic Regression, Support Vector Machines, and Random Forest. Data were split into training (70%) and testing (30%) datasets with equivalent ratios of success and behavioral elimination status as the parent dataset. Following training of the model, metrics were reported for the quality of the model as described in the Methods. Prediction of success for the Airport Terminal tests yielded consistently high accuracies between 70 and 87% (Table 2). The ability to predict successful dogs improved over time, with the best corresponding to 12-months based on F1 and AUC scores. Notably, this pattern occurred with an overall reduction in both the number of dogs and the ratio of successful to eliminated dogs (Supplemental Table 1). The top performance observed was for the Random Forest model at 12-months: accuracy of 87%, AUC of 0.68, and harmonic mean of recall and precision F1 of 0.92 and 0.53 for accepted and eliminated dogs, respectively. The Logistic Regression model performed marginally worse at 12-months. Taking the mean of the four time points for accuracy, AUC, and accepted and eliminated F1, Logistic Regression was slightly better than Random Forest for the first three elements and vice versa for the fourth. The Support Vector Machines model had uneven results largely due to poor recall for eliminated dogs (0.09 vs. 0.32 and 0.36 for the other models).
Prediction of success from the Environmental tests yielded worse and more variable results (Table 2). A contributing factor for the poorer performance may have been the smaller mean number of dogs with testing data compared to the Airport Terminal test (56% vs. 73% of the cohort). Overall, the Logistic Regression model was most effective at predicting success based on F1 and AUC scores. That model showed a pattern of improving performance with advancing months. At 12-months, accuracy was 80%, the AUC was 0.60, and F1 were 0.88 and 0.36 for accepted and eliminated dogs, respectively. The best scores, seen at 12-months, coincided with the lowest presence of dogs eliminated for behavioral reasons. Support Vector Machines had extremely low or zero F1 for eliminated dogs at all time points. All three models had their highest accuracy (0.820.84) and the highest or second highest F1 for accepted dogs (0.900.91) at 3-months. However, all three models had deficient performance in predicting elimination at 3-months (F10.10).
To maximize predictive performance, a forward sequential predictive analysis was employed with the combined data. This analysis combined data from both the Airport Terminal and Environmental at the 3-month timepoint and ran the three ML models, then added the 6-month timepoint and so on. The analysis was designed to use all available data to determine the earliest timepoint for prediction of a dogs success (Table 3). Overall, the combined datasets did not perform much better than the individual datasets when considering their F1 and AUC values. The only instances where the combined datasets performed slightly better were M03 RF over the Environmental M03, M03+M06+M09 LR over both Environmental and Airport Terminal M09, all data SVM over Airport Terminal M12, and all data LR over Environmental M12. The F1 and AUC scores for the instances where the combined sequential tests did not perform better showed that the ML models were worse at distinguishing successful and eliminated dogs when the datasets were combined.
Two feature selection methods were employed to identify the most important traits for predicting success at each time point: Principal Components Analysis (PCA) and Recursive Feature Elimination using Cross-Validation (RFECV). The PCA was performed on the trait data for each test and no separation was readily apparent between accepted and eliminated dogs in the plot of Principal Components 1 and 2 (PC1/2). Scree plots were generated to show the percent variance explained by each PC, and heatmaps of the top 2 PCs were generated to visualize the impact of the traits within those. Within the heatmaps, the top- or bottom-most traits were those that explained the most variance within the respective component. RFECV was used with Random Forest classification for each test with 250 replicates, identifying at least one feature per replicate. In addition, 2500 replicates of a Nave Bayes Classifier (NB) and Random Forest Model (RF) were generated to identify instances where RF performed better than a nave classification.
Scree plots of the Airport Terminal tests showed a steep drop at PC2, indicating most of the trait variance is explained by PC1. The variance explained by the top two PCs ranged from 55.2 to 58.2%. The heatmaps (Fig.2a) showed the PC1/2 vectors with the strongest effects were H1/2 at 3- and 6- months, and PP at 9- and 12-months, both of which appeared in the upper left quadrant (i.e., negative in PC1 and positive in PC2). Several traits showed temporal effects within PCs: (i) at 3-months, PC1 had lower H1 than H2 scores, but that reversed and its effect increased at the other time points; (ii) at 3- and 6-months, PC2 had positive signal for H1/2, but both became negative at 9- and 12-months; (iii) at 3-months, HG was negative, but that effect was absent at other time points; (iv) at 3- and 6- months, PC2 had negative signal for PP, but it changed to strongly positive at 9- and 12-months. When the RFECV was run on the same Airport Test data, a similar pattern of increasing number of selected traits with advancing time points was observed as in the PCA (Table 4). Like the PCA results, H2 was among the strongest at all time points except for the 6-month, although it first appeared among the replicates at 9-months. Means of the NB and RF models were compared (Supplemental Table 2) and showed the M06 and M12 results were the most promising for classification. This suggested that shared traits such as all possession traits (MP, IP, and PP) and the second hunt test (H2) are the most important in identifying successful dogs during these tests, however the distinct nature of the assessment in each time point does not allow for a longitudinal interpretation.
Principal Component Analysis (PCA) results for airport terminal (a) and environmental (b) tests. Each time point displays a heatmap displaying the relative amount of variance captured by each trait within the top 2 components.
The PCA results for the Environmental tests yielded scree plots that had a sharp drop at PC2 for all time points except 9-months (Fig.2b). The amount of variation explained by the top two components decreased with the increasing time points from 62.7 to 49.8. The heatmaps showed the PC1/2 vector with the strongest effect was for the toy possession trait IP, which appeared in the upper left quadrant at all time points (CR and PP had a similar effect at reduced magnitudes). Within PC observations included the following: (i) in PC1, Confidence and Initiative were negative at all time points, and (ii) in PC2, Concentration and Excitability were positive at 3-months, and increased at 6- and at 9- and 12-months. When the RFECV was run on the Environmental test scores (Table 4), all traits for both 9- and 12- months were represented in the results. At 3-months, only Confidence and Initiative were represented and at 6-months, only those and Responsiveness. Means of the NB and RF models were also compared (Supplemental Table 2) and demonstrated M03 and M12 were the most significant for classification. These tests correspond to the earliest test at the gift shop and the last test at an active airport terminal. Primary shared traits include confidence and initiative, with possession-related and concentration traits being most important at the latest time point.
Original post:
Machine learning prediction and classification of behavioral ... - Nature.com
- Predictive Analytics And Machine Learning Market: A ... - Fagen wasanni - August 4th, 2023 [August 4th, 2023]
- Photonic Neural Networks: Revolutionizing Machine Learning and AI - Fagen wasanni - August 4th, 2023 [August 4th, 2023]
- Growing Concerns Over Bias in Powerful AI and Machine Learning ... - Fagen wasanni - August 4th, 2023 [August 4th, 2023]
- Predicting BRAFV600E mutations in papillary thyroid carcinoma ... - Nature.com - August 4th, 2023 [August 4th, 2023]
- Johns Hopkins makes major investment in the power, promise of ... - The Hub at Johns Hopkins - August 4th, 2023 [August 4th, 2023]
- Postdoctoral Fellowship: Pathogenesis of High Consequence ... - Global Biodefense - August 4th, 2023 [August 4th, 2023]
- Apple's Commitment to Generative AI and Machine Learning - Fagen wasanni - August 4th, 2023 [August 4th, 2023]
- Richmond could become AI and machine learning tech hub - The Daily Progress - August 4th, 2023 [August 4th, 2023]
- Platform Reduces Barriers Biologists Face In Accessing Machine ... - Bio-IT World - August 4th, 2023 [August 4th, 2023]
- A comparative study of predicting the availability of power line ... - Nature.com - August 4th, 2023 [August 4th, 2023]
- Preventing Bias In Machine Learning - Texas A&M Today - Texas A&M University Today - August 4th, 2023 [August 4th, 2023]
- 3 Cheap Machine Learning Stocks That Smart Investors Will Snap ... - InvestorPlace - August 4th, 2023 [August 4th, 2023]
- Research Analyst/ Associate/ Fellow in Machine Learning and ... - Times Higher Education - August 6th, 2023 [August 6th, 2023]
- AI and Machine Learning: The New Frontier in Global Anti-Money ... - Fagen wasanni - August 6th, 2023 [August 6th, 2023]
- Harnessing the Power of AI and Machine Learning: Growth ... - Fagen wasanni - August 6th, 2023 [August 6th, 2023]
- Harnessing the Power of AI and Machine Learning for Enhanced ... - Fagen wasanni - August 6th, 2023 [August 6th, 2023]
- Use cases of Stereo Matching part8(Machine Learning + AI) - Medium - August 6th, 2023 [August 6th, 2023]
- Use cases of Stereo Matching part7(Machine Learning + AI) - Medium - August 6th, 2023 [August 6th, 2023]
- Use cases of Stereo Matching part9(Machine Learning + AI) - Medium - August 6th, 2023 [August 6th, 2023]
- How machine learning can expand the Landscape of Edge AI. | TDK - TDK Corporation - August 6th, 2023 [August 6th, 2023]
- Machine Learning-Trained Autonomy Tested By XQ-58 For Skyborg - Aviation Week - August 6th, 2023 [August 6th, 2023]
- Artificial Intelligence and Machine Learning in Packaging Robotics ... - Fagen wasanni - August 6th, 2023 [August 6th, 2023]
- 86-year old Hammett equation gets a machine learning update - Chemistry World - August 6th, 2023 [August 6th, 2023]
- Q & A: How A.I. and machine learning are transforming the lending ... - Digital Journal - August 6th, 2023 [August 6th, 2023]
- The Rise of AI and Machine Learning in Global E-Commerce ... - Fagen wasanni - August 6th, 2023 [August 6th, 2023]
- Machine learning-based technique for gain and resonance ... - Nature.com - August 6th, 2023 [August 6th, 2023]
- Machine learning for the development of diagnostic models of ... - Nature.com - August 6th, 2023 [August 6th, 2023]
- AI and the Heart: How Machine Learning is Changing the Face of ... - Fagen wasanni - August 6th, 2023 [August 6th, 2023]
- The Hidden Impact of AI in Photography and How Machine Learning ... - Cryptopolitan - August 6th, 2023 [August 6th, 2023]
- Machine learning identifies physical signs of stroke - Open Access Government - August 6th, 2023 [August 6th, 2023]
- Machine-learning for the prediction of one-year seizure recurrence ... - Nature.com - August 6th, 2023 [August 6th, 2023]
- Automated Machine Learning: Revolutionizing Predictive Analytics ... - Fagen wasanni - August 6th, 2023 [August 6th, 2023]
- Tim Cook says AI, machine learning are part of virtually every product Apple is building - CryptoSlate - August 6th, 2023 [August 6th, 2023]
- AI GNNs: Transforming the Landscape of Machine Learning - Fagen wasanni - August 6th, 2023 [August 6th, 2023]
- 3 Cheap Machine Learning Stocks That Smart Investors Will Snap Up Now - InvestorPlace - August 6th, 2023 [August 6th, 2023]