https://kozodoi.com Blog posts and papers on applied AI and machine learning by Nikita Kozodoi, Senior Applied Scientist at AWS. en Wed, 01 Jul 2026 00:00:00 GMT https://kozodoi.com/papers/pdf/2026-model-merging.pdf https://kozodoi.com/papers/pdf/2026-model-merging.pdf Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training. Standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model. We fine-tune experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25% to 500% of the optimal training steps and evaluating five merging methods at each duration. Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply with overfitting, while sparsification-based methods achieve their best performance well past the validation optimum. We formalize this through bias-variance decomposition analysis, drawing a parallel to random forests where averaging benefits from high-variance individual learners. These results suggest that training duration and merging method should be chosen jointly rather than independently. Paper Wed, 01 Jul 2026 00:00:00 GMT https://kozodoi.com/blog/are-we-merging-the-right-models https://kozodoi.com/blog/are-we-merging-the-right-models We benchmark Task Arithmetic, TIES, DARE, and Model Soups for merging Qwen3.5 experts across five domains. The best expert training duration turns out to depend on the merging method, with sparsification-based methods peaking well past the validation optimum. Blog Mon, 01 Jun 2026 00:00:00 GMT https://kozodoi.com/blog/medical-legal-regulatory-review-orchestration https://kozodoi.com/blog/medical-legal-regulatory-review-orchestration We orchestrate a team of specialized Amazon Bedrock agents that runs medical, legal, and regulatory review of pharmaceutical content in parallel. Promotional material is checked against scientific references, regulatory codes, and approved claims, cutting review cycles by up to 90%. Blog Mon, 01 Jun 2026 00:00:00 GMT https://kozodoi.com/blog/accelerate-medical-content-review-agentcore https://kozodoi.com/blog/accelerate-medical-content-review-agentcore An agentic pipeline on Amazon Bedrock AgentCore that checks medical marketing content against compliance guidelines. We look at agent design, tool integration, and what it takes to run the workflow at scale. Blog Fri, 01 May 2026 00:00:00 GMT https://kozodoi.com/blog/accelerate-deep-research-workflows-agentcore https://kozodoi.com/blog/accelerate-deep-research-workflows-agentcore AgentCore Deep Research is an open-source application on Amazon Bedrock AgentCore for multi-source research workflows. We cover deploying it, customizing the research agents, and adapting the pipeline to new sources. Blog Wed, 01 Apr 2026 00:00:00 GMT https://kozodoi.com/blog/agentic-maritime-anomaly-analysis https://kozodoi.com/blog/agentic-maritime-anomaly-analysis Maritime analysts drown in isolated alerts that carry no context on their own. Together with Windward, we built an agentic system that correlates them into enriched anomaly reports, shifting effort from data collection to decisions. Blog Wed, 01 Apr 2026 00:00:00 GMT https://kozodoi.com/blog/mainframe-modernization-legacy-code-documentation https://kozodoi.com/blog/mainframe-modernization-legacy-code-documentation Undocumented mainframe code is the knowledge-loss problem that stalls enterprise migration. We built agentic pipelines on Amazon Bedrock that generate documentation from legacy sources at Toyota Motor Europe. Blog Sun, 01 Mar 2026 00:00:00 GMT https://kozodoi.com/blog/boost-llm-performance-with-self-reflection https://kozodoi.com/blog/boost-llm-performance-with-self-reflection Self-reflection lets a model critique and revise its own output before returning it. We apply it at inference time on Amazon Bedrock and measure the accuracy gain against the added cost and latency across reflection depths. Blog Thu, 01 Jan 2026 00:00:00 GMT https://kozodoi.com/blog/scaling-medical-content-review-flo-health https://kozodoi.com/blog/scaling-medical-content-review-flo-health We built MACROS, a system on Amazon Bedrock with Lambda and Step Functions that reviews medical articles against guidelines and proposes revisions. The proof of concept reached 80% accuracy and over 90% recall, cutting per-guideline review from hours to minutes. Blog Thu, 01 Jan 2026 00:00:00 GMT https://kozodoi.com/blog/aumovio-agentic-coding-assistant https://kozodoi.com/blog/aumovio-agentic-coding-assistant We paired a fine-tuned Qwen3-32B with Claude Sonnet orchestration over Amazon Bedrock, SageMaker, and MCP to build a multi-model coding assistant. Fine-tuning on 7,000 annotated functions cut C++ architecture compliance violations to 0.02 against generic models. Blog Thu, 01 Jan 2026 00:00:00 GMT https://kozodoi.com/blog/embedded-automotive-development-aumovio https://kozodoi.com/blog/embedded-automotive-development-aumovio AUMOVIO's vSDV Composer pairs virtual ECUs with an Amazon Bedrock engineering assistant trained on 20+ years of automotive source code. We surface compliance violations earlier in development, with a reported 50%+ acceleration in delivery. Blog Mon, 01 Sep 2025 00:00:00 GMT https://kozodoi.com/blog/intelligent-document-processing-at-scale https://kozodoi.com/blog/intelligent-document-processing-at-scale We ship an end-to-end document processing application on Amazon Bedrock Data Automation, Step Functions, and Lambda that extracts structured data from PDFs and emails. Three extraction paths are compared: managed BDA, custom Bedrock models, and Textract OCR paired with an LLM. Blog Tue, 01 Jul 2025 00:00:00 GMT https://kozodoi.com/papers/pdf/2025-sweet-spot.pdf https://kozodoi.com/papers/pdf/2025-sweet-spot.pdf As Large Language Models (LLMs) continue to evolve, practitioners face increasing options for enhancing inference-time performance without model retraining, including budget tuning and multi-step techniques like self-reflection. While these methods improve output quality, they create complex trade-offs among accuracy, cost, and latency that remain poorly understood across different domains. This paper systematically compares self-reflection and budget tuning across mathematical reasoning, text-to-SQL generation, sentiment classification, and translation tasks. We evaluate prominent LLMs from the Amazon Nova and Anthropic Claude families under varying reflection depths and compute budgets to derive Pareto-optimal performance frontiers. Our analysis reveals substantial domain-dependent variation in self-reflection effectiveness – with performance gains up to 220% in mathematical reasoning but mixed or negative effects in translation and SQL tasks. We further investigate how reflection round depth and feedback mechanism quality influence performance across model families. Additionally, our findings were validated through a real-world case study at Lounge by Zalando, where self-reflection showed market-dependent effectiveness, reinforcing the importance of domain-specific evaluation when deploying these techniques. Our results provide actionable guidance for selecting optimal inference strategies given specific domains and resource constraints. Paper Tue, 01 Jul 2025 00:00:00 GMT https://kozodoi.com/papers/pdf/2025-sampling-bias.pdf https://kozodoi.com/papers/pdf/2025-sampling-bias.pdf Scoring models support decision-making in financial institutions. Their estimation and evaluation are based on the data of previously accepted applicants with known repayment behavior. This creates sampling bias: the available labeled data offers a partial picture of the distribution of candidate borrowers, which the model is supposed to score. The paper addresses the adverse effect of sampling bias on model training and evaluation. To improve scorecard training, we propose bias-aware self-learning - a reject inference framework that augments the biased training data by inferring labels for selected rejected applications. For scorecard evaluation, we propose a Bayesian framework that extends standard accuracy measures to the biased setting and provides a reliable estimate of future scorecard performance. Extensive experiments on synthetic and real-world data confirm the superiority of our propositions over various benchmarks in predictive performance and profitability. By sensitivity analysis, we also identify boundary conditions affecting their performance. Notably, we leverage real-world data from a randomized controlled trial to assess the novel methodologies on holdout data that represent the true borrower population. Our findings confirm that reject inference is a difficult problem with modest potential to improve scorecard performance. Addressing sampling bias during scorecard evaluation is a much more promising route to improve scoring practices. For example, our results suggest a profit improvement of about eight percent, when using Bayesian evaluation to decide on acceptance rates. Paper Tue, 01 Jul 2025 00:00:00 GMT https://kozodoi.com/papers/pdf/2025-swe-infrabench.pdf https://kozodoi.com/papers/pdf/2025-swe-infrabench.pdf Infrastructure-as-code (IaC) is critical for cloud reliability and scalability, yet LLM capabilities in this domain remain underexplored. Existing benchmarks focus on declarative tools like Terraform and full-code generation. We introduce SWE-InfraBench, a dataset of realistic incremental edits to AWS CDK repositories from real-world codebases. Each task requires modifying existing IaC based on natural language instructions, with correctness verified by passed tests. Results show current LLMs struggle: the best model (Sonnet 3.7) solves 34% of tasks, while reasoning models like DeepSeek R1 reach only 24%. Paper Sun, 01 Jun 2025 00:00:00 GMT https://kozodoi.com/blog/detect-hallucinations-for-rag-based-systems https://kozodoi.com/blog/detect-hallucinations-for-rag-based-systems RAG systems still answer confidently when the retrieved context does not support the answer. We add a detection layer that flags those cases, comparing methods such as LLM-as-judge and semantic similarity on accuracy against cost. Blog Thu, 01 May 2025 00:00:00 GMT https://kozodoi.com/blog/medical-content-creation-generative-ai https://kozodoi.com/blog/medical-content-creation-generative-ai We wire up a system on Amazon Bedrock and Lambda that drafts medical marketing content from scientific references and revises it through expert feedback loops. Textract handles document parsing and Amazon Translate the localization, cutting drafting from weeks to hours. Blog Sun, 01 Dec 2024 00:00:00 GMT https://kozodoi.com/papers/pdf/2024-demand-forecasting-gnn.pdf https://kozodoi.com/papers/pdf/2024-demand-forecasting-gnn.pdf Demand forecasting is a prominent business use case that allows retailers to optimize inventory planning, logistics, and core business decisions. One of the key challenges in demand forecasting is accounting for relationships and interactions between articles. Most modern forecasting approaches provide independent article-level predictions that do not consider the impact of related articles. Recent research has attempted addressing this challenge using Graph Neural Networks (GNNs) and showed promising results. This paper builds on previous research on GNNs and makes two contributions. First, we integrate a GNN encoder into a state-of-the-art DeepAR model. The combined model produces probabilistic forecasts, which are crucial for decision-making under uncertainty. Second, we propose to build graphs using article attribute similarity, which avoids reliance on a pre-defined graph structure. Experiments on three real-world datasets show that the proposed approach consistently outperforms non-graph benchmarks. We also show that our approach produces article embeddings that encode article similarity and demand dynamics and are useful for other downstream business tasks beyond forecasting. Paper Mon, 01 Jan 2024 00:00:00 GMT https://kozodoi.com/blog/improve-llm-responses-in-rag-use-cases https://kozodoi.com/blog/improve-llm-responses-in-rag-use-cases When retrieval returns a weak answer, most RAG systems reply anyway. We add a clarification tool so the system asks a follow-up question instead, combining Amazon Kendra retrieval, LangChain orchestration, and Amazon Bedrock. Blog Wed, 01 Nov 2023 00:00:00 GMT https://kozodoi.com/papers/pdf/2023-recency-single-data-point.pdf https://kozodoi.com/papers/pdf/2023-recency-single-data-point.pdf We investigate the predictive power of recency-based heuristics using a single data point. In many domains, simple heuristics that rely on the most recent observation can outperform complex machine learning algorithms. This study examines when and why recency-based predictions are effective, providing insights into the conditions under which less information can lead to better predictions. Paper Sat, 01 Jul 2023 00:00:00 GMT https://kozodoi.com/blog/implementing-pca-from-scratch https://kozodoi.com/blog/implementing-pca-from-scratch Principal Component Analysis, built from first principles in Python and NumPy and wrapped in a scikit-learn-style class. We work through the covariance matrix, eigendecomposition, and component selection on a toy dataset. Blog Sun, 26 Mar 2023 00:00:00 GMT https://kozodoi.com/blog/implementing-knn-from-scratch https://kozodoi.com/blog/implementing-knn-from-scratch We implement K Nearest Neighbors from first principles in Python and NumPy as a reusable class object. Distance computation, neighbor selection, and majority voting are each built up and tested on a toy dataset. Blog Sun, 19 Mar 2023 00:00:00 GMT https://kozodoi.com/papers/pdf/2022-phd-thesis.pdf https://kozodoi.com/papers/pdf/2022-phd-thesis.pdf The rise of machine learning (ML) and the rapid digitization of the economy has substantially changed decision processes in the financial industry. Financial institutions increasingly rely on ML to support decision-making. Credit scoring is one of the prominent ML applications in finance. The task of credit scoring is to distinguish between applicants who will pay back the loan or default. Financial institutions use ML to develop scoring models to estimate a borrower's probability of default and automate approval decisions. This dissertation focuses on three major challenges associated with building ML-based scorecards in consumer credit scoring: (i) optimizing data acquisition and storage costs when dealing with high-dimensional data of loan applicants; (ii) addressing the adverse effects of sampling bias on training and evaluation of scoring models; (iii) measuring and ensuring the scorecard fairness while maintaining high profitability. The thesis offers a set of tools to remedy each of these challenges and improve decision-making practices in financial institutions. Paper Fri, 01 Jul 2022 00:00:00 GMT https://kozodoi.com/blog/layer-wise-learning-rate-in-pytorch https://kozodoi.com/blog/layer-wise-learning-rate-in-pytorch Fine-tuning a pre-trained network works better when different layers move at different speeds. We set up discriminative learning rates with PyTorch parameter groups, so early layers retain what they learned while later layers adapt faster. Blog Tue, 29 Mar 2022 00:00:00 GMT https://kozodoi.com/papers/pdf/2022-fairness-credit-scoring.pdf https://kozodoi.com/papers/pdf/2022-fairness-credit-scoring.pdf The rise of algorithmic decision-making has spawned much research on fair machine learning (ML). Financial institutions use ML for building risk scorecards that support a range of credit-related decisions. Yet, the literature on fair ML in credit scoring is scarce. The paper makes three contributions. First, we revisit statistical fairness criteria and examine their adequacy for credit scoring. Second, we catalog algorithmic options for incorporating fairness goals in the ML model development pipeline. Last, we empirically compare different fairness processors in a profit-oriented credit scoring context using real-world data. The empirical results substantiate the evaluation of fairness measures, identify suitable options to implement fair credit scoring, and clarify the profit-fairness trade-off in lending decisions. We find that multiple fairness criteria can be approximately satisfied at once and recommend separation as a proper criterion for measuring the fairness of a scorecard. We also find fair in-processors to deliver a good balance between profit and fairness and show that algorithmic discrimination can be reduced to a reasonable level at a relatively low cost. The codes corresponding to the paper are available on GitHub. Paper Tue, 01 Mar 2022 00:00:00 GMT https://kozodoi.com/blog/test-time-augmentation-for-tabular-data https://kozodoi.com/blog/test-time-augmentation-for-tabular-data This post adapts test-time augmentation, a computer vision staple, to tabular data. We benchmark strategies for numeric and categorical features and measure how much accuracy they actually buy at inference. Blog Wed, 08 Sep 2021 00:00:00 GMT https://kozodoi.com/papers/pdf/2021-fairness-r-package.pdf https://kozodoi.com/papers/pdf/2021-fairness-r-package.pdf An R package for computing and visualizing algorithmic fairness metrics. The package offers a comprehensive set of fairness metrics for binary and multi-class classification problems, including demographic parity, equalized odds, and calibration metrics. It provides easy-to-use functions for bias detection and mitigation in machine learning models, with support for various protected attributes and fairness definitions. Paper Thu, 01 Jul 2021 00:00:00 GMT https://kozodoi.com/blog/extracting-intermediate-layer-outputs-in-pytorch https://kozodoi.com/blog/extracting-intermediate-layer-outputs-in-pytorch Sometimes the useful signal sits in a hidden layer rather than the output. We compare three ways to read those activations in PyTorch, from forward hooks to slicing the module list, and use them to pull out embeddings. Blog Thu, 27 May 2021 00:00:00 GMT https://kozodoi.com/blog/tracking-ml-experiments-with-neptuneai https://kozodoi.com/blog/tracking-ml-experiments-with-neptuneai Spreadsheets stop scaling once an experiment log runs to hundreds of rows. We move tracking onto Neptune.ai across a competition-style workflow, logging metrics, parameters, and artifacts, then comparing runs to decide what to keep. Blog Fri, 30 Apr 2021 00:00:00 GMT https://kozodoi.com/blog/computing-mean-std-in-image-dataset https://kozodoi.com/blog/computing-mean-std-in-image-dataset We compute channel-wise mean and standard deviation across an image dataset for normalization. The implementation batches through the data in PyTorch, so it never holds the full dataset in memory. Blog Mon, 08 Mar 2021 00:00:00 GMT https://kozodoi.com/blog/gradient-accumulation-in-pytorch https://kozodoi.com/blog/gradient-accumulation-in-pytorch Large models often will not fit a useful batch size into GPU memory. We use gradient accumulation to reach a large effective batch anyway, walking through the modified training loop and its interaction with learning rate scaling and batch norm. Blog Fri, 19 Feb 2021 00:00:00 GMT https://kozodoi.com/blog/training-pytorch-models-on-tpu https://kozodoi.com/blog/training-pytorch-models-on-tpu This post covers training PyTorch models on Cloud TPUs with PyTorch/XLA, on both Kaggle and Colab. We go through the XLA device model, multi-core distributed training, and the changes needed to port an existing GPU pipeline. Blog Fri, 30 Oct 2020 00:00:00 GMT https://kozodoi.com/papers/pdf/2020-shallow-self-learning.pdf https://kozodoi.com/papers/pdf/2020-shallow-self-learning.pdf Credit scoring models support loan approval decisions in the financial services industry. Lenders train these models on data from previously granted credit applications, where the borrowers' repayment behavior has been observed. This approach creates sample bias. The scoring model is trained on accepted cases only. Applying the model to screen applications from the population of all borrowers degrades its performance. Reject inference comprises techniques to overcome sampling bias through assigning labels to rejected cases. This paper makes two contributions. First, we propose a self-learning framework for reject inference. The framework is geared toward real-world credit scoring requirements through considering distinct training regimes for labeling and model training. Second, we introduce a new measure to assess the effectiveness of reject inference strategies. Our measure leverages domain knowledge to avoid artificial labeling of rejected cases during evaluation. We demonstrate this approach to offer a robust and operational assessment of reject inference. Experiments on a real-world credit scoring data set confirm the superiority of the suggested self-learning framework over previous reject inference strategies. We also find strong evidence in favor of the proposed evaluation measure assessing reject inference strategies more reliably, raising the performance of the eventual scoring model. Paper Tue, 01 Sep 2020 00:00:00 GMT https://kozodoi.com/blog/pre-training-with-surrogate-labels https://kozodoi.com/blog/pre-training-with-surrogate-labels When labeled training data is scarce but unlabeled test data is plentiful, the test set is still useful. We pre-train on surrogate labels derived from it, then fine-tune on the real labels to lift network performance. Blog Sun, 30 Aug 2020 00:00:00 GMT https://kozodoi.com/blog/profit-driven-demand-forecasting https://kozodoi.com/blog/profit-driven-demand-forecasting We forecast retail demand with gradient boosted trees in our Data Mining Cup 2020 solution. Optimizing a profit-based objective rather than plain forecast error aligns the model with the underlying inventory cost and revenue trade-offs. Blog Mon, 27 Jul 2020 00:00:00 GMT https://kozodoi.com/blog/detecting-blindness-with-deep-learning https://kozodoi.com/blog/detecting-blindness-with-deep-learning A convolutional network that grades diabetic retinopathy severity from retina photographs, built for a Kaggle competition and the Udacity ML Engineer program. We cover retina-specific preprocessing, augmentation, and the ordinal grading setup. Blog Sat, 11 Jul 2020 00:00:00 GMT https://kozodoi.com/papers/pdf/2020-mopso-feature-selection.pdf https://kozodoi.com/papers/pdf/2020-mopso-feature-selection.pdf Credit scoring refers to the use of statistical models to support loan approval decisions. An ever-increasing availability of data on potential borrowers emphasizes the importance of feature selection for scoring models. Traditionally, feature selection has been viewed as a single-objective task. Recent research demonstrates the effectiveness of multi-objective approaches. We propose a novel multi-objective feature selection framework for credit scoring that extends previous work by taking into account data acquisition costs and employing a state-of-the-art particle swarm optimization algorithm. Our framework optimizes three fitness functions: the number of features, data acquisition costs and the AUC. Experiments on nine credit scoring data sets demonstrate a highly competitive performance of the proposed framework. Paper Wed, 01 Jul 2020 00:00:00 GMT https://kozodoi.com/papers/pdf/2020-multiplier-heuristic.pdf https://kozodoi.com/papers/pdf/2020-multiplier-heuristic.pdf We investigate the effectiveness of simple heuristics for revenue prediction in business contexts. The multiplier heuristic, which uses a simple multiplication rule based on historical data, is compared against complex machine learning algorithms. Our findings suggest that in many business environments, simple heuristics can match or outperform sophisticated predictive models while being more interpretable and easier to implement. Paper Wed, 01 Jul 2020 00:00:00 GMT https://kozodoi.com/blog/algorithmic-fairness-in-r https://kozodoi.com/blog/algorithmic-fairness-in-r This post introduces the fairness R package for computing and visualizing algorithmic fairness metrics. We walk through demographic parity, proportional parity, and equalized odds, and show how the three diverge on the very same classifier. Blog Fri, 01 May 2020 00:00:00 GMT https://kozodoi.com/papers/pdf/2019-profit-driven-feature-selection.pdf https://kozodoi.com/papers/pdf/2019-profit-driven-feature-selection.pdf In credit scoring, feature selection aims at removing irrelevant data to improve the performance of the scorecard and its interpretability. Standard techniques treat feature selection as a single-objective task and rely on statistical criteria such as correlation. Recent studies suggest that using profit-based indicators may improve the quality of scoring models for businesses. We extend the use of profit measures to feature selection and develop a multi-objective wrapper framework based on the NSGA-II genetic algorithm with two fitness functions: the Expected Maximum Profit (EMP) and the number of features. Experiments on multiple credit scoring data sets demonstrate that the proposed approach develops scorecards that can yield a higher expected profit using fewer features than conventional feature selection strategies. Paper Mon, 01 Jul 2019 00:00:00 GMT https://kozodoi.com/papers/pdf/2019-profit-oriented-feature-selection.pdf https://kozodoi.com/papers/pdf/2019-profit-oriented-feature-selection.pdf In credit scoring, feature selection aims at removing irrelevant data to improve the performance of the scorecard and its interpretability. Standard feature selection techniques are based on statistical criteria such as correlation. Recent studies suggest that using profit-based indicators for model evaluation may improve the quality of scoring models for businesses. We extend the use of profit measures to feature selection and develop a wrapper-based framework that uses the Expected Maximum Profit measure (EMP) as a fitness function. Experiments on multiple credit scoring data sets provide evidence that EMP-maximizing feature selection helps to develop scorecards that yield a higher expected profit compared to conventional feature selection strategies. Paper Mon, 01 Jul 2019 00:00:00 GMT https://kozodoi.com/papers/pdf/2018-recency-smart-data.pdf https://kozodoi.com/papers/pdf/2018-recency-smart-data.pdf Since the early 1910s, managers have been using a simple recency-based decision strategy, the hiatus heuristic, to identify valuable customers. This study analyses the role of recency using a library of 60 data sets from business and other areas including weather, sports, and medicine. We find that the hiatus heuristic outperforms complex algorithms from machine learning, stochastic and econometric models in many of these environments. Moreover, if one includes further variables apart from recency in the complex algorithms, their performance does not improve. We show that the results are not so much driven by limited sample size than by the dominant role that recency plays in most of these environments. We conclude that less can be more, that is, relying on smart data such as recency can yield powerful predictions. Paper Sun, 01 Jul 2018 00:00:00 GMT