Senior Research Engineer (Agentic Behavior)

&lt;p&gt;At JetBrains, code is our passion. Ever since we started, back in 2000, we&#39;ve been striving to make the strongest, most effective developer tools on earth. Today, AI-powered coding agents are becoming a core part of how developers write Kotlin – and we want to make sure they write it well.&lt;/p&gt; &lt;p&gt;The Kotlin AI Value Stream team is responsible for how AI agents understand, generate, and improve Kotlin code across all platforms: Android, Kotlin Multiplatform, server-side, web, desktop, and others. We build the evaluation infrastructure, error analysis tools, and post-training pipelines that measure and improve agent behavior on real Kotlin developer tasks.&lt;/p&gt; &lt;p&gt;As a Research Engineer on this team, you&#39;ll own the end-to-end loop: Analyze how agents fail on Kotlin → build evals that capture those failures → research and implement methods to fix them → measure the improvement. Your work will directly shape how millions of developers experience Kotlin through AI coding agents.&lt;/p&gt; &lt;h2&gt;&lt;strong&gt;As part of our team, you will:&lt;/strong&gt;&lt;/h2&gt; &lt;p&gt;&lt;strong&gt;Build tools for agentic error analysis&lt;/strong&gt;&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Design and implement tooling to systematically capture, classify, and analyse errors that AI coding agents make when generating Kotlin code.&lt;/li&gt; &lt;li&gt;Build observability pipelines over agentic traces – mining patterns from agent sessions in JetBrains IDEs, Junie, Claude Code, Cursor, and other coding agents.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;&lt;strong&gt;Build evaluation pipelines&lt;/strong&gt;&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Design, implement, and maintain evaluation pipelines that measure Kotlin code generation quality across dimensions, including correctness, idiomaticity, build success, framework usage, and test coverage.&lt;/li&gt; &lt;li&gt;Build simulation environments where coding agents can be measured on realistic Kotlin developer tasks – from greenfield KMP projects and Gradle dependency management to migrating Spring applications from Java to Kotlin.&lt;/li&gt; &lt;li&gt;Own evaluation infrastructure: metrics, experiment tracking, automated regression checks, and reproducible benchmarking.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;&lt;strong&gt;Research methods for improving agent and model behavior on Kotlin&lt;/strong&gt;&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Experiment with post-training techniques (SFT, DPO, GRPO) to improve how models handle Kotlin-specific patterns, idioms, and frameworks.&lt;/li&gt; &lt;li&gt;Investigate context engineering approaches: CLAUDE.md/AGENTS.md files, compiler-as-verifier feedback loops, Kotlin LSP integration, and MCP-based tooling.&lt;/li&gt; &lt;li&gt;Run experiments to measure impact: A/B comparisons, benchmark suites, and before/after analyses on real codebases.&lt;/li&gt; &lt;li&gt;Collaborate with model providers (Anthropic, OpenAI, and Google) to translate Kotlin-specific findings into model improvements.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;&lt;strong&gt;Build public Kotlin benchmarks&lt;/strong&gt;&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Design and build open-source benchmarks that measure AI coding agent performance on Kotlin tasks and eventually become the standard reference for the ecosystem.&lt;/li&gt; &lt;li&gt;Create task datasets covering the breadth of Kotlin usage: the server side (Spring, Ktor), multiplatform projects (KMP), build systems (Gradle), Android, library development, and others.&lt;/li&gt; &lt;li&gt;Include both mined real-world tasks and carefully designed synthetic tasks that test specific Kotlin capabilities.&lt;/li&gt; &lt;li&gt;Maintain and evolve benchmarks as models improve, ensuring they remain challenging, relevant, and contamination-resistant.&lt;/li&gt; &lt;/ul&gt; &lt;h2&gt;&lt;strong&gt;We&#39;ll be happy to have you on board if you have:&lt;/strong&gt;&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;Hands-on experience building evaluation or analysis pipelines for LLMs or AI coding agents in a research or production setting.&lt;/li&gt; &lt;li&gt;Strong Python engineering skills (at least three years), with the ability to write clean, maintainable code in data-heavy and ML-adjacent codebases.&lt;/li&gt; &lt;li&gt;Experience with data analysis at scale: querying large datasets (SQL/Athena), building data pipelines, and performing statistical analysis of experimental results.&lt;/li&gt; &lt;li&gt;The ability to own projects end to end – from identifying a problem in agent traces to designing an eval, running experiments, and shipping a fix.&lt;/li&gt; &lt;li&gt;A product-aware mindset: You care about how agents are actually used by developers and can translate real failure modes into evaluation and training work.&lt;/li&gt; &lt;li&gt;Familiarity with Kotlin or a strong willingness to develop deep Kotlin expertise (you&#39;ll be living in Kotlin codebases daily).&lt;/li&gt; &lt;/ul&gt; &lt;h2&gt;&lt;strong&gt;Our ideal candidate would also have experience with:&lt;/strong&gt;&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;Post-training LLMs: SFT, RLHF, DPO, GRPO – either hands-on training or designing the data and reward pipelines that feed into training.&lt;/li&gt; &lt;li&gt;Modern deep learning frameworks (PyTorch) and LLM training stacks (TRL, verl, Megatron, or similar).&lt;/li&gt; &lt;li&gt;AI agent development: tool-using agents, multi-step coding workflows, agentic frameworks.&lt;/li&gt; &lt;li&gt;Evaluation frameworks and tools: Inspect AI, Promptfoo, LM-evaluation-harness, or custom eval pipelines.&lt;/li&gt; &lt;li&gt;Experiment tracking and observability: Weights &amp;amp; Biases, MLflow, Langfuse, or similar.&lt;/li&gt; &lt;li&gt;The Kotlin ecosystem: Android, Gradle, KMP, Spring, Ktor – with an understanding of the developer workflows that agents need to support.&lt;/li&gt; &lt;li&gt;Contributing to or maintaining open-source projects, especially benchmarks or evaluation tools.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Don&#39;t check every box? That&#39;s okay – if you&#39;re excited about this work and bring strong fundamentals, we&#39;d love to hear from you. We&#39;re happy to talk and provide the training you need to grow into the role.&lt;/p&gt; &lt;h2&gt;&lt;strong&gt;Why join JetBrains?&amp;nbsp;&lt;/strong&gt;&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;Strong base salary. We offer competitive pay that reflects your skills and experience.&lt;/li&gt; &lt;li&gt;Flexible work location. Enjoy the freedom to work from home or from the office.&lt;/li&gt; &lt;li&gt;Remote work. Spend up to 30 days per year working remotely from abroad.&lt;/li&gt; &lt;li&gt;Extra time off. More days to relax, recharge, and do the things you love.&lt;/li&gt; &lt;li&gt;Medical insurance allowance. Enjoy peace of mind for you and your family&lt;/li&gt; &lt;li&gt;Learning and development opportunities. Access to conferences, courses, and language classes.&lt;/li&gt; &lt;li&gt;Relocation support. We help make your move as smooth and stress-free as possible.&amp;nbsp;&lt;/li&gt; &lt;li&gt;Language classes. Pick up the local language or sharpen your English skills.&lt;/li&gt; &lt;li&gt;Fuel your day. Enjoy a hot meal or receive a lunch allowance on workdays.&lt;/li&gt; &lt;li&gt;Mental health support. To help you feel your best, we provide easy access to professional mental health services.&lt;/li&gt; &lt;li&gt;Sports benefit. Enjoy an on-site gym or sports club stipend.&lt;/li&gt; &lt;li&gt;Internal events. Join company-wide celebrations and team gatherings.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;*Some benefits may vary depending on location.&lt;/p&gt; &lt;p&gt;&lt;br&gt;&lt;span style=&quot;color: rgb(255, 255, 255);&quot;&gt;#LI-KP1&lt;/span&gt;&lt;/p&gt;&lt;div class=&quot;content-conclusion&quot;&gt;&lt;p&gt;&lt;strong&gt;We are an equal opportunity employer&lt;/strong&gt;&lt;br&gt;&lt;br&gt;We know great ideas can come from anyone, anywhere. That’s why we do our best to create an open and inclusive workplace – one that welcomes everyone regardless of their background, identity, religion, age, accessibility needs, or orientation.&lt;/p&gt; &lt;p&gt;&lt;em data-stringify-type=&quot;italic&quot;&gt;We process the data provided in your job application in accordance with the &lt;a href=&quot;https://www.jetbrains.com/legal/docs/privacy/privacy-recruitment/&quot;&gt;Recruitment Privacy Policy.&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;&lt;/div&gt;<p>Find more <a href="https://www.arbeitnow.com/english-speaking-jobs">English Speaking Jobs in Germany</a> on Arbeitnow</a>