2023-08-23 14:48:55

Applied AI Engineer, Operations Automation

Job Description

We're looking for an experienced Applied AI Engineer to join our Growth team and help build operations automation systems for our global mobility platform. If you're excited about applying machine learning to customer support, document verification, and fraud detection challenges across Bolt's 200M+ customers, you'll find plenty of impactful work here.

Every new customer and order at Bolt can trigger an operational decision: a refund, a document check, an account review, a ruling in a dispute. Our teams already use AI on top of existing processes to make them faster and more accurate. Many of those processes need a redesign from the ground up, and a design that gets better with every decision it makes.

You start with customer support for all users, then move to document verification & fraud detection. After that, any process with high decision volume and a recorded outcome is in scope.

You are hands-on. You ship the product and own the result.

  • Redesign processes. Mine an area's systems of record, interview the people who run it, and decide which steps become software, agents or human checkpoints.
  • Define what great looks like. Turn real cases and human decisions into test sets, and turn every new failure into a regression test.
  • Build checks that are hard to game. Calibrate LLM judges against human labels, keep them apart from programmatic checks, and guard cost and payouts.
  • Run the improvement loop across policies, prompts, tools and models. Add fine-tuning where a better policy is not enough.
  • Own the rollout gates. Decide what ships, run the A/B test, and feed the live result back into the test sets.
  • Codify what works so other teams can run it without you.

You are measured on how many decisions you automate while raising their quality.

Requirements

  • You come from software engineering, ML or data science, and you have shipped and operated a system that calls LLMs.
  • You have built an evaluation suite for an LLM or agent system and used it to decide what ships.
  • You have spent dozens of hours in agent traces finding where decisions went wrong, and fixed them.
  • You have designed an LLM judge, measured its agreement with human labels, and fixed it when the two disagreed.
  • You know how verifiers get gamed, and have caught a case of reward hacking in your work.
  • You can design an A/B test, read its result, and explain to an operations lead why the offline tests predicted something different.
  • You know when a rule, a classic ML model or a human beats an LLM, and you can show the cost trade-off.
  • You write clearly in English. Your reports pair a ceiling estimate with a realistic one.

Nice to have

  • Hands-on post-training: RLVR, GRPO or DPO, reward modelling, LoRA etc.
  • Program optimisation frameworks such as GEPA, DSPy or TextGrad.
  • RL environment or gym design with programmatic grading.
  • Operations automation at a marketplace, fintech or trust and safety team.
  • Strong SQL, and comfort working in codebases you did not write.

Company offers

  • Accelerate your professional growth with unique career opportunities.
  • Enjoy a rewarding salary and stock options, knowing that as Bolt succeeds, so do you.
  • Take care of your physical and mental health with our wellness perks.
  • Celebrate 5 years at Bolt with a 1-month paid sabbatical to recharge.
  • Connect with colleagues at annual company events and smaller team gatherings.
  • Work on-site at our Tallinn HQ, next to the operations and engineering teams whose decisions you automate.