Karve
Browse all jobs

Senior ML Systems Engineer, Frameworks & Tooling

Cohere · London

10 months ago

RemoteFull_timellm trainingdistributed systemshpc infrastructure

Workplace

Remote

Employment

Full_time

Location

London

Posted

10 months ago

Summary

Cohere is hiring a senior ml systems engineer to build and evolve the training framework for frontier-scale language models. The role focuses on distributed training abstractions, performance and stability on multi-node hpc clusters, and tooling for monitoring, logging, debugging, and reproducible runs, in a remote-friendly environment.

Responsibilities

  • Build and own the llm training framework
  • Design distributed training abstractions and strategies
  • Improve training throughput and stability on clusters
  • Develop tooling for monitoring, logging, and debugging
  • Collaborate with infra teams on cluster and hardware support
  • Investigate and resolve ml systems performance bottlenecks
  • Build reproducible and debuggable large-scale run systems
jaxcudanccldistributed trainingfsdpzerocheckpointingmemory managementslurmraykubernetesdockersingularityapptainerpytorchdeepspeedmegatronxformersvllmtensorrt-llmkv cachesprofilingdebuggingdata pipelinesfault-tolerant distributed checkpointingreproducibilityregression testinghpc
llm trainingdistributed systemshpc infrastructuregpu cluster orchestrationml systems performance engineering