Senior ML Systems Engineer, Frameworks & Tooling
Cohere · London
10 months ago
RemoteFull_timellm trainingdistributed systemshpc infrastructure
10 months ago
Workplace
Remote
Employment
Full_time
Location
London
Posted
10 months ago
Summary
Cohere is hiring a senior ml systems engineer to build and evolve the training framework for frontier-scale language models. The role focuses on distributed training abstractions, performance and stability on multi-node hpc clusters, and tooling for monitoring, logging, debugging, and reproducible runs, in a remote-friendly environment.
Responsibilities
- Build and own the llm training framework
- Design distributed training abstractions and strategies
- Improve training throughput and stability on clusters
- Develop tooling for monitoring, logging, and debugging
- Collaborate with infra teams on cluster and hardware support
- Investigate and resolve ml systems performance bottlenecks
- Build reproducible and debuggable large-scale run systems
jaxcudanccldistributed trainingfsdpzerocheckpointingmemory managementslurmraykubernetesdockersingularityapptainerpytorchdeepspeedmegatronxformersvllmtensorrt-llmkv cachesprofilingdebuggingdata pipelinesfault-tolerant distributed checkpointingreproducibilityregression testinghpc
llm trainingdistributed systemshpc infrastructuregpu cluster orchestrationml systems performance engineering
