Semester and Thesis Projects

This project is set to limited visibility by its publisher. To see the project description you need to log in at SiROP. Please follow these instructions: - Click link "Open this project..." below. - Log in to SiROP using your university login or create an account to see the details. If your affiliation is not created automatically, please follow these instructions: http://bit.ly/sirop-affiliate

**Background Information:** The control of road transportation systems aims to improve traffic efficiency by reducing congestion. It involves intersection management with traffic lights, perimeter control on a urban network level, and variable speed limit control and ramp metering for highway. Intelligent road transportation systems are hard to model, and different descriptions on macroscopic state level, down to microscopic vehicle levels exist, which make their control challenging. Besides analytic control methods, a growing branch of research is concerned with control based on deep and reinforcement learning, to leverage measurement data and learn the complex, non-linear dynamics of road transportation systems. **Task Description:** In this assignment, we would like to systematically reproduce, implement and benchmark reinforcement learning based control systems. A project typically involves following tasks: • Getting familiar with open source traffic simulator SUMO and Python • Building a network and creating a pipeline to train a control model on it • Choose one control problem and related papers, reimplement and reproduce their results • Documenting challenges for reproducibility, and assumptions made on the way **Goal Statement:** The goal of this assignment is to better understand the challenges, obstacles, and ways of working when reproducing reinforcement learning based controllers from the literature, and to derive guidance for facilitation of future reproducibility studies. Moreover, best practices for publishing more reproducible research are of interest. Ultimately, we would like to: • Score the reproducibility of a paper (1-10) with a report justifying why (max 3 pages) • List missing information or assumptions that were crucial for reproduction • Document reproducibility evolution (1st attempt, 2nd attempt, etc.) • Guidelines / lessons learned when trying to reproduce works **Related Literature (Papers to Reproduce):** - **Ramp Metering:** - Pan, Tianlu, et al. "Integrated optimal control strategies for freeway traffic mixed with connected automated vehicles: A model-based reinforcement learning approach." Transportation research part C: emerging technologies 123 (2021): 102987. - Wang, Chong, et al. "Integrated traffic control for freeway recurrent bottleneck based on deep reinforcement learning." IEEE Transactions on Intelligent Transportation Systems 23.9 (2022): 15522-15535. - Sun, Dingshan, Anahita Jamshidnejad, and Bart De Schutter. "A novel framework combining MPC and deep reinforcement learning with application to freeway traffic control." IEEE Transactions on Intelligent Transportation Systems 25.7 (2024): 6756-6769. - Belletti, Francois, et al. "Expert level control of ramp metering based on multi-task deep reinforcement learning." IEEE Transactions on Intelligent Transportation Systems 19.4 (2017): 1198-1207. - Han, Yu, et al. "A physics-informed reinforcement learning-based strategy for local and coordinated ramp metering." Transportation Research Part C: Emerging Technologies 137 (2022): 103584. - Fares, Ahmed, and Walid Gomaa. "Freeway ramp-metering control based on reinforcement learning." 11th IEEE International Conference on Control & Automation (ICCA). IEEE, 2014. - Schmidt-Dumont, Thorsten, and Jan H. van Vuuren. "Decentralised reinforcement learning for ramp metering and variable speed limits on highways." IEEE Transactions on Intelligent Transportation Systems 14.8 (2015): 1. - Lu, Chao, et al. "Coordinated ramp metering with equity consideration using reinforcement learning." Journal of Transportation Engineering, Part A: Systems 143.7 (2017): 04017028. - Fares, Ahmed, and Walid Gomaa. "Multi-agent reinforcement learning control for ramp metering." Progress in Systems Engineering: Proceedings of the Twenty-Third International Conference on Systems Engineering. Cham: Springer International Publishing, 2015. - **Intersection Management:** - Antonio, Guillen-Perez, and Cano Maria-Dolores. "Multi-agent deep reinforcement learning to manage connected autonomous vehicles at tomorrow's intersections." IEEE Transactions on Vehicular Technology 71.7 (2022): 7033-7043. - Klimke, Marvin, Benjamin Völz, and Michael Buchholz. "Automatic intersection management in mixed traffic using reinforcement learning and graph neural networks." 2023 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2023. - Karthikeyan, P., Wei-Lun Chen, and Pao-Ann Hsiung. "Autonomous intersection management by using reinforcement learning." Algorithms 15.9 (2022): 326. - Gunarathna, Udesh, et al. "Real-time intelligent autonomous intersection management using reinforcement learning." 2022 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2022. - **Variable Speed Limit:** - Ke, Zemian, et al. "Enhancing transferability of deep reinforcement learning-based variable speed limit control using transfer learning." IEEE Transactions on Intelligent Transportation Systems 22.7 (2020): 4684-4695. - Kušić, Krešimir, et al. "An overview of reinforcement learning methods for variable speed limit control." Applied Sciences 10.14 (2020): 4917. - Li, Zhibin, et al. "Reinforcement learning-based variable speed limit control strategy to reduce traffic congestion at freeway recurrent bottlenecks." IEEE transactions on intelligent transportation systems 18.11 (2017): 3204-3217. - Kušić, Krešimir, et al. "Extended variable speed limit control using multi-agent reinforcement learning." 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2020. - Gregurić, Martin, Krešimir Kušić, and Edouard Ivanjko. "Impact of deep reinforcement learning on variable speed limit strategies in connected vehicles environments." Engineering applications of artificial intelligence 112 (2022): 104850. - **Perimeter Control:** - Chen, C., et al. "Data efficient reinforcement learning and adaptive optimal perimeter control of network traffic dynamics." Transportation Research Part C: Emerging Technologies 142 (2022): 103759. - Zhou, Dongqin, and Vikash V. Gayah. "Model-free perimeter metering control for two-region urban networks using deep reinforcement learning." Transportation Research Part C: Emerging Technologies 124 (2021): 102949. - Zhou, Dongqin, and Vikash V. Gayah. "Scalable multi-region perimeter metering control for urban networks: A multi-agent deep reinforcement learning approach." Transportation Research Part C: Emerging Technologies 148 (2023): 104033. Links: - Additional remarks: Group work possible, can also be divided into multiple projects Minimum credits: 9 / 11 or 24 ECTS (depending on project/thesis) Recommended lectures: Road Transport Systems (Verkehr 3), Transport Systems, Traffic Engineering, Microscopic Modelling and Simulation of Traffic Operations Additional information: Good skills in programming language Python are required for algorithm implementation and data analysis.

The goal of this assignment is to better understand the challenges, obstacles, and ways of working when reproducing reinforcement learning based controllers from the literature, and to derive guidance for facilitation of future reproducibility studies. Moreover, best practices for publishing more reproducible research are of interest. Ultimately, we would like to: - Score the reproducibility of a paper (1-10) with a report justifying why (max 3 pages) - List missing information or assumptions that were crucial for reproduction - Document reproducibility evolution (1st attempt, 2nd attempt, etc.) - Guidelines / lessons learned when trying to reproduce works

Kevin Riehl Michail A. Makridis Anastasios Kouvelas

The newly founded **Agentic Systems Lab at ETH Zurich** is offering Master Thesis opportunities as part of our **Genesis Batch 2025**, a collaborative cohort of **10-20 highly motivated students** working on agentic AI. You can select from three distinct tracks: 🎯 **Explorer Track – Applied AI Discovery & Prototyping** - Collaborate with startups and industry partners. - Identify opportunities for AI-driven automation. - Build and evaluate agentic system prototypes. - Current partners include Browser-Use (San Francisco) and nunu.ai (Zürich), with more to come. 🔨 **Builder Track – Advancing AI in Academia & Healthcare** - Contribute to AI projects in education, science, and healthcare. - Collaborate with lab researchers on impactful real-world applications. - Example projects include AI-driven solutions for student evaluation, health coaching, and reviews of academic research. 🚀 **Bring Your Own Project** - Advance Your AI Venture - Ideal for students with an existing prototype or early-stage AI venture. - Use your thesis to strengthen your product, validate it, and prepare for publication or launch. The **emerging field of agentic AI** offers exciting but still largely unexplored research territory. A core outcome of your thesis should be the design, implementation, and evaluation of a working agentic prototype, deployed in a real-world context. You will be expected to evaluate its effectiveness through empirical methods — such as controlled experiments, user studies, or analysis of key performance indicators — and frame your findings in relation to relevant academic literature. Central **research questions** may include: - Which types of agentic system designs are more effective under specific conditions? - How do different user contexts (e.g. education, healthcare, entrepreneurship) influence agent behavior and impact? - What trade-offs emerge between automation, control, and user trust? - How can agentic systems be designed to balance initiative and user control across different domains (e.g., healthcare vs. entrepreneurship)? - What design patterns or interaction modalities (e.g., chat, voice, embedded agents) foster the highest engagement or task completion rates in agentic systems? - Which architectural choices (e.g., reactive vs. goal-based agents, centralised vs. decentralised control) lead to better performance in dynamic environments? Students are expected to formulate such questions based on prior research and address them using their own system evaluations. We strongly support and encourage participants to aim for high-quality conference or journal publications as a tangible outcome of their thesis work. We’re launching this opportunity as a cohort model to promote peer learning, collaboration, and community under the **Genesis Batch 2025**. **Application deadline:** August 31, 2025. **Start date:** No later than October 2025. Duration: 6 months Application details and criteria: https://www.agenticsystemslab.org/

Our mission is to empower Master’s students to build and validate impactful AI systems in applied, academic, or entrepreneurial contexts. Participants will gain hands-on experience and strategic insight into the architecture and deployment of agentic systems — while learning from each other within a supportive cohort.

📧 rjakob@ethz.ch Dr. Robert Jakob 📧 kosullivan@ethz.ch Dr. Kevin O'Sullivan Agentic Systems Lab, ETH Zurich 🌐 www.agenticsystemslab.org

This project is set to limited visibility by its publisher. To see the project description you need to log in at SiROP. Please follow these instructions: - Click link "Open this project..." below. - Log in to SiROP using your university login or create an account to see the details. If your affiliation is not created automatically, please follow these instructions: http://bit.ly/sirop-affiliate

This project is set to limited visibility by its publisher. To see the project description you need to log in at SiROP. Please follow these instructions: - Click link "Open this project..." below. - Log in to SiROP using your university login or create an account to see the details. If your affiliation is not created automatically, please follow these instructions: http://bit.ly/sirop-affiliate

**Work packages** Literature research Understand the training pipeline of the paper Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics. Explore the possibility of using a first-order gradient in optimizing the policy. **Requirements** Strong programming skills in Python Experience in machine learning frameworks, especially model-based reinforcement learning. **Publication** This project will mostly focus on simulated environments. Promising results will be submitted to machine learning conferences, where the method will be thoroughly evaluated and tested on different systems (e.g., simple Mujoco environments to complex systems such as quadrupeds and bipeds). **Related literature** Hafner, D., Lillicrap, T., Ba, J. and Norouzi, M., 2019. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603. Hafner, D., Lillicrap, T., Norouzi, M. and Ba, J., 2020. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193. Hafner, D., Pasukonis, J., Ba, J. and Lillicrap, T., 2023. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104. Li, C., Stanger-Jones, E., Heim, S. and Kim, S., 2024. FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning. arXiv preprint arXiv:2402.13820. Song, Y., Kim, S. and Scaramuzza, D., 2024. Learning Quadruped Locomotion Using Differentiable Simulation. arXiv preprint arXiv:2403.14864. Li, C., Krause, A. and Hutter, M., 2025. Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics. arXiv preprint arXiv:2501.10100.

Please include your CV and transcript in the submission. **Chenhao Li** https://breadli428.github.io/ chenhli@ethz.ch

**Work packages** Literature research Implement the teacher-student training baseline. Implement the proposed active system identification method. Systematic analysis and comparison between the methods, with visualization of the latent environment embedding. Hardware deployment. **Requirements** Strong programming skills in Python Experience in machine learning frameworks, especially reinforcement learning. **Publication** This project will mostly focus on simulated environments. Promising results will be submitted to machine learning conferences, where the method will be thoroughly evaluated and tested on different systems. **Related literature** https://www.science.org/doi/10.1126/scirobotics.aau5872 https://www.science.org/doi/10.1126/scirobotics.abc5986 https://www.science.org/doi/full/10.1126/scirobotics.abk2822

Please include your CV and transcript in the submission. **Chenhao Li** https://breadli428.github.io/ chenhli@ethz.ch

**Work packages** Use video diffusion models to generate reference trajectories with semantic conditions. Train a local navigation policy by learning from demonstrations using the generated trajectories. Deployment on hardware. **Requirements** Strong programming skills in Python Experience in reinforcement learning and imitation learning frameworks **Publication** This project will mostly focus on algorithm design and system integration. Promising results will be submitted to robotics or machine learning conferences where outstanding robotic performances are highlighted. **Related literature** Peng, Xue Bin, et al. "Deepmimic: Example-guided deep reinforcement learning of physics-based character skills." ACM Transactions On Graphics (TOG) 37.4 (2018): 1-14. Li, C., Vlastelica, M., Blaes, S., Frey, J., Grimminger, F. and Martius, G., 2023, March. Learning agile skills via adversarial imitation of rough partial demonstrations. In Conference on Robot Learning (pp. 342-352). PMLR. Li, Chenhao, et al. "FLD: Fourier latent dynamics for Structured Motion Representation and Learning." Serifi, A., Grandia, R., Knoop, E., Gross, M. and Bächer, M., 2024, December. Vmp: Versatile motion priors for robustly tracking motion on physical characters. In Computer Graphics Forum (Vol. 43, No. 8, p. e15175). Fu, Z., Zhao, Q., Wu, Q., Wetzstein, G. and Finn, C., 2024. Humanplus: Humanoid shadowing and imitation from humans. arXiv preprint arXiv:2406.10454. He, T., Luo, Z., Xiao, W., Zhang, C., Kitani, K., Liu, C. and Shi, G., 2024, October. Learning human-to-humanoid real-time whole-body teleoperation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 8944-8951). IEEE. He, T., Luo, Z., He, X., Xiao, W., Zhang, C., Zhang, W., Kitani, K., Liu, C. and Shi, G., 2024. Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. arXiv preprint arXiv:2406.08858. Albaba, M., Li, C., Diomataris, M., Taheri, O., Krause, A. and Black, M., 2025. Nil: No-data imitation learning by leveraging pre-trained video diffusion models. arXiv preprint arXiv:2503.10626.

Please include your CV and transcript in the submission. **Chenhao Li** https://breadli428.github.io/ chenhli@ethz.ch

**Work packages** Literature research Understand the training pipeline of the paper Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics. Explore the possibility of using a first-order gradient in optimizing the policy. **Requirements** Strong programming skills in Python Experience in machine learning frameworks, especially model-based reinforcement learning. **Publication** This project will mostly focus on simulated environments. Promising results will be submitted to machine learning conferences, where the method will be thoroughly evaluated and tested on different systems (e.g., simple Mujoco environments to complex systems such as quadrupeds and bipeds). **Related literature** Hafner, D., Lillicrap, T., Ba, J. and Norouzi, M., 2019. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603. Hafner, D., Lillicrap, T., Norouzi, M. and Ba, J., 2020. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193. Hafner, D., Pasukonis, J., Ba, J. and Lillicrap, T., 2023. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104. Li, C., Stanger-Jones, E., Heim, S. and Kim, S., 2024. FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning. arXiv preprint arXiv:2402.13820. Song, Y., Kim, S. and Scaramuzza, D., 2024. Learning Quadruped Locomotion Using Differentiable Simulation. arXiv preprint arXiv:2403.14864. Li, C., Krause, A. and Hutter, M., 2025. Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics. arXiv preprint arXiv:2501.10100.

Please include your CV and transcript in the submission. **Chenhao Li** https://breadli428.github.io/ chenhli@ethz.ch

**Work packages** Understand why SAC underperforms in legged locomotion and establish baselines. Make SAC work with massively parallel simulation frameworks. Validate performance of improved SAC policies on real hardware. Provide a rigorous, reproducible comparison between SAC and PPO. Hardware validation encouraged **Requirements** Strong programming skills in Python Experience in reinforcement learning frameworks, including SAC and PPO **Publication** This project will mostly focus on algorithm design and system integration. Promising results will be submitted to robotics or machine learning conferences. **Related literature** PPO SAC

Please include your CV and transcript in the submission. **Chenhao Li** https://breadli428.github.io/ chenhli@ethz.ch

This project is set to limited visibility by its publisher. To see the project description you need to log in at SiROP. Please follow these instructions: - Click link "Open this project..." below. - Log in to SiROP using your university login or create an account to see the details. If your affiliation is not created automatically, please follow these instructions: http://bit.ly/sirop-affiliate