Master Thesis

Leveraging LLMs for Automated Feedback Generation on Exercises

Integrating LLMs for feedback generation into Artemis and setting up a research playground for quick experimentation and iteration.

AgilePythonJavaTypeScriptSQLAngularReactNLPLLMLaTeXOSS

Athena Playground: Conduct experiment module results screenshot

Work in Progress

This page is still under construction.

Project Overview

This project is a part of my Master’s thesis. I focused on integrating LLMs into Artemis, a platform for automated exercise assessment, and setting up a research and development playground for efficient experimentation.

Key Contributions

R&D Playground: Developed a React-based playground for fast experimentation. This included defining, configuring, and conducting experiments, as well as analyzing results.
Athena Restructuring: Helped Paul Schwind plan and implement a restructuring of Athena’s codebase to allow for multiple feedback generation modules.
Feedback Generation Modules: Created modules in a FastAPI backend to generate feedback for text and programming exercises using LLMs.
Database Management: Implemented database persistence with sqlalchemy (PostgreSQL or SQLite) to store experimental data and other relevant information.
LLM Integration: Integrated various OpenAI and Replicate models (including PT-3, GPT-3.5, GPT-4, Llama-2, and CodeLlama) using LangChain.
LLM Monitoring: Set up token usage and response time logging using LangSmith for LLM performance monitoring during experiments.
Data Handling: Developed scripts for easy data export and import between Artemis and Athena using large SQL queries.

Collaboration and Contributions

Athena

Worked with Paul Schwind on making Athena’s codebase modular, allowing integration of multiple feedback generation modules. I primarily handled the implementation of LLM modules and enhanced the Playground for better module iteration and experimentation. Athena Documentation.

github.com

Subsystem decomposition of Athena

LLM module subsystem decomposition of Athena

Artemis

Contributed to Artemis mainly for the purpose of integrating Athena.

github.com

Additional work on Artemis:

Code Coverage Table Generator: Developed a Python CLI script for automatic generation of code coverage tables from Bamboo CI artifacts, simplifying the review process.