Topics related to «Trustworthy Agentic AI and Multi-Agent Systems»

About This List

Language-model agents no longer just answer questions. They call tools, read and modify code, run commands, and operate inside real environments over many steps. Coding agents are the most visible form of this shift, but the same transition puts pressure on everything around the model. A long-running agent needs a harness that drives its execution loop reliably, memory and skills that survive across sessions, ways to coordinate with other agents, and controls that keep autonomous action within bounds. On each of these pieces, the field is still more open questions than settled answers.

This page collects the research directions I believe are worth pursuing over the next two years. They are organized into four submodules — engineering and infrastructure; intelligence, adaptation, and collaboration; evaluation and efficiency; and trustworthy persistent systems — spanning eleven topics in total. Each topic lists potential research focus areas and a few references to start from.

Submodule 1: Agent Engineering and Infrastructure

Submodule Overview

This submodule examines the engineering foundations that turn language models into agents capable of sustained interaction with software and external environments. It covers the execution infrastructure surrounding a model, the architecture of coding agents, and the mechanisms needed to support reliable tool use, state management, verification, observability, and iterative execution. Projects may focus on architectural analysis, system construction, controlled experimentation, or reliability evaluation.

Topic 1.1: Agent Harness Architectures and Evolution

Suitable For: Master Students

As large language model (LLM)-based agents evolve from conversational systems into long-running systems capable of using tools, executing code, and interacting with complex environments, the design of their surrounding execution infrastructure becomes increasingly important. An agent harness is the engineering layer that connects a foundation model to external environments by coordinating execution loops, tools, context, memory, verification, observability, and feedback. Such harnesses must accommodate uncertain decisions, dynamic tool selection, long-horizon execution, and changes in models, tasks, and environments. This research direction examines the architectural principles of reliable agent harnesses, the contribution of individual harness components to system behavior, and mechanisms through which harnesses may be improved using execution evidence and system feedback.

Potential Research Focus Areas.

• Architectural patterns and component models for modern agent harnesses.
• Effects of harness components on agent reliability, efficiency, maintainability, and scalability.
• Observability-driven or feedback-driven methods for adapting agent harnesses.
• Experimental platforms and protocols for comparing alternative harness architectures.

Recommended Reading.

1. Lin, J., et al. (2026). Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses. arXiv:2604.25850. Link
2. Ning, X., et al. (2026). Code as Agent Harness. arXiv:2605.18747. Link
3. Banu, B. (2026). Harness Engineering as Categorical Architecture. arXiv:2605.12239. Link
4. OpenAI. (n.d.). OpenAI Agents SDK. Documentation. Link

Topic 1.2: Coding Agent Architecture and Reliability for Autonomous Software Engineering

Suitable For: Undergraduate and Master Students

Coding agents are a prominent application of agentic AI, extending language models from code-generation assistants into systems that can inspect repositories, modify multiple files, execute commands, run tests, and iteratively revise solutions. Their effectiveness depends on the underlying model as well as the agent architecture, repository and environment interfaces, execution workflow, state management, verification mechanisms, and operational safeguards. This research direction studies how these components influence the ability of coding agents to complete complex software-engineering tasks and how system-level design can improve effectiveness, robustness, recoverability, and practical deployment.

Potential Research Focus Areas.

• Architectural patterns and execution workflows in coding-agent systems.
• Reliability factors and failure modes in repository-level software-engineering tasks.
• Verification, testing, feedback, recovery, and human-review mechanisms for coding agents.
• Experimental platforms for analyzing coding-agent behavior in realistic development environments.

Recommended Reading.

1. Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv:2405.15793. Link
2. Wang, X., et al. (2024). OpenHands: An Open Platform for AI Software Developers as Generalist Agents. arXiv:2407.16741; accepted at ICLR 2025. Link
3. Wang, X., et al. (2025). The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents. arXiv:2511.03690; accepted at MLSys 2026. Link
4. Jarmak, S. (2026). Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model. arXiv:2608.13867. Link
5. OpenCode contributors. (n.d.). OpenCode: The open source coding agent [Software repository]. GitHub. Link

Submodule 2: Agent Intelligence, Adaptation, and Collaboration

Submodule Overview

This submodule studies mechanisms through which agents retain experience, acquire reusable procedures, adapt across tasks, and collaborate with other agents. The emphasis is on externalized memory and skills, their reliability over time, and the organizational strategies that enable multiple agents to solve tasks jointly. Projects may investigate representations, learning and management mechanisms, collaboration structures, or empirical behavior over extended interactions.

Topic 2.1: Long-Term Memory Architecture, Management, and Reliability in LLM Agents

Suitable For: Master Students

Memory enables LLM agents to maintain continuity across interactions and to reuse information or experience beyond a single context window. Long-running agents continually write, retrieve, revise, consolidate, and sometimes forget stored information while operating in changing environments. These processes can introduce error propagation, stale or contradictory records, uncontrolled storage growth, privacy risks, and poorly grounded behavior. This research direction studies memory architectures and lifecycle operations together with methods for evaluating and improving the reliability, provenance, interpretability, and behavioral effects of long-term agent memory.

Potential Research Focus Areas.

• Architectures and representations for episodic, semantic, procedural, or hybrid agent memory.
• Policies for memory writing, retrieval, consolidation, updating, and forgetting.
• Measurement of long-term memory reliability and its effects on agent decisions and task performance.
• Verification, provenance, privacy, and contradiction handling in persistent agent memory.

Recommended Reading.

1. Zhang, Z., et al. (2025). A Survey on the Memory Mechanism of Large Language Model based Agents. ACM Transactions on Information Systems. Link
2. Xiong, Z., et al. (2025). How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior. arXiv:2505.16067. Link
3. Du, P. (2026). Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers. arXiv:2603.07670. Link

Topic 2.2: Agent Skill Acquisition and Lifecycle Management

Suitable For: Master Students

As agents move from isolated tasks toward repeated and long-running operation, the ability to acquire, represent, retrieve, validate, and reuse skills becomes an important mechanism for adaptation. Agent skills can encode externalized procedural knowledge learned from demonstrations, execution trajectories, feedback, or reinforcement. A growing skill library also creates management challenges involving duplication, transferability, quality control, security, versioning, retirement, and changing task distributions. This research direction studies how agents can transform experience into reusable skills and how skill libraries can be governed and evaluated over time.

Potential Research Focus Areas.

• Acquisition of reusable skills from demonstrations, trajectories, and interaction experience.
• Representations, indexing, composition, and retrieval of transferable agent skills.
• Validation, refinement, versioning, security, and retirement of skills.
• Effects of skill libraries on adaptation, transfer, robustness, and long-term agent capability.

Recommended Reading.

1. Wang, G., et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv:2305.16291. Link
2. Wang, Z. Z., Mao, J., Fried, D., & Neubig, G. (2025). Agent Workflow Memory. Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, 63897–63911. Link
3. Xu, R., & Yan, Y. (2026). Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward. arXiv:2602.12430. Link
4. Shen, J., Zhang, T., Zhao, X., & Cheng, H. (2026). Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning. arXiv:2605.10923. Link
5. Li, Y. (2026). Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries. arXiv:2607.10113. Link

Topic 2.3: Multi-Agent Task Decomposition and Collaboration Strategies

Suitable For: Undergraduate and Master Students

Multi-agent systems distribute responsibilities among agents through task decomposition, role assignment, communication, and coordinated execution. Depending on task structure and coordination overhead, this organization can improve specialization, parallelism, error checking, or coverage, but adding agents does not by itself guarantee better performance. Poor decomposition, redundant work, inconsistent state, communication cost, and error propagation may offset expected benefits. This research direction examines how complex objectives can be decomposed, how roles and interaction structures can be selected, and how collaboration strategies affect reliability, performance, and resource use.

Potential Research Focus Areas.

• Task decomposition and role-assignment strategies for multi-agent systems.
• Centralized, decentralized, hierarchical, or dynamically formed collaboration structures.
• Communication, shared-state, conflict-resolution, and cross-agent verification mechanisms.
• Evaluation and optimization of collaboration under quality, latency, and resource constraints.

Recommended Reading.

1. Li, X., et al. (2024). A Survey on LLM-based Multi-Agent Systems: Workflow, Infrastructure, and Challenges. Vicinagearth, 1, Article 9. Link
2. Wang, H., Zhao, S., Wang, J., Qiang, Z., Qin, B., & Liu, T. (2025). Beyond Frameworks: Unpacking Collaboration Strategies in Multi-Agent Systems. Proceedings of ACL 2025. Link
3. Tran, K.-T., Dao, D., Nguyen, M.-D., Pham, Q.-V., O’Sullivan, B., & Nguyen, H. D. (2025). Multi-Agent Collaboration Mechanisms: A Survey of LLMs. arXiv:2501.06322. Link
4. Wu, Q., et al. (2023). AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv:2308.08155. Link

Submodule 3: Agent Evaluation and Efficiency

Submodule Overview

This submodule focuses on measuring agent capabilities and improving the resource efficiency of agent execution. Agent evaluation must account for interaction trajectories, tool use, planning, recovery, reliability, cost, and environmental variability in addition to final-task outcomes. Efficiency research considers how models, tools, memory, communication, and execution policies can be selected or adapted under practical budgets. Projects may develop benchmarks, metrics, evaluation protocols, cost models, routing policies, or resource-aware execution methods.

Topic 3.1: Capability Evaluation and Benchmarking for Interactive Tool-Using LLM Agents

Suitable For: Undergraduate and Master Students

Interactive, tool-using LLM agents make decisions over multiple steps and receive observations from external environments. Their performance therefore depends on planning, tool selection, state tracking, adaptation, error recovery, and execution reliability as well as final-output quality. This research direction studies how agent capabilities can be defined, measured, and compared through reproducible methodologies and realistic benchmark environments. It also considers how evaluation design, environmental variance, trajectory analysis, cost accounting, and potential benchmark contamination affect claims about practical readiness.

Potential Research Focus Areas.

• Capability taxonomies and evaluation frameworks for interactive, tool-using agents.
• Reproducible benchmarks for long-horizon tasks and stateful environment interaction.
• Trajectory-level metrics and analyses of planning, tool use, recovery, and failure behavior.
• Evaluation of reliability, cost, efficiency, generalization, and deployment readiness.

Recommended Reading.

1. Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025). Evaluation and Benchmarking of LLM Agents: A Survey. arXiv:2507.21504. Link
2. Yin, G., et al. (2025). MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains. Findings of the Association for Computational Linguistics: NAACL 2025, 4752–4780. Link
3. Starace, G., et al. (2025). PaperBench: Evaluating AI’s Ability to Replicate AI Research. OpenAI. Link
4. Liu, X., et al. (2024). AgentBench: Evaluating LLMs as Agents. International Conference on Learning Representations (ICLR 2024). Link

Topic 3.2: Cost-Aware Execution and Resource Optimization in LLM Agent Systems

Suitable For: Master Students

LLM agent systems incur dynamic costs across extended reasoning, model inference, tool calls, memory operations, external services, environment interaction, and multi-agent communication. These costs vary across tasks and over the course of execution, so efficient operation requires system-level decisions rather than model compression alone. This research direction studies how agents can estimate the value and cost of alternative actions, adapt model and tool choices, control execution depth, and balance task quality, reliability, latency, and resource consumption under explicit budgets.

Potential Research Focus Areas.

• Cost models for tokens, latency, computation, memory, tool calls, and inter-agent communication.
• Budget-aware control of reasoning depth, retries, verification, and execution termination.
• Dynamic model routing, tool-selection policies, and value-of-information estimation.
• Resource-aware scaling strategies for single-agent and multi-agent deployments.

Recommended Reading.

1. Kim, Y., et al. (2025). Towards a Science of Scaling Agent Systems. arXiv:2512.08296. Link
2. Chen, L., Zaharia, M., & Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176. Link
3. Ong, I., et al. (2024). RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665. Link
4. Wu, Q., et al. (2026). To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling. arXiv:2605.00737. Link

Submodule 4: Trustworthy and Persistent Agent Systems

Submodule Overview

This submodule addresses the controls and evidence needed for agents that can act autonomously, maintain state, and interact with users over extended periods. It brings together permission management, action control, provenance, adaptive human oversight, personalization, and persistent operation. Projects may study technical safeguards, accountability mechanisms, human–agent interaction, or system architectures that preserve reliability, privacy, security, and user control over time.

Topic 4.1: Agent Action Control and Capability-Based Permission Management

Suitable For: Undergraduate and Master Students

Tool-using agents dynamically select actions from goals, observations, model-generated plans, and environmental feedback. When those actions access files, services, code-execution environments, or communication channels, conventional static application permissions may not capture task-specific intent or changing execution context. This research direction studies how agent capabilities can be represented, granted, constrained, monitored, and revoked. It includes least-privilege policies, action validation, policy enforcement, approval mechanisms, and context-sensitive authorization intended to preserve useful autonomy within explicit operational boundaries.

Potential Research Focus Areas.

• Capability models and least-privilege representations for tool-using agents.
• Runtime action validation, policy enforcement, confinement, and revocation mechanisms.
• Context-sensitive permission updates and approval strategies.
• Evaluation of security–utility trade-offs in agent action control.

Recommended Reading.

1. Shi, T., He, J., Wang, Z., Li, H., Wu, L., Guo, W., & Song, D. (2025). Progent: Securing AI Agents with Privilege Control. arXiv:2504.11703. Link
2. Xu, S., et al. (2025). LLM-Based Agents for Tool Learning: A Survey. Data Science and Engineering. Link
3. Tang, Y., Liu, Y., Lan, J., Yan, Z., & Gelenbe, E. (2026). Security of LLM-based Agents Regarding Attacks, Defenses, and Applications: A Comprehensive Survey. Information Fusion, 126, 103941. Link
4. Ling, Y., Yu, S., Chen, Z., & Fang, C. (2026). Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation. arXiv:2606.10749. Link

Topic 4.2: Agent Decision Provenance and Dependency Analysis

Suitable For: Undergraduate and Master Students

Execution logs record events in an agent trajectory but may not represent how decisions depend on observations, retrieved evidence, memory records, tool outputs, intermediate artifacts, or prior actions. Decision provenance provides structured representations of these relationships so that researchers and operators can inspect how an outcome was produced and where failures or unsupported transitions entered the process. This research direction studies provenance models, dependency extraction, trace completeness, and applications to explanation, debugging, auditing, recovery, and reliability assessment.

Potential Research Focus Areas.

• Provenance schemas and dependency graphs for agent states, decisions, actions, and evidence.
• Extraction or reconstruction of dependencies from execution traces and artifacts.
• Metrics and tests for provenance completeness, correctness, granularity, and overhead.
• Provenance-based debugging, auditing, failure localization, and recovery.

Recommended Reading.

1. Wang, Y., et al. (2026). From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents. arXiv:2606.04990. Link
2. Souza, R., et al. (2025). PROV-AGENT: Unified Provenance for Tracking AI Agent Interactions in Agentic Workflows. arXiv:2508.02866. Link
3. Souza, R., et al. (2025). LLM Agents for Interactive Workflow Provenance: Reference Architecture and Evaluation Methodology. arXiv:2509.13978. Link
4. She, R., et al. (2026). Safeguarding LLM Agents from Misalignment through Provenance Analysis. arXiv:2607.01236. Link

Topic 4.3: Adaptive Human-in-the-Loop Intervention and Feedback Integration in LLM Agent Systems

Suitable For: Master Students

Human involvement is particularly important when agent actions are high-risk, ambiguous, irreversible, or difficult to verify. Fixed or overly conservative supervision policies can request unnecessary intervention, while insufficient supervision can allow consequential errors to propagate. This research direction studies adaptive mechanisms that allow agents to proceed independently in routine situations and request human input when uncertainty, risk, novelty, or expected value warrants intervention. It also examines how human feedback can be incorporated into current execution and longer-term agent adaptation while maintaining clear responsibility and user control.

Potential Research Focus Areas.

• Risk-, uncertainty-, or value-sensitive policies for requesting human intervention.
• Allocation of limited human attention across agent tasks and execution stages.
• Integration of corrections, preferences, demonstrations, and approvals into agent behavior.
• Evaluation of collaboration quality, workload, calibration, reliability, and responsibility sharing.

Recommended Reading.

1. Zou, H., et al. (2025). LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey. arXiv:2505.00753. Link
2. Takerngsaksiri, W., et al. (2024). Human-In-the-Loop Software Development Agents. arXiv:2411.12924. Link
3. He, J., et al. (2025). Enabling Self-Improving Agents to Learn at Test Time With Human-In-The-Loop Guidance. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. Link
4. Huang, X., et al. (2025). How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond. Proceedings of ACL 2025. Link

Topic 4.4: Persistent Personal AI Agent Systems

Suitable For: Undergraduate and Master Students

Persistent personal AI agents are intended to support users across tasks, sessions, devices, and extended periods rather than operate as isolated interactions. Such systems may maintain user-approved long-term state, integrate communication or productivity channels, invoke reusable skills, and provide personalized or proactive assistance. Persistent operation introduces challenges in memory quality, identity and preference consistency, privacy, access control, recovery, transparency, and safe autonomy. This research direction studies architectures and mechanisms that support continuity and personalization while preserving user control and dependable behavior.

Potential Research Focus Areas.

• Architectures and lifecycle models for persistent, multi-session personal agents.
• Long-term state, memory quality, identity consistency, and preference management.
• User adaptation, proactive assistance, multi-channel interaction, and skill integration.
• Privacy, security, consent, transparency, recovery, and user-control mechanisms.

Recommended Reading.

1. Li, X., et al. (2024). Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security. arXiv:2401.05459. Link
2. Park, J. S., et al. (2023). Generative Agents: Interactive Simulacra of Human Behavior. arXiv:2304.03442. Link
3. Du, P. (2026). Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers. arXiv:2603.07670. Link
4. Liu, R., et al. (2026). PEPA: a Persistently Autonomous Embodied Agent with Personalities. arXiv:2603.00117. Link
5. OpenClaw contributors. (n.d.). OpenClaw — Your assistant, on your devices, in your chats [Software repository]. GitHub. Link