Grounded in Relationship: Ethics, Emotional Intelligence, and the Responsibility of Building with AI
I’ve been building AI voice agents for healthcare and long-term care in Ontario for the past several months. And somewhere in the middle of that work — writing prompts for a system that would answer calls from people trying to reach their elderly parents, or designing intake flows for mental health navigation — I started feeling the weight of what it means to get this wrong.
This piece is my attempt to think through that weight clearly. It’s part ethics paper, part practical guide, and entirely grounded in the actual work.
The regulatory reference at the bottom is yours to keep.
Abstract
Co-creation between humans and artificial intelligence holds genuine promise for building tools that serve communities more effectively, more equitably, and with more care than any single human builder could achieve alone. But the act of building with AI does not automatically produce tools that are ethical, accessible, or emotionally intelligent.
This paper argues that responsible AI development requires practitioners to understand both the legal frameworks that govern their tools and the deeper human dimensions those frameworks attempt to protect: dignity, belonging, safety, and the right to be seen clearly. It examines why emotional intelligence — the capacity to recognize, understand, and thoughtfully respond to human emotional states and needs — must be treated as a design principle, not an afterthought. And it makes the case that grounded awareness of who we are reaching with the tools we build is not a constraint on innovation but its most necessary foundation.
I. The Promise and the Risk of Co-Creation
There is something genuinely exciting about the moment we are in. AI tools are no longer confined to research labs or large enterprise systems. Small teams, solo practitioners, community health workers, and local business owners can now build tools that would have required entire software departments a decade ago. The barrier to entry has shifted from capital and engineering headcount to clarity of purpose and quality of judgment.
This democratization is real and meaningful. But it carries a shadow that must be named directly: the ease of building does not guarantee the quality of what is built.
When a single developer or small team can deploy a voice agent that handles thousands of conversations, or an AI intake system that routes vulnerable people toward care — or away from it — the stakes of getting things wrong become very high, very quickly. The scale of AI amplifies both the good and the harm that a tool embeds. A discriminatory assumption written once into a prompt can be replicated across every interaction the system has, indefinitely, without anyone noticing.
Co-creation with AI builds a better future. But only if the humans doing the building remain grounded in who they are actually reaching.
II. What the Law Is Actually Protecting
The regulatory frameworks that govern AI use in Ontario — PHIPA, PIPEDA, AODA, and the Ontario Human Rights Code — are often encountered as compliance checklists. Organizations ask: what do we have to do to stay out of trouble? That framing, while understandable, misses the point.
These laws are not bureaucratic obstacles. They are the legal expression of values that communities have decided matter enough to protect by force of law. Understanding what each framework is actually trying to protect changes how a builder relates to it.
Health Information (PHIPA)
PHIPA exists because health information is among the most intimate data a person carries. It can reveal diagnosis, addiction history, reproductive choices, mental health treatment, and genetic risk. When that information is mishandled or exposed, the harm is not abstract. People lose jobs, relationships, and safety. The law is protecting the right to be treated as a patient seeking care, not as a data point to be processed.
An AI builder working in any health-adjacent context must ask: does this tool treat the person’s health information as theirs, or as ours?
Personal Information (PIPEDA)
PIPEDA is protecting the principle that when a person shares information with an organization, they are extending trust. That trust has conditions. It should be used for the purpose it was given, for the duration it was needed, and with care proportional to its sensitivity. AI systems that collect, aggregate, and repurpose data without meaningful consent erode that trust at scale.
The question for builders: does this tool honour the implicit agreement the person made when they handed over their information?
Accessibility (AODA)
AODA exists because the default design choices of most built environments — physical and digital — have historically excluded people with disabilities. Accessibility legislation is the legal recognition that this exclusion is not neutral. It is a choice, and it can be made differently.
AI tools that are designed for a neurotypical, abled user and then checked against an accessibility scanner at the end are not truly accessible. Accessibility built in from the beginning — with patience for non-linear communication, clear language, multiple modalities, and genuine alternatives — looks different and functions better for everyone.
Dignity and Discrimination (Ontario Human Rights Code)
The Human Rights Code is protecting something fundamental: the right to be encountered as a full human being regardless of race, disability, age, gender, sexual orientation, or any of the other grounds the Code names. When AI systems make decisions or filter access based on patterns that correlate with protected characteristics — even if those characteristics are never named in the model — they are perpetuating discrimination in a form that is harder to see and therefore harder to challenge.
The question every AI builder must be willing to ask: who does this tool see clearly, and who does it flatten?
III. Why Emotional Intelligence Must Be a Design Principle
Emotional intelligence — the capacity to recognize, name, and appropriately respond to human emotional states — is not a soft feature. In the contexts where AI tools are increasingly deployed, it may be the most important functional requirement a tool has.
Consider the contexts most relevant to health and community-facing AI in Ontario. A person calling a dental clinic voice agent may be anxious about cost or pain. A person interacting with an intake chatbot for mental health services may be in crisis. A senior navigating an AI-assisted long-term care intake form may be frightened and grieving. A newcomer to Canada using an AI-powered service directory may be exhausted by bureaucratic barriers and quietly desperate for an interaction that feels human.
If the AI tool in each of these moments responds only to the literal content of what is said — and ignores register, pacing, distress signals, and the emotional reality underneath the words — it fails the person even when it technically completes its task.
What Emotionally Intelligent AI Design Actually Requires
Awareness of emotional register in language: tools should be designed to recognize when a person’s tone, word choice, or response pattern signals distress, confusion, or urgency, and to respond with appropriate care — slowing down, softening language, offering a human transfer.
Transparent limits: an emotionally intelligent tool is honest about what it cannot do. It does not pretend to understand more than it does, and it never substitutes for human judgment in situations that require it.
Pacing and patience: voice agents and chatbots that interrupt, time out quickly, or require rapid response disadvantage people with cognitive disabilities, language processing differences, anxiety, or any communication style that differs from the assumed default.
Non-coercive design: emotionally intelligent tools do not exploit emotional states to drive conversions, reduce support costs, or push users toward outcomes that serve the organization rather than the person.
Human dignity in error states: when an AI tool fails to understand a person — misrecognizes speech, fails to parse a request, reaches the limit of its capability — the way it handles that moment matters enormously. It should communicate care, not system failure.
IV. Knowing Who You Are Reaching
The most important shift in perspective available to an AI builder is to move from thinking about users in the abstract to thinking about the actual people who will encounter this tool at the actual moments they will encounter it.
This requires disciplined specificity. Not: “our users include older adults.” But: a 74-year-old woman with early cognitive changes and significant pride about her independence, calling an automated system for the first time, with her adult daughter listening in the background. What does she need from this moment? What would make this interaction feel safe rather than humiliating?
Not: “some users may have mental health needs.” But: a 29-year-old man who has called a support line three times and hung up before speaking, who is now trying a text-based intake tool at 2 a.m. because talking feels impossible. What does he need? What would make continuing feel possible?
This specificity is not sentimentality. It is the most rigorous form of design thinking available. Personas built this way produce better tools. They reveal gaps in the design that abstract user models never surface. And they keep the human stakes visible throughout the build process, when the pull toward efficiency and elegant code can make it easy to forget that real people are on the other end.
Intersectionality and the Limits of Single-Axis Thinking
People do not occupy one protected characteristic at a time. The person who needs an accessible voice agent is also potentially a newcomer for whom English is a second language, and also potentially a person navigating a mental health condition, and also potentially a person with deep distrust of institutions due to prior harm. These dimensions interact.
AI tools built with single-axis thinking — designed for disability accessibility but not language accessibility, or for privacy but not emotional safety — will fail people at their most complex. Ethical AI design holds multiple dimensions simultaneously and builds for the intersections, not just the individual categories.
V. The Ethics of Scale
There is a particular ethical weight that comes with deploying AI tools at scale. A human worker who gives a wrong answer or responds without adequate empathy to one person in a difficult moment can be corrected, trained, or supported. The harm is contained. When an AI tool makes the same error in its design — builds in a dismissive response pattern, or excludes a demographic, or fails to recognize a crisis signal — it makes that error with every person who encounters it, indefinitely, until someone identifies and fixes the problem.
This is not an argument against building AI tools. It is an argument for building them with appropriate seriousness. The ease of deployment should increase the rigor of design, not decrease it. The faster you can ship, the more important it becomes to ask: who did we forget to think about?
The Accountability Gap
One of the structural risks of AI deployment is the diffusion of accountability. When a vendor builds the model, a developer integrates it, a business deploys it, and a user encounters it, who is responsible when something goes wrong? Under current Canadian law, the organization that collects the data and deploys the tool is accountable, regardless of where in the supply chain the problem originated.
Practically, this means that every AI builder working in Ontario must be willing to take responsibility for what their tool does in the world — not just what it was designed to do, but what it actually does with the full range of people who encounter it. This requires ongoing monitoring, accessible feedback channels, genuine willingness to fix problems that are reported, and humility about the limits of initial design.
VI. Co-Creation as an Ethical Practice
The co-creation paradigm — humans and AI building together, each contributing what they do best — offers something important: the possibility of tools that are smarter, more responsive, and more nuanced than either humans or AI could produce alone.
But co-creation is not ethically neutral by virtue of being collaborative. The quality of what co-creation produces depends entirely on what the human brings to it: the quality of their questions, the breadth of their imagination about who the tool will serve, the depth of their commitment to building something that genuinely helps rather than something that merely functions.
AI can generate a voice agent script in minutes. It cannot know, without being told with care and specificity, that the person calling is likely to be frightened, or proud, or ashamed, or exhausted. That knowledge comes from the builder — from research, from community relationship, from clinical experience, from the discipline of imagining the person on the other end of the wire.
The ethical practice of co-creation means bringing that knowledge into the process consistently and insisting that the tools produced reflect it.
What Grounded Awareness Looks Like in Practice
Conducting genuine community consultation before designing tools for communities you do not belong to.
Involving people with lived experience in design review — not as a checkbox, but as a genuine influence on design decisions.
Being willing to slow down or not ship when a tool is not ready to serve its intended population with appropriate care.
Building explicit human escalation into every AI tool — a real path to a real person when the tool reaches its limits.
Treating user feedback about harm or failure as high-priority information, not as edge cases to be filtered.
Auditing for bias, accessibility, and emotional adequacy as a regular practice, not a one-time pre-launch event.
Being honest in marketing and documentation about what the tool does and does not do, what populations it has and has not been tested with.
VII. Conclusion: Intelligence Is Not Enough
The AI systems being built today are technically remarkable. They process language, recognize patterns, generate content, and navigate complexity in ways that would have been science fiction a generation ago. But technical intelligence, however sophisticated, is not sufficient to build tools that genuinely serve human beings.
What is required alongside intelligence is something older and more demanding: the willingness to see people clearly. To understand the conditions of their lives, the shape of their fears, the specific ways they have historically been failed by systems that were designed without them in mind. To build with that seeing — not despite it.
PHIPA, PIPEDA, AODA, and the Ontario Human Rights Code are not the ceiling of ethical AI development. They are a floor: the minimum a just society has decided must be guaranteed. The builders who will create AI tools that genuinely matter will not stop at the floor. They will ask what it means to build with dignity, with care, and with honest accountability to every person the tool reaches.
Co-creation builds a better future. But only if the humans in that collaboration bring their full humanity to the work: their knowledge of who is on the other end, their willingness to be responsible for what they make, and their commitment to building tools worthy of the trust people extend when they engage with them.
Reference Guide: Building with AI in Ontario — The Regulatory Landscape
A practical breakdown of four frameworks every Ontario AI practitioner should know
PHIPA — Personal Health Information Protection Act
Health Information / Privacy
Who It Covers: Health information custodians: hospitals, clinics, physicians, dentists, pharmacists, mental health providers, and their agents.
What It Governs: How personal health information (PHI) is collected, used, disclosed, retained, and disposed of. PHI includes any information that identifies a person and relates to their physical or mental health, health history, or healthcare payments.
What This Means for AI Builders:
Voice agents handling patient inquiries, appointment booking, or symptom triage must collect only the minimum PHI necessary.
AI tools must be able to identify, isolate, and delete PHI on request.
Any AI vendor processing PHI on your behalf is considered an agent and must operate under a written agreement.
Data cannot be stored outside of Canada without the patient’s express consent.
Consent must be informed and meaningful. Patients must understand that an AI system is involved.
AI-generated summaries, transcriptions, or notes that contain PHI are subject to the same rules as paper records.
Security safeguards must be reasonable and in place: encryption, access controls, and audit logs.
A privacy breach involving PHI must be reported to the Information and Privacy Commissioner of Ontario (IPC).
Practice Note: Before deploying any AI tool in a health context, conduct a Privacy Impact Assessment (PIA). Map every point where PHI could be captured, stored, or transmitted.
PIPEDA — Personal Information Protection and Electronic Documents Act
Personal Information / Privacy
Who It Covers: Private-sector organizations that collect, use, or disclose personal information in the course of commercial activity. For health information in Ontario, PHIPA takes precedence.
What It Governs: Personal information beyond the health context: names, addresses, email addresses, purchasing behaviour, employment records, financial information, IP addresses, and any data that can identify an individual.
What This Means for AI Builders:
AI tools that collect user data must identify the purpose of collection before or at the time of collection.
Individuals have the right to access their personal information and have inaccuracies corrected.
Consent must be meaningful: pre-checked boxes and buried fine print do not meet the standard.
Retention limits must be defined: do not keep data longer than necessary for the identified purpose.
AI training on customer data requires careful consent mapping; assumed consent is not sufficient.
Automated decision-making that significantly affects individuals must have human oversight available.
If your AI vendor is located outside Canada, you are still accountable for how they protect the data.
PIPEDA is being replaced by the Consumer Privacy Protection Act (CPPA) under Bill C-27; monitor transition timelines.
Practice Note: Build a data flow map for every AI tool deployed. Know what data enters, where it goes, how long it is held, and who can access it.
AODA — Accessibility for Ontarians with Disabilities Act
Accessibility / Barriers
Who It Covers: All public and private sector organizations in Ontario with at least one employee.
What It Governs: The removal of barriers for people with disabilities across customer service, information and communications, employment, transportation, and the built environment.
What This Means for AI Builders:
AI-powered web tools, chatbots, and voice agents must meet WCAG 2.0 Level AA standards at minimum.
Voice agents must be designed so that callers with speech differences, cognitive disabilities, or hearing impairments can still successfully interact. Offer alternatives (text, human transfer) explicitly.
AI onboarding processes, forms, and output displays must be screen-reader compatible and keyboard navigable.
Do not design AI tools that assume neurotypical communication patterns: allow for longer pause times, varied sentence structures, and non-linear information requests.
If your AI tool produces documents or reports, they must be accessible: tagged PDFs, alt text on images, logical heading structure.
Feedback mechanisms for accessibility issues must be provided and genuinely monitored.
Practice Note: Run every AI-facing user interface through an accessibility audit before deployment. Test with users who have disabilities, not just automated scanners.
Ontario Human Rights Code
Dignity / Discrimination / Accommodation
Who It Covers: All organizations providing services, facilities, goods, accommodation, contracts, or employment in Ontario. No size threshold.
What It Governs: Prohibits discrimination and harassment based on race, colour, ancestry, place of origin, ethnic origin, citizenship, creed, sex, sexual orientation, age, marital status, family status, and disability. Requires accommodation to the point of undue hardship.
What This Means for AI Builders:
AI tools can encode and amplify bias. If a model was trained on data that underrepresents certain groups, its outputs may systematically disadvantage those groups. This constitutes adverse effect discrimination under the Code.
AI used in hiring, eligibility screening, or service triage must be audited for discriminatory patterns, including proxy discrimination (using postal code, name, or language as a stand-in for a protected ground).
Accommodation requests must be treated individually: AI cannot substitute for a human determination of whether an accommodation would cause undue hardship.
Voice agents must not treat individuals differently based on accent, name, or communication style in ways that map onto protected grounds.
Gendered or binary assumptions built into AI prompts or intake flows may violate protections for gender identity and expression.
Duty to accommodate applies in service delivery: if a person cannot effectively use your AI tool due to disability, you must provide an equally effective alternative.
Complaints about AI-mediated discrimination are handled by the Human Rights Tribunal of Ontario (HRTO).
Practice Note: Treat bias auditing as a compliance activity, not an optional quality check. Document your auditing process and outcomes. Have a clear human escalation path for anyone the AI system fails to serve adequately.
This document is informational and does not constitute legal advice. Consult a qualified legal professional for guidance specific to your organization and use case.
Cherie Young | Ajax Web AI | Durham Region, Ontario | June 2026