Trust Me, I’m an Algorithm

Executive Summary Algorithms make consequential decisions about credit, jobs and healthcare at scale in India. Almost none have been audited for bias.

The few audits that exist are alarming . Only 10% of AI-using financial entities had bias mitigation protocols in place.

In every case, the bias was inherited from training data and design assumptions , and was not deliberate.

India is the only major AI economy where binding algorithmic accountability covers just one sector . Everything else is ungoverned

Three risks are converging for businesses : regulatory exposure, flawed operational pipelines and reputational damage.

Organisations that audit now , before regulation compels them, will be significantly better positioned than those that wait.

. Algorithms influence millions of business decisions each day: who deserves a home loan, what someone’s credit limit should be, which candidates should be shortlisted for a job, how products are recommended and how risks should be weighted. Yet, few of these algorithms have been examined for the biases they carry.

Many decision systems rely on reams of personal and behavioural data, from transaction histories and repayment records to browsing activity, location data and patterns of digital engagement. From this information, they infer characteristics, preferences and levels of risk that can materially affect opportunities and outcomes. The logic behind these decisions is often inaccessible, difficult to explain and rarely subject to review. As a result, biases embedded within data, models or design choices can remain undetected while operating at scale.

Governance norms have not kept pace with AI adoption. This matters acutely in India, which ranked third globally in Stanford University's 2025 AI Vibrancy Index, a composite measure of AI research, investment, talent and policy activity, behind only the United States and China. While the EU has enacted comprehensive, binding legislation, and China has established robustAI oversight frameworks, India remains without dedicated AI legislation. MeitY’s November 2025 AI Governance Guidelines, is a step forward, but the guidelines remain voluntary and there are no penalties for non-compliance.

This paper examines what algorithmic opacity looks like in practice across Indian sectors, how regulators are beginning to respond and why algorithmic audit capability is becoming a business issue rather than a purely technical one.

What Audits have Found Only a handful of audits on algorithmic systems have been conducted so far, but they reveal recurring and potentially disturbing patterns.

In the realm of hiring, a 2025 working paper from IIM Bangalore tested ChatGPT on a structured shortlisting exercise: identical qualifications, equal proportions of scheduled caste (SC), scheduled tribe (ST) and general category candidates, repeated across a thousand iterations. The model selected dominant-caste candidates 84% of the time, even though the output, in all cases, looked like any other shortlist. Nothing in the interface indicated what led to these decisions. In May 2025, a US federal court certified the first class-action against an AI hiring platform, Workday , for discriminating against Black candidates, older applicants and people with disabilities. India has no mechanism that would catch an equivalent failure today, despite AI hiring tools getting embedded across most major recruiters.

In lending, the RBI’s FREE-AI framework, published in August 2025, found that Indian banks and fintechs were deploying AI across credit assessment and fraud detection with minimal oversight. Of the entities using AI, only 35% had validated their systems for bias and fairness, but only at the development stages, not in terms of actual deployment. Only 15% used any tools to interpret how their models were making decisions, just 14% conducted regular audits and only 10% had bias mitigation protocols in place.

NASSCOM, the technology industry's leading trade body, has called out algorithmic discrimination in digital lending as an active concern. The most common mechanism for perpetuating bias is proxy discrimination : algorithms use postcodes, mobile recharge frequency and behavioural data as inputs. These proxies correlate with income and community in ways that the companies deploying them have not explicitly mapped. NASSCOM has documented cases of people, including a self-employed contractor in Punjab, receiving lower credit limits than peers with identical financial profiles, mainly because of variables that the algorithm treated as relevant, but which nobody had audited for fairness. In each case, the companies involved had no mechanism to know this was happening. The bias was inherited from training data, default settings and design assumptions that encoded the worldview of the people who built the system.

In healthcare , a 2025 meta-analysis of over 70,000 clinical images found that AI dermatology diagnostic tools perform measurably worse on darker skin tones, resulting in markedly lower diagnostic accuracy. The cause, again, is training data: the datasets used to build these tools are composed predominantly of lighter-skinned patients. Every health-tech platform deploying these tools, including in India, is thus inheriting a performance gap without necessarily knowing it.

In the booming matrimonial services industry , an audit of Jeevansathi and Shaadi.com by IIT Delhi researchers found that both platforms amplify social norms around income, age and religion even when users explicitly indicate that those dimensions do not matter to them. More striking is what the platforms do before users make any choice at all. Jeevansathi automatically sets a female profile's desired partner-income either equal to or higher than hers, while leaving the same field blank for male profiles. The platform is thus encoding an ideological position in its default settings. Nearly three-quarters of surveyed users said they would prefer recommendation systems free from such biases. The platforms, meanwhile, have no obligation to inform users that the defaults exist, or to explain how the recommendation algorithm works.

Across these cases, the structure of the problem is identical: The bias was not ‘designed in’, but instead, inherited from training data, default settings, or from design assumptions that reflect the worldview of the people who built the system. In each case, it was invisible until someone specifically looked for it.

The Regulatory Patchwork India's regulators have begun to move on this issue, but mainly around the financial-services sector. The RBI’s FREE-AI framework, covering banks and NBFCs, requires board-approved AI policies, mandatory incident reporting and algorithmic audit processes for all regulated financial entities. A February 2025 SEBI circular, covering capital markets, goes further in one specific respect: it formally distinguishes between white box algorithms, where the logic is transparent and replicable, and black box algorithms, where it is not, and requires providers of the latter to register as Research Analysts and maintain detailed documentation for each algorithm. The fact that two of India's most powerful regulators arrived at the same conclusion about opacity as a governance risk, independently and from different starting points, is itself a signal worth noting.

Effectively, then, India currently has a binding framework in only one domain: financial services. Every other areas of business activity – from hiring and healthcare, to matrimony, e-Commerce and agriculture – remains entirely ungoverned from an algorithmic accountability perspective. In comparison, the EU AI Act , effective August 2024, establishes binding risk-based obligations across all sectors. Similarly, China has enacted broad algorithm-recommendation regulations requiring user transparency and control.

Potentially, this creates two types of issues. First, as research from Google finds, conventional ‘fairness frameworks’ are West-centric in their design, and may fail to account for India's specific social hierarchies, data reliability problems and the distance between AI systems and the communities they affect. Simply importing, say, the EU model would be mere window dressing in a country where the ground conditions are fundamentally different. Indeed, a joint study by the Singapore Institute of Management and Christ University found that Indian AI developers and policymakers have limited awareness of the biases embedded in the systems they are building and regulating. The study's most pointed observation is that addressing bias reactively, after harm has occurred, risks entrenching it further. Second, patchy implementation may later create material exposure for businesses operating outside sectors where the issues are being addressed.

Why this is a Board-Level Risk The business case for acting now comes down to three risks:

The regulatory risks are the most immediate. India's sectoral AI governance will not stay fragmented forever. When a national framework does arrive, organisations that have already audited their systems will have ready answers. Those who have not will be building them from scratch under scrutiny. Goldman Sachs did not know its Apple Card algorithm was producing different credit limits for men and women until customers noticed. That ignorance made the situation considerably worse when regulators inquired.

The operational risks are less overt, but may compound over time. A hiring algorithm filtering by caste proxies is effectively excluding over 30% of India's workforce without anyone in the organisation knowing. Such distortions compound silently across hiring cycles, producing no visible signals until someone audits the funnel. By that point, years of flawed decisions have already been made.

The reputational risks may prove the most insidious. Public responses to algorithmic failures tend to focus less on technical limitations and more on organisational responsibility. Once an issue becomes visible, companies are often expected to explain not only what happened, but whether adequate oversight existed in the first place.

India's DPDP Act is in force and a broader AI governance framework is in development. Before it arrives, CXOs can pay attention to the following:

Inventory what your algorithms are deciding, and on whose behalf

Test whether outputs differ systematically across demographic groups in ways the decision criteria do not justify

Require audit documentation from every vendor whose algorithmic system you license

Rather than focusing on sophisticated tooling, the need of the hour is to ask the right questions about your decision-making algorithms. Most organisations have not.