AI and Cybersecurity – Everything You Wanted to Know, But Were Afraid to Ask
To better understand the current state of artificial intelligence (AI) in cybersecurity, SecurityWeek spoke with dozens of security practitioners, researchers, vendors, analysts, and AI experts. The result is a comprehensive snapshot of how AI is being used across the security landscape today. Organized into five key topic areas, this report examines the role of AI through multiple lenses: whether it can be trusted, how organizations are using it, how it can be misused by legitimate insiders, how it is being exploited by cyber adversaries, and where the technology is likely headed next. The five topics are: Generative AI (gen-AI) Agentic AI Shadow AI Machine learning (ML) Artificial general intelligence (AGI) Taken together, these perspectives provide a practical assessment of AI’s opportunities, risks, and likely evolution in cybersecurity. Generative AI Generative AI (gen-AI) is the bedrock of contemporary AI, although it is technically and potentially born out of earlier machine learning (ML, see below). It does what it says: it generates new content (most commonly text) from an AI model (most usually a large language model or LLM). Chatbots are the users’ interface to the LLM, enabling questions (known as prompts) to be applied and responses received in natural language, and answers to be received in natural language. Chatbots are the interface, and LLMs are the reasoning engine. For most people in most direct use the two seem inseparable – just one big gen-AI application. “Gen-AI trains on massive data sets, learns statistical and relationship patterns, and then uses those patterns to synthesize original output from a prompt,” explains Ahmad Shadid, co-founder and CEO at ORGN.com. This is important. It does not create factually correct answers to prompts; it predicts probable answers based on the relationship patterns it has learned – but it does create linguistically correct and compelling responses. Four deep learning architectures power the training for modern gen-AI variants. Transformer architecture (the ‘T’ in GPT and BERT) is used for the LLMs such as ChatGPT, BERT and Claude. Diffusion training generates the variants that focus on creating high quality images and also audio and video. Fundamentally, this process starts with random noise. Mathematically (guided by the user’s prompt) it reduces and reshapes the noise into the required clear result. Diffusion reverses the process of destruction. The generated result is again based on probability – in this case, the probably correct distribution of pixels. Classic diffusion is evolving into diffusion transformer technology (Sora) and ‘flow matching’ (DALL-E 3 and Midjourney) which can be described as next-gen diffusion. Generative adversarial networks (GANs) are trained via two adversarial networks locked in a feedback loop. One creates fake data, while the other learns to detect flaws by repeatedly suggesting flaws and feeding them back to the creation. Both improve until the detector can find no more flaws in the creation. This approach is good at creating images, video and audio, but has largely been superseded by diffusion technology for business use. However, criminals still use GAN-based simple, fast, real‑time face‑swap and voice‑clone models to create deepfakes. The fourth architecture, variational autoencoders (VAEs), use an encoder-decoder architecture for synthetic data generation, data compression, and anomaly detection. “Their main applications are in medical imaging and molecular generation for drug discovery,” comments Shadid. Trust in gen-AI “Gen-AI is a prediction engine. It generates what’s statistically plausible based on patterns it has seen before,” explains Emanuel Salmona, CEO and co-founder at Nagomi Security. “This makes it good at exploration: generating exploit hypotheses, trying different inputs, and connecting a strange behavior to known vulnerability patterns,” expands Albert Ziegler, head of AI at XBOW. “It’s a tool companies can use to automate creative labor,” adds David Karandish, CEO and founder at Capacity. And because of this, “It is becoming closely embedded into security teams’ workflows, from summarizing incident reports to helping draft response plans,” continues Devvret Rishi, general manager of AI at Rubrik. Galina Kho, chief strategy officer at Cyberbay, describes the advent of gen-AI as an efficiency revolution. “It’s not that entirely new capabilities have emerged; it’s that existing ones have become dramatically easier to execute at scale.” The biggest question in the use of AI is whether you can trust an output that is based on probability rather than grounded in known truth. The answer here is 56 shades of ‘No’. “It can be considered both trustworthy and not trustworthy, depending on the intent, the models used and the overall data flow involved,” comments Melissa Ruzzi, senior director of AI at AppOmni. “Gen-AI is not inherently trustworthy,” says Yichuan Zhang, CEO and co-founder of Boltzbit. “It is prone to hallucinations (confident but false statements) and data leakage (reproducing the training content or the context content exactly).” Trever Falconi, director of security and IT operations at HOPPR, explains, “Deploying a gen-AI model is not like installing software. A model trained at one institution will behave differently at another because it learned from a specific set of data and workflows. Move it somewhere new and you’ve introduced a distribution shift: the real-world data it now encounters no longer matches what it was built on, and performance quietly degrades.” Trustworthiness is a complicated question, suggests Aaron Sant-Miller, VP of AI at Booz Allen. “The model is making its best guess at the right response, but it’s not perfect.” Since gen-AI is the bedrock of all AI, there is a trickle-down effect of its strengths and weaknesses into both agentic AI and shadow AI discussed later. Cyber defenders should always be aware that gen-AI can produce errors; but that should not prevent its use. However, as Ruzzi stresses in quoting from Henri Thiel’s 1971 book (Principles of Econometrics), “Models are to be used, not believed. AI should assist analysts, not replace judgment.” The danger is that human nature drives people to believe anything that is said with confidence, and gen-AI can outright lie with confidence. Randell McNair, an adjunct professor at Florida Polytechnic university, explains on LinkedIn, “[Gen-AI] is for all practical purposes a ‘smart’ kid that has been told its whole life it is ‘brilliant’ when in fact, it is just a nearly-8 year old that has never experienced (felt the pain of) a single tangible consequence for being wrong, and has no memory of having ever truly failed someone and had to genuinely regret the shame and embarrassment that should be part of the ‘learning from failure’ process.” Gen-AI use Zhang suggests three areas where gen-AI use offers benefit: SOC productivity (summarizing complex incident logs and writing initial draft reports); secure coding (assisting developers with boilerplate code that adheres to security standards); and vibe coding (assisting non-developers with coding software applications from scratch). “Many enterprises use these models to generate documents, write articles, generate software, or replicate the messages a human would send when orchestrating a larger workflow,” says Sant-Miller. “It helps draft emails, summarize information and reduce manual effort,” adds Travis Springer, president at Sagiss. “Medical imaging teams are piloting vision-language models to surface findings from imaging studies,” says Falconi, “and researchers use synthetic data generation to fill gaps where real patient data is scarce or sensitive to use at scale.” New uses for gen-AI are continually being developed, but within cybersecurity, the most effective use comes from agentic-AI (see below) which can transform gen-AI from a passive responder into an active engager. Gen-AI misuse The misuse of gen-AI within enterprises is usually unintentional: it emanates from a failure of governance around the technology. Ungoverned use of gen-AI is always a misuse of AI. Individuals begin to rely on AI to provide quick (but not necessarily accurate) answers to questions or problems. If an AI model is deployed across the company without adequate control over its use, this can lead to a degradation of personal skill levels and an ungoverned increase in costs (the idea that AI is cheap is wrong). If access to a chatbot is not provided, employees will use external services with even less control (see shadow AI below). The problem comes from both individuals and management treating AI as a solution rather than an assistant. For example, there is potential to use AI’s coding capability to reduce the number of expensive qualified programmers. Anyone who can prompt an AI can now produce a program – but such programs will inevitably introduce new vulnerabilities. This problem goes away if qualified people use AI as an assistant, a tool to improve performance, rather than a means to reduce expensive headcount. Governance is the key to preventing the misuse of gen-AI. Gen-AI abuse By abuse, we mean bad actor use. In cybersecurity, bad actors always adopt new technology at a faster rate than legitimate business. This has certainly been true with AI. The primary reason is the power and complexity of AI. When an enterprise develops an internal AI application, it must be certain to get it right or face a possible self-inflicted catastrophe. This takes time. Criminals don’t have this concern. If something they implement doesn’t work perfectly, they just start again at no disruptive cost. The result is that new attacks tend to appear before adequate defense appears – the defenders may expect the attacks but have no detailed knowledge of them before they start. Zhang highlights three primary examples of gen-AI abuse: hyper-realistic phishing (eliminating the grammar/spelling ‘tells’ of traditional phishing); polymorphic malware (using gen-AI to subtly rewrite malware code to bypass signature-based detection); and vibe coded phishing websites and/or aggressive attacking software (using gen-AI to subtly rewrite apps that look like the original apps, but steal the user’s sensitive data). Gino Sciretta, CEO at BranditScan, warns, “Generating a convincing fake identity now takes seconds. Detecting one reliably still requires specialized tools and trained analysts. Most platforms and most users are not equipped for that. The technology has outpaced the safeguards, and the gap is widening, not closing.” Gen-Ai has introduced a step change in the quality of adversarial social engineering. It can be used to profile an individual by analyzing any social media footprint, and to then develop a targeted lure. It can build a compelling backstory to the attack, and prepare a false or disguised website to capture personal data. “Gen-AI makes mass targeted phishing, malware iteration and vulnerability research much more accessible to bad actors. Tools like WormGPT strip out the safety guardrails entirely, so attackers get the same speed advantages as regular GenAI but without the friction,” comments Harshit Agarwal, co-founder and CEO at Appknox. Image and voice cloning, and video generation is creating a deepfake scenario that increases a BEC and VEC threat that will only escalate in scale and sophistication. “Ninety-four per cent of AI-generated images had visual artifacts, but those artifacts were so subtle that the majority of targets never noticed them,” adds Sciretta. “The telltale signs are there if you know where to look, such as inconsistent light reflections in the eyes, where one pupil reflects a window and the other reflects something entirely different. But consumers are not trained to look for that, and the generators are improving faster than public awareness.” But he adds, “The most dangerous development is not the fake photos. It is the fake conversations. AI-driven chat systems can now sustain emotionally convincing dialogue over days or weeks, accelerating emotional manipulation roughly 300% faster than a human operator could.” As Ted Miracco, CEO at Approov, says, “The danger isn’t just what AI can do; it’s how fast it acts before anyone notices.” For now, criminals are primarily using AI to improve what they already do: more efficient social engineering, discovery of vulnerabilities in code, and generation of exploits. The next step will be automating the complete process of attack through agentic AI systems. Gen-AI future Amara’s law (Wikipedia) states, “We overestimate the impact of technology in the short run and underestimate the effect in the long run.” The difficulty with AI is that the short run could be next week, while the long run is probably just a few months. By the time most people really understand what is happening, what is happening has already changed. Nevertheless, some brave experts have held a finger to the wind and given their predictions. Ronan Murphy, chief data strategy officer at Forcepoint, believes, “Gen Al will be embedded in everything – every spreadsheet, every video, every workflow. The distinction between ‘using AI’ and simply ‘doing your job’ will essentially dissolve. For security teams, that means the surface you’re trying to protect keeps expanding, probably faster than your policy framework can keep up.” Zhang sees a future with SLMs (small rather than large language models). “We are moving toward ‘small language models’ that are hyper-specialized for specific domains (like a model trained exclusively on Linux kernel vulnerabilities) to reduce noise and increase accuracy.” Sant-Miller is more circumspect, wondering if the very nature of current AI makes its future indeterminable. “The future of gen-AI is a complicated one,” he says. “Models continue to get larger and, accordingly, more powerful. But there are two oppositional forces. Larger models are more expensive – both to train and to use – so capability comes at a cost. And models are trained off human generated content that provides a proxy on human reasoning. What then when most of the content is AI generated and no longer provides that proxy. These are the big questions we need to resolve as an industry.” Agentic AI Agentic AI is an evolutionary extension of chatbot gen-AI. Simplistically, a user asks the chatbot a question and then behaves in accordance with the answer received. With agentic AI, the gen-AI returns its answer to an agent, which can then instruct other organizational tools to fulfill the required behavior. But agentic AI is far more complex than this simple view – it is a task controller (or decision-maker) that uses an LLM as the primary cognitive source. The agent, or agents, are dynamic, stateful and adaptive, goal-driven and aware of the tools it or they can use to fulfill the goal. “Agentic AI converts LLMs that answer questions into software that automates the execution of work,” explains Eric Syphard, executive lead for AI at Booz Allen. “Think LLMs with hands.” Technical breakthroughs in long-memory context, tool use and evaluation enable agentic systems to complete complicated multi-step processes with little to no human oversight. “This isn’t just a new version of AI,” he continues, “It’s an entirely new operating and economic model for delivering labor: ‘labor as software’.” Miracco adds, “Agentic AI doesn’t just answer questions, it can also act autonomously on your behalf. By calling APIs, running code, managing workflows, making decisions, the LLM is now the brain controlling anything it has been granted access to, such as your phone.” Agentic AI is a meaningful step beyond gen-AI. “Rather than responding to a single prompt, an agent reasons, plans and acts. This often happens across multiple tools, data sources and application integrations, with minimal human involvement at each step,” explains Murphy. “You give the agent a goal, not an instruction, and it figures out how to get there.” Trust in agentic AI Since agentic AI uses gen-AI for cognition, it inherits the gen-AI trust issues, but with greater danger from direct access to company assets coupled with the potential for autonomous action on those assets. Can it be trusted? “It depends entirely on how it’s built. An autonomous agent with unrestricted access to your systems is a liability (see OpenClaw). An autonomous agent with scoped permissions, human approval workflows, audit trails, and budget controls is a tool you can actually rely on. Agentic systems must be transparent to more than one user,” says Marcel Folaron, CEO at Cochat. He adds, “The trust question isn’t binary. It’s architectural. Can you see what the agent did? Can you control what it’s allowed to do? Can you review its work before it takes consequential action? If yes, you have a trustworthy system. If no, you have a risk.” Can it be trusted? “Only if it is actively governed, which most organizations are not today equipped to do. Agents need broad access to function, and once that access is granted, the output rarely gets reviewed or reduced,” warns Agarwal “They bypass the UI layer entirely, interfacing directly with APIs in ways that don’t generate the session data or behavioral signals that security teams use to detect anomalies. This removes traditional visibility, especially in API-driven and mobile-first environments. Their traffic also looks legitimate, often not appearing in the logs anyone is actually monitoring.” Syphard adds, “Establishing trust requires robust identity and access governance, treating agents as nonperson entities with unique identities that must be continuously authenticated, authorized, audited, and monitored – much like human users – as agents can introduce insider threat-like risks.” Kho suggests, “Trust in agentic AI comes from how well it’s constrained. It doesn’t come from how good the model is. Therefore, the most important question is not how accurate it is, but what is the worst thing it can do if it’s wrong. Because in practice, risk is defined more by permissions than by performance.” You Mon Tsang, founder and CEO at ChurnZero, agrees with the need for governance to ensure trustworthy agentic AI. “Trust has to be earned through containment: considered data access, human-in-the-loop checkpoints, and logging everything” Zhang adds, “Trust is a major hurdle. Because agents can execute actions (like deleting a user or changing a firewall rule), they require strict guardrails and ‘human-in-the-loop’ checkpoints to prevent runaway processes… it is very important to have the guardrails in place for any agentic AI.” A ‘human in the loop’ is an important part of the governance mechanism that allows trust in agentic AI. But it’s a moving target. As the quality of AI, the speed of doing business, and the volume and pace of attacks all increase, so the pressure to reduce the level of human constraint over agentic action also grows. Continuously ensuring the correct balance between human and autonomous action is an important factor in maintaining maximum performance with maximum trust. Agentic AI use Current implementation of agentic AI is cautious but accelerating: cautious because most organizations understand this is a beast that is difficult to tame; accelerating because the benefits are real. The ability to act at machine speed with minimal human activity is a boon to cybersecurity. Zhang cites the potential for ‘autonomous patching’. “An agent identifies a vulnerability, finds the patch, tests it in a sandbox and deploys it,” he says. “It’s being used for workflow automation, assistants, ticket triage, and research support. In security teams, AI is primarily helping analysts gather context, not make decisions,” adds Kho. “Enterprises use agentic AI for tasks that are repeatable, time-consuming, and currently falling through the cracks – monitoring, reporting, data aggregation, alert triage, content generation, compliance checks… The value is highest for small and mid-size teams that lack the headcount to do everything manually,” says Folaron. “Enterprises are already deploying agentic AI to write code, triage security alerts, manage infrastructure, and automate workflows that previously required entire teams. The productivity gains are very real,” adds Jim Sherlock, VP of AI & cybersecurity R&D at ProCircular. “For enterprises, the real opportunity is closing the gap between finding a problem and actually resolving it. That gap – the investigation, the cross-team coordination, the verification that a fix actually held – has been almost entirely manual for decades,” says Salmona. “Agentic systems are starting to absorb that work. But the ones doing it well are grounded in deep environmental context. The ones that aren’t grounded in context are generating activity without reducing exposure. Those are very different things.” Agentic AI misuse Misuse of agentic AI is generally accidental, rooted in its lack of governance, guardrails, and careful design, and exacerbated by the unpredictability of machine reasoning. A never-ending logic loop (continuously striving but never succeeding) could keep the agent running effectively forever unless manually halted. Such never-ending loops could be caused by hallucination, bad design, or a failure of the agent/LLM to recognize that its goal has already been achieved. “An important aspect of trust in AI agents is training them to know their limits. No one wants to be stuck in an endless loop when a human could easily step in and solve the problem,” comments Karandish. “The problem that keeps me up at night is simple: an agent is only as good as the context it operates on,” adds Salmona. “Give it an accurate, correlated view of your environment – your assets, your controls, your exposures, your threat landscape – and it can make decisions that genuinely reduce risk. Give it incomplete data and it will still act. Confidently. Quickly. Incorrectly. Automation without verified context is just a faster way to be wrong at scale.” It is this type of accidental misuse of agentic AI that feeds the widespread trust problems. “For agentic AI to succeed in the future, safety, governance and recoverability must be top priorities,” warns Rishi. “The unintended consequences are real. Agents can drift – taking actions that technically follow their instructions but produce outcomes nobody intended. They can accumulate permissions over time if nobody’s auditing them,” adds Folaron. Agentic AI abuse The complex reality of agentic AI is that while it benefits enterprises, it simultaneously increases their attack surface – and that bad actors both attack agentic systems and use their own agentic systems to speed and scale their attacks. To make matters worse, enterprises are often unaware of the expanded attack surface – downloaded apps are sometimes provided with built-in, but unspecified, agentic systems. “As for the bad actors, right now, the most common ways they are using AI are to conduct old-school attacks at a much faster rate or publish malicious AI projects to infect early adopters riding the hype. But over time, as AI gets into more critical systems and companies give their internal agents more authority, we are going to see a lot more prompt injections used to manipulate these systems,” adds Amit Chita, field CTO at Mend.io. Bad actors target the agentic attack surface in multiple ways, most commonly via prompt injections. “Prompt injection attacks can quietly redirect an agent to exfiltrate data or build delayed-execution payloads from inputs that looked harmless on arrival,” warns Agarwal. “In my view,” comments Shadid, “cyber offenses will become near-fully agentic within one to two years. Defense will have to become autonomous to match.” Ron Longo, CEO at TrustLogix, suggests, “Cybercriminals will leverage the sheer scale and intelligence of agentic AI to launch more advanced and overwhelming phishing and malware attacks. Because agentic AI can be autonomous, it can automate and orchestrate these attacks with greater sophistication than in previous eras of cybercrime.” We’re not there yet. “They don’t need full automation. Even a partial automation significantly increases their efficiency and return on effort,” explains Kho. But it’s going to get worse. “Agentic AI enables attackers to automate reconnaissance, vulnerability discovery, exploitation attempts, and adaptation across many targets at once. Instead of just generating content, it can pursue an objective across multiple steps,” says Ziegler. “In other words, the shift is from AI generating artifacts to AI conducting operations.” Murphy adds, “The concern looking ahead is agentic systems that can identify a weakness autonomously, exploit it, exfiltrate data and cover their tracks, all without a human in the loop on the attacker’s side. We’re not fully there yet. But the trajectory is obvious, and the security industry is not moving fast enough to get ahead of it.” That trajectory has already been confirmed by Anthropic’s discovery of a largely automated attack from a China-linked state-sponsored threat actor in November 2025. “Instead of a human hacker manually probing a system, an AI agent can scan for vulnerabilities, test exploits, exfiltrate data, and cover its tracks – all without human intervention. Spear-phishing campaigns that adapt in real time based on the target’s responses. Automated reconnaissance at a scale that wasn’t possible before. The same autonomy that makes agents useful for defenders makes them dangerous in the wrong hands,” says Folaron. Agentic AI future “Given the potential productivity impact of agentic AI, I am confident it will become a core part of how businesses are run,” says Tsang. Within cybersecurity, “We will likely see the rise of ‘agentic orchestration’, where multiple specialized agents (a ‘detection agent’ and a ‘remediation agent’) collaborate to manage entire security lifecycles” suggests Zhang. But the complexity of agentic will need to be matched by complex controls. “What I’m certain about is that static, rule-based controls won’t keep pace. You need data security that understands context – what the agent is doing, what data it’s touching, what risk that represents – and adapts dynamically. This idea of adaptive security is so critical for today and for tomorrow. The old ‘block or allow’ binary doesn’t work when an AI agent is making hundreds of data decisions per minute,” warns Murphy. “The future is agents that run continuously, learn from their results, collaborate with each other, and only surface to humans when a decision requires judgment. But that future only works if we solve governance first. Autonomy without accountability is a disaster waiting to happen,” says Folaron. One area that is still heavily debated is the degree of autonomy that will be allowed in future agentic systems. “In the future, agentic AI will be successful where governance is strong, but risky where automation is mistaken for maturity. It should be supervised automation, backed by clear boundaries and continuous validation. The future isn’t autonomous security,” says Kho. “I expect more systems built from many short-lived agents with narrow goals, persistent coordination, strict policy controls, and independent validation,” says Ziegler. “Agentic performance is not just about picking one favorite model forever; sometimes different models contribute different strengths at different points in the loop. The real frontier is not ‘more autonomy at all costs.’ It is autonomy that remains auditable, evidence-driven, and safe to operate in production.” Shadow AI Shadow AI is AI installed within the enterprise but unknown to the IT and security departments, or external AI used by an employee without reference to the IT and security department. “Shadow AI is the cybersecurity version of shadow IT, except the blast radius is orders of magnitude larger. It enters enterprises the same way every unsanctioned tool does,” comments Sherlock. Agentic shadow AI usually enters when an employee finds an open source tool and installs it to improve his or her work performance. However, “Unlike shadow IT, shadow AI operates inside workflows, not outside them. That makes it harder to detect and easier to trust” warns Agarwal. This is especially pertinent when the AI is included but undisclosed agents within a downloaded cloud SaaS app. If these apps are installed, they can arrive with one or more pre-approved valid OAuth tokens granting access to different parts of the customers’ infrastructure. If an attacker gains access to such an OAuth token, that attacker gains easy access to frequently sensitive information. The potential extent of this access can be massive – as seen in the Salesloft Drift compromise in 2025. Drift is an AI chatbot and website engagement tool for Salesforce. Attackers stole its OAuth tokens, gaining access to organizations that installed Drift. Subsequently, more than 700 organizations were compromised via the shadow AI within Drift. If any of those organizations were unaware of the agentic AI within Drift, they were effectively compromised by shadow AI. A further example of shadow AI occurs when an employee uses an external chatbot, without the security department’s knowledge, to access gen-AI. That employee could then perform actions inside the organization based on incorrect, inadequate or simply hallucinated information. Nevertheless, “The productivity benefits [of using shadow AI] are real, and I want to be clear about that,” says Murphy. “People aren’t using these tools because they’re reckless. They’re using them because they work. The problem is that productivity gains and data risk are happening simultaneously, and organizations frequently have visibility into neither.” Shadow AI trust Ultimately, there can be no trust in shadow AI. What cannot be seen, cannot be trusted. “Models that haven’t been vetted can be manipulated, can inherit biases from unvetted training data, or can behave unpredictably when they encounter inputs outside their training distribution. And because no one is monitoring them, that unpredictability goes undetected,” warns Falconi. Shadow AI use “Shadow AI is what happens when good intentions meet convenience. An employee discovers a new AI tool – a browser plugin, a code assistant, a productivity app – and starts using it because it genuinely helps them get work done faster,” explains Murphy. In the short term, shadow AI can benefit the company. But in the long term, “The risk of a massive data breach or regulatory fine (GDPR/CCPA) far outweighs the efficiency gains,” adds Zhang. “People start using shadow AI because it genuinely helps them work faster. The problem is that the productivity gain comes with unquantified risk. You’re trading speed for control, and you often don’t realize what you’ve given up until something goes wrong,” says Folaron. Shadow AI misuse Strictly speaking, any and all use of shadow AI is a misuse of AI, simply because it hasn’t been sanctioned by the company. This misuse can cause serious problems, albeit accidental. “In healthcare, the scenario plays out regularly,” comments Falconi. “A radiologist finds an open-source model and starts using it to help triage scans. A researcher pipes imaging data through a consumer AI tool to accelerate analysis. Nobody in IT, security, or compliance knows it exists. There’s no audit trail, no documented provenance on the data it’s touched, and no version control.” The compliance issue is magnified within shadow AI – it can cause serious regulatory issues. “When something goes wrong, there is no way to trace it, contain it, or demonstrate to a regulator that reasonable precautions were taken.” The regulatory exposure varies by industry, but the accountability gap is consistent. “In healthcare, that’s a compliance failure, a potential HIPAA liability. In financial services, it implies SEC and FINRA oversight. In any organization handling EU data, GDPR applies. Across all of these regulations, the legal position is the same: an enterprise that cannot document how an AI system was built, validated and monitored has no defensible posture when that system causes harm,” he adds. However, “The security implications go beyond compliance gaps. When employees use unsanctioned AI tools, sensitive data often leaves the organization’s perimeter entirely, fed into external APIs or platforms with opaque data retention policies. Unlike traditional shadow IT, the exposure isn’t just a misconfigured tool. It’s proprietary or protected data potentially being ingested into systems that the organization has no visibility into and no contractual control over,” continues Falconi. “These tools often use ‘public’ settings, meaning any sensitive data entered (like proprietary source code or customer PII) becomes part of the vendor’s training set, effectively leaking it to the public,” explains Zhang, adding, “Bad actors look for exposed API keys or ‘leaked’ company secrets within public AI datasets to gain a foothold in the target network.” And Geoff Mattson, CEO at SecureAuth, warns, “When someone configures an MCP server on their laptop to give Claude access to internal databases, that’s shadow AI with real teeth.” Shadow AI abuse “Shadow AI is more of an attack surface than an attack tool,” comments Folaron. The biggest problem introduced by shadow is this larger attack surface: “Every unsanctioned AI integration is a potential data leak, a potential compliance violation, a potential entry point. The risk is structural, not just behavioral,” continues Murphy. Unsanctioned tools running inside an enterprise perimeter are, by definition, unmonitored. “That makes them a viable vector for malicious insiders. An employee using an unvetted tool to exfiltrate data, manipulate outputs, or conduct competitive intelligence with little risk of detection is risky. Because shadow AI exists outside formal IT systems, the usual tripwires aren’t in place,” expands Falconi. Bad actors also attempt to enlarge and manipulate this hidden attack surface by tricking employees into downloading deliberately poisoned open source models. “Someone can download and deploy a model that has been tampered with upstream, and it operates invisibly inside the enterprise. Without provenance documentation or a validation process, there’s no way to know what you’re running or whether it’s been manipulated,” continues Falconi. “In cybersecurity, problems rarely start with attacks; they start with blind spots. The issue with shadow AI isn’t in trusting it but not having full awareness of it,” explains Kho. Shadow AI future The correct future for shadow AI is known, but whether it is achievable is moot. “Shadow AI will grow before it shrinks. The tools are too accessible and the productivity incentives too strong for the trend to reverse on its own. Locking everything down doesn’t work either. Overly restrictive policies don’t eliminate shadow AI; they just drive it further underground where it becomes even harder to detect and govern,” argues Falconi. Despite the problems involved in ridding companies of their shadow AI, many practitioners believe it can and will happen. “The trajectory is predictable,” says Tsang. “IT will catch up. Organizations that move fastest to offer sanctioned, secure AI tooling will have the least shadow AI problem, because the incentive to go around IT disappears when IT is actually delivering.” Falconi agrees with this. “When organizations provide practitioners with secure, auditable platforms that offer the speed and flexibility they’re looking for, the appeal of unsanctioned tools diminishes. Shadow AI exists because the governed alternative is too slow, cumbersome, or unavailable. Fix that, and you address the root case rather than the symptom.” Salmona adds, “The organizations that solve this won’t do it by banning tools. They’ll do it by making the approved path faster than the shadow path. That’s a design problem, not a policy problem.” History, however, begs to differ. Our shadow IT (that is, IT without any AI) has been with us for many years. Not only has industry failed to solve shadow IT, but the problem is also bigger than ever. The idea that we will in time solve the shadow AI problem, which is likely to be more intransigent than shadow IT, is decidedly moot. Machine Learning (ML) Industry has been using machine learning AI systems for many years – long before gen-AI found popular usage. ML and gen-AI are related. Both are trained on data. But while gen-AI is trained on mass data scraped from the internet, ML is trained on data constrained to its primary, usually local, task. Because of the more constrained source data, the output is deterministic while gen-AI’s output is probabilistic. “Text recognition tools/systems (OCR) are a good example,” says Folaron. “They are ML tools that have been trained on thousands and thousands of papers. When you scan a document, they identify the text fairly accurately. If you scan the same page twice it will most likely give you the same output.” ML uses statistical algorithms to find anomalies in data. “In security,” says Zhang, “it is primarily used for pattern recognition and behavioral analysis.” That behavioral analysis is used to locate indications of compromise by highlighting deviations from the norm. ML trust “In general, ML is more trustworthy than gen-AI as it is used to analyze existing content, not generate new content,” says Ruzzi. However, “ML is only as reliable as what it was trained on, and models trained on incomplete or outdated data will miss threats that don’t look like past threats. Attackers know this. They study detection logic to craft inputs that stay inside the boundaries of what looks normal, effectively teaching themselves to evade the systems designed to catch them. ML systems also fail silently. When they miss, they do not alert. They normalize,” warns Agarwal. “Within its trained domain, ML can be exceptionally reliable, often more consistent than human analysts who tire, get distracted, or become overwhelmed by volume. But it has blind spots. ML models are only as good as their training data. If the training data doesn’t include a particular type of attack, the model won’t catch it,” agrees Folaron. “The honest answer is that ML is trustworthy as one layer of defense, not as the only layer. It’s excellent at reducing noise and surfacing what matters. It’s not a replacement for human judgment on critical decisions.” “ML in cybersecurity is trustworthy when it is used to augment human decision-making, not replace it,” adds Sciretta. “The risk comes when organizations treat ML as a set-and-forget solution. Models degrade over time as the threat landscape shifts. If you are not continuously retraining, validating, and auditing your models, you are building on a foundation that is slowly crumbling underneath you.” Rishi says, “The key challenge is making machine learning decisions observable and explainable. Organizations need a clear understanding of how outcomes are derived, what signals they depend on, and where those decisions can be validated or overridden, or they risk relying on decisions they don’t fully understand or control.” ML use “Machine learning applications in cyber look toward risk analysis, behavioral analysis, and threat detection. Each machine learning approach carries different tradeoffs based on the method selected under the hood. Some are less powerful but generalize to new use cases better and are more transparent (easy to explain why they said or did things). Others are more focused and black box. Those are tradeoffs an AI team balances when delivering these features,” says Sant-Miller. “ML is the foundation of many AI applications in cybersecurity. It involves using models that are trained on past data to identify patterns, spot unusual activities and highlight behavior that differs from what’s considered normal,” continues Agarwal. “Enterprises use it for threat detection, malware analysis, risk scoring and behavioral monitoring across endpoints and networks. Its value comes from operating at a scale no human team can match.” Folaron adds, “Threat detection – spotting anomalies in network traffic, endpoint behavior, or user activity. Email filtering. Fraud detection. Vulnerability prioritization – figuring out which of your 10,000 vulnerabilities actually matters. User and entity behavior analytics (UEBA) – learning what normal looks like for each user and flagging deviations. Log analysis at scales no human team could process manually.” Zhang provides a specific UEBA example: “Flagging when a user suddenly downloads 5GB of data at 3:00 am.” Ruzzi says it is used “For deep analysis of numerical content, large volumes of data, or data analysis where the intent is to be as deterministic as possible, ML is normally preferred over gen-AI, or is used as an intermediate step to analyze data to then be used by gen-AI.” Despite the potential value of ML in cyber defense, Rishi stresses, “The key challenge is making machine learning decisions observable and explainable. Organizations need a clear understanding of how outcomes are derived, what signals they depend on, and where those decisions can be validated or overridden, or they risk relying on decisions they don’t fully understand or control.” ML misuse ML doesn’t lend itself to active misuse by employees: what misuse occurs is by omission rather than commission. Salmona gives an example – model drift. “Accuracy at deployment is not accuracy six months later. Environments change, attacker behaviors evolve, and the model doesn’t automatically keep up. Most organizations have no systematic way to monitor for that degradation. They trust the tool because it worked before. That assumption will eventually cost them.” ML abuse The same advantage of automated analyses means that adversaries use their own ML systems. “Any system that helps automate selection, prioritization, or iteration can make attackers faster and more persistent.” It is a key component in the ongoing industrialization of cybercrime. “The general pattern is that cyber operations stop being bespoke and become industrialized,” says Ziegler. Attackers also use ML for evasion; “Using their own ML models to simulate a target’s security system and find ‘blind spots’ where their attacks won’t be detected,” says Zhang. “Evading detection systems by training models that learn what triggers alerts and then optimizing attacks to stay below the threshold,” expands Folaron, adding. “Automated password cracking. Generating polymorphic malware that mutates enough to bypass signature-based detection while maintaining its payload. And increasingly, using ML to prioritize targets – analyzing publicly available data to identify the most vulnerable or valuable organizations to attack.” Ruzzi adds, “Bad actors can use ML to automate reconnaissance or map network vulnerabilities.” But bad actors will also directly attack enterprise ML systems. “It is susceptible to adversarial attacks where attackers ‘poison’ the training data to make the ML model ignore specific types of malicious activity,” warns Zhang. ML future The future for machine learning is a convergence with gen-AI. “ML in cybersecurity is moving toward real-time, adaptive defense – systems that don’t just detect known patterns but continuously learn and respond to new ones. The convergence with agentic AI is where it gets interesting. Instead of ML flagging a threat and waiting for a human to respond, you’ll have ML-powered agents that detect, investigate, and contain threats autonomously within defined boundaries. Speed of response becomes the competitive advantage, because attackers are already operating at machine speed,” explains Folaron. The reasoning is clear. “Where ML is going is toward more adaptive, continuously retrained models that can keep pace with how fast attacker behavior evolves. In other words, fewer static rulesets and more real-time learning from live environments,” agrees Agarwal. “Moving from “reactive” detection to “predictive” defense where ML models can forecast where an attacker is likely to move next based on early-stage lateral movement,” confirms Zhang. However, whether ML can ingest agentic strengths without simultaneously inheriting agentic’s concerns, remains to be seen. “ML is heading toward tighter integration with agentic systems – models that both inform and trigger action. That’s where the real leverage is, and also where the risk compounds,” says Salmona. “An ML model feeding a bad signal into an automated workflow produces a wrong action at machine speed, across your entire environment, before anyone realizes something is off. Context and continuous validation aren’t optional anymore – they’re the difference between automation that reduces risk and automation that amplifies it.” Artificial General Intelligence (AGI) Many of the best known AI frontier labs, such as OpenAI, DeepMind, Anthropic, and xAI, are pursuing the idea of general artificial intelligence (AGI). “AGI is a hypothetical stage of AI capability that allows machines to replicate or exceed all dimensions of human cognitive capability. Everything from reasoning, adapting, novel concept creation, and (in theory) consciousness. With all scientific endeavors, everything feels impossible until the next breakthrough brings you closer. Two hundred years ago we didn’t have cars, and now flying across the country or to another planet feels normal. Fifty years ago, we didn’t have the internet, and now the vast majority of our communication is electronic,” explains Sant-Miller. A true and accurate definition of AGI is elusive. “I don’t really know how to define AGI, but I also don’t think it matters. Something will be built in the next few years that will leave us all in wonder. The models are improving fast enough that quibbling over the definition can’t be the point,” comments Tsang. “True AGI, with the ability to learn from experience for critical decisions, is still widely considered decades away by most researchers,” adds Zhang. Many people believe AGI will be achieved, others are less certain, but most agree that the task is daunting and the timeline obscure. “It’s neither inevitable nor impossible, and anyone who tells you they know the timeline is guessing,” says Folaron. “We’ve made remarkable progress in narrow AI, but the gap between what current systems do and what AGI requires is often understated.” AGI trust Trust in any future AGI entity will be a moral dilemma: should we trust the decisions of an entity that has vastly more knowledge and deeper intelligence than ourselves? Will it make the right decision between taking an action that would benefit hundreds while harming dozens? Everyday life is full of such dilemmas for everyone. But should we be willing to delegate the power of choice to something that is ultimately a machine? The answer will be the answer to almost all cybersecurity questions: ‘It depends’. But on what, we don’t yet know. AGI use, misuse and abuse “The practical stance is use it, and build guardrails as it gets more capable. The dangerous scenarios aren’t ones where some pundit or AI CEO declares AGI achieved. They’re ones where a highly capable system makes a catastrophic decision or is turned against targets by a sophisticated adversary. And the technology we have today is powerful enough to be very ready,” suggests Tsang. Catastrophic decisions already occur with current AI, which seems to be moving inexorably toward increasing autonomy. Guardrails are the safeguard, but they haven’t yet prevented all catastrophes. “I think the more practical question for security leaders isn’t whether we achieve artificial general intelligence, but whether we’re ready for artificial general authority. We’re already giving AI agents meaningful decision-making power over sensitive systems. The governance frameworks, identity architectures, and trust models are what need to be built right now, not after some theoretical singularity arrives,” warns Mattson. But here is another of cybersecurity’s moral dilemmas. Should we hobble a racehorse so that it cannot cause collateral bystander harm, or should we set it free to run fast and break things? Safety or potential greater business profit? “The big risk in AGI is similar to gen-AI, where the focus on functionality clouds proper cybersecurity due diligence,” comments Zhang. “By trying to make AI as powerful as it can be, organizations may misconfigure settings, leading to over-permissions and data exposure. They may also grant too much power, creating a major single point of failure,” warns Ruzzi. If AGI is ever fully realized, the effect on cybersecurity will be profound. “If AGI is achieved, cybersecurity as we know it fundamentally changes for both sides. On defense, you’d have systems that can genuinely reason about novel attacks, understand attacker intent, and adapt defenses in real time without human guidance. On offense, you’d have attackers with access to systems that can find and exploit vulnerabilities faster than any human team could patch them,” comments Folaron. “If achieved, it would be the ultimate zero day event,” warns Zhang. “An AGI could find and exploit vulnerabilities in every system simultaneously. Conversely, an AGI-based defense could theoretically create a ‘perfect’ security posture that adapts in real-time to any threat, effectively ending the era of human-driven hacking.” Rishi adds, “In an AGI world, recovery and resilience are the primary safety nets. Since even the best governance cannot predict every move a general intelligence might make, organizations must have rewind capabilities.” AGI future “We are still many major breakthroughs away from AGI. I’m far from an expert to estimate when those breakthroughs will be realized, how far they will move us forward, and what they will change. But the beauty of science is that things often thought impossible are proven to be possible,” says Sant-Miller “We keep debating whether machines can truly think. Meanwhile, they’re beating the MIT math team in problem solving, passing the bar exam, writing exploits, and running sophisticated operations. The philosophical debate is becoming irrelevant. The distance between very impressive narrow AI and AGI is narrowing faster than most people are prepared to accept. Our take is that AGI isn’t far off, but it still remains a fuzzy threshold. We may cross that threshold without realizing we’ve even crossed it,” comments Miracco. Today, the idea of AGI is magic. Tomorrow it may be science. It is not a new concept in literature, but unless we learn from today’s mistakes made in current AI’s development and use, tomorrow’s genuine AGI may really become a world of machine-versus-machine, with ever-decreasing human relevance. Amara’s law applies. The arrival of true AGI is likely to be further off than most people predict, but when it comes it will be far more beneficial and far more dangerous than we can currently imagine. Related: Shadow AI Risk: How SaaS Apps Are Quietly Enabling Massive Breaches Related: The Wild West of Agentic AI – An Attack Surface CISOs Can’t Afford to Ignore Related: AI Fuels ‘Industrial’ Cybercrime as Time-to-Exploit Shrinks to Hours Related: Cyber Insights 2025: Social Engineering Gets AI Wings
securityweek.comJun 16, 2026extracted