What is generative AI?

A generative model estimates how data is generated—formally, it models a probability distribution p(x) (or conditional p(x|y) when you condition on a prompt, label, or image). At inference time you sample or search that distribution to create new examples that plausibly belong to the training domain.

That contrasts with discriminative models, which learn boundaries between classes: p(y|x) (“given this email, is it spam?”). Both use neural networks; the difference is the objective and what you do at deploy time—generate vs classify.

Generative AI is not:

• Rules-based templating — no learned distribution; brittle but predictable.
• Retrieval alone — fetching documents is not generation (though retrieval often pairs with generators in RAG).
• Magic — outputs are statistical; they can be wrong, biased, or unsafe without engineering guardrails.

For how this era fits into seventy years of AI, see From Symbols to Foundation Models. For the transformer-centric “foundation model” stack in detail, see AI Foundation Models in Depth. For LLM-specific mechanics (tokens, KV cache, decoding), see Large Language Models in Depth.

Generative vs discriminative vs predictive ML

Mature products often combine all three: a generative assistant drafts text, a discriminative classifier routes tickets, and a time-series model forecasts load. Choosing generative AI for a problem that needs exact retrieval or deterministic rules is a common early mistake.

A short map of how we got here

1. 2014–2018 — GANs and VAEs made realistic images feasible; training instability and mode collapse kept teams cautious.
2. 2017+ — Transformers replaced RNNs for sequence modeling; scale and data unlocked language.
3. 2018–2022 — GPT-style LMs showed emergent in-context learning; ChatGPT made the UX mainstream.
4. 2020–present — Diffusion dominated image/video quality (Stable Diffusion, DALL·E, Midjourney class systems).
5. 2023+ — Multimodal and agents — vision+audio+text in one interface; tool use and orchestration layers on top of generators.

The hype cycle focuses on the newest modality; the engineering cycle focuses on evaluation, cost, and safety—which lag behind demos in most organizations.

Core model families (how they actually work)

1. Autoregressive models (text and code)

Decoder-only transformers (GPT family, Llama, Mistral, Claude, Gemini text modes) factor language as a chain rule: predict the next token given all previous tokens. Training minimizes cross-entropy on huge corpora; generation samples token by token (greedy, top-k, or nucleus sampling).

• Strengths — flexible prompts, few-shot behavior, unified API for many tasks.
• Weaknesses — hallucination, context limits, sensitivity to prompt phrasing, token cost at scale.
• Production note — treat the model as a completion engine; structure outputs with JSON schema, function calling, or post-parse validation.

2. Diffusion models (images, audio, video)

Diffusion learns to reverse a noise process: start from Gaussian noise and iteratively denoise toward a data sample. Conditioning on text (CLIP embeddings, cross-attention) drives text-to-image systems.

• Strengths — high perceptual quality, stable training relative to early GANs for many teams.
• Weaknesses — slow inference (many steps unless distilled); copyright and likeness risks in training data.
• Production note — batch offline jobs vs real-time UX; consider smaller distilled schedulers and GPU queues.

3. GANs (generative adversarial networks)

A generator and discriminator play a minimax game: the generator fools the discriminator, the discriminator learns real vs fake. Still used in some image enhancement, style transfer, and research settings.

• Strengths — fast sampling once trained; sharp images in favorable setups.
• Weaknesses — training instability, mode collapse, hard to scale to text; largely eclipsed by diffusion for general image gen.

4. VAEs and latent models

Variational autoencoders encode inputs into a latent distribution and decode samples—useful for representation learning, anomaly detection, and as building blocks in larger systems (e.g. latent diffusion).

5. Encoder–decoder and seq2seq

T5, BART, and original translation stacks map input sequences to outputs—summarization, translation, captioning. Many production pipelines still use encoder–decoder or encoder-only embedders beside decoder-only chat models.

6. Multimodal fusion

Vision encoders (ViT, CLIP) plus language decoders align images and text in a shared space—powering “describe this image,” document AI, and video understanding. The product surface is one chat box; under the hood it is multiple specialists.

Modalities and what “good” looks like

The generative lifecycle (data → train → align → deploy)

Data

Generative quality ceilings are set by data: licensing, deduplication, toxicity filters, PII scrubbing, and domain coverage. For enterprise RAG, your private corpus matters as much as the base model—chunking, metadata, and refresh cadence determine answer quality.

Pre-training

Expensive, one-time (for most teams): next-token prediction, denoising objectives, or contrastive alignment across modalities. Scale laws suggest loss improves with compute, data, and parameters—until infrastructure or diminishing returns stop you.

Alignment and instruction tuning

Raw generators are not polished products. Supervised fine-tuning on (instruction, response) pairs, plus preference optimization (RLHF, DPO, ORPO), shapes tone, refusals, and format—not guaranteed factual accuracy.

Adaptation without full retraining

• Prompt engineering — version prompts like code; A/B against eval sets.
• RAG — retrieve grounded chunks before generation; essential for proprietary knowledge.
• Fine-tuning / LoRA — lock in style, tool formats, or domain jargon.
• Tool use and agents — model calls APIs; observe results; loop with timeouts and allow lists.

See Foundation Models in Depth for expanded treatment of RAG, agents, and inference economics.

Inference and serving

Generative systems are latency- and memory-sensitive:

• Text — tokens/sec, time-to-first-token, context window, batching for offline jobs.
• Image/video — GPU memory, step count, resolution; queue-based workers for bursts.
• Cost — API per-token pricing vs self-hosted GPU hours; FinOps applies (see cloud platform evolution).
• Quantization — INT8/INT4 for edge and cost-sensitive paths with measured quality regression.

From demo to product: architecture around the model

A generative feature in production is never “just an API call.” Typical layers:

• Gateway — authentication, rate limits, routing to model versions, prompt caching where safe.
• Grounding — vector DB, hybrid search, citation requirements for answers.
• Safety — input/output filters, PII redaction, blocklists, human review for high-impact flows.
• Observability — log prompts/responses with retention policy, trace token usage, regression alerts.
• Eval harness — golden questions, human rubrics, automated checks (JSON validity, citation overlap).
• Governance — risk classification, model cards, change control; frameworks like ISO/IEC 42001 (AI management systems).

On AWS, patterns from Generative AI Foundations map to Bedrock (managed models), SageMaker (custom train/host), and platform primitives (IAM, KMS, CloudWatch). Classical ML vocabulary from Machine Learning Foundations still applies to eval and hybrid systems.

Use-case patterns (where generative AI earns its cost)

• Draft-and-review — emails, specs, tickets; human approves before send.
• Semantic search + answer — RAG over docs, support KB, internal wikis.
• Extraction and structuring — PDF → JSON, log → incident fields; validate schema.
• Code assistance — completion and refactor in IDE; CI still runs tests.
• Creative acceleration — mood boards, variants; legal review for likeness and IP.
• Synthetic data — test fixtures and edge cases; watch for privacy leakage from training on real PII.

Poor fits: high-stakes decisions without human oversight, strict deterministic compliance text, or problems where a small discriminative model is cheaper and more explainable.

Risks, limits, and responsible deployment

Hallucination — fluent falsehoods; mitigate with RAG, citations, confidence thresholds, and “I don’t know” policies.

Prompt injection and tool abuse — untrusted content in context can hijack agents; least-privilege tools, output filtering, separation of system vs user content.

Bias and fairness — outputs reflect training data and feedback loops; test across languages, demographics, and edge cases.

Copyright and consent — training data, generated assets, and user uploads need clear policy; RAG corpora need rights to index.

Environmental impact — large training and inference consume energy; right-size models and batch workloads (ties to FinOps/GreenOps themes on this site).

Transparency — disclose AI-generated content where regulations or user trust require it.

Technical mitigations without organizational policy rarely suffice. Course-level foundations (e.g. AWS Generative AI Foundations) emphasize vocabulary and boundaries; standards like ISO 42001 address how organizations govern systems in operation.

Choosing an approach (decision guide)

• Need fluent language over public knowledge, moderate risk — frontier API + monitoring; iterate prompts.
• Answers must use your documents — RAG + evals; smaller embed model + capable LLM often beats largest model alone.
• Strict brand voice or API format — fine-tune or constrained decoding; golden tests on every release.
• Data cannot leave your network — open weights on private GPUs or VPC endpoints.
• Images at scale, brand-controlled — diffusion with approved styles; human review for external publish.
• Tabular fraud or churn — classical ML may win on cost and explainability.

Open vs closed models

Closed APIs (vendor-hosted frontier models) optimize time-to-market and elastic scale. Open weights (Llama, Mistral, Qwen, Stable Diffusion class) trade operational burden for control, auditability, and air-gapped deploy. Hyperscaler mappings: AWS vs GCP vs Azure service mapping (Bedrock, Vertex AI, Azure OpenAI).

Where the field is heading

• Smaller, capable specialists — route easy queries to small models; reserve frontier for hard tasks.
• Longer context + better retrieval hybrids — “million token” research vs practical RAG in production.
• Multimodal defaults — voice, vision, and document in one product surface.
• Agents with guardrails — more automation demands security and ops maturity (same as any platform).
• Regulation and procurement — AI management systems and sector rules becoming checklist items.

Generative AI is infrastructure. Teams that treat it like databases or Kubernetes—versioned, measured, secured, and owned—ship durable value; teams that treat it as magic ship demos.

Learning path on this site

1. Historical paradigms — narrative context.
2. This guide — generative AI breadth across modalities and lifecycle.
3. Foundation models in depth — transformers, training, RAG, production.
4. How to become an AI developer — stack and career framing.
5. AWS Generative AI Foundations — cloud and responsible-use vocabulary.

Further reading

• Goodfellow, Bengio, Courville — Deep Learning (generative chapters)
• Ho, Jain, Abbeel — diffusion model foundations
• Goodfellow et al. — original GAN paper (read historically)
• Bommasani et al. — On the Opportunities and Risks of Foundation Models (Stanford CRFM)
• Major cloud and model-provider documentation on RAG, evals, and safety

Blog index · LLMs in depth · Foundation models · AI history · AI developer guide · Generative AI Foundations · ML Foundations · ISO 42001

Back to blog list