Threats in AI Data Engineering

The first article in this series introduced a lifecycle view of AI security. This second article zooms into the first major stage: Data Engineering. This is where teams acquire raw data, clean and normalize it, analyze or transform it into useful features, and finally prepare trusted datasets for training, validation, and testing.

In conventional software, data is often an input handled by code. In AI systems, data is more than an input: it helps define the behavior of the final model. A poisoned source, a compromised cleaning step, a biased feature selection process, or a contaminated train-test split can silently reshape the model long before anyone deploys it.

Why the Data Engineering Stage Matters

Every downstream AI control depends on the assumptions created here: what data is trusted, which records are included or excluded, how labels are assigned, how identifiers are sanitized, and how datasets are split for evaluation. If those assumptions are wrong, later model validation may only confirm a false sense of security.

Strong data engineering security therefore requires more than storage encryption or access control. It requires source vetting, ingestion validation, reproducible transformations, privacy review, provenance tracking, and checks that detect statistical drift, imbalance, leakage, and suspicious records before training starts.

1. Data Acquisition: Trust Starts at the Source

Data acquisition covers first-party collection, third-party feeds, public datasets, web-scraped content, user-generated telemetry, and direct queries against internal systems. The choices made here determine the raw material from which the model will learn.

The central risk is that an attacker, insider, or weakly governed data supplier can influence the source before the security team ever sees it. Poisoned records, malicious labels, planted web content, or manipulated API feeds can enter the pipeline looking like normal data.

• Data poisoning at source: malicious samples or incorrect labels are inserted into crowdsourced, open, partner, or internal datasets.
• Tampering during ingestion: unsecured ETL jobs, API pulls, or file transfers allow records to be modified in transit.
• Dataset supply-chain compromise: reused public datasets, mirrors, or benchmark corpora may contain hidden backdoors or poisoned subsets.
• Tainted scraping streams: adversaries publish content designed to be scraped into training data, introducing misinformation, bias, or embedded instructions.
• Raw-data exposure: weak permissions on buckets, data lakes, or shared workspaces allow theft, replacement, or unauthorized manipulation.

Security objective: treat every data source as a supply-chain dependency. Record provenance, verify integrity, validate schemas, scan for anomalies, and require explicit acceptance criteria before raw data enters trusted storage.

2. Data Pre-processing: Cleaning Can Become Corruption

Pre-processing includes cleaning, deduplication, imputation, encoding, normalization, generalization, filtering, and sanitization. These steps are supposed to reduce noise and privacy risk, but they also create powerful points of control over what the model eventually sees.

A compromised notebook, pull request, ETL script, or deserialization routine can quietly flip labels, inject adversarial noise, alter scaling statistics, or weaken anonymization. Because pre-processing is often automated and reused, one bad transformation can propagate across many experiments.

• Label-flipping attacks: small targeted changes to labels alter the meaning of training examples while preserving the appearance of a clean dataset.
• Adversarial noise injection: perturbations are hidden inside normalization or denoising steps and later trigger misclassification.
• Pipeline code exploitation: insecure scripts, notebooks, dependency updates, or unsafe deserialization expose data or run malicious code.
• Privacy regression: weak sanitization leaves quasi-identifiers or rare combinations that can re-identify individuals.
• Feature-scale tampering: manipulated outliers distort summary statistics so attacker-favored records appear normal.

Security objective: make pre-processing reproducible and reviewable. Version transformation code, pin dependencies, inspect label distributions before and after cleaning, and repeat privacy checks after sanitization.

3. Data Analysis: Feature Engineering Is an Attack Surface

Data analysis and feature engineering turn cleaned data into usable signal. This can include exploratory analysis, extraction, transformation, manual or assisted labeling, augmentation, and synthetic data generation. These activities can improve model performance, but they can also create new privacy, integrity, and bias risks.

Derived features may reveal sensitive information that was removed from raw data. Augmentation may introduce backdoor triggers. Synthetic data may reproduce rare real records. Feature selection may be manipulated so the model depends on spurious signals.

• Backdoor triggers in features: rare patterns introduced during augmentation cause targeted model behavior later.
• Labeling compromise: malicious annotators or flawed model-assisted labeling create systematic errors.
• Synthetic data leakage: generated records resemble sensitive originals closely enough to expose private information.
• Statistical manipulation: attackers skew feature selection, correlation analysis, or importance measures toward weak or attacker-controlled signals.
• Shared analytics exfiltration: cloud notebooks or SaaS analytics platforms leak intermediate features, embeddings, or summaries.

Security objective: audit derived data as new data, not merely a cleaned copy of the original. Validate feature provenance, test augmentation policies, review annotation quality, and run privacy assessments on engineered features.

4. Data Preparation: The Last Gate Before Training

Data preparation packages engineered data into training, validation, and test sets. It may include stratified or time-aware splitting, dataset cards, manifests, hashes, storage in object repositories, and release into an MLOps pipeline. This is the final opportunity to catch data integrity issues before model training begins.

Attackers can target this stage by poisoning validation data, creating train-test leakage, skewing class balance, or swapping final dataset shards in storage. Even accidental contamination can inflate evaluation metrics and hide overfitting.

• Validation and test-set poisoning: crafted evaluation records distort metrics or make weak models appear safe.
• Train-test leakage: duplicated or overlapping records inflate performance and conceal memorization.
• Class-balance skewing: manipulated splits bias decision boundaries and reduce representativeness.
• Packaging and registry attacks: final shards, manifests, or dataset artifacts are replaced or modified before training.
• Embedded exfiltration risks: identifiers left in prepared sets create later privacy exposure through inference attacks.

Security objective: make prepared datasets tamper-evident. Hash artifacts, enforce immutable storage for approved splits, document lineage, and gate training on leakage, balance, privacy, and integrity checks.

Key Threats to Prioritize

From Data Security to Model Security

Data Engineering is the foundation of AI security because it determines what the model learns, what it ignores, and what evaluation signals are trusted. Secure AI teams should therefore treat each data handoff as a control point: source to raw data, raw data to pre-processed data, pre-processed data to engineered features, and engineered features to versioned training assets.

The next article will move into Experiment and Development, where versioned data becomes trained models and where automation, validation gaps, dependencies, and MLOps pipelines create a new set of AI security risks.

Back to blog ↑