PII masking package

Highlights

• Policy-as-config — a single YAML file declares which fields get masked and how, so the rules live with the data contract, not scattered across code
• Multiple masking strategies — pseudonymization (stable, joinable tokens via keyed HMAC), redaction, number bucketing, and date truncation, each matched to the sensitivity of the field
• Free-text scanning — detects and masks PII buried in unstructured fields like ticket descriptions or calendar titles, not just structured columns
• Re-identification risk reporting — measures whether combinations of “harmless” fields (floor + role + time of day) can still single someone out, and flags which rows to suppress before publishing
• Secrets-first key management — HMAC keys are fetched from your cloud provider’s Secret Manager, never hardcoded

Why?

Working at Snap there were multiple instances where I had to check my data or change my pipelines for PII compliance. My idea behind this package was to handle privacy upstream in data pipelines and to make privacy more measurable. Most masking tools focus on obvious fields like email addresses or ID numbers. Quasi-identifiers, details such as a job title, a building floor, a timestamp that together can uniquely identify a person, are more challenging to catch.

This package was built to handle both layers: mask direct identifiers correctly (with tokens that still work for analytics joins), and measure re-identification risk from combinations of fields before any data is shared or published.