Documentation

text-mallet is a toolkit for transforming text into obfuscated or derived formats while preserving utility for downstream NLP tasks such as classification, retrieval, and topic modelling.

We primarily focus on preventing the risk of copyright infringement by degrading the information content of the text sufficiently to prevent reconstructability (an increasingly difficult task) while also preserving some utility of the texts for model training, particularly encoder-only.

What is a potential use case?

For instance, a use case is at the Deutsche National Bibliothek (German National Library), where they have a significant amount of text that is copyright protected and wish to use it in an obfuscated, non-human-readable form to build open-source classification models.

 Text Mallet Logo

Examples

Algorithm: Part-of-Speech Filtering
Original
Data obfuscation is the process of modifying sensitive data in such a way that it is of no or little value to unauthorized intruders while still being usable by software or authorized personnel. Data masking can also be referred as anonymization, or tokenization, depending on different context.
↓  pos-filter  ↓
Obfuscated
Data obfuscation AUX DET process ADP VERB ADJ data ADP DET DET way SCONJ PRON AUX ADP DET CCONJ ADJ value ADP ADJ intruders SCONJ ADV AUX ADJ ADP software CCONJ ADJ personnel PUNCT Data masking AUX ADV AUX VERB ADP anonymization PUNCT CCONJ tokenization PUNCT VERB ADP ADJ context PUNCT
Algorithm: Mutual Information Filter
Original
Data obfuscation is the process of modifying sensitive data in such a way that it is of no or little value to unauthorized intruders while still being usable by software or authorized personnel. Data masking can also be referred as anonymization, or tokenization, depending on different context.
↓  shannon  ↓
Obfuscated
Data obfuscation _ _ process _ modifying sensitive data _ such _ _ _ _ _ _ _ _ value _ unauthorized intruders while _ _ usable _ _ _ authorized personnel _ Data masking _ _ _ _ anonymization _ _ tokenization _ depending _ _ context _
Algorithm: Hierarchical Scrambling
Original
Data obfuscation is the process of modifying sensitive data in such a way that it is of no or little value to unauthorized intruders while still being usable by software or authorized personnel. Data masking can also be referred as anonymization, or tokenization, depending on different context.
↓  scramble-hier  ↓
Obfuscated
in or software obfuscation that such it value Data to data personnel little. of while the no is way by process usable still a sensitive unauthorized authorized being modifying intruders , different tokenization can Data or masking also. anonymization as context depending referred be on
Algorithm: Bag of Words
Original
Data obfuscation is the process of modifying sensitive data in such a way that it is of no or little value to unauthorized intruders while still being usable by software or authorized personnel. Data masking can also be referred as anonymization, or tokenization, depending on different context.
↓  scramble-BoW  ↓
Obfuscated
intruders unauthorized Data referred modifying usable as being of or Data is process software it still little sensitive be that value context. different or depending masking a no tokenization, authorized of to or personnel. anonymization, the can data while by is obfuscation on way such also in

Contents