Course contents
Lesson 27 — What the audit trail must not show
By Dorian Chávez · co-founder and Chief Executive Officer of Hábil ·
What you will achieve: making the audit trail of an order (what the database keeps for diagnosis and what the journey from lesson 26 displays) not show who the customer is: not her name, not her email, not her phone number, not her street. And having it, masked, still be useful for understanding what happened. To the destination, the order keeps going out complete.
Before you start you need: lesson 26 finished and the environment from lesson 00 turned on.
What is new today, in one sentence
In lesson 26 you built the journey: anyone who can query it sees each attempt with what went out, in full. That includes the person who bought. Today the test order is 4008, which carries a customer's name, email, phone number and address (made up, like everything in this course). Whoever handles support needs to know what happened to the order; they do not need to know where the customer lives. Today you mask the personal data before it enters the audit trail.
The picture: your card statement in the bank's app. It shows you **** **** **** 0428: enough for you to know
which one it is, not enough for someone to use it. The bank does have the full number, and uses it to charge; what it
masks is what is displayed.
1. Set up your folder
cd lecciones/27-lo-que-el-rastro-no-debe-mostrar
cp -R inicio mio
Open datos/ordenes/tramo-2/orden-4008.json and mark with a pencil each piece of data that says who the customer
is. Then open mio/orden-router/reparto.camel.yaml and, in the intentar route, find the comment Desde la 27
("since lesson 27"). Before being stored in payload, the body goes through a new mapping:
lo que sale ──→ al canal (completo)
└─→ mappings/rastro/mascara/v1 ──→ variable rastro ──→ payload (tapado)
| Piece | What it stores |
|---|---|
| The body that goes to the channel | The complete order: the API, the ERP and the partner need it in order to deliver |
payload.cuerpo |
What went out, masked |
intento.sha256 |
The fingerprint of what really went out, unmasked: it still serves to recognize repeats (lesson 18) |
The route is already wired. Your job is the mapping: mio/orden-router/mappings/rastro/mascara/v1/mapeo.jsonata.
Today it returns everything just as it arrives ($). And it has its golden test cases in cases/, like the mappings from
lesson 03.
2. Predict
Answer on paper:
- In the journey of order 4008, which of the customer's data would you see today? Who can query that journey?
- If you mask everything about the customer, could you still tell whether her email was spelled correctly or whether the postal code was valid?
- The
ordentable also stores order 4008 in full. Why is that one allowed to have the data?
Look at how it is today:
./verifica inicio
The first check comes out red: the golden test cases expect the data masked and the mapping lets it through. And further down, the search finds the customer's name, email, phone number and street in the audit trail and in the journey.
3. Mask just enough
Before writing, open the three golden test cases: cases/orden-con-cliente, cases/embarque and cases/entrega. Each
one has its entrada.json and its esperado.json. Compare them and fill in this table:
| Data | Arrives | Is stored | What it still says |
|---|---|---|---|
cliente.nombre |
Ana Ejemplo Prueba |
A. E. P. |
Her initials |
cliente.correo |
ana.prueba@example.com |
a***@example.com |
Whether the domain was valid |
cliente.telefono |
+52 55 0000 0428 |
*** **** **28 |
The last two digits, to recognize it |
cliente.direccion.calle |
Calle de Ejemplo 28 |
*** |
Nothing |
recibio (in a delivery) |
Ana Ejemplo Prueba |
A. E. P. |
Her initials |
tienda, renglones, cp, colonia… |
The same | Everything that is useful for diagnosis |
The mapping has two TODOs.
TODO 1: the $mascara function
In JSONata you define a function with function($v, $modo) { … } and store it in a variable with :=. Yours
receives a value and a mode, and returns the masked value. This is what its shape looks like, with each mode as a
? : question:
(
$mascara := function($v, $modo) {
$not($exists($v)) ? $v
: $modo = 'iniciales' ? …
: $modo = 'correo' ? …
: $modo = 'telefono' ? …
: '***'
};
… (aquí va el TODO 2)
)
The first line takes care of a case that gets forgotten: if the value is not there, let it stay not there. A
shipment does not carry a customer; without that line, a made-up cliente would appear.
The JSONata functions you need for each mode:
| Mode | Idea | Functions |
|---|---|---|
iniciales |
Split the name into words, take the first letter of each one with a period, and join them with a space | $split($trim($v), ' '), $map(…, function($p) { $substring($p, 0, 1) & '.' }), $join(…, ' ') |
correo |
The first letter, ***@ and what comes after the at sign |
$substring($v, 0, 1), $substringAfter($v, '@'), and & to join texts |
telefono |
*** **** ** and the last two digits |
$replace($v, /[^0-9]/, '') leaves only the digits; $substring(…, -2) takes the last two |
TODO 2: applying it where it belongs
JSONata has an operator to change only one part of a document and leave the rest the same, the transform
operator: ~> | ruta | { cambios } |. It reads "in whatever is at ruta, put these changes".
$
~> | cliente | {
"nombre": $mascara(nombre, 'iniciales'),
"correo": $mascara(correo, 'correo'),
"telefono": $mascara(telefono, 'telefono')
} |
~> | cliente.direccion | { "calle": $mascara(calle, 'todo') } |
~> | $ | { "recibio": $mascara(recibio, 'iniciales') } |
They are three chained transforms: one on cliente, one on cliente.direccion and one on the root ($), for the
recibio of a delivery. What you do not name stays the same: that is why the postal code and the order lines pass through
untouched.
Test your mapping before bringing anything up:
../../herramientas/probar-mapeo mio/orden-router/mappings/rastro/mascara/v1
4. Break one on purpose
With your mapping finished, add the postal code to what gets masked, in the transform on cliente.direccion:
"cp": $mascara(cp, 'todo'). Run ./verifica.
First, predict: the audit trail now masks more. Is that better?
The first check comes out red (the golden test case expected the postal code) and so does the last: the audit trail no longer says the postal code. If a delivery fails because the postal code does not exist, the audit trail no longer lets you see it. Masking too much is also a mistake: the audit trail exists for diagnosis, and one that says nothing is useless. A postal code alone, without street or name, hardly points to the customer. Put it back the way it was.
5. Check
./verifica
verifica tests your golden test cases, sends order 4008, waits for its three channels to answer and then searches
for the customer's four pieces of data in the database audit trail, in the journey and in the logs of all the services. It
is done when you see the seven green checks:
✅ El mapeo rastro/mascara/v1 pasa sus casos dorados
✅ Arrancaron el router (que tapa los datos personales del rastro), los tres canales y el sondeo
✅ Control: la orden guardada sí tiene los 4 datos personales (la búsqueda encuentra lo que hay)
✅ En el rastro de la base (payload, intentos, acuses, avisos, conflictos): 0 datos personales
✅ GET /ordenes/4008/recorrido: la orden sí sale, y sin sus datos personales
✅ En los logs: la 4008 aparece (…) veces y sus datos personales 0
✅ Lo tapado sirve para diagnosticar: (…)
Look at the third one. Before counting zeros, verifica searches for the same four pieces of data where they do
have to be, in the orden table, and finds them. If it did not find them there, the "0" in the following checks
would prove nothing: it could be that the search was written wrong. That is called a positive control: before
believing a zero, check that the search knows how to find. And read the zero with its scope: 0 datos personales
("0 personal data") means 0 occurrences of the four pieces of data that verifica searches for, in the places where
it searches for them.
Not working? ./verifica solucion tests ours, and
diff mio/orden-router/mappings/rastro/mascara/v1/mapeo.jsonata solucion/orden-router/mappings/rastro/mascara/v1/mapeo.jsonata
tells you where they differ.
| What you see | What to do |
|---|---|
no pasa sus casos dorados |
Run probar-mapeo as in step 3: it tells you which case and which field does not match |
The embarque case fails with a cliente that should not be there |
The first line of $mascara is missing: if the value is not there, let it stay not there |
En el rastro de la base hay … apariciones |
Some transform is not being applied: check the path of each `~> |
GET /ordenes/4008/recorrido: … de los 4 datos personales a la vista |
The same: the journey reads from payload |
Lo que guarda el rastro de la 4008 without the postal code or without the domain |
You masked too much: the postal code and the email domain stay |
El puerto 8080 está ocupado |
The services from another run were left up: DONDE=mio docker compose -f compose.yaml down |
What this teaches beyond the course
You mask what is displayed, not what is used. The destinations need the complete data in order to deliver: the carrier needs the address. The audit trail is for diagnosis, and it is read by more people than should know the customer. That is why the masking happens there and not in the send.
Masking is a design, not a deletion. The initials, the email domain, the last two digits and the postal code say enough to recognize and diagnose, and much less than the full data. Deciding what stays is the hard part. And masking fields is not anonymizing: together, or with other sources, some data can point back to the person (exercise 2 explores it). A real audit trail also needs a decision on who can read it and how long it is kept.
The fingerprint is still that of the real thing. The audit trail stores what is masked, but the fingerprint from lesson 18 is taken from what really went out. That way a repeat is still recognized without storing the data.
Before believing a zero, search where it is. A 0 datos personales can be a clean audit trail or a search that does
not know how to search. The positive control is what tells them apart, and it works for any measurement that gives
you zero.
Exercise: what is seen and what is not
- Ask for the journey of order 4008 (
curl -s localhost:8080/ordenes/4008/recorrido) and fill in two columns: what you can know about the order and what you cannot know about the customer. - In
orden-4008.json, find another piece of data that, together with the ones left unmasked, could help identify the customer. Would you mask it? What would you lose for diagnosis if you mask it? - Someone in support needs to call the customer because her delivery failed. Where should they get the phone number from: the audit trail or somewhere else? Who should be able to see it?
Why it was done this way
What we decided in this lesson, along with what we ruled out and what would make us change our mind:
- The audit trail is stored masked
Review
- Which data in order 4008 is personal, and which is not?
- Why is the masking done in the audit trail and not in what goes to the destination?
- What does JSONata's transform operator,
~> | ruta | { … } |, do? - Why are the postal code and the email domain left in view?
- The closing question:
verificasearches for the data in theordentable before counting zeros in the audit trail. What is that for, and what would happen to the test if it did not?
Next: in lesson 28 the integration stops talking only over HTTP and files: the lifecycle events of the order travel through a topic, and each integration reads them with its own group.
Prefer email? Write to us at hola@habil.mx