Home labs & practical projects

Build a rerunnable pipeline for late-arriving records

A project to do with your own tools, with guidance on what to create and capture.

Vendor environment Suggested time: 2–3 hoursYour own authorised environment

What you could create

A small events dataset arrives in batches containing duplicates and late records. The downstream table must remain trustworthy when a batch is rerun.

Example evidence format

Project evidence map (example)

Pipeline code and synthetic input batchesScope and inputs

Rerun and data-quality verification resultsDecisions and reasoning

Evidence record (example)

Evidence	What to capture
Pipeline code and synthetic input batches	Show the inputs, scope and decisions behind this record. Reprocessing does not duplicate the intended business records.
Rerun and data-quality verification results	Include the evidence supporting your conclusion. Invalid and late data have explicit treatment and visible counts.

What you'll need

Vendor environment

Use an authorised Databricks environment with synthetic event IDs, timestamps and values. Prepare two batches with a deliberate duplicate and a late event.

Project record

Keep the working files for pipeline code and synthetic input batches and record assumptions as you go.

Current reference material

Use the linked provider resources to verify the features, access terms and version relevant to your project.

Access & cost

Access may require a licence, trial or training subscription. Confirm availability and permitted use before committing; no free tenant is assumed.

Use an authorised training tenant with test identities and synthetic data. If access is unavailable, document the plan and label it untested.

Suggested milestones

1
Plan and investigate
Define the raw and cleaned data layers and a rule for record identity.
2
Develop your approach
Implement ingestion and transformation with explicit quality checks.
3
Verify and explain
Rerun a batch and process the late event, documenting the resulting table state.

Evidence worth keeping

Keep clear, organised evidence of your design and any testing you carry out.

1
Pipeline code and synthetic input batches
Show the inputs, scope and decisions behind this record. Reprocessing does not duplicate the intended business records.
2
Rerun and data-quality verification results
Include the evidence supporting your conclusion. Invalid and late data have explicit treatment and visible counts.

Label design proposals and observed results clearly.

Explain your work

Use these prompts to reflect on your decisions and what you learned.

Which part of your pipeline code and synthetic input batches needed the most judgement?
What observation or assumption changed your approach?
What would you improve, and what have you not verified?

Describe this as a personal project in your portfolio.

Useful resources

Use these resources to deepen your understanding and check key details.

Official certification objectives

Review the current provider requirements and relevant subject areas.

Explore external learning resources

Find relevant providers and check features, access requirements and costs.

A useful starting point

Use an authorised Databricks environment with synthetic event IDs, timestamps and values. Prepare two batches with a deliberate duplicate and a late event.

What a good result looks like

Reprocessing does not duplicate the intended business records.
Invalid and late data have explicit treatment and visible counts.

Finish the project

Remove test identities, policies and data, restore the initial configuration, and check trial expiry or renewal settings.

Independent home-project guidance, not an official vendor lab or a claim of complete exam coverage. Use your own tools and label the work accurately.