Senior Data Engineer: Firewall Automation Program
- Contract: 6-12 months (Pty Ltd only)
- $1,150-$1,200/day + GST
- Sydney based (Hybrid)
- Urgent start – interviewing immediately
About the role Build MS SQL ETL pipelines that consolidate 5 million network flows down to 5,000-10,000 automated firewall rules for a tier-1 bank’s CoreMod migration. You’ll enrich NetFlow data with CMDB/DNS/VMware metadata, produce JSON/CSV deployment feeds for APIC/CSM, and optimise performance at scale. This role suits a data engineer who thrives on genuinely interesting technical problems in collaborative mob programming environments.
About the engagement Major infrastructure transformation migrating 400 applications across ~3,000 servers. Three-person specialist team (you, Python dev, network engineer) automating firewall implementation across Legacy and CoreMod infrastructure. Fast-tracked delivery with high autonomy and creative problem-solving.
What you’ll own Data Pipeline: Consolidate 5M network flows stored in MS SQL into contextualised, validated firewall rules using high-performance T-SQL ETL jobs with deduplication, enrichment and sliding-window aggregations.
Deployment Feeds: Produce controller-ready JSON/CSV exports conforming to CSM import formats and APIC object model payloads that enable the Python dev to automate rule implementation via Cisco APIs.
Orchestration: Create Tidal job definitions, retries, alerts and exception queues. Implement lineage tracking from aggregated tuples back to raw flows for audit purposes.
This is collaborative data engineering – mob programming setup where all three specialists work in lockstep. You’ll optimise partitioning, indexing and query plans for very large NetFlow tables while navigating access problems and data challenges with a proactive mindset.
What we need Must have: Expert MS SQL Server (T-SQL, indexing, partitioning, window functions, query tuning), high-performance ETL pipelines for large-scale datasets (5M+ rows), controller-ready export formats (JSON/CSV) for network automation platforms, Tidal or similar orchestration tools, data engineering mindset (NOT DBA), collaborative team player, Pty Ltd contractor.
Nice to have: Dask/Spark experience, Power BI, understanding of network flows/NetFlow data, migration project experience.
Won’t work: Pure DBA backgrounds, expecting cloud migration work, needing extensive banking experience.
The opportunity Real infrastructure impact enabling automated firewall deployment at enterprise scale. Strong likelihood of 12-month engagement.