Pipelines and Python
Loading, cleaning and scheduling data with Python and plain SQL: what broke and how you fixed it.
Rules for this space
- A small example that fails beats a long script.
- Say what a correct result would look like.
Loading, cleaning and scheduling data with Python and plain SQL: what broke and how you fixed it.
A script runs every 15 minutes from cron. Last week one run took 25 minutes, the next one started while it was still going, and both wrote to the same file. How do I stop a second…
Question#cron#schedulingby tom_hartley2 replies
I read a CSV exported from our shop. The order_date column looks like 03/11/2025 (day first), but pandas says the type is object and sorting puts 25/12/2025 before 03/11/2025. A…
Question#dates#pandasby lucia_fernandez2 replies
My load job inserts the previous day's sales into a reporting table. One night it failed halfway, I re-ran it, and now that day has twice the rows and the totals are wrong. I do…
Question#idempotency#pipelinesby chloe_dubois3 replies