Skip to content
PodSubs
ShowsPeopleGuestsJobsNewsEvents
More
BlogPodCred
Menu
ShowsPeopleGuestsJobsNewsEventsBlogPodCredNetworksSign in
Join nowSign in
  • Shows
  • People
  • Jobs
  • News
  • More
    • Guests
    • Events
    • Blog
    • PodCred
    • Networks
    Join nowSign in

Daily Python Projects Podcast

Daily Python Projects Podcast artwork

Read a 1 GB Log File with Python Generators: Day 2 - Generator Pipelines and a Live Log Watcher

Oct 21, 20261 minAudio

This is a free preview of a paid episode. To hear more, visit dailypythonprojects.substack.com Yesterday we read a 1 GB log file with one generator. Today we connect small generators into a pipeline, and point it at a log that a web server is still writing. Here is today’s program watching a live log. A fake web server adds about 50 requests a second, and the watcher prints each error as it appears: It ran for 45 seconds, saw 2,248 requests and 9 errors, and never used more than 0.03 MB of memory. Projects in this week’s series: Day 1: Yesterday, we read a 1 GB log file with a generator and measured how much memory it saves. Day 2 (Today): We connect generators into a pipeline, and build a live watcher that prints new errors while the log is still being written. View All Projects This Week What you get today * A pipeline of four small generators that read, count, parse and filter a log * The first 5 errors of the 1 GB file in 0.002 seconds * A live watcher that works like tail -f and prints errors as they happen * A fake web server, so you can test the watcher without a real website The key idea Each step is a generator that takes the previous one as its input: lines = read_lines("access.log") requests = parse(lines) errors = only_errors(requests) for when, page, status in errors: print(when, status, page) The first three lines do not read anything. They only connect the steps. The work starts when the for loop asks errors for a value. Then only_errors asks parse, parse asks read_lines, and one line travels through all the steps. To watch a live log, we change only the first step. follow() reads to the end of the file and waits for new lines. The rest stays the same. Because nothing runs ahead, the pipeline can also stop early. Here it asks yesterday’s 1 GB file for the first 5 server errors: It read only 772 of the 5 million lines, so it finished in 0.002 seconds. Worth knowing * islice() from itertools is like slicing a list, but for a generator. islice(errors, 5) takes the first 5 values and then stops asking, so the whole pipeline stops too. * A generator that never ends is normal. follow() has a while True loop, so the watcher runs until you press Ctrl+C. * The watcher can catch a line that is only half written. follow() keeps the half and waits for the rest, so the parser only ever sees whole lines. Run it Save the code as logwatch.py next to yesterday’s access.log, then run: python logwatch.py access.log python logwatch.py --demo The second command starts the live demo. It writes a small file called live.log and runs until you press Ctrl+C. There is nothing to install. Try this Change only_errors() so it also lets status 404 through, and run python logwatch.py access.log 20. One page fills the whole list. Which page is it, and who do you think keeps asking for it? Reply and tell me. Solution The full code is below. It is one file, and every step has a comment.

Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.

Hosts and Guests

Nobody is credited on this episode yet.

Credit the people on this episode

Be the first to rate this episode

More episodes

All episodes
  1. Build a Private PDF Toolkit with Python and CustomTkinter: Day 2 - Compress, Rotate and Protect PDFs, and Build an .exe

    Nov 4, 20261 minAudio

    This is a free preview of a paid episode. To hear more, visit dailypythonprojects.substack.com Yesterday we built a desktop app that merges and splits PDFs on your own computer. Today it gets three more tools: Compress,...

    Credit the people on this episode

  2. Build a Sales Report Robot with Python and Polars: Day 2 - Email the Report and Run It Every Monday

    Oct 28, 20261 minAudio

    This is a free preview of a paid episode. To hear more, visit dailypythonprojects.substack.com Yesterday we turned four Excel files into a PDF report with Polars. Today the robot sends that PDF by email, and the...

    Credit the people on this episode

  3. How to use Daily Python Projects

    Aug 21, 20242 minAudio

    This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dailypythonprojects.substack.com/subscribe

    Credit the people on this episode

  4. How to Learn Python

    Aug 16, 20246 minAudio

    This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dailypythonprojects.substack.com/subscribe

    Credit the people on this episode

  5. For-loops thoroughly explained for beginners

    Aug 9, 202418 secAudio

    This is a free preview of a paid episode. To hear more, visit dailypythonprojects.substack.com

    Credit the people on this episode