Automate the Boring Stuff with Python

Automate the Boring Stuff with Python: Practical Programming for Total Beginners
Dedication
About the Author
About the Tech Reviewer
Acknowledgments
Introduction
    Whom Is This Book For?
    Conventions
    What Is Programming?
        What Is Python?
        Programmers Don’t Need to Know Much Math
        Programming Is a Creative Activity
    About This Book
    Downloading and Installing Python
    Starting IDLE
        The Interactive Shell
    How to Find Help
    Asking Smart Programming Questions
    Summary
I. Python Programming Basics
    1. Python Basics
        Entering Expressions into the Interactive Shell
        The Integer, Floating-Point, and String Data Types
        String Concatenation and Replication
        Storing Values in Variables
            Assignment Statements
            Variable Names
        Your First Program
        Dissecting Your Program
            Comments
            The print() Function
            The input() Function
            Printing the User’s Name
            The len() Function
            The str(), int(), and float() Functions
        Summary
        Practice Questions
    2. Flow Control
        Boolean Values
        Comparison Operators
        Boolean Operators
            Binary Boolean Operators
            The not Operator
        Mixing Boolean and Comparison Operators
        Elements of Flow Control
            Conditions
            Blocks of Code
        Program Execution
        Flow Control Statements
            if Statements
            else Statements
            elif Statements
            while Loop Statements
                An Annoying while Loop
            break Statements
            continue Statements
            for Loops and the range() Function
                An Equivalent while Loop
                The Starting, Stopping, and Stepping Arguments to range()
        Importing Modules
            from import Statements
        Ending a Program Early with sys.exit()
        Summary
        Practice Questions
    3. Functions
        def Statements with Parameters
        Return Values and return Statements
        The None Value
        Keyword Arguments and print()
        Local and Global Scope
            Local Variables Cannot Be Used in the Global Scope
            Local Scopes Cannot Use Variables in Other Local Scopes
            Global Variables Can Be Read from a Local Scope
            Local and Global Variables with the Same Name
        The global Statement
        Exception Handling
        A Short Program: Guess the Number
        Summary
        Practice Questions
        Practice Projects
            The Collatz Sequence
            Input Validation
    4. Lists
        The List Data Type
            Getting Individual Values in a List with Indexes
            Negative Indexes
            Getting Sublists with Slices
            Getting a List’s Length with len()
            Changing Values in a List with Indexes
            List Concatenation and List Replication
            Removing Values from Lists with del Statements
        Working with Lists
            Using for Loops with Lists
            The in and not in Operators
            The Multiple Assignment Trick
        Augmented Assignment Operators
        Methods
            Finding a Value in a List with the index() Method
            Adding Values to Lists with the append() and insert() Methods
            Removing Values from Lists with remove()
            Sorting the Values in a List with the sort() Method
        Example Program: Magic 8 Ball with a List
        List-like Types: Strings and Tuples
            Mutable and Immutable Data Types
            The Tuple Data Type
            Converting Types with the list() and tuple() Functions
        References
            Passing References
            The copy Module’s copy() and deepcopy() Functions
        Summary
        Practice Questions
        Practice Projects
            Comma Code
            Character Picture Grid
    5. Dictionaries and Structuring Data
        The Dictionary Data Type
            Dictionaries vs. Lists
            The keys(), values(), and items() Methods
            Checking Whether a Key or Value Exists in a Dictionary
            The get() Method
            The setdefault() Method
        Pretty Printing
        Using Data Structures to Model Real-World Things
            A Tic-Tac-Toe Board
            Nested Dictionaries and Lists
        Summary
        Practice Questions
        Practice Projects
            Fantasy Game Inventory
            List to Dictionary Function for Fantasy Game Inventory
    6. Manipulating Strings
        Working with Strings
            String Literals
                Double Quotes
                Escape Characters
                Raw Strings
                Multiline Strings with Triple Quotes
                Multiline Comments
            Indexing and Slicing Strings
            The in and not in Operators with Strings
        Useful String Methods
            The upper(), lower(), isupper(), and islower() String Methods
            The isX String Methods
            The startswith() and endswith() String Methods
            The join() and split() String Methods
            Justifying Text with rjust(), ljust(), and center()
            Removing Whitespace with strip(), rstrip(), and lstrip()
            Copying and Pasting Strings with the pyperclip Module
        Project: Password Locker
            Step 1: Program Design and Data Structures
            Step 2: Handle Command Line Arguments
            Step 3: Copy the Right Password
        Project: Adding Bullets to Wiki Markup
            Step 1: Copy and Paste from the Clipboard
            Step 2: Separate the Lines of Text and Add the Star
            Step 3: Join the Modified Lines
        Summary
        Practice Questions
        Practice Project
            Table Printer
II. Automating Tasks
    7. Pattern Matching with Regular Expressions
        Finding Patterns of Text Without Regular Expressions
        Finding Patterns of Text with Regular Expressions
            Creating Regex Objects
            Matching Regex Objects
            Review of Regular Expression Matching
        More Pattern Matching with Regular Expressions
            Grouping with Parentheses
            Matching Multiple Groups with the Pipe
            Optional Matching with the Question Mark
            Matching Zero or More with the Star
            Matching One or More with the Plus
            Matching Specific Repetitions with Curly Brackets
        Greedy and Nongreedy Matching
        The findall() Method
        Character Classes
        Making Your Own Character Classes
        The Caret and Dollar Sign Characters
        The Wildcard Character
            Matching Everything with Dot-Star
            Matching Newlines with the Dot Character
        Review of Regex Symbols
        Case-Insensitive Matching
        Substituting Strings with the sub() Method
        Managing Complex Regexes
        Combining re.IGNORECASE, re.DOTALL, and re.VERBOSE
        Project: Phone Number and Email Address Extractor
            Step 1: Create a Regex for Phone Numbers
            Step 2: Create a Regex for Email Addresses
            Step 3: Find All Matches in the Clipboard Text
            Step 4: Join the Matches into a String for the Clipboard
            Running the Program
            Ideas for Similar Programs
        Summary
        Practice Questions
        Practice Projects
            Strong Password Detection
            Regex Version of strip()
    8. Reading and Writing Files
        Files and File Paths
            Backslash on Windows and Forward Slash on OS X and Linux
            The Current Working Directory
            Absolute vs. Relative Paths
            Creating New Folders with os.makedirs()
        The os.path Module
            Handling Absolute and Relative Paths
            Finding File Sizes and Folder Contents
            Checking Path Validity
        The File Reading/Writing Process
            Opening Files with the open() Function
            Reading the Contents of Files
            Writing to Files
        Saving Variables with the shelve Module
        Saving Variables with the pprint.pformat() Function
        Project: Generating Random Quiz Files
            Step 1: Store the Quiz Data in a Dictionary
            Step 2: Create the Quiz File and Shuffle the Question Order
            Step 3: Create the Answer Options
            Step 4: Write Content to the Quiz and Answer Key Files
        Project: Multiclipboard
            Step 1: Comments and Shelf Setup
            Step 2: Save Clipboard Content with a Keyword
            Step 3: List Keywords and Load a Keyword’s Content
        Summary
        Practice Questions
        Practice Projects
            Extending the Multiclipboard
            Mad Libs
            Regex Search
    9. Organizing Files
        The shutil Module
            Copying Files and Folders
            Moving and Renaming Files and Folders
            Permanently Deleting Files and Folders
            Safe Deletes with the send2trash Module
        Walking a Directory Tree
        Compressing Files with the zipfile Module
            Reading ZIP Files
            Extracting from ZIP Files
            Creating and Adding to ZIP Files
        Project: Renaming Files with American-Style Dates to European-Style Dates
            Step 1: Create a Regex for American-Style Dates
            Step 2: Identify the Date Parts from the Filenames
            Step 3: Form the New Filename and Rename the Files
            Ideas for Similar Programs
        Project: Backing Up a Folder into a ZIP File
            Step 1: Figure Out the ZIP File’s Name
            Step 2: Create the New ZIP File
            Step 3: Walk the Directory Tree and Add to the ZIP File
            Ideas for Similar Programs
        Summary
        Practice Questions
        Practice Projects
            Selective Copy
            Deleting Unneeded Files
            Filling in the Gaps
    10. Debugging
        Raising Exceptions
        Getting the Traceback as a String
        Assertions
            Using an Assertion in a Traffic Light Simulation
            Disabling Assertions
        Logging
            Using the logging Module
            Don’t Debug with print()
            Logging Levels
            Disabling Logging
            Logging to a File
        IDLE’s Debugger
            Go
            Step
            Over
            Out
            Quit
            Debugging a Number Adding Program
            Breakpoints
        Summary
        Practice Questions
        Practice Project
            Debugging Coin Toss
    11. Web Scraping
        Project: mapit.py with the webbrowser Module
            Step 1: Figure Out the URL
            Step 2: Handle the Command Line Arguments
            Step 3: Handle the Clipboard Content and Launch the Browser
            Ideas for Similar Programs
        Downloading Files from the Web with the requests Module
            Downloading a Web Page with the requests.get() Function
            Checking for Errors
        Saving Downloaded Files to the Hard Drive
        HTML
            Resources for Learning HTML
            A Quick Refresher
            Viewing the Source HTML of a Web Page
            Opening Your Browser’s Developer Tools
            Using the Developer Tools to Find HTML Elements
        Parsing HTML with the BeautifulSoup Module
            Creating a BeautifulSoup Object from HTML
            Finding an Element with the select() Method
            Getting Data from an Element’s Attributes
        Project: “I’m Feeling Lucky” Google Search
            Step 1: Get the Command Line Arguments and Request the Search Page
            Step 2: Find All the Results
            Step 3: Open Web Browsers for Each Result
            Ideas for Similar Programs
        Project: Downloading All XKCD Comics
            Step 1: Design the Program
            Step 2: Download the Web Page
            Step 3: Find and Download the Comic Image
            Step 4: Save the Image and Find the Previous Comic
            Ideas for Similar Programs
        Controlling the Browser with the selenium Module
            Starting a Selenium-Controlled Browser
            Finding Elements on the Page
            Clicking the Page
            Filling Out and Submitting Forms
            Sending Special Keys
            Clicking Browser Buttons
            More Information on Selenium
        Summary
        Practice Questions
        Practice Projects
            Command Line Emailer
            Image Site Downloader
            2048
            Link Verification
    12. Working with Excel Spreadsheets
        Excel Documents
        Installing the openpyxl Module
        Reading Excel Documents
            Opening Excel Documents with OpenPyXL
            Getting Sheets from the Workbook
            Getting Cells from the Sheets
            Converting Between Column Letters and Numbers
            Getting Rows and Columns from the Sheets
            Workbooks, Sheets, Cells
        Project: Reading Data from a Spreadsheet
            Step 1: Read the Spreadsheet Data
            Step 2: Populate the Data Structure
            Step 3: Write the Results to a File
            Ideas for Similar Programs
        Writing Excel Documents
            Creating and Saving Excel Documents
            Creating and Removing Sheets
            Writing Values to Cells
        Project: Updating a Spreadsheet
            Step 1: Set Up a Data Structure with the Update Information
            Step 2: Check All Rows and Update Incorrect Prices
            Ideas for Similar Programs
        Setting the Font Style of Cells
        Font Objects
        Formulas
        Adjusting Rows and Columns
            Setting Row Height and Column Width
            Merging and Unmerging Cells
            Freeze Panes
        Charts
        Summary
        Practice Questions
        Practice Projects
            Multiplication Table Maker
            Blank Row Inserter
            Spreadsheet Cell Inverter
            Text Files to Spreadsheet
            Spreadsheet to Text Files
    13. Working with PDF and word Documents
        PDF Documents
            Extracting Text from PDFs
            Decrypting PDFs
            Creating PDFs
                Copying Pages
                Rotating Pages
                Overlaying Pages
                Encrypting PDFs
        Project: Combining Select Pages from Many PDFs
            Step 1: Find All PDF Files
            Step 2: Open Each PDF
            Step 3: Add Each Page
            Step 4: Save the Results
            Ideas for Similar Programs
        Word Documents
            Reading Word Documents
            Getting the Full Text from a .docx File
            Styling Paragraph and Run Objects
            Creating Word Documents with Nondefault Styles
            Run Attributes
            Writing Word Documents
            Adding Headings
            Adding Line and Page Breaks
            Adding Pictures
        Summary
        Practice Questions
        Practice Projects
            PDF Paranoia
            Custom Invitations as Word Documents
            Brute-Force PDF Password Breaker
    14. Working with CSV Files and JSON Data
        The CSV Module
            Reader Objects
            Reading Data from Reader Objects in a for Loop
            Writer Objects
            The delimiter and lineterminator Keyword Arguments
        Project: Removing the Header from CSV Files
            Step 1: Loop Through Each CSV File
            Step 2: Read in the CSV File
            Step 3: Write Out the CSV File Without the First Row
            Ideas for Similar Programs
        JSON and APIs
        The JSON Module
            Reading JSON with the loads() Function
            Writing JSON with the dumps() Function
        Project: Fetching Current Weather Data
            Step 1: Get Location from the Command Line Argument
            Step 2: Download the JSON Data
            Step 3: Load JSON Data and Print Weather
            Ideas for Similar Programs
        Summary
        Practice Questions
        Practice Project
            Excel-to-CSV Converter
    15. Keeping Time, Scheduling Tasks, and Launching Programs
        The time Module
            The time.time() Function
            The time.sleep() Function
        Rounding Numbers
        Project: Super Stopwatch
            Step 1: Set Up the Program to Track Times
            Step 2: Track and Print Lap Times
            Ideas for Similar Programs
        The datetime Module
            The timedelta Data Type
            Pausing Until a Specific Date
            Converting datetime Objects into Strings
            Converting Strings into datetime Objects
        Review of Python’s Time Functions
        Multithreading
            Passing Arguments to the Thread’s Target Function
            Concurrency Issues
        Project: Multithreaded XKCD Downloader
            Step 1: Modify the Program to Use a Function
            Step 2: Create and Start Threads
            Step 3: Wait for All Threads to End
        Launching Other Programs from Python
            Passing Command Line Arguments to Popen()
            Task Scheduler, launchd, and cron
            Opening Websites with Python
            Running Other Python Scripts
            Opening Files with Default Applications
        Project: Simple Countdown Program
            Step 1: Count Down
            Step 2: Play the Sound File
            Ideas for Similar Programs
        Summary
        Practice Questions
        Practice Projects
            Prettified Stopwatch
            Scheduled Web Comic Downloader
    16. Sending Email and Text Messages
        SMTP
        Sending Email
            Connecting to an SMTP Server
            Sending the SMTP “Hello” Message
            Starting TLS Encryption
            Logging in to the SMTP Server
            Sending an Email
            Disconnecting from the SMTP Server
        IMAP
        Retrieving and Deleting Emails with IMAP
            Connecting to an IMAP Server
            Logging in to the IMAP Server
            Searching for Email
                Selecting a Folder
                Performing the Search
                Size Limits
            Fetching an Email and Marking It As Read
            Getting Email Addresses from a Raw Message
            Getting the Body from a Raw Message
            Deleting Emails
            Disconnecting from the IMAP Server
        Project: Sending Member Dues Reminder Emails
            Step 1: Open the Excel File
            Step 2: Find All Unpaid Members
            Step 3: Send Customized Email Reminders
        Sending Text Messages with Twilio
            Signing Up for a Twilio Account
            Sending Text Messages
        Project: “Just Text Me” Module
        Summary
        Practice Questions
        Practice Projects
            Random Chore Assignment Emailer
            Umbrella Reminder
            Auto Unsubscriber
            Controlling Your Computer Through Email
    17. Manipulating Images
        Computer Image Fundamentals
            Colors and RGBA Values
            Coordinates and Box Tuples
        Manipulating Images with Pillow
            Working with the Image Data Type
            Cropping Images
            Copying and Pasting Images onto Other Images
            Resizing an Image
            Rotating and Flipping Images
            Changing Individual Pixels
        Project: Adding a Logo
            Step 1: Open the Logo Image
            Step 2: Loop Over All Files and Open Images
            Step 3: Resize the Images
            Step 4: Add the Logo and Save the Changes
            Ideas for Similar Programs
        Drawing on Images
            Drawing Shapes
                Points
                Lines
                Rectangles
                Ellipses
                Polygons
                Drawing Example
            Drawing Text
        Summary
        Practice Questions
        Practice Projects
            Extending and Fixing the Chapter Project Programs
            Identifying Photo Folders on the Hard Drive
            Custom Seating Cards
    18. Controlling the Keyboard and Mouse with GUI Automation
        Installing the pyautogui Module
        Staying on Track
            Shutting Down Everything by Logging Out
            Pauses and Fail-Safes
        Controlling Mouse Movement
            Moving the Mouse
            Getting the Mouse Position
        Project: “Where Is the Mouse Right Now?”
            Step 1: Import the Module
            Step 2: Set Up the Quit Code and Infinite Loop
            Step 3: Get and Print the Mouse Coordinates
        Controlling Mouse Interaction
            Clicking the Mouse
            Dragging the Mouse
            Scrolling the Mouse
        Working with the Screen
            Getting a Screenshot
            Analyzing the Screenshot
        Project: Extending the mouseNow Program
        Image Recognition
        Controlling the Keyboard
            Sending a String from the Keyboard
            Key Names
            Pressing and Releasing the Keyboard
            Hotkey Combinations
        Review of the PyAutoGUI Functions
        Project: Automatic Form Filler
            Step 1: Figure Out the Steps
            Step 2: Set Up Coordinates
            Step 3: Start Typing Data
            Step 4: Handle Select Lists and Radio Buttons
            Step 5: Submit the Form and Wait
        Summary
        Practice Questions
        Practice Projects
            Looking Busy
            Instant Messenger Bot
            Game-Playing Bot Tutorial
A. Installing Third-Party Modules
    The pip Tool
    Installing Third-Party Modules
B. Running Programs
    Shebang Line
    Running Python Programs on Windows
    Running Python Programs on OS X and Linux
    Running Python Programs with Assertions Disabled
C. Answers to the Practice Questions
    Chapter 1
    Chapter 2
    Chapter 3
    Chapter 4
    Chapter 5
    Chapter 6
    Chapter 7
    Chapter 8
    Chapter 9
    Chapter 10
    Chapter 11
    Chapter 12
    Chapter 13
    Chapter 14
    Chapter 15
    Chapter 16
    Chapter 17
    Chapter 18
D. Resources
Index
Copyright