Reading and Writing Files

File operations are essential in programming as they allow your applications to store and retrieve data persistently. Python provides simple and powerful tools for working with files of various types.

Opening and Closing Files

Before you can read from or write to a file, you need to open it. In Python, this is done using the built-in open() function:

# Basic syntax
file = open(filename, mode)

The open() function takes two main parameters:

filename: The name or path of the file you want to open
mode: A string that specifies how the file will be opened

After you’re done with the file, it’s crucial to close it to free up system resources:

file.close()

However, the preferred approach in Python is to use a context manager (the with statement), which automatically handles closing the file for you:

# Recommended approach using context manager
with open(filename, mode) as file:
    # File operations here
    # No need to explicitly call file.close()

Important: Always use the with statement when working with files. It ensures that resources are properly managed even if exceptions occur during file operations.

File Modes

When opening a file, you specify a mode that determines what operations you can perform:

Mode	Description
`'r'`	Read mode (default). Opens file for reading.
`'w'`	Write mode. Creates a new file or truncates (empties) an existing file.
`'a'`	Append mode. Opens for writing, creating the file if it doesn’t exist, but keeps existing content.
`'x'`	Exclusive creation. Creates a new file, fails if the file already exists.
`'b'`	Binary mode (added to other modes, e.g., `'rb'`, `'wb'`).
`'t'`	Text mode (default, added to other modes, e.g., `'rt'`).
`'+'`	Update mode (both reading and writing, added to other modes, e.g., `'r+'`).

# Examples of opening files with different modes
with open('data.txt', 'r') as file:  # Open for reading (text mode)
    # Read operations

with open('output.txt', 'w') as file:  # Open for writing (text mode)
    # Write operations

with open('log.txt', 'a') as file:  # Open for appending (text mode)
    # Append operations

with open('image.jpg', 'rb') as file:  # Open for reading (binary mode)
    # Binary read operations

Note: The default mode is 'r' (read text) if not specified. When working with text files, Python automatically handles line ending conversions based on your operating system. When working with binary files (like images or executables), always use binary mode ('b').

Reading from Files

Python offers several methods to read from files:

Reading the Entire File

# Read the entire file content as a single string
with open('example.txt', 'r') as file:
    content = file.read()
    print(content)

# Limit how much to read by specifying a size
with open('example.txt', 'r') as file:
    content = file.read(10)  # Read the first 10 characters
    print(content)

Reading Line by Line

# Read a single line
with open('example.txt', 'r') as file:
    first_line = file.readline()
    second_line = file.readline()
    print(first_line)
    print(second_line)

# Read all lines into a list
with open('example.txt', 'r') as file:
    lines = file.readlines()
    print(lines)  # List where each element is a line from the file

# Iterate through the file line by line (memory efficient)
with open('example.txt', 'r') as file:
    for line in file:
        print(line.strip())  # strip() removes the trailing newline

Important: The strip() method removes whitespace characters (including newlines) from the beginning and end of a string. It’s commonly used when reading lines from a file to remove the trailing newline character.

File Position

Files have a current position that determines where the next read or write operation will occur. You can control this position:

with open('example.txt', 'r') as file:
    # Read the first 5 characters
    content = file.read(5)
    print(content)
    
    # Get the current position
    position = file.tell()
    print(f"Current position: {position}")
    
    # Move to a different position (seeking)
    file.seek(0)  # Move back to the beginning
    new_content = file.read(5)
    print(new_content)

Writing to Files

Python provides several methods to write to files:

Basic Writing

# Write a string to a file
with open('output.txt', 'w') as file:
    file.write("Hello, World!\n")
    file.write("This is a new line of text.\n")

Writing Multiple Lines

# Write multiple lines at once
lines = ["First line\n", "Second line\n", "Third line\n"]
with open('output.txt', 'w') as file:
    file.writelines(lines)

# Alternative approach with a loop
lines = ["First line", "Second line", "Third line"]
with open('output.txt', 'w') as file:
    for line in lines:
        file.write(line + '\n')

Note: The write() method doesn’t automatically add a newline character at the end of the string. If you want each write to start on a new line, you need to add \n explicitly.

Appending to Files

If you want to add content to an existing file without overwriting it, use append mode:

# Append to an existing file
with open('log.txt', 'a') as file:
    file.write("New log entry at 2023-05-08 10:30\n")

Working with Different File Types

CSV Files

CSV (Comma-Separated Values) files are commonly used for tabular data:

import csv

# Reading CSV
with open('data.csv', 'r', newline='') as file:
    csv_reader = csv.reader(file)
    
    # Skip the header row
    header = next(csv_reader)
    print(f"Headers: {header}")
    
    # Process the data rows
    for row in csv_reader:
        print(row)

# Writing CSV
data = [
    ['Name', 'Age', 'Country'],  # Header
    ['Alice', 30, 'USA'],
    ['Bob', 25, 'Canada'],
    ['Charlie', 35, 'UK']
]

with open('users.csv', 'w', newline='') as file:
    csv_writer = csv.writer(file)
    csv_writer.writerows(data)

Note: The newline='' parameter is important when working with CSV files to ensure consistent line endings across different platforms.

Reading and Writing CSV with Dictionaries

The csv module also supports working with dictionaries, which can be more intuitive:

import csv

# Reading CSV into dictionaries
with open('data.csv', 'r', newline='') as file:
    csv_reader = csv.DictReader(file)
    for row in csv_reader:
        print(f"Name: {row['Name']}, Age: {row['Age']}, Country: {row['Country']}")

# Writing dictionaries to CSV
data = [
    {'Name': 'Alice', 'Age': 30, 'Country': 'USA'},
    {'Name': 'Bob', 'Age': 25, 'Country': 'Canada'},
    {'Name': 'Charlie', 'Age': 35, 'Country': 'UK'}
]

with open('users.csv', 'w', newline='') as file:
    fieldnames = ['Name', 'Age', 'Country']
    csv_writer = csv.DictWriter(file, fieldnames=fieldnames)
    
    csv_writer.writeheader()  # Write the header row
    csv_writer.writerows(data)  # Write the data rows

JSON Files

JSON (JavaScript Object Notation) is a lightweight data interchange format:

import json

# Reading JSON
with open('data.json', 'r') as file:
    data = json.load(file)
    print(data)

# Writing JSON
user = {
    'name': 'Alice',
    'age': 30,
    'is_active': True,
    'courses': ['Python', 'Data Science', 'Web Development']
}

with open('user.json', 'w') as file:
    # The indent parameter makes the output more readable
    json.dump(user, file, indent=4)

JSON is particularly useful for storing structured data that includes nested objects and arrays.

Working with File Paths

When working with files, you often need to handle file paths. Python’s os and pathlib modules provide cross-platform solutions:

Using `os.path`

import os

# Current working directory
cwd = os.getcwd()
print(f"Current working directory: {cwd}")

# Join path components (works on all platforms)
data_dir = os.path.join('data', 'user_files')
file_path = os.path.join(data_dir, 'example.txt')
print(f"File path: {file_path}")

# Check if a file exists
if os.path.exists(file_path):
    print(f"The file {file_path} exists")
else:
    print(f"The file {file_path} does not exist")

# Get file information
if os.path.exists(file_path):
    size = os.path.getsize(file_path)
    print(f"File size: {size} bytes")
    
    modified_time = os.path.getmtime(file_path)
    print(f"Last modified: {modified_time}")

Using `pathlib` (Python 3.4+)

The pathlib module provides an object-oriented approach to file paths:

from pathlib import Path

# Current working directory
cwd = Path.cwd()
print(f"Current working directory: {cwd}")

# Create a path
data_dir = Path('data') / 'user_files'
file_path = data_dir / 'example.txt'
print(f"File path: {file_path}")

# Check if a file exists
if file_path.exists():
    print(f"The file {file_path} exists")
else:
    print(f"The file {file_path} does not exist")

# Get file information
if file_path.exists():
    size = file_path.stat().st_size
    print(f"File size: {size} bytes")
    
    modified_time = file_path.stat().st_mtime
    print(f"Last modified: {modified_time}")

# Creating directories
data_dir.mkdir(parents=True, exist_ok=True)  # Create parent directories if needed

Note: pathlib is generally preferred in modern Python code for its simplicity and readability. It treats paths as objects with methods and properties rather than strings.

Error Handling in File Operations

File operations can fail for various reasons, such as missing files, permission issues, or disk errors. Proper error handling is crucial:

try:
    with open('non_existent_file.txt', 'r') as file:
        content = file.read()
        print(content)
except FileNotFoundError:
    print("Error: The file does not exist.")
except PermissionError:
    print("Error: You don't have permission to access this file.")
except Exception as e:
    print(f"An unexpected error occurred: {e}")

Here are common file-related exceptions:

Exception	Description
`FileNotFoundError`	The specified file doesn’t exist
`PermissionError`	You don’t have the required permissions
`IOError`	Input/Output operation failed
`IsADirectoryError`	You tried to open a directory as a file
`UnicodeDecodeError`	Error decoding the file with the specified encoding

Working with File Encodings

Text files use character encodings to represent text. The most common encoding is UTF-8, but you might encounter others:

# Specify encoding when opening a file
with open('data.txt', 'r', encoding='utf-8') as file:
    content = file.read()
    print(content)

# Try different encodings if necessary
encodings = ['utf-8', 'latin-1', 'cp1252']
for encoding in encodings:
    try:
        with open('data.txt', 'r', encoding=encoding) as file:
            content = file.read()
            print(f"Successfully read with {encoding} encoding")
            break
    except UnicodeDecodeError:
        print(f"Failed to decode with {encoding} encoding")

Important: Always specify the encoding explicitly when opening text files to ensure consistent behavior across different platforms and Python versions. UTF-8 is a good default choice for most applications.

Practical Examples

Example 1: Log File Parser

This example reads a log file and extracts specific information:

def parse_log_file(log_file_path):
    """
    Parse a log file to extract error messages.
    
    Args:
        log_file_path (str): Path to the log file
        
    Returns:
        list: A list of error messages found in the log
    """
    error_messages = []
    
    try:
        with open(log_file_path, 'r', encoding='utf-8') as file:
            for line_number, line in enumerate(file, 1):
                if 'ERROR' in line:
                    # Extract and store the error message
                    error_messages.append(f"Line {line_number}: {line.strip()}")
    except FileNotFoundError:
        print(f"Error: The log file '{log_file_path}' does not exist.")
    except Exception as e:
        print(f"An error occurred while parsing the log file: {e}")
    
    return error_messages

# Example usage
log_errors = parse_log_file('application.log')
print(f"Found {len(log_errors)} error messages:")
for error in log_errors:
    print(error)

Example 2: CSV Data Analyzer

This example reads a CSV file containing sales data and performs some analysis:

import csv
from datetime import datetime

def analyze_sales_data(csv_file_path):
    """
    Analyze sales data from a CSV file.
    
    Args:
        csv_file_path (str): Path to the CSV file
        
    Returns:
        dict: A dictionary containing the analysis results
    """
    results = {
        'total_sales': 0,
        'average_sale': 0,
        'best_selling_product': '',
        'best_selling_quantity': 0,
        'product_sales': {}
    }
    
    try:
        with open(csv_file_path, 'r', newline='', encoding='utf-8') as file:
            reader = csv.DictReader(file)
            
            sale_count = 0
            
            for row in reader:
                product = row['Product']
                quantity = int(row['Quantity'])
                price = float(row['Price'])
                sale_amount = quantity * price
                
                # Update total sales
                results['total_sales'] += sale_amount
                sale_count += 1
                
                # Track product sales
                if product in results['product_sales']:
                    results['product_sales'][product]['quantity'] += quantity
                    results['product_sales'][product]['amount'] += sale_amount
                else:
                    results['product_sales'][product] = {
                        'quantity': quantity,
                        'amount': sale_amount
                    }
                
                # Check for best selling product
                if results['product_sales'][product]['quantity'] > results['best_selling_quantity']:
                    results['best_selling_product'] = product
                    results['best_selling_quantity'] = results['product_sales'][product]['quantity']
            
            # Calculate average sale if there were any sales
            if sale_count > 0:
                results['average_sale'] = results['total_sales'] / sale_count
    
    except FileNotFoundError:
        print(f"Error: The CSV file '{csv_file_path}' does not exist.")
    except Exception as e:
        print(f"An error occurred during analysis: {e}")
    
    return results

# Example usage
sales_analysis = analyze_sales_data('sales_data.csv')
print(f"Total Sales: ${sales_analysis['total_sales']:.2f}")
print(f"Average Sale: ${sales_analysis['average_sale']:.2f}")
print(f"Best Selling Product: {sales_analysis['best_selling_product']} ({sales_analysis['best_selling_quantity']} units)")
print("\nProduct Sales Breakdown:")
for product, data in sales_analysis['product_sales'].items():
    print(f"{product}: {data['quantity']} units, ${data['amount']:.2f}")

Example 3: Config File Manager

This example creates a simple configuration file manager using JSON:

import json
import os

class ConfigManager:
    """A class to manage configuration files in JSON format."""
    
    def __init__(self, config_file_path):
        """
        Initialize the ConfigManager.
        
        Args:
            config_file_path (str): Path to the configuration file
        """
        self.config_file_path = config_file_path
        self.config = {}
        self.load_config()
    
    def load_config(self):
        """Load the configuration from the file."""
        if os.path.exists(self.config_file_path):
            try:
                with open(self.config_file_path, 'r') as file:
                    self.config = json.load(file)
                print(f"Configuration loaded from {self.config_file_path}")
            except json.JSONDecodeError:
                print(f"Error: The config file {self.config_file_path} is not valid JSON.")
            except Exception as e:
                print(f"Error loading configuration: {e}")
        else:
            print(f"Config file {self.config_file_path} not found. Using default settings.")
            self.config = {}
    
    def save_config(self):
        """Save the configuration to the file."""
        try:
            # Ensure the directory exists
            os.makedirs(os.path.dirname(self.config_file_path), exist_ok=True)
            
            with open(self.config_file_path, 'w') as file:
                json.dump(self.config, file, indent=4)
            print(f"Configuration saved to {self.config_file_path}")
        except Exception as e:
            print(f"Error saving configuration: {e}")
    
    def get(self, key, default=None):
        """
        Get a configuration value.
        
        Args:
            key (str): The configuration key
            default: The default value if the key doesn't exist
            
        Returns:
            The configuration value or the default
        """
        return self.config.get(key, default)
    
    def set(self, key, value):
        """
        Set a configuration value.
        
        Args:
            key (str): The configuration key
            value: The value to set
        """
        self.config[key] = value
    
    def delete(self, key):
        """
        Delete a configuration key.
        
        Args:
            key (str): The configuration key to delete
            
        Returns:
            bool: True if the key was deleted, False if it didn't exist
        """
        if key in self.config:
            del self.config[key]
            return True
        return False

# Example usage
config = ConfigManager('app_config.json')

# Get configuration values
theme = config.get('theme', 'light')
font_size = config.get('font_size', 12)
print(f"Current theme: {theme}, Font size: {font_size}")

# Update configuration
config.set('theme', 'dark')
config.set('font_size', 14)
config.set('auto_save', True)

# Save changes
config.save_config()

Example 4: Text File Backup Utility

This example creates a backup of a text file before modifying it:

import os
import shutil
from datetime import datetime

def backup_and_modify_file(file_path, new_content):
    """
    Create a backup of a file and then modify its content.
    
    Args:
        file_path (str): Path to the file to backup and modify
        new_content (str): New content to write to the file
        
    Returns:
        bool: True if successful, False otherwise
    """
    # Check if the file exists
    if not os.path.exists(file_path):
        print(f"Error: The file {file_path} does not exist.")
        return False
    
    try:
        # Create a backup filename with timestamp
        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
        file_name = os.path.basename(file_path)
        backup_name = f"{file_name}.{timestamp}.bak"
        backup_path = os.path.join(os.path.dirname(file_path), backup_name)
        
        # Create the backup
        shutil.copy2(file_path, backup_path)
        print(f"Backup created at {backup_path}")
        
        # Modify the original file
        with open(file_path, 'w', encoding='utf-8') as file:
            file.write(new_content)
        print(f"File {file_path} has been updated")
        
        return True
    
    except Exception as e:
        print(f"An error occurred: {e}")
        return False

# Example usage
file_to_modify = 'important_document.txt'

# Read the current content
try:
    with open(file_to_modify, 'r') as file:
        current_content = file.read()
        print(f"Current content:\n{current_content}")
except FileNotFoundError:
    # Create the file if it doesn't exist
    with open(file_to_modify, 'w') as file:
        file.write("Initial content")
    current_content = "Initial content"
    print(f"File created with initial content")

# Modify the content
new_content = current_content + "\nThis line was added on " + datetime.now().strftime("%Y-%m-%d %H:%M:%S")

# Backup and modify
success = backup_and_modify_file(file_to_modify, new_content)
if success:
    print("Backup and modification successful")
else:
    print("Backup and modification failed")

Best Practices for File Operations

Always use context managers (with statement)

# Good practice
with open('file.txt', 'r') as file:
    content = file.read()

# Avoid this
file = open('file.txt', 'r')
content = file.read()
file.close()  # Might not execute if an exception occurs

Specify file encoding explicitly

# Good practice
with open('file.txt', 'r', encoding='utf-8') as file:
    content = file.read()

Handle exceptions properly

try:
    with open('file.txt', 'r') as file:
        content = file.read()
except FileNotFoundError:
    print("File not found. Creating a new one.")
    with open('file.txt', 'w') as file:
        file.write("New file content")

Use appropriate file modes

# Reading only
with open('file.txt', 'r') as file:
    # Read operations

# Writing (overwrites existing content)
with open('file.txt', 'w') as file:
    # Write operations

# Appending (preserves existing content)
with open('file.txt', 'a') as file:
    # Append operations

Process large files line by line

# Memory efficient for large files
with open('large_file.txt', 'r') as file:
    for line in file:
        process_line(line)

# Avoid this for large files
with open('large_file.txt', 'r') as file:
    content = file.read()  # Loads the entire file into memory

Use appropriate libraries for specific file formats

# For CSV files
import csv
with open('data.csv', 'r', newline='') as file:
    reader = csv.reader(file)
    # Process CSV

# For JSON files
import json
with open('data.json', 'r') as file:
    data = json.load(file)
    # Process JSON

Create backup before overwriting important files

import shutil

# Create a backup before modifying
shutil.copy2('important_file.txt', 'important_file.txt.bak')

# Modify the original file
with open('important_file.txt', 'w') as file:
    file.write("New content")

Use path manipulation libraries for cross-platform compatibility

# Good practice with pathlib
from pathlib import Path
data_dir = Path('data')
file_path = data_dir / 'example.txt'

# Alternative with os.path
import os
data_dir = os.path.join('data')
file_path = os.path.join(data_dir, 'example.txt')

Exercises

Exercise 1: Write a program that reads a text file and counts the occurrences of each word, then writes the results to a new file in the format “word: count”.

Exercise 2: Create a function that takes a CSV file containing student data (name, score1, score2, score3) and calculates the average score for each student. Write the results to a new CSV file with columns for name and average score.

Exercise 3: Build a simple text file encryption/decryption program that reads a file, applies a Caesar cipher (shifting each letter by a fixed number of positions), and writes the result to a new file.

Exercise 4: Create a log file analyzer that reads a log file (you can create a sample one), extracts entries between two specified dates, and saves those entries to a new file.

Hint for Exercise 1: Use a dictionary to keep track of word counts. Remember to strip punctuation and normalize word case (e.g., convert to lowercase) for accurate counting.

# Exercise 1 solution outline
def count_words(input_file, output_file):
    word_counts = {}
    
    with open(input_file, 'r', encoding='utf-8') as file:
        for line in file:
            words = line.strip().lower().split()
            for word in words:
                # Remove punctuation
                word = word.strip('.,!?":;()[]{}')
                if word:
                    word_counts[word] = word_counts.get(word, 0) + 1
    
    with open(output_file, 'w', encoding='utf-8') as file:
        for word, count in sorted(word_counts.items()):
            file.write(f"{word}: {count}\n")

In the next section, we’ll explore more file operations, focusing on working with CSV and JSON files in more detail.