You are previewing File System Forensic Analysis.
O'Reilly logo
File System Forensic Analysis

Book Description

The Definitive Guide to File System Analysis: Key Concepts and Hands-on Techniques

Most digital evidence is stored within the computer's file system, but understanding how file systems work is one of the most technically challenging concepts for a digital investigator because there exists little documentation. Now, security expert Brian Carrier has written the definitive reference for everyone who wants to understand and be able to testify about how file system analysis is performed.

Carrier begins with an overview of investigation and computer foundations and then gives an authoritative, comprehensive, and illustrated overview of contemporary volume and file systems: Crucial information for discovering hidden evidence, recovering deleted data, and validating your tools. Along the way, he describes data structures, analyzes example disk images, provides advanced investigation scenarios, and uses today's most valuable open source file system analysis tools—including tools he personally developed. Coverage includes

  • Preserving the digital crime scene and duplicating hard disks for "dead analysis"

  • Identifying hidden data on a disk's Host Protected Area (HPA)

  • Reading source data: Direct versus BIOS access, dead versus live acquisition, error handling, and more

  • Analyzing DOS, Apple, and GPT partitions; BSD disk labels; and Sun Volume Table of Contents using key concepts, data structures, and specific techniques

  • Analyzing the contents of multiple disk volumes, such as RAID and disk spanning

  • Analyzing FAT, NTFS, Ext2, Ext3, UFS1, and UFS2 file systems using key concepts, data structures, and specific techniques

  • Finding evidence: File metadata, recovery of deleted files, data hiding locations, and more

  • Using The Sleuth Kit (TSK), Autopsy Forensic Browser, and related open source tools

  • When it comes to file system analysis, no other book offers this much detail or expertise. Whether you're a digital forensics specialist, incident response team member, law enforcement officer, corporate security specialist, or auditor, this book will become an indispensable resource for forensic investigations, no matter what analysis tools you use.

    Table of Contents

    1. Copyright
      1. Dedication
    2. Foreword
    3. Preface
      1. Roadmap
      2. Scope of Book
      3. Resources
    4. Acknowledgments
    5. I. Foundations
      1. 1. Digital Investigation Foundations
        1. Digital Investigations and Evidence
        2. Digital Crime Scene Investigation Process
          1. System Preservation Phase
            1. Preservation Techniques
          2. Evidence Searching Phase
            1. Search Techniques
          3. Event Reconstruction Phase
          4. General Guidelines
        3. Data Analysis
          1. Analysis Types
          2. Essential and Nonessential Data
        4. Overview of Toolkits
          1. EnCase by Guidance Software
          2. Forensic Toolkit by AccessData
          3. ProDiscover by Technology Pathways
          4. SMART by ASR Data
          5. The Sleuth Kit / Autopsy
        5. Summary
        6. Bibliography
          1. Bibliography
      2. 2. Computer Foundations
        1. Data Organization
          1. Binary, Decimal, and Hexadecimal
          2. Data Sizes
          3. Strings and Character Encoding
          4. Data Structures
          5. Flag Values
        2. Booting Process
          1. Central Processing Units and Machine Code
          2. Boot Code Locations
        3. Hard Disk Technology
          1. Hard Disk Geometry and Internals
          2. ATA / IDE Interface
          3. Types of Sector Addresses
            1. Interface Standards
            2. Disk Commands
            3. Hard Disk Passwords
            4. Host Protected Area
            5. Device Configuration Overlay
            6. Serial ATA
          4. BIOS versus Direct Access
            1. Direct Access to Controller
            2. BIOS Access to Controller
          5. SCSI Drives
            1. SCSI versus ATA
            2. Types of SCSI
            3. Connector Types
            4. Size Barriers
        4. Summary
        5. Bibliography
          1. Bibliography
      3. 3. Hard Disk Data Acquisition
        1. Introduction
          1. General Acquisition Procedure
          2. Data Acquisition Layers
          3. Acquisition Tool Testing
        2. Reading the Source Data
          1. Direct versus BIOS Access
          2. Dead Versus Live Acquisition
          3. Error Handling
          4. Host Protected Area
          5. Device Configuration Overlay
          6. Hardware Write Blockers
          7. Software Write Blockers
        3. Writing the Output Data
          1. Destination Location
          2. Image File Format
          3. Compressing the Image File
          4. Network-based Acquisition
          5. Integrity Hashes
        4. A Case Study Using dd
          1. Input Sources
          2. HPA
          3. Output Destinations
          4. Error Handling
          5. Cryptographic Hashes
        5. Summary
        6. Bibliography
          1. Bibliography
    6. II. Volume Analysis
      1. 4. Volume Analysis
        1. Introduction
        2. Background
          1. Volume Concepts
          2. General Theory of Partitions
          3. Usage of Volumes in UNIX
          4. General Theory of Volume Assembly
          5. Sector Addressing
        3. Analysis Basics
          1. Analysis Techniques
          2. Consistency Checks
          3. Extracting the Partition Contents
          4. Recovering Deleted Partitions
        4. Summary
      2. 5. PC-based Partitions
        1. DOS Partitions
          1. General Overview
            1. Basic MBR Concepts
            2. Extended Partition Concepts
            3. Putting the Concepts Together
            4. Boot Code
            5. Summary
          2. Data Structures
            1. MBR Data Structure
            2. Extended Partition Data Structures
            3. Example Image Tool Output
        2. Analysis Considerations
          1. Summary
        3. Apple Partitions
          1. General Overview
          2. Data Structures
            1. Partition Map Entry
            2. Example Image Tool Output
          3. Analysis Considerations
          4. Summary
        4. Removable Media
        5. Bibliography
          1. Bibliography
      3. 6. Server-based Partitions
        1. BSD Partitions
          1. General Overview
            1. FreeBSD Overview
            2. NetBSD and OpenBSD Overview
            3. Boot Code
          2. Data Structures
            1. Disk Label Data Structure
            2. OpenBSD Example Image
            3. FreeBSD Example Image
          3. Analysis Considerations
          4. Summary
        2. Sun Solaris Slices
          1. General Overview
          2. Sparc Data Structures
          3. i386 Data Structures
          4. Analysis Considerations
          5. Summary
        3. GPT Partitions
          1. General Overview
          2. Data Structures
          3. Analysis Considerations
        4. Summary
        5. Bibliography
          1. Bibliography
      4. 7. Multiple Disk Volumes
        1. RAID
          1. RAID Levels
          2. Hardware RAID
            1. Background
            2. Acquisition and Analysis
          3. Software RAID
            1. Background
            2. Acquisition and Analysis
          4. General Analysis Comments
          5. Summary
        2. Disk Spanning
          1. Overview
          2. Linux MD
            1. Background
            2. Acquisition and Analysis
          3. Linux LVM
            1. Background
            2. Acquisition and Analysis
          4. Microsoft Windows LDM
            1. Dynamic Disks
            2. The LDM Database
            3. Acquisition and Analysis
        3. Bibliography
          1. Bibliography
    7. III. File System Analysis
      1. 8. File System Analysis
        1. What is a File System?
          1. Data Categories
          2. Essential and Non-Essential Data
          3. Analysis by Category
        2. File System Category
          1. Analysis Techniques
        3. Content Category
          1. General Information
            1. Logical File System Address
            2. Allocation Strategies
            3. Damaged Data Units
          2. Analysis Techniques
            1. Data Unit Viewing
            2. Logical File System-Level Searching
            3. Data Unit Allocation Status
            4. Data Unit Allocation Order
            5. Consistency Checks
          3. Wiping Techniques
        4. Metadata Category
          1. General Information
            1. Logical File Address
            2. Slack Space
            3. Metadata-based File Recovery
            4. Compressed and Sparse Files
            5. Encrypted Files
          2. Analysis Techniques
            1. Metadata Lookup
            2. Logical File Viewing
            3. Logical File Searching
            4. Unallocated Metadata Analysis
            5. Metadata Attribute Searching and Sorting
            6. Data Structure Allocation Order
            7. Consistency Checks
            8. Wiping Techniques
        5. File Name Category
          1. General Information
            1. File Name-based File Recovery
          2. Analysis Techniques
            1. File Name Listing
            2. File Name Searching
            3. Data Structure Allocation Order
            4. Consistency Checks
          3. Wiping Techniques
        6. Application Category
          1. File System Journals
        7. Application-level Search Techniques
          1. Application-based File Recovery (Data Carving)
          2. File Type Sorting
        8. Specific File Systems
        9. Summary
        10. Bibliography
          1. Bibliography
      2. 9. FAT Concepts and Analysis
        1. Introduction
        2. File System Category
          1. General Concepts
          2. Essential Boot Sector Data
          3. Non-essential Boot Sector Data
          4. Boot Code
          5. Example Image
          6. Analysis Techniques
          7. Analysis Considerations
          8. Analysis Scenario
        3. Content Category
          1. Finding the First Cluster
          2. Cluster and Sector Addresses
          3. Cluster Allocation Status
          4. Allocation Algorithms
          5. Analysis Techniques
          6. Analysis Considerations
          7. Analysis Scenario
        4. Metadata Category
          1. Directory Entries
          2. Cluster Chains
          3. Directories
          4. Directory Entry Addresses
          5. Example Image
          6. Allocation Algorithms
            1. Directory Entry Allocation
            2. Time Value Updating
          7. Analysis Techniques
          8. Analysis Considerations
          9. Analysis Scenarios
            1. File System Creation Date
            2. Searching for Deleted Directories
        5. File Name Category
          1. Allocation Algorithms
          2. Analysis Techniques
          3. Analysis Considerations
          4. Analysis Scenarios
            1. File Name Searching
            2. Directory Entry Ordering
        6. The Big Picture
          1. File Allocation Example
          2. File Deletion Example
        7. Other Topics
          1. File Recovery
          2. Determining the Type
          3. Consistency Check
        8. Summary
        9. Bibliography
          1. Bibliography
      3. 10. FAT Data Structures
        1. Boot Sector
        2. FAT32 FSINFO
        3. FAT
        4. Directory Entries
        5. Long File Name Directory Entries
        6. Summary
        7. Bibliography
          1. Bibliography
      4. 11. NTFS Concepts
        1. Introduction
        2. Everything is a File
        3. MFT Concepts
          1. MFT Entry Contents
          2. MFT Entry Addresses
          3. File System Metadata Files
        4. MFT Entry Attribute Concepts
          1. Attribute Headers
          2. Attribute Content
          3. Standard Attribute Types
        5. Other Attribute Concepts
          1. Base MFT Entries
          2. Sparse Attributes
          3. Compressed Attributes
          4. Encrypted Attributes
            1. Cryptography Basics
            2. NTFS Implementation
        6. Indexes
          1. B-Trees
          2. NTFS Index Attributes
        7. Analysis Tools
        8. Summary
        9. Bibliography
          1. Bibliography
      5. 12. NTFS Analysis
        1. File System Category
          1. $MFT File Overview
          2. $MFTMirr File Overview
          3. $Boot File Overview
          4. $Volume File Overview
          5. $AttrDef File Overview
          6. Example Image
          7. Analysis Techniques
          8. Analysis Considerations
          9. Analysis Scenario
        2. Content Category
          1. Clusters
          2. $Bitmap File Overview
          3. $BadClus File Overview
          4. Allocation Algorithms
          5. File System Layout
          6. Analysis Techniques
          7. Analysis Considerations
          8. Analysis Scenario
        3. Metadata Category
          1. $STANDARD_INFORMATION Attribute
          2. $FILE_NAME Attribute
          3. $DATA Attribute
          4. $ATTRIBUTE_LIST Attribute
          5. $SECURITY_DESCRIPTOR Attribute
          6. $Secure File
          7. Example Image
          8. Allocation Algorithms
            1. MFT Entry and Attribute Allocation
            2. Time Value Updating
          9. Analysis Techniques
          10. Analysis Considerations
          11. Analysis Scenario
        4. File Name Category
          1. Directory Indexes
          2. Root Directory
          3. Links to Files and Directories
          4. Object Identifiers
          5. Allocation Algorithms
          6. Analysis Techniques
          7. Analysis Considerations
          8. Analysis Scenario
        5. Application Category
          1. Disk Quotas
            1. Analysis Considerations
          2. Logging—File System Journaling
            1. Analysis Considerations
          3. Change Journal
            1. Analysis Considerations
        6. The Big Picture
          1. File Allocation Example
          2. File Deletion Example
        7. Other Topics
          1. File Recovery
          2. Consistency Check
        8. Summary
        9. Bibliography
          1. Bibliography
      6. 13. NTFS Data Structures
        1. Basic Concepts
          1. Fixup Values
          2. MFT Entries (File Records)
          3. Attribute Header
        2. Standard File Attributes
          1. $STANDARD_INFORMATION Attribute
          2. $FILE_NAME Attribute
          3. $DATA Attribute
          4. $ATTRIBUTE_LIST Attribute
          5. $OBJECT_ID Attribute
          6. $REPARSE_POINT Attribute
        3. Index Attributes and Data Structures
          1. $INDEX_ROOT Attribute
          2. $INDEX_ALLOCATION Attribute
          3. $BITMAP Attribute
          4. Index Node Header Data Structure
          5. Generic Index Entry Data Structure
          6. Directory Index Entry Data Structure
        4. File System Metadata Files
          1. $MFT File
          2. $Boot File
          3. $AttrDef File
          4. $Bitmap File
          5. $Volume File
          6. $VOLUME_NAME Attribute
          7. $VOLUME_INFORMATION Attribute
          8. $ObjId File
          9. $Quota File
          10. $LogFile File
          11. $UsrJrnl File
        5. Summary
        6. Bibliography
          1. Bibliography
      7. 14. Ext2 and Ext3 Concepts and Analysis
        1. Introduction
        2. File System Category
          1. Overview
            1. Superblock
            2. Block Group Descriptor Tables
            3. Boot Code
            4. Example Image
          2. Analysis Techniques
          3. Analysis Considerations
          4. Analysis Scenario
        3. Content Category
          1. Overview
            1. Blocks
            2. Allocation Status
          2. Allocation Algorithms
          3. Analysis Techniques
          4. Analysis Considerations
          5. Analysis Scenario
        4. Metadata Category
          1. Overview
            1. Inodes
            2. Block Pointers
            3. Attributes
            4. Example Image
          2. Allocation Algorithms
            1. Inode Allocation
            2. Time Value Updating
          3. Analysis Techniques
          4. Analysis Considerations
          5. Analysis Scenario
        5. File Name Category
          1. Overview
            1. Directory Entries
            2. Links and Mount Points
            3. Hash Trees
          2. Allocation Algorithms
          3. Analysis Techniques
          4. Analysis Considerations
          5. Analysis Scenarios
            1. Source of a Moved File
            2. File Deletion Order
        6. Application Category
          1. File System Journaling
            1. Overview
            2. Analysis Techniques
            3. Analysis Considerations
          2. Analysis Scenario
        7. The Big Picture
          1. File Allocation Example
          2. File Deletion Example
        8. Other Topics
          1. File Recovery
          2. Consistency Check
        9. Summary
        10. Bibliography
          1. Bibliography
      8. 15. Ext2 and Ext3 Data Structures
        1. Superblock
        2. Group Descriptor Tables
        3. Block Bitmap
        4. Inodes
        5. Extended Attributes
        6. Directory Entry
        7. Symbolic Link
        8. Hash Trees
        9. Journal Data Structures
        10. Summary
        11. Bibliography
          1. Bibliography
      9. 16. UFS1 and UFS2 Concepts and Analysis
        1. Introduction
        2. File System Category
          1. Overview
            1. Superblock
            2. Cylinder Group Descriptor
            3. Boot Code
            4. Example Image
          2. Analysis Techniques
          3. Analysis Considerations
        3. Content Category
          1. Overview
          2. Allocation Algorithms
          3. Analysis Techniques
          4. Analysis Considerations
        4. Metadata Category
          1. Overview
            1. Inodes
            2. Extended Attributes
            3. Example Image
          2. Allocation Algorithms
          3. Analysis Techniques
          4. Analysis Considerations
        5. File Name Category
          1. Overview
          2. Allocation Algorithms
          3. Analysis Techniques
          4. Analysis Considerations
        6. The Big Picture
          1. File Allocation Example
          2. File Deletion Example
        7. Other Topics
          1. File Recovery
          2. Consistency Check
        8. Summary
        9. Bibliography
          1. Bibliography
      10. 17. UFS1 and UFS2 Data Structures
        1. UFS1 Superblock
        2. UFS2 Superblock
        3. Cylinder Group Summary
        4. UFS1 Group Descriptor
        5. UFS2 Group Descriptor
        6. Block and Fragment Bitmaps
        7. UFS1 Inodes
        8. UFS2 Inodes
        9. UFS2 Extended Attributes
        10. Directory Entries
        11. Summary
        12. Bibliography
      11. A. The Sleuth Kit and Autopsy
        1. The Sleuth Kit
          1. Disk Tools
          2. Volume System Tools
          3. File System Tools
            1. File System Category
            2. Content Category
            3. Metadata Category
            4. File Name Category
            5. Application Category
            6. Multiple Category
          4. Searching Tools
        2. Autopsy
          1. Analysis Modes
        3. Bibliography
          1. Bibliography