When setting up a new computer, there's no need to spend money on most commercial programs - Freeware equivalents exist. Download what you need and enjoy the savings.
This list is not meant to be all-inclusive. It's a list of software I find essential when saving money and replacing commercial software with freeware. (Freeware game recommendations are elsewhere.) While multiple free solutions exist for all categories, I only recommend one to keep things simple, the one I prefer myself.
Productivity applications
OpenOffice: A package consisting of a word processor, spreadsheet, presentation maker, and other handy programs, it can read and write Microsoft Office files (like Word and Excel), and can export to PDF. More detail and specifics on OpenOffice here.
CutePDF Writer: Create PDFs from any "printable" application with this PDF creator tool. If you can print it to a printer, you can create a PDF from it. More detail here. Replaces Adobe Acrobat.
ConTEXT: An advanced text editor, great for quick text editing, programmer-specific formatting, highlighting, and macros.
Power Calculator: Advanced calculator with graphing capabilities.
KPlato: Part of the KOffice Project, this project management application works similarly to Microsoft Project.
Open Clip Art Library: This collection of user-created, public domain clip art is available for anyone to use for free.
Antivirus and antispyware security software (more detail here)
Avast! Antivirus: Excellent antivirus software for the home or non-profit user. Replaces components of McAfee and Symantec Norton antivirus solutions.
Microsoft Defender: Spyware, Adware, and pop-up protection. Replaces components of McAfee and Symantec security suites.
Zonealarm: The basic version of Zonealarm gives you a free firewall (not needed with newer versions of Windows - Windows XP Service Pack 2 and newer come with built-in firewalls). Replaces components of McAfee and Symantec security suites.
Communications and Internet Freeware
Firefox: For web browsing, use Firefox instead of Microsoft's Internet Explorer. It's better in many ways.
Thunderbird: An excellent email client, equivalent to but better than the (also free) Outlook Express. For those just needing email functionality, it replaces Microsoft Outlook.
Gmail: If you don't need or want a locally-installed email client, use Google's version of free web-based email.
Trillian: Among the best in multi-user, universal instant messaging.
Skype: Talk to anyone, anywhere, for free. All you need is an Internet connection. Read more here.
Filezilla: For those needing to manage FTP transfers, Filezilla is hard to beat - It's got an easy GUI interface, ability to manage multiple FTP sites with different security protocols and ability to see hidden files, and edit / open / download / upload directly from the GUI interface.
Free Backup and Archiving Tools
7-Zip: Use it to create and read ZIP files and other types of file archives. Like WinZip, but with many more features. For example, use Z-Zip as a free ISO reader.
Syncback Freeware (scroll down the page to find it) For those who need a more advanced automatic data synchronization and free backup software solution.
ISO Recorder and Daemon Tools: These allow you to write and read ISO CD and DVD images on your computer. Saving you time and money instead of viewing disk ISO images, you can work with them without ever inserting a disk.
Multimedia Design and Manipulation Freeware
Audacity: I can't say enough good things about Audacity, so I'll just point you at two of its many applications: Ripping any audio for editing, and converting, importing and manipulating audio streams and files. Don't forget you'll need the LAME MP3 encoder plugin in order to export your work to MP3.
VLC Media Player: Play almost any media file type with this media player, including DVDs.
Nvu: For web designers, this is a free website creation and authoring tool, similar to Macromedia Dreamweaver and Microsoft FrontPage.
GIMP: Image manipulation a la Photoshop. (Click for more free graphic editing and creation tools.)
ZS4 Video Editor: Edit video and audio streams for video export, similar to Adobe Premiere.
Email updates