Usually, you retrieve a huge log file and search through it, making selections so that you can get useful data out of it.  This is done by searching the file for all occurrences of something and moving that information to another file, of which is smaller, and repeating as needed.  However, this doesn’t seem to work for several hundred megabyte text files which I run across from time to time.

The first technique was to use Notepad++.  You can go to the Search menu, then Find, then you can find something in all opened documents or the current opened document.  It would find all occurrences and put them in a results area below.  You can copy that to a new document and save it if you want or search again.  The only problem is that Notepad++ doesn’t like to open huge files.

Still, this program does amazing things such as context highlighting, regex searching, and displays most documents you would ever want to display.  Too bad the menus can be quite confusing at times.  Notepad++ is free and can be downloaded at << this website >>.

Being on Windows, I wanted a program to “trim” a huge file down, just give me the tail end of it.  So, an idea was to get a program that can read the file, scroll somewhere, hit CTRL – SHIFT – END to select to the end of the document, copy it, and paste it into Notepad++.  A program called ConText can load huge files, and fit this role well… but it wasn’t an ideal solution.  Context can be downloaded << here >>.

Finally, today I thought of a Linux idea… why not cat the file, grep to find a string, and pipe output as needed?  Well, I’m on Windows…

Luckly, Cigwin is “Linux on Windows” or simply put, a port of Linux programs to Windows.  Some basic Linux CLI (Command Line Interface) knowledge is required.  You download setup.exe, run it, and just install, no need to customize any of the settings (except where files are downloaded, maybe).  So, it does its’ thing, and installs.  It by default installs Bash, and core utilities which include man, grep, and cat.  So, with that done, I drop the huge file in c:\cygwin\home\MyUsername\ and then start up Cygwin Bash Shell from the start menu.  Then I issue a command…

cat file | grep SearchString > output

The last part (> output) is optional, without that piece it will dump to screen.  I can choose to right click on the menu bar and go to Edit, Mark, left click to select an area, right click to copy if I want.  Also, you can go to Properties in the same right click menu and change the Window layout to make it take up more of your screen.  In the end, I got a scanner for a huge text file that is far faster than the other options.

Cygwin is free and available at << this site >>.