DigiFlawFinder Documentation

(updated March 14, 2009)

Contents Structure

Overview

       Purpose/ Usage

       Requirements

       Inputs

       Outputs

       Installation/Updates/Uninstall/Discussion

Usage Details

       Inputs

       Outputs

       Grid

       Options

       Multiple Instances and Performance

       Command Line Processing

Overview

Purpose/ Usage

 

DigiFlawFinder finds and reports potential digital flaws in 16bit pc wav files of the type originating from dat. It does not detect analog flaws or digital flaws that have been greatly transformed and manipulated through additional sound processing.

 

DigiFlawFinder was developed for personal use. Because it seemed it would be beneficial to others, it is now being made available here free to others for noncommercial personal use. All rights are retained by the developer.

 

It was developed for two purposes:

 

• The digital transfer of a dat to a pc wav file can result in flaws caused by dat read and transfer errors.  These flaws manifest themselves as drops, square wav static, digipops, various glitches and discontinuities.  Normally a careful transferor would listen carefully for these before proceeding with additional processing and distribution and evaluate found flaws for potential repair or retransfer. DigiFlawFinder was developed as a tool to aid that process and not to replace it.

 

• Prior errors from dat to dat, dat to wav, and cdr audio transfers will accumulate in a pc wav version of a recording. Given multiple pc wav versions of a recording from different transfer paths, listening to each carefully to determine the least flawed is not always an option. So having a tool to report the flaws can help with that determination and comparison.

 

This is not for everyone. It is for people who care and are into the details of finding and documenting flaws.

 

Hopefully how to use this program is intuitive, but the below documentation may fill in some missing details.

Requirements

 

Microsoft.Net Framework 3.5 SP1 is required. Since this is an optional update from Microsoft in Windows Update and required for many other programs, it or prior versions may already be installed on the computer.  If it has not been installed yet on your computer, you may need 485meg or more in disk space.  Less disk space will be required if you already have version 2 or 3 installed. Installation may require 10 to 15 minutes with broadband and requires a reboot.

The .NET Framework 3.5 requires one of the following operating systems on the target computer:

·      Microsoft Windows XP Home or Microsoft Windows XP Professional, both with Service Pack 2 or later.

·      Microsoft Windows Server 2003 family with Service Pack 1 or later.

·      Microsoft Vista.

·      Microsoft Windows Server 2008.

Hardware Requirements

The following table lists the hardware requirements for running the .NET Framework 3.5.

	CPU required	RAM required
Minimum	Pentium 400 MHz	96 MB
Recommended	Pentium 1 GHz or higher	256 MB or more

 

The program does not use much memory or disk io other than reading the wav files once. But it is cpu intensive and will max out a cpu when it runs.  So the faster the cpu the quicker the find runs. Timings for some sample cpus are in the Usage Details Performances and Multiple Instances section below. It does not seem to dominate a processor but makes use of it and shares it, so you can still run other applications at the same time.

 

This has worked well with Screen Resolutions as low as 1280 by 768. At 1024 by 768 the grid no longer fit the screen but was still usable with the use of the horizontal scroll bar.

 

You will need a default browser to view the Html reports.

Inputs

 

DigiFlawFinder works on16bit pc wav files which can be at 32k, 44k, or 48k that follow wav standards. (There are some programmers who wrote programs that wrote wav files that did not follow standards. The program will tell you when it finds one of those. Generally the wav file can be repaired sometimes with shntool or read into a wav editor and saved correctly. Testing a wide variety of shn and flac downloads has so far only found 3 wav files out of thousands requiring repair)

 

Wav files may be selected for find  flaw processing by individual wav file, folder of wav files, or by drag and drop.

Outputs

 

The output is a summary of the potential or possible flaws categorized by drops, clips, horizontals, and verticals. The flaw detection is not perfect, but the developer is satisfied with the current sensitivity of detection so that most major digital flaws are found, plus a good number that cannot be heard and not many false positives are reported. However human interpretation is still necessary. Careful listening may help determine whether a repair or notation of a found flaw is desired. The find flaw process has been tested successfully on thousands of wav files that were previously documented for flaws based on careful listening.

 

The output is in table format both to an html file and to a tab delimited txt file. The html file is a report viewed in your browser and the txt file can be viewed in a DigiFlawFinder grid or imported into another program such as a spreadsheet. The DigiFlawFinder grid allows for interface to a wav editor to browse through and view each of the flaws.

 

An example of an html output file is here. (This is a wav file produced from one track of a flac download sold by Rhino. Obviously they should have used more care in its production.)

 

The output table columns are described in the Details Section below with some guidance on how to interpret the report

Installation/Updates/Uninstall/Discussion

The support for this software is from this yahoo newsgroup:

http://tech.groups.yahoo.com/group/DigiFlawFinder/

 

For installation, join that group and go to the files section and Install folder at

http://tech.groups.yahoo.com/group/DigiFlawFinder/files/Install/

Click on the file DigiFlawFinder.zip and save it to your computer.

 

Decide where you want to install the software from on your computer and create a directory for it. For example create a folder called “c:\DigiFlawFinderInstall” or on some other drive.  Right click on the DigiFlawFinder.zip file you downloaded and choose “Extract All” from the menu. In the extraction wizard, choose the install directory you created. If this is an update you will want to replace all existing files.

 

In the install folder on your computer, run the setup.exe program. If you do not have the required Microsoft software such as Microsoft.Net Framework 3.5 SP1, you will be asked to install it. On a broadband connection it can take 10 to 15 minutes and requires a reboot. After the reboot, the software installation will continue. You should wait until all your computer startup software such as firewall and security has loaded so that you have internet access before continuing. Follow the directions in the setup program to complete the install. To run the program use the DigiFlawFinder start menu item added in the installation.

 

For DigiFlawFinder updates, it is recommended before starting to go to the DigiFlawFinder Options window and save your settings to a file. It is possible on an update that your settings may be automatically reset by an uncontrollable Microsoft “feature”. If that happens, you will want to reload the settings file from the Options windows.

 

The update install is similar, except you should already have Microsoft.Net Framework 3.5 SP1 and will not need it again, but it is possible it will want to update to a newer version of that too. Make sure you close any instances of DigiFlawFinder that are currently running or the update will not happen. If the previous DigiFlawFinder.zip is still on your computer (you do not need to keep it after the installation unless you think you might want to revert to a prior version of the software), then overwrite it on the save file from the yahoo group. However make sure you extract all files to the same directory as you did on your initial install and make sure you select to replace all existing files when prompted. If you do not use the same install directory, you cannot do an update install, and must use Control Panel’s Add or Remove Programs to remove the current DigiFlawFinder. Then you can install. You will not lose output files, but you will lose settings unless you have saved them to a file from the Options window so you can reload them after the install. (The requirement of the same install directory is an uncontrollable Microsoft “feature”.)

 

To remove or uninstall DigiFlawFinder from your computer use Control Panel’s Add or Remove Programs. If you generated any output data files you can manually delete them from the folder \My Documents\DigiFlawFinder\.

 

If you do not have internet access on the target computer, you can still download the Microsoft.Net Framework from a different computer to a cdr and install it. Search the Microsoft Downloads for the Microsoft.Net Framework 3.5 Full Redistributable Package to get a file called dotnetfx35.exe. You will also need to first download and install Windows Installer 3.1 Redistributable (v2) package. Both of these are available off this web page: http://www.microsoft.com/downloads/details.aspx?FamilyId=333325FD-AE52-4E35-B531-508D977D32A6&displaylang=en

 

The yahoo newsgroup is also available for discussions and feedback and problems.

Usage Details

Inputs

Menu Items

 

• File > Open Grid – allows for selection of a previously generated grid file and displays it

 

• File > Open Html  – allows for selection of a previously generated html file and displays it in the default browser

 

• File > Exit – closes this program

 

• Show Grid – displays the grid

 

• Options – displays the Options window

 

• Help > Documentation – displays the Help Documentation in the default browser. The most current file is here.

 

• Help > About – displays the about window

 

Buttons

 

• Clear All – clears all items in the large listbox

 

• Clear Selected – clears selected  items in the large listbox

 

• Get Wavs – allows for input of selected wav files into the large listbox

 

• Get Folder – allows for input of wav files into the large listbox from a folder. All wav files in the selected folder and one level of subfolders are selected but no further down in subfolder levels.

 

• Find Flaws – starts the Find Flaws process for all items in the large listbox. This disables selection of more files until complete; progress can be monitored in the progress bar.

 

• Cancel Find – if the Find Flaws process is active, this will stop it. There is no restart, but you can suspend the find process by not responding to the cancel confirmation message and then resuming it by responding “No” to the cancel confirmation message.

 

Listboxes

 

• Small – the items in this listbox are entered on the Options window. The selected item is the starting folder for the Get Wavs and Get Folder buttons. This allows for quicker access to regularly used locations for getting wav files.

 

• Large – This listbox is populated from the Get Wavs and Get Folder buttons and through drag and drop of wav files or folders containing wav files. Only files with extension of “wav” will be input into this listbox. Other file types are ignored.  The Clear and Clear Selected buttons are used to remove items. This listbox is filled with the sorted list of the wav files that the Find Flaws button will process. When processing multiple wav files it is important to use good naming standards so wav files sort in the recording order to avoid additional discontinuities between wav files when the music is continuous across wav files.

 

There are two modes for the large listbox based on how it was filled. They are directory mode and file mode. You are in directory mode if the listbox was populated from either a single “Get Folder” or single drag drop of a folder. The window title will include the folder name followed by “(d)” to indicate directory mode. File mode occurs when the listbox was populated by all other methods. The window title will include the first filename in the listbox plus two levels of folder names with special characters replaced by underscores followed by “(f)” to indicate file mode. Output files generated from the Find Flaws process will use the window title in their filenames. So good naming is important for future file retrieval.

Outputs

The Find Flaws process produces 2 outputs – an html report and a tab delimited txt file which can be imported into this programs grid or into a spreadsheet.  The output files are stored in \MyDocuments\DigiFlawFinder\ folder. Their filenames are determined by their wav folder or wav filenames. Refer to the description of the Large Listbox in the Inputs section on how the filenames are derived. The user is responsible for archiving or deleting these files for their convenience.

 

The two types of output files contain much of the same information but are formatted a little differently. The output is tabular and colors are used to highlight certain conditions. Some columns are repeated for the left and right channel. The times indicate the first occurrence of that type of flaw for the channel.

 

Each row represents a summary of flaws for each second starting from the beginning of the first wav file. Only seconds that have a flaw are shown. Times to locate the flaws are reported both globally from 0:00 of the first track to the end of the last track. These global times are also converted to track times. Fractional times are reported in decimal format and represent the first occurrence of the flaw type on that channel.

 

The column types are

 

• Flaw – type of flaw. This table cell is colored cyan when the certain horizontal flaw types that have more potential to be a square wav static flaw.
• GSec – the global second of the flaw being reported
• GFrac– the global fraction of the second in decimal format of the first flaw for this channel of this global second being reported
• TSec – the track second of the first flaw for this channel being reported. This table cell along with TFrac is colored green when it occurs within one second of a track boundary. This is useful on wavs which have gone through cdr generations because drops and discontinuity flaws can be generated through poor cdr copying techniques and show up there.
• TFrac – the track fraction of the second in decimal format of the of the first flaw for this channel being reported
• n – next track . This is rarely seen and when the value is “n” indicates the first flaw for the global second belongs to the next track and so does its track times. This condition occurs because the table row refers to the global second and tracks are not cut on one second boundaries so sometimes the first flaw on the channel is in the next track.
• Len – the length in seconds of the all the flaws of this type for this channel for this global second. This table cell is colored cyan when the length is considered long.
• Occ – the number of occurrences of the type of flaw for this global second
• Value – the value for the flaw for this global second. For a horizontal flaw, this represents the absolute maximum sample value of all the flaws in that global second. For a vertical flaw it represents the magnitude of the vertical change. This table cell is colored cyan when the value is high on a vertical flaw indicating a big discontinuity or low on a horizontal flaw indicating possible square wavs.
• Filename – the filename associated with the beginning of this global second

 

The output is categorized by 4 flaw types of drops, clips, horizontals, and verticals.

 

• Drops – these are sustained digital drops of wav sample value at 0.  The location is important as many recordings have drops at the beginning and end, and some between cdr splits, and some between tracks (as well as discontinuities). The length and number of occurrences are important.

 

• Clips – these are sustained digital clips. The length and number of occurrences are important. Whether you can hear it, depends a lot whether it is in the vocals which can turn scratchy or in the bass.

 

• * Horizontals – in a wav editor these appear as sustained horizontal lines. They may be fuzzy and not a constant sample value. They generally occur near drops, near clips, clipping which has been lowered in volume, heavy   limiting, heavy compression, and square wav static. Occasionally a drawn out peak or trough will produce a false positive. If the flaws were already reported as a drop or clip, it is not reported also as a horizontal. Multiple subtypes are used to describe horizontals. Horizontals may occur at an absolute peak or a trough, or undetermined. The flaw type is listed in the format of “left channel subtype-right channel subtype”. A “p” is used to designate a peak where both sides of the horizontal go towards the zero level. A “d“ is used to designate a trough where both sides of the horizontal go away from the zero level. A “ “ (space) is used to designate a horizontal where one side increases and the other side decreases. And an “m” is used to indicate a mixed combination of the other subtypes across all the occurrences for that second for that channel. Both “ ” and “m” flaw types are highlighted because they sometimes indicate square wavs. Also values where the peak sample value is low are highlighted because they also indicate possible square wav static or drops not at the zero value. High value peaks are generally clipping where the volume has been reduced, heavy compression, or heavy limiting. The value for a horizontal indicates the maximum sample value for that global second. Sample values for 16bit audio range from 32767 to -32768.

 

• Verticals – in a wav editor these appear as vertical lines between samples. They occur at digipops, discontinuities and on square wav static. Occasionally a clap or sudden sound change will produce a false positive.  The value is highlighted when it is high indicating a big discontinuity.

 

Whether you will be able to hear a flaw not only depends on its severity but also your stereo, your ears, your digital to analog converter (high end ones will reproduce the flaw in its fullness where low ends one may fuzz it out), and the surrounding music.

 

There are totals at the bottom of the Html report which can be useful for comparing two different versions of the same recording, but the severity of a flaw is still in the human interpretation of that flaw and can be obscured in the totals.

Grid

Reading through the Html report and looking up most of the flaws reported in the wav editor became time consuming. So a grid was constructed to interface with the wav editor.  The grid is similar to the Html report except one can double click on a grid flaw and then see it displayed in the wav editor. One can also browse through the grid seeing each flaw displayed in the wav editor.

 

Refer to the documentation on the Html report for what the fields on the grid mean.  The grid is shown through the menu File > Open Grid selection, the Show Grid menu item, or automatically after a Find Flaws process completes if the option to do so was selected in the Options window.

 

On displaying the grid you may find the column sizes and window size do not work well on your computer because your display settings are different. So you may adjust the grid window size and each column size by dragging its edge to the desired width. The filename is in the last displayed column of the grid. As the window is resized, this column’s width will expand or narrow. If the total column width is too large for the window, a horizontal scroll bar will appear to let you scroll over and see the other columns. If you are satisfied with new column and window size settings that you made, you can save these setting using the grid menu item Sizing > Save Current. You can also restore the original setting using the grid menu item Sizing > Restore Default.

 

When you click on a grid cell, it will become the current grid cell and highlighted in blue. You can also navigate the grid with the cursor keys. That current cell (highlighted blue) based on its position on the row will establish whether you are browsing the left channel or the right channel. If you press the Show menu item, the wav editor will display the first flaw for that channel and global second. It will try to center the flaw in the wav editor display and give an appropriate viewing range. Double clicking on a cell will do the same. The current global second and fraction that was last displayed are highlighted in yellow. You can browse down or up the grid by selecting the menu items Next and Previous. The grid may scroll as you do this to keep the current row displayed centered. If a grid row only has one channel with a flaw in it, then that channel’s first flaw will be displayed even if it is not the current channel.

 

The interface to each wav editor will probably be different and will require custom programming as all wav editor developers did all follow a common interface standard. Only wav editors that can display a time range from a distinct set of keystrokes can be interfaced to. Cooledit is one such editor. For example the keystrokes sent to a Cooledit window titled with the first wav filename of the find set are “(alt)VV(time1){TAB}(time2) {TAB}{ENTER}”. Those keystrokes open the menu View, then the window Viewing Range, where the first time is typed in, then tab to the next time field where it typed in, then a tab to the OK button, and then the Enter to display that time window magnified and centered on the flaw for easy recognition.

 

The wav editors for Feurio, Eac, and CDWave did not seem to have such a similar wav to interface to them that was workable. There are many wav editors and many are expensive, but if someone who owns one is motivated enough to figure out both the method to repetitively send keystrokes and the window title to send the keystrokes to, then the developer may add the interface to the DigiFlawFinder options.

 

Using the Cooledit interface requires, selecting Cooledit on the DigiFlawFinder options window. All the same wav files that were scanned for flaws must be loaded into Cooledit in the same order they were scanned. This involves opening the first wav file and then appending all the rest of the files. That way the Cooledit window is titled with the first filename which is used by the grid to find the Cooledit session to send the keystrokes to. Following good file naming standards is helpful here because if you are dealing with multiple Cooledit windows all titled “Track01.wav”, the grid will have trouble choosing the correct one to send keystrokes to. The time range is sent in decimal format so make sure that option is selected under the Cooledit View Menu Display Time Format option.

 

Upon showing a flaw in Cooledit, some flaws will be very small and hard to see. So you may need to magnify the sample values to better see the flaw, or you may want to view the flaw in spectral view to help find it. In spectral view most glitches become bright vertical lines.

Options

• Autoshow Grid – check this so that when the find process completes the grid output is loaded and displayed

 

• Autoshow Html – check this so that when the find process completes the Html output is displayed

 

• Bright Grid Colors – some monitors do not display light colors well enough to easily be seen so select this if you prefer or need brighter colors on the grid

 

• Get Wav Folder Values Browse – select up to 3 folders to populate the small listbox on the main window. These folders allow a shortcut to selecting items for the large listbox.

 

• Wav Editor – the interface from the grid to some wav editors has been preprogrammed. Select your wav editor if it present, otherwise leave blank, or try “custom”.

 

• Custom Title – if in Wav Editor you have selected “custom” and you enter a Custom Title, the keystrokes will be sent to an application with this title. If you have selected “custom” and you leave Custom Title blank, then the keystrokes will be sent to an application titled with the filename of the first file in the find process.

 

• Custom Keys - if in Wav Editor you have selected “custom”, these are the keystrokes that will be sent to the wav editor. The sequence of keys sent are what is entered on the first line followed by the beginning time range followed by the second line followed by the ending time range followed by the third line. To learn more about how to represent special keys, visit Microsoft support and search for documentation on Sendkeys Method at: http://msdn.microsoft.com/en-us/library/8c6yea83(VS.85).aspx

 

• Save Settings – this will save any changes made (including grid settings) and they will be available for this instance’s session and all future instances, but not concurrently running instances

 

• Cancel Settings – closes the options window but does not do any saving

 

• Save to File – this will save any settings changes made and store them in the file \MyDocuments\DigiFlawFinder\Settings\DigiFlawFinder.ini. This provides a backup for your settings. It is possible on future program updates or a reinstall that your settings will be lost. They may then be reloaded from this file. Also you may wish to transfer your settings to another computer or temporarily try other settings knowing you can reload the saved ones.

 

• Reload File – if you have previously saved your settings to a file, this will reload your settings from \MyDocuments\DigiFlawFinder\Settings\DigiFlawFinder.ini

 

Multiple Instances and Performance

 

You can run multiple instances of DigiFlawFinder at the same time. On multi-processor systems, it is recommended to leave one processor available for other tasks. So on a 4 cpu system it is best to run at most 3 find processes at once and you can have many more idle instances. More find instances can run at once but it will just slow down performance of the other instances.

 

Processing times will still vary for different wav sets being processed.  The more flaw like the wavs are, the longer it will take.

 

These times were reported on a sample 136 minute wav files set:

• An Intel(R) Xeon(R) CPU 5130 @ 2.00GHz  (4 processor, 2 years old) took 15 minutes with one processor maxed.
• An Intel(R) Celeron(R) CPU  @ 2.53GHz  (1 processor, 2 years old,  budget computer 512meg) took 54 minutes with the one processor maxed out.
• An Intel(R) Xeon(R) CPU  @ 2.00GHz  (2 processor, 6 years old) took 56 minutes with one processor maxed out.

 

These times were reported on a sample 125 minute wav files set that was heavily compressed:

• An Intel(R) Xeon(R) CPU 5130 @ 2.00GHz  (4 processor, 2 years old) took 21 minutes with one processor maxed.
• An Intel(R) Celeron(R) CPU  @ 2.53GHz  (1 processor, 2 years old,  budget computer 512meg) took 47 minutes with the one processor maxed out
• An Intel(R) Xeon(R) CPU  @ 2.00GHz  (2 processor, 6 years old) took 55 minutes with one processor maxed out

 

So basically there is a wide range in performance between computers. The higher end and more recent processors performed significantly better.

 

Command Line Processing

This is an advanced feature most will not need to use. You can locate the DigiFlawFinder.exe file and run it from the command line with parameters to tell it what files or folders to process flaws for that session. This allows for batch execution and control by other programs.

 

Finding DigiFlawFinder.exe is not easy as the Microsoft install program hides it and buries it deep in subdirectories. To find it search for it looking for hidden files. Once you find it you will probably want to copy it to a more accessible location like your install directory. However you will need to do this every time you update the program.

 

The command line parameters are space delimited strings of folders or filenames. If your folders or filenames have imbedded spaces in your names, you will need to put them in quotes. If your folders or filenames have imbedded quotes or apostrophes or foreign characters, it may not work and you will have to rename them.

 

On execution of the command line, an instance of DigiFlawFinder is displayed, and each parameter is loaded as a folder or filename into the large listbox as if it were manually selected using get folder or get wavs buttons and then it is processed the same as if the Find Flaws button was pressed. This is repeated sequentially for each parameter and then the program is closed. If you are running this from a bat or cmd file, control is then returned to that process. While this is running you may use the Cancel button the same as in an active session to cancel this find and all subsequently scheduled ones. If any errors occur, they will not generally appear in message boxes but will be logged to the log file \MyDocuments\DigiFlawFinder\log\cmdlinelog.txt. Messages are appended to the end. Successful completed messages are of the form “0=datetimestamp = parameter = success” while unsuccessful messages are of the form “1=datetimestamp = parameter = errormessage”.

 

An example command line may look like:

 

C:\DigiFlawFinderInstall\DigiFlawFinder.exe c:\wavs\folder1 “c:\disk 1\file1.wav”

 

This runs the DigiFlawFinder find process just like from the window for c:\wavs\folder1 and then “c:\disk 1\file1.wav”.

 

In addition to folders and wav files, command line parameters can also include one “.bat” and one “.txt” file. If a “.bat” is included, it will be launched after processing all finds just before the window closes. If a “.txt” file is included, all logging will go to that file instead of the default log file.